GuidesUnderstanding AI

Quantization: fitting an AI model into less memory

By the Banzai Labs teamUpdated on 2 min read

16 bits8 bits4 bitsMemory neededPrecision of each weightFewer bits per weight: a lighter model, slightly less precise
Fewer bits per weight: the model takes less memory, at the cost of some precision.

Calculating memory

  1. 1Number of parametersFor example 8 billion.
  2. 2× bits per weight ÷ 8To get bytes.
  3. 3+ a marginFor the context and computation.
Precision 8-billion-parameter model (weights only)
16 bits about 16 GB
8 bits about 8 GB
4 bits about 4 GB

The trade-off

Fewer bits per weight
What you gain
  • Memory divided by two to four
  • Runs on more modest hardware
  • Often faster
What you lose
  • Some precision
  • More sensitive below 4 bits
  • To check on your tasks

The right reflex

A larger model quantized to 4 bits often does better than a smaller model in 16 bits, for the same memory. Compare both on your cases before choosing.

The use case: a local AI model for sensitive data. The choice: open source model or API.

Frequently asked questions

How much quality do you lose?
At 8 bits, the difference is usually hard to see. At 4 bits, it stays small on most tasks, more visible on long reasoning or code. Below that, quality drops clearly.
Which format should I choose?
GGUF for llama.cpp and Ollama, on CPU or a small GPU. GPTQ or AWQ for serving on GPU with tools such as vLLM.
Does quantization apply to APIs?
No, the provider manages its own models. It matters when you run a model yourself.

Read next

All guides