GuidesUnderstanding AI
Quantization: fitting an AI model into less memory
Calculating memory
- 1Number of parametersFor example 8 billion.
- 2× bits per weight ÷ 8To get bytes.
- 3+ a marginFor the context and computation.
| Precision | 8-billion-parameter model (weights only) |
|---|---|
| 16 bits | about 16 GB |
| 8 bits | about 8 GB |
| 4 bits | about 4 GB |
The trade-off
What you gain
What you lose
The right reflex
A larger model quantized to 4 bits often does better than a smaller model in 16 bits, for the same memory. Compare both on your cases before choosing.
The use case: a local AI model for sensitive data. The choice: open source model or API.
Frequently asked questions
- How much quality do you lose?
- At 8 bits, the difference is usually hard to see. At 4 bits, it stays small on most tasks, more visible on long reasoning or code. Below that, quality drops clearly.
- Which format should I choose?
- GGUF for llama.cpp and Ollama, on CPU or a small GPU. GPTQ or AWQ for serving on GPU with tools such as vLLM.
- Does quantization apply to APIs?
- No, the provider manages its own models. It matters when you run a model yourself.