GuidesAI technical choices
Long context or RAG: give the model everything, or search first?
The two approaches
Long context
RAG
How to choose
| Criterion | Rather long context | Rather RAG |
|---|---|---|
| Volume | A few documents | Thousands |
| Updates | Rare | Frequent |
| Access rights | Same for everyone | Per document or per person |
| Questions | Cross-cutting (“compare these contracts”) | Targeted (“what is the deadline?”) |
| Call volume | Low, or with a cache | High |
The hybrid approach
- 1SearchThe relevant documents, not fragments.
- 2LoadThose whole documents into the context.
- 3AnswerWith the big picture and the citations.
The concepts: the context window, prompt caching and RAG on your documents. The other choice: RAG or fine-tuning.
Frequently asked questions
- Does the model read a very long context just as well?
- Recent models do well, but attention is not perfect: a piece of information buried in a huge context can be used less well. Test on your cases.
- Does prompt caching change the math?
- Yes. When the same corpus comes back on every call, reading it from the cache costs a fraction of the normal price and responds faster.
- Can you combine both?
- Yes, and it is common: retrieval selects whole documents rather than small chunks, and the model reads them in a long context.