GuidesAI technical choices

Long context or RAG: give the model everything, or search first?

By the Banzai Labs teamUpdated on 3 min read

Long contextEverything is read, every callRAGOnly the useful passagesSmall stable corpus: read it all. Large changing corpus: search.
Long context: every document on every call. RAG: only the useful passages.

The two approaches

Read everything, or search first
Long context
  • Every document in every call
  • Nothing to index, nothing to chunk
  • Cost and latency grow with size
RAG
  • Only the useful passages
  • Corpus with no size limit
  • A retrieval pipeline to build

How to choose

Criterion Rather long context Rather RAG
Volume A few documents Thousands
Updates Rare Frequent
Access rights Same for everyone Per document or per person
Questions Cross-cutting (“compare these contracts”) Targeted (“what is the deadline?”)
Call volume Low, or with a cache High

The hybrid approach

  1. 1SearchThe relevant documents, not fragments.
  2. 2LoadThose whole documents into the context.
  3. 3AnswerWith the big picture and the citations.

The concepts: the context window, prompt caching and RAG on your documents. The other choice: RAG or fine-tuning.

Frequently asked questions

Does the model read a very long context just as well?
Recent models do well, but attention is not perfect: a piece of information buried in a huge context can be used less well. Test on your cases.
Does prompt caching change the math?
Yes. When the same corpus comes back on every call, reading it from the cache costs a fraction of the normal price and responds faster.
Can you combine both?
Yes, and it is common: retrieval selects whole documents rather than small chunks, and the model reads them in a long context.

Read next

All guides