GuidesUnderstanding AI

Context window: what an AI model reads at once

By the Banzai Labs teamUpdated on 2 min read

Context window: everything the model readsOutside: forgotten
The context window: instructions, documents and conversation. Whatever overflows is forgotten.

What is in the window

Everything the model reads, on every call
  1. System promptRole, rules, format.
  2. ToolsTheir description.
  3. DocumentsThe attached passages.
  4. ConversationThe messages exchanged.
  5. AnswerIt takes room too.

Size is counted in tokens. Recent models accept hundreds of thousands, but room is never free.

Three ways to manage it

  • SummariseOld messages are replaced by a short summary.
  • SearchOnly the useful passages of your documents are sent: that is RAG.
  • OrderInstructions first, the question last, tagged documents in between.

The middle trap

Models use the beginning and the end of a long context better than its middle. An important instruction goes first, and is repeated close to the question.

To choose between a large context and search, see RAG or fine-tuning.

Frequently asked questions

Does the model remember our previous exchanges?
No. It only sees what the application sends back on each call. An assistant’s “memory” is the history the application puts back in the window.
Is a large window enough to give it everything?
It lets you give a lot, but every token is paid for and attention gets diluted: information buried in the middle of a long text is used less well.
What happens when the window is full?
The call fails, or the application has to remove something: the oldest messages, or a summary in their place.

Read next

All guides