GuidesUnderstanding AI
Context window: what an AI model reads at once
What is in the window
- System promptRole, rules, format.
- ToolsTheir description.
- DocumentsThe attached passages.
- ConversationThe messages exchanged.
- AnswerIt takes room too.
Size is counted in tokens. Recent models accept hundreds of thousands, but room is never free.
Three ways to manage it
- SummariseOld messages are replaced by a short summary.
- SearchOnly the useful passages of your documents are sent: that is RAG.
- OrderInstructions first, the question last, tagged documents in between.
The middle trap
Models use the beginning and the end of a long context better than its middle. An important instruction goes first, and is repeated close to the question.
To choose between a large context and search, see RAG or fine-tuning.
Frequently asked questions
- Does the model remember our previous exchanges?
- No. It only sees what the application sends back on each call. An assistant’s “memory” is the history the application puts back in the window.
- Is a large window enough to give it everything?
- It lets you give a lot, but every token is paid for and attention gets diluted: information buried in the middle of a long text is used less well.
- What happens when the window is full?
- The call fails, or the application has to remove something: the oldest messages, or a summary in their place.