Foundation

Context Window

The context window is the total amount of information a large language model can hold in active consideration at one time, encompassing the conversation history, retrieved documents and instructions that shape its response to any given query.
DATE UPDATED: 11 Sep 2026
|
FIRST PUBLISHED: 22 Mar 2026

What the context window is

A large language model does not have persistent memory in the way a human does. When it generates a response, it works with whatever information has been provided to it in that moment: the current query, the conversation history, any documents or data that have been retrieved and passed to it, and any instructions that govern how it should behave. All of this constitutes the context window, and the model can only work with what is in it.

The size of the context window is measured in tokens, a unit roughly corresponding to a word or part of a word. Early language models had context windows of a few thousand tokens, enough for a short conversation or a brief document. Current models have context windows of hundreds of thousands of tokens, sufficient to accommodate lengthy documents, extensive conversation histories and substantial retrieved content simultaneously.

Everything the model produces is a function of what is in its context window at the moment of generation. A model with access to the right information in its context window will produce a relevant, accurate and appropriately nuanced response. A model working with incomplete, poorly structured or irrelevant context will produce a response that reflects those limitations, sometimes confidently.

Why the context window matters for system design

Understanding the context window is useful for anyone designing or commissioning AI systems because it clarifies a constraint that shapes many of the most important architectural decisions. You cannot simply point a language model at everything the organisation knows and expect it to use all of it. The model works with what fits in the window at any given moment. This is why retrieval systems, RAG architectures and careful prompt design matter: they are the mechanisms by which the right information is selected and placed in the context window so the model has what it needs.

As context windows have grown, some of the complexity of retrieval design has reduced. A model that can hold an entire lengthy contract in its context window can analyse it without requiring the contract to be chunked and selectively retrieved. But even very large context windows have limits, and the quality of reasoning often degrades when the window is densely packed, because the model’s attention is distributed across a larger volume of material. Good system design is still required regardless of context window size.

Context windows and conversational AI

In a conversational AI application, the context window includes the history of the conversation. As a conversation extends, earlier parts of it may be compressed or removed to make room for new exchanges, which can affect the coherence of longer interactions. This is a design consideration for AI systems intended to support extended client conversations or complex multi-turn workflows, where the continuity of the conversational context matters for the quality of the output.

Insight

For professional services organisations, the context window is the practical explanation for why AI systems behave differently when given complete, well-structured information versus incomplete or poorly organised information. A document review system that receives a well-prepared contract with clear structure and relevant precedents in its context window will produce substantially better output than the same system receiving a poorly formatted document with irrelevant material included. This is not a limitation of the AI. It is a reflection of the principle that the quality of AI output is determined by the quality of the inputs, and that structuring those inputs well is a design discipline, not an afterthought.

Misconceptions

A larger context window does not mean better performance. A model working with a densely packed context window may produce lower quality reasoning than one working with a focused, well-structured set of relevant information. Selecting what goes into the context window matters as much as the size of the window itself.

The context window is not the same as the model's memory. A large language model does not remember previous conversations once the context window is cleared. Each new session starts without knowledge of previous ones unless that history is explicitly included in the new context.

Context window size is not the primary determinant of whether RAG is needed. Even with very large context windows, retrieval systems remain valuable for selecting the most relevant content from a large knowledge base rather than including everything and relying on the model to find what matters.

BACK TO GLOSSARY
RELATED POSTS