Let's talk

LLM Essentials

What is a Context Window?

The context window determines how much information a model can process and remember during a single interaction.

By Divyaprakash JhaUpdated 4 min readTopic: Large language models

Photo for What is a Context Window?

Defining the Context Window

A context window is the maximum number of tokens a large language model can consider at once, including instructions, history, and data. It acts as the model's "active" workspace during a single interaction.

Think of the context window as the amount of text a person can hold in their working memory at one time. If you are a shop owner reading a long contract, the context window is the amount of text you can keep in your head before you start forgetting the beginning of the page.

  • The context window is the limit on how many tokens a model can consider at once.
  • It includes your initial instructions, any uploaded documents, the conversation history, and the model's own generated reply.
  • While early models handled only a few thousand tokens, modern models can process hundreds of thousands or more.

When you ask a model to draft a GST reminder for a customer, it is not looking up a stored letter. Instead, it produces the reply one token at a time, choosing the most likely next piece of text based on everything currently inside that window.

How It Works: Tokens and Attention

To understand the window, you must understand how the model "sees" your data. Models do not read letters or whole words; they process text as tokens, which are chunks of words or characters.

The transformer architecture, which is the foundation of most modern LLMs, uses a mechanism called "attention." When processing a word, the model looks at every other word in the context and decides how much each one matters. For example, in the sentence "The clinic called the patient because she was late," attention is what lets the model work out who "she" most likely refers to.

Every layer of the model refines a token's meaning by looking at the other tokens available in the current context. For every token, a layer computes three specific vectors:

  1. A query: What am I looking for?
  2. A key: What do I offer?
  3. A value: What information do I carry?

The model compares the query of one token with the keys of all earlier tokens to decide which information is most important to include in the final answer.

The Limits of Large Windows

It is a common misconception that a larger window makes a model "smarter." While a large window allows the model to read more text at once, it does not guarantee the model will use every part of the text equally well. Key information is most effective when placed clearly near the actual question.

There are also technical and financial trade-offs to consider. Using very long contexts requires significant GPU memory to store the KV cache. This is a specialized storage system that keeps the "keys and values" of previous tokens so the model doesn't have to recompute them every single time it generates a new word. As the context grows, the memory required to maintain this cache grows as well.

Why Context Window Size Matters

For a business owner, the size of the context window directly impacts the complexity of tasks you can automate. The choice of window size often dictates which technical approach you should take to solve a problem.

Feature Small Context Window Large Context Window
Best Use Case Short chats, simple drafting, or single-task commands. Analyzing long legal documents, entire books, or long-running customer support threads.
Primary Method Often requires RAG (Retrieval-Augmented Generation) to find relevant snippets. Can process the entire dataset directly in the prompt.
Cost/Complexity Lower cost per interaction; easier to manage. Higher cost per interaction; requires more memory/compute.

If you want a model to answer questions about your specific company policy, you have two main paths. You can either use a large window to paste the whole policy every time, or you can use RAG to find only the most relevant paragraphs and put those into a smaller window. For most business applications, RAG is the preferred way to "give the model your data."

Context Window vs. Model Knowledge

It is vital to distinguish between what a model "knows" and what a model "sees" in a conversation. These two concepts are often confused by new users of AI technology.

Concept Analogy Description
Model Parameters Long-term Knowledge The information the model learned during its massive training phase. This is its "brain."
Context Window Short-term Memory The specific information provided in the current prompt or conversation. This is its "notepad."

You use the context window to provide specific, real-time data that the model didn't know during its training. For example, if you want a model to use a new fee list for a medical clinic, you don't need to "retrain" the model. You simply paste the fee list into the context window. The model uses its "long-term" understanding of language to process your "short-term" fee list and provide an accurate answer.

Frequently Asked Questions

Does a larger context window make a model smarter?
Not necessarily. While it allows the model to read more text at once, a large window doesn't guarantee the model will use all that information effectively; placement of key info still matters.
What happens when a conversation exceeds the context window?
The model begins to "forget" the earliest parts of the conversation to make room for new tokens, as it can only consider tokens within its defined limit.
How does context window size affect cost?
Since pricing is often based on the number of tokens processed, larger context windows (sending more data in each prompt) can increase the cost per interaction.
What is the relationship between context window and memory?
Longer contexts require more GPU memory to store the KV cache, which is the data used to speed up the generation process.
Can I use a small context window for large documents?
Yes, by using RAG (Retrieval-Augmented Generation) to find and insert only the most relevant parts of the document into the window.
Close-up of a computer screen displaying lines of CSS code with color swatches.
Photo: Free computer code screen image by unknown, CC0

Questions people ask

Does a larger context window make a model smarter?

Not necessarily. While it allows the model to read more text at once, a large window doesn't guarantee the model will use all that information effectively; placement of key info still matters.

What happens when a conversation exceeds the context window?

The model begins to 'forget' the earliest parts of the conversation to make room for new tokens, as it can only consider tokens within its defined limit.

How does context window size affect cost?

Since pricing is often based on the number of tokens processed, larger context windows (sending more data in each prompt) can increase the cost per interaction.

What is the relationship between context window and memory?

Longer contexts require more GPU memory to store the KV cache, which is the data used to speed up the generation process.

Can I use a small context window for large documents?

Yes, by using RAG (Retrieval-Augmented Generation) to find and insert only the most relevant parts of the document into the window.