What is LLM Inference?
Inference is the stage where a trained model uses its learned parameters to generate an answer to a prompt. It produces text one token at a time, choosing the most likely next piece based on everything that came before it.
When you ask a model to draft a GST reminder for a customer, it does not retrieve a stored letter from a database. Instead, it generates the reply dynamically, predicting each word or part of a word as it goes. This is why LLMs can be so creative, but it is also why they can hallucinate—producing plausible-sounding but factually incorrect text if the model is making a best-sounding guess rather than verifying a fact.
The Mechanics: Attention and Transformers
Most modern LLMs are built on the transformer architecture. This architecture relies on a mechanism called "attention," which allows the model to weigh the importance of every other word in a sentence when processing a specific word.
For example, in the sentence, "The clinic called the patient because she was late," attention is what allows the model to work out that "she" most likely refers to the patient. Each layer of the transformer refines the representation of a token by using this context, building up complex patterns of grammar, facts, and reasoning through dozens of stacked layers.
At a deeper level, for every token, a layer computes three specific vectors:
- Query: What am I looking for?
- Key: What do I offer?
- Value: What information do I carry?
The model compares the query of one token with the keys of all earlier tokens to decide how much weight to give them, resulting in a weighted mix of their values.
Tokens and the Context Window
Models do not read letters or whole words. Instead, they use a tokenizer to split text into "tokens," which are common words or parts of words. This has significant practical implications for cost and speed, as pricing and limits are often measured in tokens rather than words.
The "context window" is the maximum number of tokens the model can consider at one time. This includes your initial instructions, any documents you have provided, the conversation history, and the model's own growing response. While windows have grown from a few thousand tokens to hundreds of thousands in current models, it is important to note that a larger window does not guarantee the model will use every part of the information equally well.
To keep generation efficient, models use a KV cache. During the generation process, the model would waste significant effort recomputing the attention keys and values for every earlier token at every single step. The KV cache stores these values, making generation much faster, though it does consume more memory as the context grows longer.
Sampling: Controlling Output Variety
When a model generates text, it actually outputs a probability for every possible next token. Sampling settings allow builders to decide which of those tokens is actually picked, allowing you to balance creativity against accuracy.
Common settings include:
- Temperature: This adjusts the probability distribution. A low temperature (near 0) makes the model's choices very predictable and consistent, which is ideal for tasks like writing code or extracting data. A higher temperature makes the output more varied and "creative," which is better for brainstorming ideas.
- Top-p (Nucleus Sampling): This method limits the model to a small set of tokens whose cumulative probability reaches a specific threshold "p." This effectively cuts off the "long tail" of unlikely tokens, preventing the model from choosing nonsense.
Optimizing Speed and Memory
Running large models requires significant hardware resources. To make models faster or able to run on cheaper equipment, developers use several optimization techniques:
| Technique | How it Works | Primary Benefit |
|---|---|---|
| Quantization | Reduces the precision of weights (e.g., from 16-bit to 4-bit). | Massively reduces memory footprint; allows running on CPUs or Apple silicon. |
| Mixture-of-Experts (MoE) | Uses a router to send tokens to only a few "expert" sub-networks. | High total parameters for knowledge, but low "active" compute per token for speed. |
For example, the GGUF file format is a popular way to package quantized weights so they can run efficiently on consumer-grade hardware like a laptop rather than requiring a massive GPU cluster.
Inference vs. Training
It is common to confuse training with inference, but they are very different stages of a model's lifecycle.
Training is the massive, expensive process where a model is shown trillions of tokens of text and code to learn patterns. This results in a "base model" that is knowledgeable but mostly just good at continuing text.
Inference is the much faster, lightweight process of using those existing, learned parameters to answer a specific user query. While training requires massive compute power, inference can be optimized to run on everything from giant cloud GPU clusters to offline mobile phones.
In short, training is how the model learns the rules of language, and inference is the model actually using those rules to talk to you.
Frequently Asked Questions
- Why do LLMs hallucinate during inference?
- Hallucinations occur because models are trained to produce the most plausible next token rather than to verify facts. If the training data is thin on a topic, the model will still generate a best-sounding guess.
- What is the difference between a large model and a small model in inference?
- Larger models have more parameters and can handle complex reasoning, but they require more memory and compute. Smaller models are faster and can often run locally on devices like phones.
- How does context window size affect inference?
- The context window determines how much information the model can "see" at once. Larger windows allow for more complex tasks but require more GPU memory due to the KV cache.
- Does quantization make a model less accurate?
- Quantization reduces the precision of the model's weights to save memory. While it can lead to a noticeable quality loss in very small models, the loss is usually modest for large models.
- What is test-time compute in reasoning models?
- Reasoning models use extra compute during the inference stage to "think" through a problem, creating an internal chain of reasoning before providing a final answer.

Questions people ask
Why do LLMs hallucinate during inference?
Hallucinations occur because models are trained to produce the most plausible next token rather than to verify facts. If the training data is thin on a topic, the model will still generate a best-sounding guess.
What is the difference between a large model and a small model in inference?
Larger models have more parameters and can handle complex reasoning, but they require more memory and compute. Smaller models are faster and can often run locally on devices like phones.
How does context window size affect inference?
The context window determines how much information (documents, history, instructions) the model can 'see' at once. Larger windows allow for more complex tasks but require more GPU memory due to the KV cache.
Does quantization make a model less accurate?
Quantization reduces the precision of the model's weights to save memory. While it can lead to a noticeable quality loss in very small models, the loss is usually modest for large models.
What is test-time compute in reasoning models?
Reasoning models use extra compute during the inference stage to think through a problem, creating an internal chain of reasoning before providing a final answer.
