Let's talk

AI Architecture Explained

How Do Large Language Models Work?

At their core, Large Language Models (LLMs) are sophisticated prediction engines. By scaling up the simple task of predicting the next word, they gain the ability to reason, code, and converse.

By Divyaprakash JhaUpdated 6 min readTopic: Large language models

Photo for How Do Large Language Models Work?

The Core Mechanism: Next-Token Prediction

Large Language Models (LLMs) are neural networks trained to predict the most likely next "token" in a sequence. By scaling this simple prediction task across massive amounts of data, they develop the ability to reason, code, and engage in complex conversation.

At its simplest level, an LLM is a prediction engine. It does not look up facts in a database like a traditional search engine; instead, it generates text one piece at a time based on patterns it learned during its training phase.

When you ask a model to draft a GST reminder for a customer, it does not retrieve a stored letter from a folder. It produces the reply one token at a time, each time choosing the most likely next piece given everything that came before it.

The "Large" in LLM refers to the number of parameters—the adjustable numbers the model learns during training. These can range from a few hundred million in small models to hundreds of billions or more in the largest, most capable models.

From Text to Numbers: Tokens and Embeddings

Models do not read letters or whole words. To process language, they use a process called tokenization.

Tokens

A tokenizer splits text into "tokens." Common words might become a single token, while rare words or complex characters are split into several smaller pieces. You can read more about the specifics in our guide on what is a token in LLM.

This has practical implications for business owners. For example, languages that were under-represented in a tokenizer's training data—including many Indian languages written in their own scripts—often require more tokens to represent the same sentence. This can make these languages slower and more costly to process on certain models.

Embeddings

Once the text is split into tokens, each token is converted into an "embedding." An embedding is a list of numbers, also known as a vector, with hundreds or thousands of dimensions. This mathematical representation captures the meaning of the text.

Tokens with similar meanings are mapped to vectors that sit close together in a high-dimensional space. This concept is what powers semantic search: by embedding both your documents and a user's question, a system can find the documents whose vectors are closest to the question.

The Transformer and the Power of Attention

Most modern LLMs use the Transformer architecture. The breakthrough of the Transformer is a mechanism called "attention," which allows the model to understand context by looking at how words relate to one another.

When processing a word, the model looks at every other word in the sentence to decide which ones are most relevant. For example, in the sentence, "The clinic called the patient because she was late," the attention mechanism is what allows the model to work out that "she" most likely refers to the patient.

On a technical level, for every token, a layer computes three distinct vectors:

  • A Query: What am I looking for?
  • A Key: What do I offer?
  • A Value: What information do I carry?

The model compares the query of one token with the keys of all earlier tokens to create weights. These weights determine how much of the "value" information is passed along. Dozens of these layers are stacked together to build up the model's ability to handle grammar, facts, style, and reasoning patterns.

How Models are Trained

Building a model happens in distinct stages, moving from general knowledge to specific helpfulness.

1. Pretraining

In the pretraining phase, the model views trillions of tokens of text and code. The goal is simple: predict what comes next. Every time the model makes a wrong guess, it nudges its billions of parameters slightly to improve. The result is a "base model" that is incredibly knowledgeable but often just continues text rather than following instructions.

2. Instruction Tuning

To make a model useful for humans, it undergoes instruction tuning. The model is trained on examples of specific instructions and the ideal responses that follow. This teaches the model the format of being an assistant—knowing when to answer a question, how to follow a requested format, and when to stop talking.

3. Preference Tuning (RLHF/RLAIF)

The final polish comes from preference tuning, which shapes the model's tone and safety. In Reinforcement Learning from Human Feedback (RLHF), people rank different versions of an answer. A "reward model" learns these human preferences, and the LLM is optimized to match them. In other cases, AI feedback (RLAIF) is used to guide the model based on written principles.

The Rise of Reasoning Models

A new family of models has emerged that focuses on "thinking" before they speak. While standard models are designed for speed and fluency, reasoning models are trained using reinforcement learning on problems with checkable answers, such as mathematics or computer code.

These models use what is known as "test-time compute." Instead of giving an immediate answer, they produce a long, internal chain of reasoning. They try different approaches, catch their own mistakes, and only provide a final answer once they have worked through the logic. This makes them ideal for complex, multi-step problems, though they are generally slower and more expensive than standard models.

Understanding Hallucinations

A hallucination occurs when a model generates a response that is fluent and confident, but factually false. This might look like an invented citation, a wrong GST rate, or a company policy that does not exist.

Hallucinations happen because the model is fundamentally designed to produce the most plausible next token, not necessarily the most truthful one. If the training data was thin on a specific topic, the model will still attempt to generate a best-sounding guess.

Making LLMs Useful for Business

To move from a general chatbot to a specialized business tool, you can use three primary methods. We recommend trying them in this order of increasing complexity.

Method What it changes Best for... Effort Level
Prompting The instructions and examples you send Format, tone, and simple tasks Low
RAG The documents you add to the context at question time Answering from your own, changing data Medium
Fine-tuning The model's actual internal weights Consistent style or narrow, specialized skills Higher

For most businesses wanting to "make the model know our data," Retrieval-Augmented Generation (RAG) is the standard solution. It keeps facts current by providing the model with external documents to read at the moment the question is asked. Fine-tuning is better suited for when you need the model to behave in a very specific, consistent way or to master a highly specialized skill.

If you want to understand the broader landscape of how these models fit into the AI ecosystem, see our guide on what is LLM in AI.

Frequently Asked Questions

Why do LLMs hallucinate?
Hallucinations happen because models are trained to predict the most plausible next token rather than verifying facts. If training data is thin on a topic, the model will still generate a best-sounding guess.
What is the difference between a token and a word?
A token is the basic unit of text a model processes. While common words might be a single token, rare words or complex characters are often split into multiple tokens. You can find more details in our guide on tokens.
What is a context window?
The context window is the maximum number of tokens a model can consider at one time, including your instructions, conversation history, and the model's own previous replies. For more, see what is a context window.
What is the difference between RAG and fine-tuning?
RAG provides the model with external documents to read at question time to keep facts current. Fine-tuning updates the model's internal weights to change its behavior or style.
What are parameters in an LLM?
Parameters are the adjustable numbers the model learns during training. The number of parameters is a primary measure of a model's size and capacity.
Close-up of a computer screen displaying lines of CSS code with color swatches.
Photo: Free computer code screen image by unknown, CC0

Questions people ask

Why do LLMs hallucinate?

Hallucinations happen because models are trained to predict the most plausible next token rather than verifying facts. If training data is thin on a topic, the model will still generate a best-sounding guess.

What is the difference between a token and a word?

A token is the basic unit of text a model processes. While common words might be a single token, rare words or complex characters are often split into multiple tokens.

What is a context window?

The context window is the maximum number of tokens a model can consider at one time, including your instructions, conversation history, and the model's own previous replies.

What is the difference between RAG and fine-tuning?

RAG provides the model with external documents to read at question time to keep facts current. Fine-tuning updates the model's internal weights to change its behavior or style.

What are parameters in an LLM?

Parameters are the adjustable numbers the model learns during training. The number of parameters (e.g., 7B or 70B) is a primary measure of a model's size and capacity.