Let's talk

Training Stages

How are LLMs trained?

Large Language Models (LLMs) are built through a multi-stage process that transforms raw text into an intelligent, instruction-following assistant.

By Divyaprakash JhaUpdated 5 min readTopic: Large language models

Photo for How are LLMs trained?

Large Language Models (LLMs) are trained in three main stages: pretraining on massive text datasets, instruction tuning for following directions, and preference tuning (RLHF). This multi-stage process transforms raw text into an intelligent, instruction-following assistant.

1. Pretraining: The Foundation of Knowledge

The journey begins with pretraining. During this phase, the model is shown trillions of tokens of text and code. The core task is simple but massive in scale: next-token prediction. The model is essentially playing a high-stakes game of "guess what comes next" in a sequence of text.

Every time the model makes a wrong guess, the system nudges billions of parameters—the adjustable numbers that make up the model—to improve accuracy for the next time. As it processes more data, it learns the patterns of human language, facts about the world, and even how to write code.

The result of this stage is what we call a base model. A base model is incredibly knowledgeable, but it has a significant limitation: it doesn't know how to be an assistant. If you ask a base model, "What is the GST rate for services?", it might respond by simply continuing the text with more questions or unrelated facts rather than giving you a direct answer. It is a text-completer, not a conversationalist.

2. Instruction Tuning: Learning to be an Assistant

To turn a base model into something useful for business—like a customer service bot or a drafting tool—it undergoes instruction tuning. This is a form of supervised fine-tuning where the model is trained on specific examples of instructions paired with high-quality, correct responses.

This stage teaches the model a specific format for interaction. It learns to:

  • Answer questions directly rather than just completing a sentence.
  • Follow requested formats, such as "write this in a list" or "output this in JSON."
  • Know when to stop talking so it doesn't ramble indefinitely.

For a small business owner, this is the difference between a model that just predicts text and a model that can actually draft a polite reminder for a customer who has missed a payment.

3. Preference Tuning: Shaping Tone and Safety

The final major layer of training involves refining the model based on human preferences. Even after instruction tuning, a model might be too blunt, too wordy, or might accidentally provide unsafe information. Preference tuning solves this.

There are several ways this is achieved:

  • RLHF (Reinforcement Learning from Human Feedback): Humans rank different model responses from best to worst. A "reward model" learns these preferences and then optimizes the LLM to favor the types of answers humans liked.
  • RLAIF (Reinforcement Learning from AI Feedback): Instead of humans, a second AI provides labels based on a set of written principles (often called "Constitutional AI"). This is faster and more scalable than using humans.
  • Direct Preference Optimization: A simpler method where the model learns directly from pairs of preferred vs. non-preferred answers.

This stage is critical for shaping the model's helpfulness, its tone, and the safety guardrails that prevent it from generating harmful content.

4. Reasoning Models and Test-Time Compute

A new family of models has emerged in late 2024. These are reasoning models trained specifically to handle complex, multi-step problems like advanced mathematics or complex coding tasks. Unlike standard models, these are often trained using reinforcement learning on problems that have "checkable" answers.

These models use a concept called test-time compute. Instead of instantly spitting out the first likely word, they spend extra computational power to generate a long internal "chain of reasoning" before they reply. They essentially "think" through the problem, catch their own mistakes, and try different approaches before giving you the final answer.

For a builder, this is a trade-off. These models are much better for hard, logical problems, but they are slower and more expensive than standard models. You wouldn't use a reasoning model to "rewrite this email politely," but you might use it to "debug this complex logistics algorithm."

The Engine: The Transformer Architecture

Underneath all these training stages is the Transformer, the neural network architecture that makes modern AI possible. The "secret sauce" of the Transformer is a mechanism called attention.

Attention allows the model to look at every word in a sentence and decide which other words are most relevant to understanding the current one. For example, in the sentence, "The clinic called the patient because she was late," the attention mechanism helps the model understand that "she" refers to the "patient" and not the "clinic."

The process works through several layers:

  1. Query, Key, and Value: For every token, the model calculates what it is looking for (Query), what it offers (Key), and what information it carries (Value).
  2. Attention Heads: Multiple "heads" run in parallel, each tracking different types of relationships (like grammar, subject-verb agreement, or context).
  3. Feed-Forward Networks: After attention, the information passes through blocks that hold much of the model's stored knowledge.

By stacking dozens of these layers, the model builds up a deep understanding of grammar, facts, style, and reasoning patterns. If you want to dive deeper into how these models function, see our guide on how large language models work.

Why Training Leads to Hallucinations

One of the biggest challenges for business owners is "hallucination"—when a model provides a fluent, confident, but completely false answer. This might manifest as an invented legal citation, a wrong GST rate, or a policy that your company never actually implemented.

It is vital to understand why this happens: LLMs are trained to produce the most plausible next token, not necessarily the most factual one.

If a model's training data was thin on a specific topic, or if you ask it a question about something it has never seen, it won't necessarily say "I don't know." Instead, it will use its training to generate the most likely-sounding sequence of words, which results in a "hallucination."

Method What it changes Best for... Effort Level
Prompting The instructions you send Format, tone, simple tasks Low
RAG Documents added to the context at question time Answering from your own, changing data Medium
Fine-tuning The model's actual weights Consistent style, narrow skills Higher

To minimize these errors in a business setting, builders should use RAG (Retrieval-Augmented Generation). Instead of relying on the model's internal memory, you provide it with specific, factual source material (like your company's handbook or price list) and instruct it to answer only using that information. Always keep a human in the loop for any legal, medical, or financial tasks.

Close-up of a computer screen displaying lines of CSS code with color swatches.
Photo: Free computer code screen image by unknown, CC0

Questions people ask

What are parameters in an LLM?

Parameters are the adjustable numbers the model learns during training. A model's size is often described by its parameter count, ranging from millions to hundreds of billions.

What is the difference between pretraining and fine-tuning?

Pretraining is the massive-scale process of learning from trillions of tokens to gain general knowledge. Fine-tuning is a more targeted process used to teach the model specific tasks or styles.

What is RLHF?

RLHF stands for Reinforcement Learning from Human Feedback. It is a method where humans rank different model responses to help the AI learn which answers are most helpful and safe.

Why do LLMs hallucinate?

Hallucinations happen because models are trained to predict the most likely next piece of text rather than looking up facts in a database. They prioritize plausibility over truth.

What is a token?

A token is the basic unit of text a model reads and writes. It can be a whole word or a part of a word, depending on how the tokenizer processes the text.