Let's talk

AI Explained

How Does ChatGPT Work?

ChatGPT is a large language model that uses a transformer architecture to predict the most likely next piece of text in a conversation.

By Divyaprakash JhaUpdated 5 min readTopic: Large language models

Photo for How Does ChatGPT Work?

The Core: Next-Token Prediction

ChatGPT is a large language model that uses a transformer architecture to predict the most likely next piece of text in a conversation. It works by calculating the probability of the next word or part of a word based on the patterns it learned during training.

At its heart, ChatGPT is a neural network trained to predict the next token in a sequence. It is important to understand that the model does not look up facts in a database like a traditional search engine. Instead, it generates text one piece at a time based on probability.

A token can be a whole word or just a part of a word. This process is known as tokenization. For example, if you ask a model to draft a GST reminder for a customer, it does not retrieve a stored letter from a folder. It produces the reply one token at a time, choosing the most likely next piece given everything that came before it.

To understand how these building blocks fit together, you might want to read our guide on tokens.

The Transformer and Attention Mechanism

Modern chat models use the transformer architecture, which relies on a mechanism called "attention." This architecture is what allows the model to understand the relationship between different words in a long sentence.

Attention allows the model to look at every other word in a sentence to decide which ones are most relevant to the current word. For instance, in the sentence, "The clinic called the patient because she was late," attention is what lets the model work out that "she" most likely refers to the patient, not the clinic.

Each layer of the transformer refines how the model understands these tokens. By stacking dozens of these layers together, the model builds up a complex understanding of grammar, facts, style, and reasoning patterns. This is the fundamental technology behind Large Language Models (LLMs).

From Base Model to Chat Assistant

Building a model like ChatGPT involves several distinct stages of training to turn a raw mathematical engine into a helpful assistant.

1. Pretraining

In the first stage, the model is shown trillions of tokens of text and code. The goal is simple: predict what comes next. Every time the model makes a wrong guess, it nudges its internal parameters slightly to correct itself. The result is a "base model" that is incredibly knowledgeable but is not very good at following instructions; it simply wants to continue the text you started.

2. Instruction Tuning

To make the model useful for business, it undergoes instruction tuning. The model is trained on specific examples of instructions and good responses. This teaches the model the format of being an assistant—knowing when to answer a question, how to follow a requested format, and when to stop talking.

3. Preference Tuning

Finally, the model learns from human preferences. In a process called Reinforcement Learning from Human Feedback (RLHF), people rank different versions of an answer. A reward model learns what humans prefer, and the LLM is optimized to align with those preferences for helpfulness and safety.

The Rise of Reasoning Models

Newer reasoning models represent a shift in how AI handles complex tasks. While standard models are designed to be fast and conversational, reasoning models are designed to solve difficult problems through a process of "thinking."

These models use reinforcement learning on problems with checkable answers, such as math or code. They produce a long internal chain of reasoning, where they try different approaches, catch their own mistakes, and only then provide a final reply. This process is known as "test-time compute," meaning the model spends more computational power at the moment of answering to ensure accuracy.

Feature Standard Chat Model Reasoning Model
Primary Use Case Rewriting messages, brainstorming, simple chat. Complex math, coding, multi-step logic.
Speed Fast, immediate response. Slower, due to "thinking" time.
Cost Generally lower per task. Higher due to increased compute.

For builders, the trade-off is clear: use reasoning models for hard, multi-step problems rather than simple tasks like "rewrite this message politely."

Understanding Hallucinations

One of the biggest challenges for business owners is "hallucination." A hallucination occurs when a model generates a fluent, confident answer that is actually false—such as an invented legal citation, a wrong tax rate, or a policy your clinic never actually had.

This happens because the model is designed to produce plausible text rather than to verify truth. If the training data was thin on a specific topic, the model will still attempt to generate its "best-sounding guess."

If you want to understand the broader context of how these models are built, see our guide on LLM mechanics.

Context Windows and Speed

When using an AI, you are working within a "context window." This is the limit of how many tokens the model can consider at one time. This includes your current instructions, any documents you have uploaded, the conversation history, and the model's own previous replies.

While context windows have grown significantly, it is important to remember that a larger window does not guarantee the model will use every part of it equally well. Key information is most effective when placed clearly near the question.

To keep these interactions fast, models use a "KV cache." This technology stores previously computed information so the model doesn't have to recompute every single word in a long conversation every time you send a new message. This makes generation much faster, though it requires more memory.

Finally, developers can control the "creativity" of the model using sampling settings:

  • Temperature: Controls how predictable the output is. Low temperature (near 0) is best for structured tasks like extracting data from a WhatsApp message. High temperature is better for creative brainstorming.
  • Top-p: A method of narrowing down the possible next words to only the most likely ones, cutting off unlikely "noise."

Frequently Asked Questions

Does ChatGPT search the internet for answers?
By default, LLMs generate text based on patterns learned during training rather than looking up facts. However, they can be connected to tools or RAG to access external, real-time data.
Why does the AI sometimes lie?
This is called hallucination. It happens because the model is trained to predict the most likely next word, which can result in a response that sounds confident but is factually incorrect.
What is the difference between a token and a word?
Tokens are the basic units of text a model reads. While common words might be one token, rare words or complex characters are often split into several smaller pieces.
How can I make ChatGPT more accurate for my business?
You can use RAG (Retrieval-Augmented Generation) to give the model your specific documents, or fine-tuning to teach it a specific style or narrow skill.
What is the 'context window' in a chat?
The context window is the maximum amount of text (instructions, conversation history, and documents) the model can process at one time.
Close-up of a computer screen displaying lines of CSS code with color swatches.
Photo: Free computer code screen image by unknown, CC0

Questions people ask

Does ChatGPT search the internet for answers?

By default, LLMs generate text based on patterns learned during training rather than looking up facts. However, they can be connected to tools or RAG to access external data.

Why does the AI sometimes lie?

This is called hallucination. It happens because the model is trained to predict the most likely next word, which can result in a response that sounds confident but is factually incorrect.

What is the difference between a token and a word?

Tokens are the basic units of text a model reads. While common words might be one token, rare words are often split into several smaller pieces.

How can I make ChatGPT more accurate for my business?

You can use RAG (Retrieval-Augmented Generation) to give the model your specific documents, or fine-tuning to teach it a specific style or narrow skill.

What is the 'context window' in a chat?

The context window is the maximum amount of text (instructions, conversation history, and documents) the model can process at one time.