Let's talk

Guide · LLMs

What is an LLM? How large language models work (2026 guide)

A large language model is a neural network trained to predict the next piece of text. This guide explains, step by step, how that simple idea becomes a system that writes, reasons, reads images and uses tools.

By Divyaprakash JhaUpdated 13 min read

What is an LLM?

TL;DR: A large language model (LLM) is a neural network, usually a transformer, trained on enormous amounts of text to predict the next token. That one skill, scaled up and then tuned to follow instructions, lets it answer questions, write, translate, code and reason. LLMs do not look facts up; they generate likely text, which is why they can hallucinate. You make them useful with good prompts, retrieval of your own data (RAG), occasional fine-tuning, and tools that turn them into agents.

"Large" refers to the number of parameters, the adjustable numbers the model learns during training, which run from a few hundred million in small models to hundreds of billions or more in the largest ones. "Language model" means a model of which text is likely to come next.

When you ask a model to draft a GST reminder for a customer, it does not retrieve a stored letter. It produces the reply one token at a time, each time choosing a likely next piece given everything before it. The rest of this guide explains how that works and how to use it well.

Tokens, tokenizers and embeddings

Tokens

Models do not read letters or whole words. A tokenizer splits text into tokens: common words become one token, rare words split into several pieces. Most tokenizers use a variant of byte-pair encoding, which learns frequent chunks from training data.

This has practical effects. Pricing, speed and context limits are counted in tokens. Languages that were under-represented in a tokenizer's training data, including many Indian languages written in their own scripts, often need more tokens for the same sentence, which makes them slower and costlier on such models. Newer tokenizers with larger vocabularies have narrowed that gap.

Embeddings

Each token is mapped to an embedding: a list of numbers (a vector) with hundreds or thousands of dimensions. Tokens with related meanings end up with similar vectors. The same idea powers semantic search: embed your documents and a question, then find the documents whose vectors are closest. That is the retrieval half of RAG, covered below.

The transformer and attention

The plain-language version

Almost every modern LLM is a transformer, the architecture introduced in the 2017 paper Attention Is All You Need. Its key idea is attention: when processing a word, the model looks at every other word in the context and decides how much each one matters.

In "The clinic called the patient because she was late", attention is what lets the model work out who "she" most likely refers to. Each layer refines every token's representation using context; dozens of layers stacked together build up grammar, facts, style and reasoning patterns.

One level deeper

For every token, a layer computes three vectors: a query (what am I looking for?), a key (what do I offer?) and a value (what information do I carry?). The query of one token is compared with the keys of all earlier tokens; the scores become weights; the output is a weighted mix of their values. Several such attention heads run in parallel, each free to track a different kind of relationship.

After attention, each token passes through a feed-forward network, a block that holds much of the model's stored knowledge. Residual connections and normalisation keep training stable across many layers. In a decoder-only model, the kind used by today's chat models, a mask ensures each token can only attend to tokens before it, so the model learns to predict what comes next.

How LLMs are trained

Pretraining: next-token prediction

Pretraining shows the model trillions of tokens of text and code and asks one question over and over: what comes next? Each wrong guess nudges billions of parameters slightly. Research on scaling laws and later compute-optimal training showed that loss falls predictably as you scale parameters, data and compute together. The result is a base model: knowledgeable, but it continues text rather than following instructions.

Instruction tuning

Next, the model is trained on examples of instructions and good responses. This supervised fine-tuning teaches the format of being an assistant: answer the question, follow the requested format, stop when done.

RLHF, RLAIF and preference tuning

Then the model learns from preferences. In reinforcement learning from human feedback (RLHF), described in OpenAI's InstructGPT paper, people rank alternative answers, a reward model learns those preferences, and the LLM is optimised toward them. RLAIF replaces some human labels with AI feedback guided by written principles, as in Anthropic's Constitutional AI. Simpler methods such as Direct Preference Optimization learn from preference pairs without a separate reward model. This stage shapes helpfulness, tone and safety.

Reasoning models and test-time compute

Since late 2024, a family of reasoning models has been trained, largely with reinforcement learning on problems with checkable answers such as maths and code, to think before answering. They produce a long internal chain of reasoning, try approaches, catch mistakes and only then reply. This spends more compute at answer time (test-time compute) in exchange for better results on hard, multi-step problems. It builds on the earlier finding that chain-of-thought prompting improves reasoning. The trade-off is cost and latency: use reasoning modes for hard problems, not for "rewrite this message politely".

Context windows, the KV cache and sampling

Context window

The context window is how many tokens the model can consider at once: your instructions, documents, conversation history and its own reply. Windows have grown from a few thousand tokens in early chat models to hundreds of thousands or more in current ones. A large window does not guarantee the model uses every part equally well; key information placed clearly, near the question, still helps.

KV cache

While generating, the model would waste effort recomputing attention keys and values for every earlier token at every step. The KV cache stores them. It makes generation fast, but it consumes memory that grows with context length and the number of simultaneous users, which is why long contexts need more GPU memory. The AI hardware guide works through that maths.

Sampling: temperature and top-p

The model outputs a probability for every possible next token. Sampling settings decide which one is picked:

  • Temperature sharpens or flattens the probabilities. Low (near 0) gives consistent, predictable output, good for extraction and code. Higher values give more varied output, useful for brainstorming.
  • Top-p (nucleus sampling) keeps only the smallest set of tokens whose probabilities add up to p, then samples from that set, cutting off unlikely tokens.

For business tasks like filling a booking form from a WhatsApp message, use a low temperature and ask for structured output.

Hallucinations and why they happen

A hallucination is a fluent, confident answer that is false: an invented citation, a wrong GST rate, a policy your clinic never had. It happens because the model is trained to produce plausible text, not to know when it does not know. If the training data was thin on a topic, or your question asks about something it never saw, it still generates its best-sounding guess.

What helps:

  • Give the model the source material (RAG) and ask it to answer only from it, quoting where it found each fact.
  • Allow and encourage "I don't know" in the prompt.
  • Verify with code or tools where you can: recompute totals, look up the real record.
  • Keep a human in the loop for anything legal, medical or financial.

These reduce errors; none removes them. Design the workflow assuming some answers will be wrong.

RAG vs fine-tuning vs prompting

There are three main ways to make a general model work for your business. Try them in this order.

PromptingRAGFine-tuning
What it changesThe instructions and examples you sendThe documents you add to the context at question timeThe model's weights
Best forFormat, tone, simple tasksAnswering from your own, changing dataConsistent style, narrow skills, smaller and cheaper models
Keeps facts currentOnly if you paste themYes, update the indexNo, needs retraining
EffortLowMedium: chunking, embeddings, searchHigher: data preparation, training, evaluation
Example"Reply in polite Hindi, under 60 words"Clinic assistant answering from the current fee list and doctor scheduleA small model that always extracts invoice fields in your exact JSON format

Retrieval-augmented generation is usually the right answer to "how do I make it know our data". Fine-tuning is the right answer to "how do I make it behave consistently and cheaply".

LoRA and QLoRA

Full fine-tuning updates every parameter and needs a lot of GPU memory. LoRA freezes the original weights and trains small low-rank adapter matrices instead, a tiny fraction of the size. QLoRA goes further by keeping the frozen base model in 4-bit precision while training the adapters, making fine-tuning of fairly large models possible on a single GPU. Adapters can be swapped per customer or per task.

Quantization and mixture-of-experts

Quantization

Each parameter is a number. Stored in 16-bit floating point (FP16 or BF16), it takes 2 bytes. Quantization stores weights in fewer bits:

  • 8-bit roughly halves memory versus FP16 with very small quality loss for most models.
  • 4-bit roughly quarters it. Quality loss is usually modest for large models and more noticeable for small ones; good methods keep the most sensitive weights at higher precision.
  • GGUF is a file format from the llama.cpp project that packages quantized weights for efficient CPU, Apple silicon and GPU inference. Ollama and many desktop and mobile apps use it.

Less memory means a model fits on cheaper hardware and often runs faster, since inference is frequently limited by how fast weights can be read from memory. For memory sizes and speed estimates, see the AI hardware guide.

Mixture-of-experts

A mixture-of-experts (MoE) model replaces each feed-forward block with many smaller "experts" and a router that sends each token to only a few of them. So a model has a large total parameter count (all experts, which must sit in memory) but a much smaller active count per token (the compute actually spent). Mixtral made the idea popular among open models, and many leading models now use it. The practical upshot: an MoE model runs at roughly the speed of a model its active size, while needing memory for its total size.

Multimodal, open-weight and closed models

Multimodal models

Many current models accept images, and some audio and video, alongside text. Images are split into patches and turned into embeddings the transformer can attend to, just like tokens. That lets one model read a photographed bill, a handwritten prescription or a screenshot of an error. Some models also generate images or speech.

Open-weight vs closed

Closed models are available only through a provider's API or apps. They are typically the most capable and easiest to start with, but your data goes to the provider and you depend on their pricing and changes.

Open-weight models publish the trained weights, under licences that vary, so you can run them on your own servers or devices, fine-tune them and keep data in-house. "Open-weight" is more precise than "open-source", since the training data and code are often not released. For data residency, offline use and high-volume work, open-weight models are often the better fit.

Tool use: how LLMs become agents

On its own, an LLM can only produce text. Tool use (function calling) changes that: you describe functions such as check_stock(sku) or create_invoice(customer, items), and the model can respond with a structured request to call one. Your code runs it and returns the result, and the model continues with real data.

Put that in a loop, where the model plans a step, calls a tool, reads the result and decides what to do next, and you have an AI agent. Agents can work through multi-step jobs like reconciling payments or following up on unpaid invoices over WhatsApp, with approvals before anything risky. The design of those systems is covered in the companion guide on agentic architecture.

How to choose an LLM

  1. Define the task and a test set. Collect 30 to 50 real examples with good answers. Public leaderboards are a starting point, not a decision.
  2. Decide on data constraints. If data cannot leave your servers or the country, shortlist open-weight models you can host.
  3. Check languages. If customers write in Hindi, Marathi or Hinglish, test in those languages; quality varies a lot between models.
  4. Start large, then shrink. Get it working with a capable model, then test whether a smaller or quantized one holds up.
  5. Weigh latency and cost per task, not per token. A reasoning model that solves the task in one try can be cheaper than a fast model that needs three.
  6. Check tool calling and structured output if you plan to build agents.

Small models on phones

Models with a few billion parameters, quantized to around 4 bits, now run fully offline on recent phones. They will not match large server models on hard reasoning, but they handle summarising, drafting, translation and simple question answering with no network, no per-query cost and no data leaving the device. In my lab we run 4-bit models entirely offline on mobile in Indian languages, which matters for users with patchy connectivity or sensitive data.

LLM glossary

Parameter
A learned number inside the model. Model size is usually quoted in parameters, for example 7B for seven billion.
Token
The unit of text a model reads and writes, often a short word or part of a word.
Embedding
A vector of numbers representing a token, sentence or document, where similar meanings sit close together.
Transformer
The neural network architecture behind nearly all modern LLMs, built around attention.
Attention
The mechanism that lets each token weigh information from other tokens in the context.
Context window
The maximum number of tokens a model can consider in one request, including its reply.
KV cache
Stored attention keys and values from earlier tokens that speed up generation at the cost of memory.
Inference
Running a trained model to produce output, as opposed to training it.
RAG
Retrieval-augmented generation: fetching relevant documents and adding them to the prompt.
Fine-tuning
Further training a model on your examples to change its behaviour.
LoRA
A fine-tuning method that trains small adapter matrices instead of all weights.
Quantization
Storing weights in fewer bits, such as 8 or 4, to save memory and speed up inference.
Mixture-of-experts
An architecture where each token uses only a few of many expert sub-networks.
Hallucination
Confident model output that is false or unsupported.
Agent
A system where an LLM repeatedly plans, calls tools and checks results to reach a goal.

Questions people ask

What is an LLM in simple terms?

An LLM is a computer program that has read a huge amount of text and learned to predict which word, or piece of a word, comes next. Repeating that prediction many times lets it write answers, summaries, code and translations.

How does a large language model work?

It splits text into tokens, turns each token into a vector of numbers, and passes those vectors through many transformer layers that use attention to mix in context. The final layer gives a probability for every possible next token, one is chosen, and the process repeats.

Is ChatGPT an LLM?

ChatGPT is a chat product built on top of large language models made by OpenAI. The LLM is the underlying model; the product adds a chat interface, safety systems, tools such as search, and memory.

Why do LLMs hallucinate?

An LLM is trained to produce likely-sounding text, not to check facts. When it lacks the right information, it can still produce a fluent, confident answer that is wrong. Retrieval, citations and verification reduce this but do not remove it.

What is the difference between RAG and fine-tuning?

RAG gives the model relevant documents at question time, so it can answer from your current data. Fine-tuning changes the model's weights with extra training, which is better for teaching a style, format or narrow skill than for adding facts that change often.

What is a token in an LLM?

A token is the unit of text a model reads and writes, often a whole short word or part of a longer word. Pricing, speed and context limits are all measured in tokens.

Can an LLM run on a phone?

Yes. Small models with a few billion parameters or fewer, compressed to around 4 bits per weight, can run fully offline on recent phones, though they are less capable than large server models.

What is the difference between open-weight and closed models?

Open-weight models publish their trained weights so you can download, run and fine-tune them on your own hardware. Closed models are only available through the provider's API or apps.