Let's talk

LLM Essentials

AI Explained: How Large Language Models Work

Large Language Models (LLMs) are neural networks trained to predict the next token in a sequence. By scaling this simple skill, they can reason, code, and converse.

By Divyaprakash JhaUpdated 5 min readTopic: Large language models

Photo for AI Explained: How Large Language Models Work

What is an LLM?

A Large Language Model (LLM) is a neural network designed to predict the next piece of text in a sequence. By scaling this simple prediction task across massive datasets, these models gain the ability to reason, code, and converse like humans.

At its core, an LLM is a mathematical engine. When you ask a model to draft a GST reminder for a customer, it does not look up a stored letter in a database. Instead, it produces the reply one token at a time, choosing the most likely next piece of text based on everything that came before it.

The "Large" in LLM refers to the number of parameters—the adjustable numbers the model learns during its training process. These can range from a few hundred million in small, efficient models to hundreds of billions or even more in the most powerful systems.

Tokens and Embeddings

To process language, models do not read letters or whole words. They use a tokenizer to break text into smaller units called tokens. Common words might become a single token, while rare or complex words are split into several pieces.

This has practical implications for business owners using models in different languages. For example, languages that were under-represented in a tokenizer's training data, such as many Indian languages written in their own scripts, often require more tokens to express the same sentence. This can make those languages slower and more expensive to process on certain models.

Once text is tokenized, each token is mapped to an embedding. An embedding is a list of numbers (a vector) that represents the meaning of the token. In this mathematical space, tokens with similar meanings result in similar vectors.

This vector math is the engine behind semantic search. By embedding your company documents and a user's question into these same numerical spaces, a system can find the documents whose vectors are closest to the question. This is the foundation of AI techniques like Retrieval-Augmented Generation (RAG).

The Transformer and Attention

Modern LLMs are built on the transformer architecture. The breakthrough of the transformer is a mechanism called attention. When a model processes a word, attention allows it to look at every other word in the context and decide how much weight or importance to give to each one.

Consider the sentence: "The clinic called the patient because she was late." Attention is what allows the model to work out that the word "she" most likely refers to the patient, not the clinic.

The architecture works through several layers:

  • Attention Heads: Several attention mechanisms run in parallel, each tracking different types of relationships between words.
  • Feed-Forward Networks: After attention, each token passes through a block that holds much of the model's stored knowledge.
  • Refinement: Each successive layer refines the token's representation using the context provided by the layers before it, building up complex patterns of grammar and reasoning.

How LLMs are Trained

Training an LLM happens in stages, moving from general knowledge to specific instruction-following behavior.

1. Pretraining

During pretraining, the model is shown trillions of tokens of text and code. It is asked one question repeatedly: "What comes next?" By making billions of guesses and adjusting its parameters based on errors, the model creates a base model. This model is incredibly knowledgeable but lacks the ability to follow instructions; it simply wants to continue the text it has been given.

2. Instruction Tuning

To make the model useful for humans, it undergoes supervised fine-tuning. It is trained on examples of instructions and the correct responses, teaching it the format of being an assistant—such as answering a question directly rather than just completing a sentence.

3. Preference Tuning

Finally, models undergo preference tuning (such as RLHF or RLAIF) to shape their tone and safety. In Reinforcement Learning from Human Feedback (RLHF), humans rank different answers, and the model is optimized to favor the higher-ranked responses. This ensures the model is helpful and avoids generating harmful content.

Reasoning and Sampling

Not all AI responses are the same. Depending on your business goal, you may need different "thinking" styles or "selection" methods.

Reasoning Models

A new family of reasoning models uses "test-time compute" to solve hard problems. Instead of replying instantly, they generate a long internal chain of reasoning, trying different approaches and catching their own mistakes before they give you a final answer. This is excellent for complex math or coding, but it is slower and more expensive than standard models.

Sampling Settings

When a model generates text, it calculates a probability for every possible next token. "Sampling" is the method used to pick the actual word. You can control this using two main settings:

Setting Low Value (Near 0) High Value
Temperature Consistent, predictable, and focused. Best for code or data extraction. Varied, creative, and diverse. Best for brainstorming.

Another common setting is Top-p (nucleus sampling). This limits the model to choosing from only the smallest set of tokens whose probabilities add up to a certain percentage, effectively cutting off the "long tail" of unlikely, nonsensical words.

Making AI Useful for Business

A general-purpose model like GPT-4 is powerful, but to make it work for your specific business, you have three main levers. You should generally try them in this order:

  1. Prompting: Changing the instructions or examples you send in the message. This is low effort and good for changing tone or format.
  2. RAG (Retrieval-Augmented Generation): Adding your own documents (like a price list or manual) to the context so the model answers from your specific data. This is the best way to keep facts current.
  3. Fine-tuning: Updating the actual weights of the model. This is higher effort and best for teaching a model a very specific, consistent style or narrow, specialized skill.
Method What it changes Best for...
Prompting The instructions and examples you send Format, tone, simple tasks
RAG The documents you add to the context Answering from your own, changing data
Fine-tuning The model's actual weights Consistent style, narrow skills

From LLMs to AI Agents

An LLM on its own is essentially a very advanced text generator. It can talk about a task, but it cannot perform it. To move from a "chatbot" to an "agent," we use tool use (also known as function calling).

Tool use allows you to describe specific functions to the model, such as check_stock(sku) or create_invoice(customer_id). The model doesn't run the code itself; instead, it outputs a structured request that your software can execute. Your software then feeds the result back to the model.

When a model can plan a multi-step job—such as checking an unpaid invoice, looking up the customer's contact details, and then drafting a follow-up message—it has become an AI agent. This allows automation of complex workflows, like reconciling payments or managing inventory, with a human in the loop to approve any final, high-risk actions.

Close-up of a computer screen displaying lines of CSS code with color swatches.
Photo: Free computer code screen image by unknown, CC0

Questions people ask

Why do AI models hallucinate?

Hallucinations happen because models are trained to produce plausible-sounding text rather than to know when they lack facts. They are predicting the most likely next word, not retrieving a database.

What is the difference between RAG and fine-tuning?

RAG is best for keeping facts current by adding documents to the context at question time. Fine-tuning is best for teaching a model a consistent style or a specific behavior.

What is a context window?

The context window is the maximum number of tokens a model can consider at once, including your instructions, conversation history, and the model's own previous replies.

What are open-weight models?

Open-weight models publish the trained weights, allowing you to host them on your own servers for data residency, offline use, or high-volume work.

Can AI run on my phone?

Yes. Small models with a few billion parameters, when quantized (compressed) to 4-bit, can run fully offline on recent mobile devices.