Generative AI is driven by Large Language Models (LLMs) that predict the next token in a sequence to generate human-like text, code, and reasoning. These models use advanced architectures to understand context and follow complex instructions.
How Generative AI Works
At its core, modern generative AI relies on a specific type of neural network called a transformer. This architecture uses a mechanism known as "attention," which allows the model to look at every word in a sentence and decide which ones are most important to the meaning. For example, in the sentence "The clinic called the patient because she was late," the attention mechanism helps the model understand that "she" refers to the patient, not the clinic.
The training process happens in several distinct stages:
- Pretraining: The model is shown trillions of pieces of text and asked to predict the next piece of text. By doing this billions of times, it learns grammar, facts, and reasoning patterns. This results in a "base model" that is knowledgeable but mostly just continues text rather than answering questions.
- Instruction Tuning: To make the model useful for humans, it undergoes supervised fine-tuning. This teaches the model the format of being an assistant—learning to follow commands and stop when a task is done.
- RLHF (Reinforcement Learning from Human Feedback): Humans rank different model responses, and a reward model is trained to understand those preferences. This shapes the model's tone, safety, and helpfulness.
For a deeper dive into the underlying math, see our guide on machine learning.
Tokens and Embeddings
It is a common misconception that models read words or letters. Instead, they use a tokenizer to split text into "tokens." A token might be a whole common word, or it might be several chunks of characters for a rare word. This has significant practical implications for business owners:
- Cost and Speed: Pricing, speed, and context limits are all measured in tokens, not words.
- Language Disparity: Languages that were under-represented in training data—including many Indian languages written in their own scripts—often require more tokens to represent the same sentence. This makes them slower and more expensive to process on many models.
Once a token is identified, it is converted into an embedding. An embedding is a list of numbers (a vector) that represents the meaning of the token. In this mathematical space, tokens with similar meanings are placed close to each other. This concept is what enables "semantic search," where a computer finds information based on meaning rather than just matching keywords.
Understanding Hallucinations
A "hallucination" occurs when a model produces a response that is fluent and confident but factually incorrect. A model might invent a fake legal citation, provide a wrong GST rate, or claim a clinic has a policy that does not exist. This happens because the model is trained to produce plausible text rather than to "know" facts. It is essentially making its best-sounding guess based on patterns.
To reduce these errors, you can use Retrieval-Augmented Generation (RAG). By providing the model with specific source material (like your company's actual handbook) and instructing it to answer only from that text, you significantly increase accuracy.
RAG vs. Fine-Tuning
Business owners often ask whether they should fine-tune a model or use RAG. The answer depends on whether you want to change the model's knowledge or its behavior. Use the following guide to decide your strategy:
| Method | What it changes | Best for... | Effort Level |
|---|---|---|---|
| Prompting | Instructions and examples in the chat | Changing tone or format (e.g., "Reply in polite Hindi") | Low |
| RAG | Documents added to the context at runtime | Answering questions from your specific, changing business data | Medium |
| Fine-Tuning | The model's internal weights | Teaching a consistent style or narrow, specialized skills | High |
In short: RAG is the best way to give a model access to your current data without the need for constant retraining.
The Rise of Reasoning Models
A new class of models, often called "reasoning models," has emerged. Unlike standard models that predict the next token immediately, these models use reinforcement learning to "think" through a problem before they reply. They use what is known as test-time compute—running an internal chain of reasoning to catch mistakes before they show the answer to the user.
These models are highly effective for complex, multi-step tasks like solving math problems or writing intricate code. However, they are not always the most efficient choice. For simple tasks like "rewrite this email to be more professional," a standard, faster model is usually better and more cost-effective.
Open-Weight vs. Closed Models
When building an AI application, you must choose between closed and open-weight models. This choice impacts your privacy, cost, and control.
- Closed Models: These are accessed via an API (like a service provided by a large tech company). They are very easy to start with and highly capable, but you must send your data to the provider, and you are subject to their pricing and availability.
- Open-Weight Models: These allow you to host the model on your own servers or devices. This is often the better fit for high-volume work, strict data residency requirements, or if you need the system to work offline.
For more on the technical differences in how these models are structured, see our page on deep learning.
From LLMs to AI Agents
An LLM on its own is just a text generator. To make it useful for business automation, we use a concept called tool use (or function calling). This allows the model to interact with the real world by requesting to run specific functions, such as check_stock(item_id) or create_invoice(customer_id).
When you combine tool use with a loop—where the model plans a step, calls a tool, reads the result, and decides on the next action—you have an AI Agent. An agent can handle multi-step workflows, such as reconciling payments or following up on unpaid invoices via WhatsApp, often requiring human approval before taking any high-risk actions.
How to Choose an LLM
Choosing the wrong model can lead to wasted budget or poor user experiences. Follow this workflow to find the right fit for your business:
- Define and Test: Start by defining your task. Collect 30 to 50 real-world examples of inputs and the "perfect" answers you expect.
- Start Large: Use a large, highly capable model to prove the concept and build your test set.
- Shrink for Scale: Once the workflow is proven, try moving to a smaller or quantized model (a compressed version) to save on cost and speed up response times.
- Check Language Quality: If your customers communicate in Hindi, Marathi, or Hinglish, test the models specifically in those languages to ensure quality.
- Evaluate Logic vs. Speed: Decide if you need a "reasoning model" for complex logic or a fast, simple model for tasks like data extraction.
To explore more about the tools available for these tasks, visit our AI tools directory.
Frequently Asked Questions
- What is a hallucination in AI?
- A hallucination is when a model generates a fluent, confident, but factually incorrect answer because it is predicting plausible text rather than retrieving specific facts.
- What is the difference between RAG and fine-tuning?
- RAG provides the model with external documents to read at the moment of the query, while fine-tuning updates the model's internal weights to learn new patterns or styles.
- What are tokens in an LLM?
- Tokens are the basic units of text (like words or parts of words) that a model reads and writes. They are the basis for measuring cost, speed, and context limits.
- What is a transformer architecture?
- A transformer is a neural network architecture that uses an "attention" mechanism to weigh the importance of different words in a sentence to understand context.
- Can AI run offline on a phone?
- Yes, small models with a few billion parameters, quantized to around 4 bits, can run fully offline on recent phones. This is useful for users with patchy connectivity or for handling sensitive data.

Questions people ask
What is a hallucination in AI?
A hallucination is when a model generates a fluent, confident, but factually incorrect answer because it is predicting plausible text rather than retrieving facts.
What is the difference between RAG and fine-tuning?
RAG provides the model with external documents to read at the moment of the query, while fine-tuning updates the model's internal weights to learn new patterns or styles.
What are tokens in an LLM?
Tokens are the basic units of text (like words or parts of words) that a model reads and writes. They are the basis for measuring cost, speed, and context limits.
What is a transformer architecture?
A transformer is a neural network architecture that uses an 'attention' mechanism to weigh the importance of different words in a sentence to understand context.
Can AI run offline on a phone?
Yes, small models with a few billion parameters can run fully offline on recent phones if they are quantized to around 4 bits.
