Let's talk

Founder's Blueprint

Building an AI Startup in India

Leverage Large Language Models to build scalable, agentic solutions for the Indian market and beyond.

By Divyaprakash JhaUpdated 5 min readTopic: Large language models

Photo for Building an AI Startup in India

The Core: Large Language Models

Building an AI startup in India requires mastering Large Language Models (LLMs) to create intelligent, automated solutions. By leveraging these neural networks, founders can build everything from simple chatbots to complex agents capable of handling real-world business workflows.

At their simplest, LLMs are neural networks—specifically using the transformer architecture—trained on massive datasets to predict the next token in a sequence. Think of it like a highly advanced version of autocomplete. When you ask a model to draft a GST reminder for a customer, it does not look up a stored letter; instead, it produces the reply one token at a time, choosing the most likely next piece of text based on everything that came before it.

The "size" of a model refers to its parameters, which are the adjustable numbers the model learns during training. These can range from a few hundred million in small, efficient models to hundreds of billions in the most capable systems. While these models can answer questions, write, translate, and code, it is important to remember that they do not "know" facts; they generate plausible text, which is why they can sometimes hallucinate incorrect information.

Key Concepts for Builders

  • Transformers: The underlying architecture that uses "attention" to weigh the importance of different words in a sentence.
  • Tokens: The basic units of text. Common words might be one token, while rare words are split into several.
  • Parameters: The internal variables that determine the model's knowledge and reasoning capabilities.

Making Models Useful for Business

A raw base model is knowledgeable but often lacks the specific context or behavior your business needs. To turn a general model into a specialized product, you have three primary levers: prompting, RAG, and fine-tuning. You should generally try them in this order of complexity.

Method What it changes Best for Effort
Prompting The instructions and examples you send Format, tone, and simple tasks Low
RAG The documents you add to the context at question time Answering from your own, changing data Medium
Fine-tuning The model's actual weights Consistent style or narrow, specialized skills Higher

For most startups, Retrieval-Augmented Generation (RAG) is the most effective way to make a model "know" your data. By embedding your documents into vectors, the model can find the most relevant information to answer a specific query. Fine-tuning, on the other hand, is better suited for teaching a model a very specific behavior, such as ensuring a small model always extracts invoice fields into a precise JSON format.

From Chatbots to AI Agents

The next frontier for AI startups is moving from passive chatbots to active agents. While a standard LLM can only produce text, "tool use" or function calling allows the model to interact with the world. By describing functions like check_stock(sku) or create_invoice(customer, items), the model can request to run specific code.

An AI agent is essentially an LLM placed in a loop. The model plans a step, calls a tool, reads the result, and decides the next step based on that information. This enables agents to execute multi-step, complex jobs that would otherwise require human intervention.

Real-World Agentic Use Cases

  • Finance: Reconciling payments by checking bank statements against internal ledgers.
  • Operations: Following up on unpaid invoices via WhatsApp and flagging those that remain unpaid.
  • Customer Support: Checking real-time shipping status and updating customers automatically.

Open-Weight vs. Closed Models

Founders must decide how to host their intelligence. This choice impacts your data privacy, costs, and ability to operate offline.

Feature Closed Models (via API) Open-Weight Models
Ease of Use Very High (Plug and play) Moderate (Requires hosting)
Data Privacy Data is sent to the provider Data stays on your servers
Customization Limited to API features High (Can be fine-tuned/quantized)

Closed models are excellent for rapid prototyping. However, open-weight models are often the better fit for high-volume work or industries requiring strict data residency. Furthermore, you can use quantization (such as 4-bit precision) to make these models run on much cheaper hardware or even directly on mobile devices without needing a constant internet connection.

Navigating the Indian Context

Building for the Indian market presents unique technical challenges that global-first models might overlook. If your product targets the next billion users, you must account for language, connectivity, and cost.

The Tokenization Tax

Most LLMs use a "tokenizer" to break text into chunks. Many tokenizers are optimized for English. For Indian languages written in their own scripts, the tokenizer often needs more tokens to represent the same sentence. This means processing Hindi or Marathi can be slower and more expensive than processing English.

Connectivity and Edge AI

In many parts of India, internet connectivity can be patchy. A massive advantage for local startups is the ability to run small, quantized models (around 4 billion parameters) offline on a smartphone. This allows for features like instant translation or document summarization without any per-query cost or reliance on a data connection.

How to Choose Your AI Stack

Choosing the right model and workflow is a balance of performance, latency, and cost. Don't get caught in "model chasing"; instead, focus on solving the specific problem for your users.

  1. Start Large, Then Shrink: Use a highly capable, large model to prove your concept works. Once the workflow is proven, try to move to a smaller, cheaper, or quantized model to save on margins.
  2. Build a Test Set: Do not rely on public leaderboards. Define a test set of 30 to 50 real-world examples (e.g., 50 different customer queries) and measure how your model handles them.
  3. Prioritize Reasoning for Complexity: For hard, multi-step problems, use "reasoning models." These models spend more compute time "thinking" through a problem before they reply. While they have higher latency, they are often cheaper than running a fast model three times to get the same result.

Quick Reference: When to use what?

  • Use RAG: When you need the model to know your specific, changing business facts.
  • Use Fine-tuning: When you need a very specific, consistent style or a tiny, specialized model.
  • Use Agents: When the task requires taking actions in other software (like sending an email or checking a database).

Frequently Asked Questions

How do I prevent AI hallucinations in my startup's product?
Use RAG to provide the model with source material and instruct it to answer only from those facts. Always keep a human in the loop for legal, medical, or financial tasks.
What is the difference between RAG and fine-tuning?
RAG is best for keeping facts current by adding documents to the context at question time. Fine-tuning is better for teaching a model a specific, consistent behavior or style.
Can I run AI models on a smartphone?
Yes, models with a few billion parameters, quantized to around 4 bits, can run fully offline on recent mobile devices.
Why are some languages more expensive to process in LLMs?
Many tokenizers use fewer tokens for English than for Indian languages in native scripts, meaning the latter requires more tokens to represent the same sentence.
What is a reasoning model?
These are models trained with reinforcement learning to "think" through a long internal chain of reasoning before replying, making them better at complex, multi-step problems.
Close-up of a computer screen displaying lines of CSS code with color swatches.
Photo: Free computer code screen image by unknown, CC0

Questions people ask

How do I prevent AI hallucinations in my startup's product?

Use RAG to provide the model with source material and instruct it to answer only from those facts. Always keep a human in the loop for legal, medical, or financial tasks.

What is the difference between RAG and fine-tuning?

RAG is best for keeping facts current by adding documents to the context at question time. Fine-tuning is better for teaching a model a specific, consistent behavior or style.

Can I run AI models on a smartphone?

Yes, models with a few billion parameters, quantized to 4 bits, can run fully offline on recent mobile devices.

Why are some languages more expensive to process in LLMs?

Many tokenizers use fewer tokens for English than for Indian languages in native scripts, meaning the latter requires more tokens to represent the same sentence.

What is a reasoning model?

These are models trained with reinforcement learning to 'think' through a long internal chain of reasoning before replying, making them better at complex, multi-step problems.