Let's talk

Infrastructure for Autonomy

Building an India Sovereign LLM: The Hardware Guide

To achieve true AI sovereignty, you must move from APIs to owned hardware. This guide explains the math and memory required to run private LLMs locally.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for Building an India Sovereign LLM: The Hardware Guide

An India sovereign LLM is a large language model run on hardware you control, ensuring data residency and independence from external API providers. To achieve this, you must understand the relationship between model parameters, VRAM, and memory bandwidth to select the right local infrastructure.

What is an India Sovereign LLM?

  • A sovereign LLM is a model run on hardware you control, ensuring data residency and independence from external API providers.
  • Achieving sovereignty requires understanding the relationship between model parameters, VRAM, and memory bandwidth.
  • For builders in India, this means moving from cloud APIs to local GPUs, workstations, or dedicated on-premise servers.

For businesses in sectors like finance or healthcare in India, sovereignty is about more than just speed; it is about ensuring that sensitive customer data never leaves your local network. By running models on your own hardware, you move away from the uncertainty of third-party API availability and pricing shifts. This transition is the foundation of sovereign AI in India.

The Golden Rule: Memory First, Bandwidth Second

To run an LLM, you need enough fast memory to hold the model weights plus the KV cache for your context length. If the model does not fit in your memory, it will not run, or it will run incredibly slowly by swapping to your system RAM.

A rough rule for sizing memory: parameters × bytes per parameter. The number of bytes depends on the precision (quantization) you use:

  • FP16/BF16: 2 bytes per parameter.
  • 8-bit: 1 byte per parameter.

For example, an 8B model needs about 16 GB at FP16, 8 GB at 8-bit, and 4 to 5 GB at 4-bit. Once the model fits, memory bandwidth determines your tokens per second (speed).

How Much VRAM Do You Need?

Model weights are a fixed cost, but the KV cache (which stores past tokens) grows with every token of context. This cache allows the model to "remember" the beginning of a long conversation without recomputing everything every time.

A model that "just fits" will fail or slow down once a long conversation fills the KV cache. You must leave headroom for the context window and the software itself. Always budget 1 to 2 GB of runtime overhead for the framework and activations.

Model Size 4-bit Weights (Approx) Recommended VRAM
3B 1.5–2 GB 6 GB+
8B 4–5 GB 8–12 GB
32B 16–20 GB 24–32 GB
70B 35–42 GB 48 GB or larger

For a 70B model at 4-bit, you need 35–42 GB of VRAM for weights, plus significant headroom for context. If you are serving many users simultaneously, the KV cache requirement grows linearly with the number of users and their context lengths.

Choosing Your AI Hardware Tier

Hardware for AI generally falls into three distinct categories based on your scale and budget.

  • Consumer GPUs (e.g., RTX 4090): Best bandwidth per rupee, but limited VRAM (24 GB GDDR6X). Good for up to 32B models at 4-bit. These are great for individual developers or small-scale testing.
  • Workstation GPUs (e.g., RTX PRO 6000 Blackwell): High VRAM (96 GB GDDR7) and ECC memory; ideal for 70B models and small-team serving. These are designed to run continuously in a professional environment.
  • Datacenter GPUs (e.g., H100/H200): Use HBM memory for massive bandwidth; built for high-throughput serving and training. These are the heavy lifters used for massive scale or training models from scratch.

For those focused on on-device AI, the choice often depends on whether you need raw capacity or raw speed.

Alternative Hardware: Apple Silicon and CPUs

Not all AI hardware is a dedicated graphics card. There are several alternative paths depending on your budget and use case.

Apple Silicon

Apple Silicon (M-series) uses unified memory, allowing a Mac to load massive models that would require multiple GPUs. A Mac with a large memory configuration can load models that would otherwise require several consumer GPUs. While bandwidth is lower than top-tier NVIDIA cards, the capacity is incredibly high and the system is efficient.

CPUs and System RAM

Running on a CPU via llama.cpp is possible but limited by memory bandwidth. For example, dual-channel DDR5-5600 has a theoretical peak of roughly 90 GB/s, which is about a tenth of a high-end GPU. A 70B model might top out at about 2 tokens per second on a typical desktop. CPU inference is best for small models, background batch jobs, or embeddings.

NPUs

NPUs in phones and laptops are designed for small (1B–4B) models for tasks like transcription or smart replies. These are highly optimized for low power rather than large-scale reasoning.

Why Bandwidth Dictates Speed

When generating text (decoding), the chip must read every active weight from memory once per token. Because the mathematical operations are so simple, the chip spends most of its time waiting for data to arrive from the memory. This is why memory bandwidth is the primary bottleneck for inference speed.

A useful ceiling to estimate your speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token.

Mixture-of-Experts (MoE) models, like Gemma 4 26B, trade memory capacity for speed by only activating a fraction of weights per token. For example, a model might have 26 billion total parameters but only activate about 3.8 billion per token. This means it requires memory like a 26B model but decodes at a speed closer to a 4B model.

The Decision: Renting vs. Buying Hardware

Deciding whether to invest in physical hardware or stick to the cloud is a critical business decision. Consider your expected usage and data sensitivity.

Situation Better Choice Why?
Exploring models, a few hours a week Rent (Cloud/API) No upfront cost; try different GPUs.
Short training or fine-tuning bursts Rent Datacenter GPUs Cheaper than buying for short bursts.
Bulk generation many hours every day Buy (On-premise) Fixed cost beats per-token/hour bills.
Sensitive data must stay on your machine Buy (On-premise) You control where data goes.
Spiky public traffic Rent or API Scale up only when demand arrives.

Software to Run Your Sovereign Stack

Once you have the hardware, you need the right software layer to manage your models and serve requests.

Ollama
The easiest way to download and run open models locally, with a simple API; built on llama.cpp.
llama.cpp
A lean C/C++ engine that runs on CPUs, NVIDIA, AMD, and Apple hardware using quantized models.
vLLM
High-throughput server with paged KV-cache management; best for teams or applications sending many concurrent requests.
SGLang
Fast serving framework that reuses shared prompt prefixes, useful for agents or structured output.
Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

How much VRAM do I need for a 7B model?

At 4-bit quantization, a 7B model needs about 3.5 GB for weights, plus 1-2 GB for overhead and additional space for the KV cache.

Why is my LLM slow on my laptop?

Speed is limited by memory bandwidth. While a GPU uses fast VRAM, a CPU uses system RAM, which has significantly lower bandwidth.

What is the difference between a GPU and a CPU for AI?

A CPU has a few powerful cores for complex logic, while a GPU has thousands of simpler cores designed for the massive parallel math required for AI.

Can I run a 70B model on a consumer GPU?

A single 4090 (24 GB) cannot fit a 70B model at high precision, but you can run a 4-bit quantized version if you have enough VRAM or use multi-GPU setups.

What is the KV cache?

The KV cache stores key and value vectors for past tokens so the model doesn't have to recompute them, allowing for faster text generation.