Let's talk

Building Local AI

Sovereign AI India: The Hardware Foundation

To achieve true AI sovereignty, India needs robust local compute. This guide explains the hardware math required to run large language models on your own infrastructure.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for Sovereign AI India: The Hardware Foundation

What is Sovereign AI Infrastructure?

Sovereign AI infrastructure provides the local compute power necessary to run large language models on hardware you control. By moving workloads from paid APIs to on-premise or dedicated local servers, businesses can ensure data residency and avoid unpredictable API costs.

For a business owner in India, "sovereignty" means having the ability to process sensitive customer data or proprietary business logic without sending it to a third-party provider's cloud. It is about moving away from a dependency on external API limits and pricing fluctuations.

The goal of building local infrastructure is to route bulk, repeatable work—such as processing thousands of support tickets or analyzing local language documents—to hardware that you own. This creates a predictable cost structure and ensures that your AI capabilities remain functional even if external services change their terms or availability.

The Memory Rule: Calculating VRAM Needs

When building local AI, memory is your first and most critical constraint. You must have enough fast memory (VRAM) to hold the model weights themselves, plus extra space for the "KV cache" (which handles the conversation history) and runtime overhead.

A useful rule of thumb for calculating the memory needed for model weights is: parameters × bytes per parameter. The number of bytes depends on the precision of the model.

Here is how the math looks for a standard 8B (8 billion parameter) model:

  • FP16 (Full precision): 8B × 2 bytes = 16 GB
  • 8-bit (Quantized): 8B × 1 byte = 8 GB

Note: Never aim for a setup that "just fits." You must always leave at least 1–2 GB of headroom for runtime overhead and activations. If you run out of VRAM, the system will either fail or slow down significantly as it tries to use your much slower system RAM.

Model Size FP16 Weights 8-bit Weights 4-bit Weights Typical GPU (at 4-bit)
3B ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB Any 6–8 GB card
8B ≈ 16 GB ≈ 8 GB ≈ 4–5 GB 8–12 GB card
32B ≈ 64 GB ≈ 32 GB ≈ 16–20 GB 24–32 GB card
70B ≈ 140 GB ≈ 70 GB ≈ 35–42 GB 48 GB or larger

Why Context Length Demands More Memory

While model weights are a fixed cost, the memory required to actually *use* the model changes as the conversation progresses. This is due to the KV cache.

As a transformer model generates text, it stores "key" and "value" vectors for every past token in the conversation. This prevents the model from having to recompute the entire history every time it generates a new word. However, this cache grows linearly with the length of the context and the number of users you are serving simultaneously.

Consider a Llama 3.1 8B style model using 16-bit precision for the cache. A context of 8,192 tokens requires about 1 GiB of cache. If you want to use a massive 128K context window, that cache alone requires about 16 GiB—which is more than the weights of the 4-bit model itself!

For businesses looking to serve many users at once, the memory demand is even higher. If you serve ten users each with an 8K context, you need an additional 10 GiB of VRAM just for the cache. To manage this, many modern models use Grouped-query attention (GQA), which allows several "query" heads to share a single "KV" head, significantly shrinking the cache size.

Memory Bandwidth: The Speed Limit

If VRAM capacity determines *if* a model can run, memory bandwidth determines *how fast* it runs. Once a model fits in your memory, the speed at which it generates text (tokens per second) is largely dictated by how quickly data can move from the memory to the processor.

During the "decoding" phase (when the model is streaming words to you), the chip spends most of its time waiting for the next weight to arrive from memory. A rough ceiling for your speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token.

For example, an RTX 4090 has a bandwidth of about 1,008 GB/s. If you are running a 4-bit 8B model (which is about 4.5 GB), the theoretical limit is roughly 220 tokens per second. In the real world, you might see 50% to 80% of that due to overhead, but it is a great way to compare hardware.

This is why high-bandwidth memory (HBM) found in datacenter cards is so much faster than the GDDR memory found in consumer graphics cards.

Choosing the Right GPU Tier

Not all GPUs are created equal. Depending on whether you are a developer testing a new idea or a company deploying a service, your choice will change.

GPU Tier Example Memory Type/Cap | Bandwidth Best Use Case
Workstation RTX PRO 6000 Blackwell 96 GB GDDR7 (w/ ECC) | ~1,792 GB/s 70B models, large MoE models, small-team serving.

Consumer cards are excellent for individual builders, but they lack the memory capacity for large models and the cooling required for 24/7 operation. Workstation cards offer ECC (Error Correction Code) memory, which is vital for stability during long tasks. Datacenter cards are built for massive scale and use HBM to achieve incredible speeds, but they require specialized server racks and power setups.

Mixture-of-Experts (MoE) for Efficiency

A new way to get high performance without massive hardware is through Mixture-of-Experts (MoE) architectures. Instead of activating every single parameter for every single word, an MoE model splits its layers into "expert" sub-networks.

When a token is processed, the model only activates a small fraction of its total weights. This means you can have a model with a huge memory footprint (so it is very smart) but a very fast decoding speed (because it only reads a few parameters per token).

A great example is Gemma 4 26B. It has roughly 25 billion total parameters, but it only activates about 3.8 billion parameters per token. This allows it to act like a much smaller, faster model while retaining the knowledge of a much larger one. This makes MoE models an excellent choice for hardware with large VRAM but limited bandwidth.

Strategic Deployment: Renting vs. Buying

For an Indian business owner, the decision to buy hardware or rent cloud GPUs is a financial one. There is no single "correct" answer; it depends on your usage patterns.

Situation Best Choice Why?
Exploring models or short bursts of training Rent No upfront cost; try different GPUs without commitment.
Bulk generation (many hours every day) Buy or Lease Fixed cost beats per-token or per-hour bills at high use.
Sensitive data must stay on-premise Buy (On-premise) You have total control over where your data goes.
Spiky public traffic Rent or API Scale up only when demand arrives.

A simple way to decide is to estimate your monthly GPU hours. If you expect your hardware to be busy most of the day for a year or more, owning a dedicated machine usually wins. If the machine will sit idle most of the time, renting is much more cost-effective.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

How much VRAM do I need for a 70B model?

At 4-bit quantization, a 70B model requires about 35–42 GB of VRAM. You should also add headroom for the KV cache and runtime overhead.

What is the difference between a CPU and a GPU for AI?

A CPU has a few powerful cores for complex logic, while a GPU has thousands of simpler cores designed for parallel matrix multiplication, which is the core of AI work.

Why is memory bandwidth important for LLMs?

During text generation, the chip must read every active weight from memory for every token. This makes the process memory-bandwidth bound rather than compute-bound.

Can I run LLMs on a Mac?

Yes, Apple Silicon uses unified memory, allowing the GPU to access a large pool of system RAM. This can load models that would require multiple consumer GPUs.

What is the benefit of quantization?

Quantization (e.g., 4-bit or 8-bit) reduces the number of bytes used per parameter, allowing larger models to fit into less VRAM and increasing inference speed.