Let's talk

Building the Infrastructure

National AI Compute: The Hardware Foundation

To scale AI at a national or enterprise level, you must master the relationship between model parameters, memory bandwidth, and VRAM capacity.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for National AI Compute: The Hardware Foundation

Memory First, Bandwidth Second

To run a large language model (LLM), you need enough fast memory to hold the model weights plus the KV cache for your context length. Once the model fits, memory bandwidth determines how many tokens per second you get.

For builders and business owners, the most important takeaway is this: Memory comes first, bandwidth second. If the model weights do not fit into your available VRAM, the system will either fail to run or will default to much slower system RAM, making the AI feel sluggish and unusable for real-time tasks.

A rough rule of thumb for calculating memory needs is: parameters × bytes per parameter. For example, if you are running a model with 8 billion parameters at 16-bit precision (2 bytes per parameter), you need at least 16 GB of memory just for the weights.

Once you have ensured the model fits, your speed is dictated by memory bandwidth. This is why GPUs are the standard for AI; their VRAM provides significantly higher bandwidth than the standard RAM found in a typical desktop or server CPU. While a CPU is great for logic and branching, a GPU is designed for the massive, parallel matrix multiplications that drive neural networks.

Calculating VRAM for Different Models

When planning your hardware, do not just aim for the exact size of the model. Model weights are a fixed cost, but you must also leave "headroom" for the KV cache and runtime overhead. If you run out of memory during a long conversation, the system will slow down significantly or crash.

The amount of memory required depends heavily on the precision (quantization) of the model. Using 4-bit or 8-bit quantization allows you to run much larger models on smaller, more affordable hardware by reducing the number of bits used to represent each weight.

Model Size FP16 / BF16 Weights 8-bit Weights 4-bit Weights
3B Model ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB
8B Model ≈ 16 GB ≈ 8 GB ≈ 4–5 GB
14B Model ≈ 28 GB ≈ 14 GB ≈ 7–9 GB
32B Model ≈ 64 GB ≈ 32 GB ≈ 16–20 GB
70B Model ≈ 140 GB ≈ 70 GB ≈ 35–42 GB

Pro Tip: When using 4-bit weights, budget roughly 4.5 to 5 bits per weight. This extra cushion accounts for scaling factors and the precision needed to maintain model quality.

The Role of the KV Cache

While model weights are static, the KV (Key-Value) cache is dynamic. As the AI generates text, it stores vectors for every past token in the conversation so it doesn't have to recompute the entire history every time it generates a new word. This is essential for "long context" capabilities.

The KV cache grows linearly based on two factors: the length of the conversation (context length) and the number of users you are serving simultaneously. For a single user, this is manageable, but for a business serving hundreds of customers, the memory requirements for the cache can eventually exceed the memory required for the model itself.

There are two main ways to manage this memory pressure:

  • Grouped-Query Attention (GQA): This architectural feature allows several "query" heads to share a single "KV" head. This can make the cache up to four times smaller without a major loss in intelligence.
  • Quantization: Many modern software tools allow you to quantize the KV cache itself (e.g., to 8-bit), which can halve its size with minimal impact on the quality of the AI's responses.

Understanding GPU Tiers

Not all GPUs are created equal. Depending on whether you are a developer experimenting with code or a company deploying a service in a data center, your hardware needs will differ significantly.

GPU Tier Example Model Memory Type/Spec Best Use Case
Consumer RTX 4090 24 GB GDDR6X Individual developers, prototyping, small models (up to 32B at 4-bit)
Workstation RTX PRO 6000 Blackwell 96 GB GDDR7 (with ECC) Small teams, large MoE models, 70B models, 24/7 server use
Datacenter H100 / H200 80–141 GB HBM3/e High-throughput enterprise serving, large-scale training

Consumer cards offer the best "bandwidth per rupee" (or dollar), making them excellent for individual builders. However, they lack the Error Correction Code (ECC) memory and the cooling designed for the constant, heavy workloads found in professional environments. Datacenter GPUs use HBM (High Bandwidth Memory) and technologies like NVLink to allow multiple cards to act as one massive, unified accelerator.

Apple Silicon and Unified Memory

Apple has taken a unique approach with its M-series chips. Instead of having separate memory for the CPU and the GPU, Apple uses a unified memory architecture. This means the CPU and GPU share a single, large pool of high-speed memory.

This is a massive advantage for running very large models. For example, a Mac with a high-end memory configuration can load massive models that would require several expensive NVIDIA consumer GPUs to hold in memory. While the memory bandwidth on a Mac is typically lower than a top-tier NVIDIA card—meaning you will see fewer tokens per second—the sheer capacity makes it a highly efficient and quiet option for local development.

If you are interested in running AI locally on your laptop, you might want to explore on-device AI capabilities.

Running AI on CPUs

Can you run AI on a standard computer CPU? Yes, but there are significant trade-offs. CPU inference is best suited for small models, background batch jobs (like processing a large pile of documents overnight), or generating text embeddings.

The primary bottleneck for CPUs is memory bandwidth. A typical dual-channel DDR5 system provides roughly 90 GB/s of bandwidth, which is about a tenth of the speed of a high-end GPU. Because the chip spends most of its time waiting for data to arrive from the RAM, the text generation will feel very slow—often only a few words per second for medium-sized models.

When a model is too large for your GPU's VRAM, tools like llama.cpp allow you to "split" the model. The layers that fit on the GPU run fast, while the remaining layers run on the CPU. This allows you to run large models on modest hardware, but the CPU portion will dictate the overall speed of the system.

Renting vs. Buying Hardware

For business owners, the ultimate question is: Should you buy your own hardware or rent it from a cloud provider?

Situation Better Choice Why?
Exploring models for a few hours a week Rent No upfront cost; you can try different GPUs without commitment.
Short training or fine-tuning bursts Rent Rent datacenter GPUs for days, rather than buying expensive hardware for occasional use.
Bulk generation (many hours every day) Buy/Lease A dedicated server becomes more cost-effective than paying hourly cloud rates.
Sensitive data requirements Buy (On-premise) You maintain complete control over where your data resides.

A simple way to decide is to estimate your monthly GPU hours. If your hardware would be busy for a significant portion of the year, owning or leasing a dedicated machine is usually the winner. If the machine would sit idle most of the time, renting is the smarter financial move.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

How much VRAM do I need for an LLM?

You need enough to hold the model weights (parameters × bytes per parameter) plus additional space for the KV cache and runtime overhead.

Why is memory bandwidth important for AI?

During decoding, the chip spends most of its time waiting on memory; higher bandwidth directly translates to more tokens per second.

What is the difference between a GPU and a CPU for AI?

A CPU has a few powerful cores for logic, while a GPU has thousands of simpler cores designed for the parallel matrix multiplication required by AI.

Yes, using tools like llama.cpp, but performance is limited by the much lower memory bandwidth of system RAM compared to GPU VRAM.

What are the benefits of Mixture-of-Experts (MoE) models?

MoE models trade memory capacity for speed; they require enough VRAM to hold all weights, but only activate a fraction of them per token.