Let's talk

Hardware Essentials

AI Inference Chips: Hardware for Running LLMs

Running large language models requires more than just raw compute; it requires a strategic balance of memory capacity and bandwidth.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for AI Inference Chips: Hardware for Running LLMs

How AI Inference Works on Silicon

To run a large language model (LLM), you need enough fast memory to hold the model weights plus the KV cache for your context length. While memory capacity determines if a model fits, memory bandwidth determines how many tokens per second you actually get.

Running an AI model is fundamentally different from running a standard office application. At its core, a neural network is mostly a series of massive matrix multiplications. When the model generates a single token, it is multiplying the current input by billions of individual weights stored in its memory.

Because each of these multiplications is independent of the others, the workload is "embarrassingly parallel." This is why the hardware choice matters so much. A standard CPU has a few dozen powerful cores designed for complex branching logic and serial tasks, making it relatively slow for these massive math problems.

In contrast, a GPU contains thousands of simpler cores designed to perform the same operation on large blocks of numbers simultaneously. This parallel architecture is exactly the shape of AI work, allowing GPUs to process the billions of calculations required for each token much faster than a traditional processor.

The Critical Role of Memory and Bandwidth

When building an AI system, you must distinguish between two different bottlenecks: capacity and speed. If you are a business owner looking at hardware specs, remember this: Memory capacity determines if a model fits; memory bandwidth determines how many tokens per second you get.

To run an LLM, you need enough fast memory (VRAM) to hold the entire set of model weights, plus extra space for the "KV cache," which tracks the conversation history.

Once the model fits into your memory, your speed is capped by how fast that memory can feed data to the chip. This is known as being "memory-bound." A useful way to estimate your maximum speed is a simple calculation: max tokens per second ≈ memory bandwidth ÷ bytes read per token.

This relationship is why high-end datacenter cards, which use HBM (High Bandwidth Memory), are so much faster than consumer cards that use standard GDDR memory. The chip spends much of its time simply waiting for the next piece of data to arrive from the memory.

Calculating VRAM and Quantization

How much memory do you actually need? A simple rule of thumb is: parameters × bytes per parameter. However, "bytes per parameter" can change depending on how you compress the model through a process called quantization.

Quantization involves storing weights in lower precision (like 4-bit or 8-bit instead of the full 16-bit). This significantly reduces the memory footprint and increases the amount of math the chip can perform per second. For a business looking to deploy locally, quantization is the key to running powerful models on affordable hardware.

3B Model ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB
8B Model ≈ 16 GB ≈ 8 GB ≈ 4–5 GB
14B Model ≈ 28 GB ≈ 14 GB ≈ 7–9 GB
32B Model ≈ 64 GB ≈ 32 GB ≈ 16–20 GB
70B Model ≈ 140 GB ≈ 70 GB ≈ 35–42 GB

When planning your hardware, always leave headroom. You cannot use 100% of your VRAM for weights; you must leave space for the KV cache and 1 to 2 GB of runtime overhead for the software framework. If a model "just fits" the capacity, it will likely fail or slow down as soon as you start a long conversation.

The KV Cache: Why Context Costs Memory

As you chat with an AI, it doesn't re-read the entire conversation every time it generates a new word. Instead, it stores "key and value" vectors for every past token in a special area called the KV cache. This allows the model to generate text efficiently without recomputing everything from scratch.

However, this cache comes with a cost: it grows linearly with the length of the conversation (context length) and the number of users you are serving simultaneously.

To combat this, modern architectures use Grouped-Query Attention (GQA). Instead of every "query head" having its own unique KV head, several query heads share a single KV head. This can make the cache significantly smaller, allowing for much longer conversations on the same hardware.

Comparing GPU Tiers: Consumer, Workstation, and Datacenter

Choosing the right hardware depends on your specific use case. Most AI hardware falls into one of three distinct tiers:

  • Consumer GPUs (e.g., RTX 4090/5090): These offer the best "bandwidth per rupee" for individual developers. They are great for testing and small-scale use, but they have limited VRAM and are designed for desktop cooling.
  • Workstation Cards (e.g., RTX PRO 6000 Blackwell): These trade some price efficiency for much higher VRAM (up to 96 GB) and Error Correction Code (ECC) memory. They are designed to run 24/7 in professional environments.
  • Datacenter Cards (e.g., H100/H200): These are the heavy lifters. They use HBM for massive throughput and feature high-speed links like NVLink, allowing multiple cards to act as one giant accelerator.
GPU Tier Example Model Capacity (at 4-bit) Best Use Case
Consumer (RTX 5090) Up to 32B model Personal research, prototyping
Workstation (PRO 6000) 70B model + large context Small team serving, MoE models
Datacenter (H200) Massive models / many users High-throughput enterprise serving

Apple Silicon, CPUs, and NPUs

While NVIDIA dominates the high-end market, other hardware options exist for different needs. Apple Silicon (M-series) uses a "unified memory" architecture. This allows the GPU to access the same large pool of RAM used by the CPU. While its bandwidth is lower than a top NVIDIA card, a Mac with 128 GB of RAM can run models that would require multiple expensive GPUs to hold.

CPUs with standard system RAM are capable of running LLMs using tools like llama.cpp, but they are much slower. Because system RAM (like DDR5) has significantly lower bandwidth than VRAM, a CPU might only generate 2 tokens per second for a large model, whereas a GPU might generate 50. This makes CPUs better suited for background batch jobs or very small models.

Finally, there are NPUs (Neural Processing Units). These are small, highly efficient accelerators found in modern laptops and phones. They are not meant for massive LLMs, but they are perfect for running compact 1B to 4B parameter models for tasks like smart replies or transcription without draining your battery.

Why Mixture-of-Experts (MoE) Models are Fast

A recent trend in AI is the "Mixture-of-Experts" (MoE) architecture. Instead of activating every single parameter for every single word, an MoE model splits its layers into many specialized "expert" sub-networks. For every token processed, the model only activates a small fraction of its total parameters.

This creates a unique hardware requirement: an MoE model needs the memory capacity of the full, massive model to hold all the experts, but it only needs the compute speed of a much smaller model because it only "reads" a few experts at a time. This makes MoE models incredibly efficient for hardware with large VRAM, such as workstation or datacenter GPUs.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

Why are GPUs better than CPUs for AI inference?

GPUs have thousands of simple cores designed for parallel matrix multiplication, whereas CPUs are designed for complex branching logic and serial tasks.

How much VRAM do I need for a 7B model?

You need about 14 GB at FP16, 7 GB at 8-bit, or 3.5 GB at 4-bit, plus extra for the KV cache and runtime overhead.

What is the difference between GDDR and HBM?

GDDR is common in consumer cards, while HBM (High Bandwidth Memory) is stacked memory used in datacenter cards to provide much higher bandwidth.

Does more VRAM make AI faster?

VRAM capacity determines if a model can fit; memory bandwidth determines how fast the tokens are generated.

Can I run LLMs on my CPU?

Yes, using tools like llama.cpp, but speed is limited by the much lower memory bandwidth of system RAM compared to a GPU.