Let's talk

Hardware Bottlenecks

The Memory Wall: Why Bandwidth Matters More Than Compute for AI

In AI inference, the speed of generating text is often limited by how fast data moves from memory to the chip, not just raw processing power.

By Divyaprakash JhaUpdated 6 min readTopic: AI hardware

Photo for The Memory Wall: Why Bandwidth Matters More Than Compute for AI

The memory wall is a hardware bottleneck where the speed of AI generation is limited by how fast data moves from memory to the processor, rather than the processor's raw math capability. To run large language models (LLMs) effectively, you must prioritize memory bandwidth and sufficient VRAM capacity to prevent the chip from sitting idle while waiting for data.

What is the Memory Wall in AI?

In the context of AI, the "memory wall" refers to the bottleneck where the processor spends more time waiting for data to arrive from the memory than it does actually performing calculations. This is a critical concept for anyone building AI applications or investing in hardware.

For Large Language Models (LLMs), the work required per weight is tiny. During the "decoding" phase—the part where the model generates text one token at a time—the chip is often memory-bound. This means the bottleneck isn't how many calculations the chip can do per second, but how fast it can pull the model's weights from the memory into the processor.

Once a model fits entirely into your available memory, the raw compute power (the TFLOPS) becomes less important than the memory bandwidth. The bandwidth is the primary factor that decides how many tokens per second your user actually sees on their screen.

Calculating VRAM: Weights and the KV Cache

When planning your hardware, you cannot simply look at the model size; you must calculate two distinct memory requirements: the model weights and the KV cache.

1. Model Weights

The weights are the fixed "knowledge" of the model. The amount of memory they require follows a simple rule: Parameters × Bytes per parameter. For example, if you are using an 8-billion parameter (8B) model, the memory needs change significantly based on precision:

  • FP16 (16-bit): 2 bytes per parameter ≈ 16 GB
  • 8-bit: 1 byte per parameter ≈ 8 GB

2. The KV Cache

The KV (Key-Value) cache is the "short-term memory" of the model. To avoid recomputing every single word in a long conversation every time a new word is generated, the model stores the mathematical representations of past tokens in the KV cache. This cache grows linearly with the length of the conversation (context length) and the number of users you are serving simultaneously.

Model Size FP16 Weights 8-bit Weights 4-bit Weights Typical GPU (4-bit)
3B ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB Any 6–8 GB card
8B ≈ 16 GB ≈ 8 GB ≈ 4–5 GB 8–12 GB card
14B ≈ 28 GB ≈ 14 GB ≈ 7–9 GB 12–16 GB card
32B ≈ 64 GB ≈ 32 GB ≈ 16–20 GB 24–32 GB card
70B ≈ 140 GB ≈ 70 GB ≈ 35–42 GB 48 GB or larger

Why Bandwidth Dictates Token Speed

During the decoding process, the chip must read every single active weight from the memory to generate just one new token. Because the math required for each weight is so small, the processor finishes the math quickly and then sits idle, waiting for the next batch of weights to arrive from the memory bus.

A useful mathematical ceiling to estimate your speed is:
max tokens per second ≈ memory bandwidth ÷ bytes read per token

While real-world speeds are often 50% to 80% of this theoretical ceiling due to software overhead, it is an excellent way to compare hardware. For example, an 8B model at 4-bit quantization on an RTX 4090 (which has 1,008 GB/s bandwidth) has a ceiling of roughly 220 tokens per second. On much slower hardware, that number will drop significantly even if the chip's "compute power" is high.

This is why high-end datacenter GPUs use HBM (High Bandwidth Memory). HBM uses stacked memory to achieve several TB/s of throughput, whereas consumer cards use GDDR memory, which is much slower.

Using Quantization to Bypass Memory Limits

Quantization is the process of reducing the precision of model weights—for example, moving from 16-bit (FP16) to 4-bit. This is one of the most effective ways to run large models on affordable hardware.

The benefits of quantization are twofold:

  • Reduced Footprint: You can fit a much larger model (like a 70B model) on a single consumer GPU by using 4-bit precision instead of 16-bit.
  • Increased Effective Bandwidth: Because each weight takes up less space, the chip can pull more "intelligence" per second through the same memory bus, leading to faster token generation.

Advanced techniques like Grouped-Query Attention (GQA) further help by shrinking the KV cache size. GQA allows several "query heads" to share a single "KV head," which can make the cache four times smaller, allowing for much longer context windows on the same hardware.

Mixture-of-Experts (MoE) and the Memory Trade-off

Mixture-of-Experts (MoE) models represent a clever way to get high performance without the massive compute costs of traditional models. Instead of activating every single parameter for every token, an MoE model splits its layers into sub-networks (experts) and only activates a small fraction of them for each token.

This creates a unique hardware requirement: You need the memory capacity of a large model, but you only need the compute speed of a small model.

For example, a model like Gemma 4 26B might have 26 billion total parameters, but it only activates about 4 billion per token. This means it requires enough VRAM to hold all 26 billion parameters, but it can decode text at a speed close to a much smaller 4B model. This makes high-VRAM GPUs, which can hold massive models, the ideal choice for MoE architectures.

Choosing the Right Hardware Tier

When deciding whether to buy hardware for your business or rent it in the cloud, you need to understand the three main tiers of AI hardware.

GPU Tier Example Hardware Memory Type/Capacity Best Use Case
Consumer GeForce RTX 4090 24 GB GDDR6X Exploring models, personal development, small-scale testing.
Workstation RTX PRO 6000 Blackwell 96 GB GDDR7 (with ECC) Large MoE models, long-term stability, small-team serving.
Datacenter H100 / H200 80–141 GB HBM3/e High-throughput serving, large-scale training, many users.

Consumer cards offer the best "bandwidth per rupee" (or dollar), but they are limited by VRAM capacity. Workstation cards add ECC (Error Correction Code) memory and much higher capacity for stability. Datacenter cards use HBM and high-speed links like NVLink to allow multiple GPUs to act as one massive accelerator.

The Alternative: Apple Silicon Unified Memory

Apple's M-series chips (like the M3 or M4) take a different approach using unified memory. In a Mac, the CPU and GPU share a single pool of system RAM.

This offers a massive advantage in capacity. A Mac with a high-end memory configuration can load models so large they would require several expensive NVIDIA GPUs to fit. For instance, an M3 Ultra can have up to 512 GB of memory. While the bandwidth is lower than a top-tier NVIDIA datacenter card, the sheer capacity allows you to run massive models that are otherwise inaccessible on a single consumer workstation.

Frequently Asked Questions

How much VRAM do I need for an 8B model?
An 8B model requires about 16 GB at FP16, 8 GB at 8-bit, and 4-5 GB at 4-bit. You should also add 1-2 GB for runtime overhead and extra for the KV cache.
What is the KV cache in LLMs?
The KV cache stores key and value vectors for past tokens so the model doesn't have to recompute them. It grows linearly with the length of the conversation (context length).
Why are GPUs better than CPUs for AI inference?
GPUs have thousands of simple cores designed for parallel matrix multiplication, whereas CPUs have fewer, more powerful cores designed for branching and logic.
What is the benefit of 4-bit quantization?
Quantization reduces the amount of memory needed to store weights, allowing larger models to fit on smaller hardware and increasing the speed of token generation.
Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

How much VRAM do I need for an 8B model?

An 8B model requires about 16 GB at FP16, 8 GB at 8-bit, and 4-5 GB at 4-bit. You should also add 1-2 GB for runtime overhead and extra for the KV cache.

What is the KV cache in LLMs?

The KV cache stores key and value vectors for past tokens so the model doesn't have to recompute them. It grows linearly with the length of the conversation (context length).

Why are GPUs better than CPUs for AI inference?

GPUs have thousands of simple cores designed for parallel matrix multiplication, whereas CPUs have fewer, more powerful cores designed for branching and logic.

More VRAM allows you to run larger models, but memory bandwidth (GB/s) is what determines how fast the tokens are generated.

More VRAM allows you to run larger models, but memory bandwidth (GB/s) is what determines how fast the tokens are generated.

What is the benefit of 4-bit quantization?

Quantization reduces the amount of memory needed to store weights, allowing larger models to fit on smaller hardware and increasing the speed of token generation.