The memory wall is a hardware bottleneck where the speed of AI generation is limited by how fast data moves from memory to the processor, rather than the processor's raw math capability. To run large language models (LLMs) effectively, you must prioritize memory bandwidth and sufficient VRAM capacity to prevent the chip from sitting idle while waiting for data.
What is the Memory Wall in AI?
In the context of AI, the "memory wall" refers to the bottleneck where the processor spends more time waiting for data to arrive from the memory than it does actually performing calculations. This is a critical concept for anyone building AI applications or investing in hardware.
For Large Language Models (LLMs), the work required per weight is tiny. During the "decoding" phase—the part where the model generates text one token at a time—the chip is often memory-bound. This means the bottleneck isn't how many calculations the chip can do per second, but how fast it can pull the model's weights from the memory into the processor.
Once a model fits entirely into your available memory, the raw compute power (the TFLOPS) becomes less important than the memory bandwidth. The bandwidth is the primary factor that decides how many tokens per second your user actually sees on their screen.
Calculating VRAM: Weights and the KV Cache
When planning your hardware, you cannot simply look at the model size; you must calculate two distinct memory requirements: the model weights and the KV cache.
1. Model Weights
The weights are the fixed "knowledge" of the model. The amount of memory they require follows a simple rule: Parameters × Bytes per parameter. For example, if you are using an 8-billion parameter (8B) model, the memory needs change significantly based on precision:
- FP16 (16-bit): 2 bytes per parameter ≈ 16 GB
- 8-bit: 1 byte per parameter ≈ 8 GB
2. The KV Cache
The KV (Key-Value) cache is the "short-term memory" of the model. To avoid recomputing every single word in a long conversation every time a new word is generated, the model stores the mathematical representations of past tokens in the KV cache. This cache grows linearly with the length of the conversation (context length) and the number of users you are serving simultaneously.
| Model Size | FP16 Weights | 8-bit Weights | 4-bit Weights | Typical GPU (4-bit) |
|---|---|---|---|---|
| 3B | ≈ 6 GB | ≈ 3 GB | ≈ 1.5–2 GB | Any 6–8 GB card |
| 8B | ≈ 16 GB | ≈ 8 GB | ≈ 4–5 GB | 8–12 GB card |
| 14B | ≈ 28 GB | ≈ 14 GB | ≈ 7–9 GB | 12–16 GB card |
| 32B | ≈ 64 GB | ≈ 32 GB | ≈ 16–20 GB | 24–32 GB card |
| 70B | ≈ 140 GB | ≈ 70 GB | ≈ 35–42 GB | 48 GB or larger |
Why Bandwidth Dictates Token Speed
During the decoding process, the chip must read every single active weight from the memory to generate just one new token. Because the math required for each weight is so small, the processor finishes the math quickly and then sits idle, waiting for the next batch of weights to arrive from the memory bus.
A useful mathematical ceiling to estimate your speed is:
max tokens per second ≈ memory bandwidth ÷ bytes read per token
While real-world speeds are often 50% to 80% of this theoretical ceiling due to software overhead, it is an excellent way to compare hardware. For example, an 8B model at 4-bit quantization on an RTX 4090 (which has 1,008 GB/s bandwidth) has a ceiling of roughly 220 tokens per second. On much slower hardware, that number will drop significantly even if the chip's "compute power" is high.
This is why high-end datacenter GPUs use HBM (High Bandwidth Memory). HBM uses stacked memory to achieve several TB/s of throughput, whereas consumer cards use GDDR memory, which is much slower.
Using Quantization to Bypass Memory Limits
Quantization is the process of reducing the precision of model weights—for example, moving from 16-bit (FP16) to 4-bit. This is one of the most effective ways to run large models on affordable hardware.
The benefits of quantization are twofold:
- Reduced Footprint: You can fit a much larger model (like a 70B model) on a single consumer GPU by using 4-bit precision instead of 16-bit.
- Increased Effective Bandwidth: Because each weight takes up less space, the chip can pull more "intelligence" per second through the same memory bus, leading to faster token generation.
Advanced techniques like Grouped-Query Attention (GQA) further help by shrinking the KV cache size. GQA allows several "query heads" to share a single "KV head," which can make the cache four times smaller, allowing for much longer context windows on the same hardware.
Mixture-of-Experts (MoE) and the Memory Trade-off
Mixture-of-Experts (MoE) models represent a clever way to get high performance without the massive compute costs of traditional models. Instead of activating every single parameter for every token, an MoE model splits its layers into sub-networks (experts) and only activates a small fraction of them for each token.
This creates a unique hardware requirement: You need the memory capacity of a large model, but you only need the compute speed of a small model.
For example, a model like Gemma 4 26B might have 26 billion total parameters, but it only activates about 4 billion per token. This means it requires enough VRAM to hold all 26 billion parameters, but it can decode text at a speed close to a much smaller 4B model. This makes high-VRAM GPUs, which can hold massive models, the ideal choice for MoE architectures.
Choosing the Right Hardware Tier
When deciding whether to buy hardware for your business or rent it in the cloud, you need to understand the three main tiers of AI hardware.
| GPU Tier | Example Hardware | Memory Type/Capacity | Best Use Case |
|---|---|---|---|
| Consumer | GeForce RTX 4090 | 24 GB GDDR6X | Exploring models, personal development, small-scale testing. |
| Workstation | RTX PRO 6000 Blackwell | 96 GB GDDR7 (with ECC) | Large MoE models, long-term stability, small-team serving. |
| Datacenter | H100 / H200 | 80–141 GB HBM3/e | High-throughput serving, large-scale training, many users. |
Consumer cards offer the best "bandwidth per rupee" (or dollar), but they are limited by VRAM capacity. Workstation cards add ECC (Error Correction Code) memory and much higher capacity for stability. Datacenter cards use HBM and high-speed links like NVLink to allow multiple GPUs to act as one massive accelerator.
The Alternative: Apple Silicon Unified Memory
Apple's M-series chips (like the M3 or M4) take a different approach using unified memory. In a Mac, the CPU and GPU share a single pool of system RAM.
This offers a massive advantage in capacity. A Mac with a high-end memory configuration can load models so large they would require several expensive NVIDIA GPUs to fit. For instance, an M3 Ultra can have up to 512 GB of memory. While the bandwidth is lower than a top-tier NVIDIA datacenter card, the sheer capacity allows you to run massive models that are otherwise inaccessible on a single consumer workstation.
Frequently Asked Questions
- How much VRAM do I need for an 8B model?
- An 8B model requires about 16 GB at FP16, 8 GB at 8-bit, and 4-5 GB at 4-bit. You should also add 1-2 GB for runtime overhead and extra for the KV cache.
- What is the KV cache in LLMs?
- The KV cache stores key and value vectors for past tokens so the model doesn't have to recompute them. It grows linearly with the length of the conversation (context length).
- Why are GPUs better than CPUs for AI inference?
- GPUs have thousands of simple cores designed for parallel matrix multiplication, whereas CPUs have fewer, more powerful cores designed for branching and logic.
- What is the benefit of 4-bit quantization?
- Quantization reduces the amount of memory needed to store weights, allowing larger models to fit on smaller hardware and increasing the speed of token generation.

Questions people ask
How much VRAM do I need for an 8B model?
An 8B model requires about 16 GB at FP16, 8 GB at 8-bit, and 4-5 GB at 4-bit. You should also add 1-2 GB for runtime overhead and extra for the KV cache.
What is the KV cache in LLMs?
The KV cache stores key and value vectors for past tokens so the model doesn't have to recompute them. It grows linearly with the length of the conversation (context length).
Why are GPUs better than CPUs for AI inference?
GPUs have thousands of simple cores designed for parallel matrix multiplication, whereas CPUs have fewer, more powerful cores designed for branching and logic.
More VRAM allows you to run larger models, but memory bandwidth (GB/s) is what determines how fast the tokens are generated.
More VRAM allows you to run larger models, but memory bandwidth (GB/s) is what determines how fast the tokens are generated.
What is the benefit of 4-bit quantization?
Quantization reduces the amount of memory needed to store weights, allowing larger models to fit on smaller hardware and increasing the speed of token generation.
