The Core Difference: Prefill vs. Decoding
In LLM inference, memory bandwidth determines token speed (tokens/sec) while compute handles prompt processing (prefill). For most large language model (LLM) tasks, the speed of text generation is limited by how fast data moves from memory to the chip, not how fast the chip can calculate.
To understand why your AI feels slow or fast, you have to distinguish between two distinct phases of inference: prefill and decoding.
Prompt processing (prefill) is compute-bound. When you paste a long document into a chat, the model processes many tokens all at once. This phase is limited by raw math capability—the "compute" power of the chip. This is why a long prompt has a brief pause before the first word appears.
Token generation (decoding) is memory-bandwidth bound. Once the model starts streaming text, it generates one token at a time. During this phase, the chip spends most of its time waiting to read model weights from memory. The math is actually quite simple; the bottleneck is simply how fast the weights can be moved into the processor.
Why Bandwidth Matters More for LLMs
For many builders, the instinct is to look for the highest "TFLOPS" or "compute" number when buying hardware. For LLM inference, this is often a mistake. Generating a single token requires reading every active weight from the model in memory exactly once.
Because the arithmetic performed per weight is tiny, the chip's math speed becomes secondary to the data transfer rate. If your memory is slow, your powerful processor will sit idle, waiting for the next piece of data to arrive. This is why GPUs with very fast VRAM, such as GDDR6X or HBM, significantly outperform CPUs for inference.
If you are looking for more information on the specific hardware types used to solve this, you can explore our guide on AI hardware.
Memory Capacity: The First Hurdle
Before you worry about how fast the text streams, you must satisfy the most basic requirement: capacity. If the model does not fit entirely within your available video memory (VRAM), it will either fail to run or slow down drastically as it swaps data to your much slower system RAM.
A rough rule for calculating the memory needed for model weights is: parameters × bytes per parameter. For example, if you use 4-bit quantization (which is very common for efficiency), an 8B model needs roughly 4 to 5 GB of space. If you use 16-bit precision, that same model needs 16 GB.
However, you cannot use 100% of your VRAM for weights. You must always leave headroom for the KV cache and runtime overhead. A model that "just fits" will often crash or become incredibly slow once a conversation grows long or multiple users start asking questions at once.
| Model Size | FP16 / BF16 Weights | 8-bit Weights | 4-bit Weights | Typical GPU (4-bit) |
|---|---|---|---|---|
| 3B | ≈ 6 GB | ≈ 3 GB | ≈ 1.5–2 GB | Any 6–8 GB card |
| 8B | ≈ 16 GB | ≈ 8 GB | ≈ 4–5 GB | 8–12 GB card |
| 14B | ≈ 28 GB | ≈ 14 GB | ≈ 7–9 GB | 12–16 GB card |
| 32B | ≈ 64 GB | ≈ 32 GB | ≈ 16–20 GB | 24–32 GB card |
| 70B | ≈ 140 GB | ≈ 70 GB | ≈ 35–42 GB | 48 GB or larger |
For a deeper dive into managing these memory requirements, see our guide on VRAM for LLMs.
The KV Cache: The Hidden Memory Consumer
As mentioned, the weights are only part of the story. To avoid recomputing everything from scratch every time a new word is generated, the model uses something called the KV cache. This stores key and value vectors for all past tokens in the conversation.
The KV cache is a "hidden" consumer because it grows linearly with two things: the length of the conversation (context length) and the number of users you are serving at once. If you are building a chatbot for a small business in India to handle customer queries, one user might only need a few gigabytes, but a thousand users will require massive amounts of extra VRAM.
To illustrate the scale, let's look at a Llama 3.1 8B style model with a 128K context window. The KV cache alone can require about 16 GiB of memory. In this scenario, the cache actually requires more memory than the 4-bit model weights themselves!
Modern techniques like Grouped-query attention (GQA) help by allowing multiple "query heads" to share the same KV heads, which can make the cache significantly smaller. You can also quantize the KV cache to 8-bit to save even more space.
Comparing Hardware Tiers
Choosing the right hardware depends on whether you need raw speed, massive capacity, or a balance of both. Hardware generally falls into three tiers:
- Consumer GPUs (e.g., RTX 4090): These offer the best bandwidth per rupee. They are excellent for a single developer or a small local setup, but they have limited VRAM capacity.
- Workstation Cards (e.g., RTX PRO 6000): These trade some price efficiency for much higher VRAM capacity and ECC (Error Correction Code) support. They are designed for professional environments where stability is key.
- Datacenter GPUs (e.g., H100): These use HBM (High Bandwidth Memory) to provide massive, industry-leading bandwidth (measured in TB/s). These are built for high-throughput serving in large data centers.
There is also a unique fourth category: Apple Silicon. Using a "unified memory" architecture, a Mac can load models that would require several expensive NVIDIA GPUs by using the system's main RAM. While the bandwidth is lower than a top-tier NVIDIA card, the capacity is incredibly high, making it a favorite for running large models locally.
If you are deciding between a dedicated GPU or a general-purpose computer, check out our AI PC guide.
How Mixture-of-Experts (MoE) Changes the Math
Mixture-of-Experts (MoE) models, such as the Gemma 4 26B, introduce a clever way to handle large models efficiently. Instead of using every single parameter for every single token, an MoE model only "activates" a small fraction of its parameters per token.
This creates a unique hardware requirement: you still need enough memory to hold all the weights (the full 26B parameters), but you get the generation speed of a much smaller model (closer to a 4B model). This makes MoE an excellent choice for businesses that have the VRAM capacity to hold the model but want the fast, snappy response times of a smaller, cheaper model.
The Builder's Rule of Thumb
When designing your inference infrastructure, follow these three steps to avoid costly mistakes:
- Calculate total memory needed: Add your Model Weights + KV Cache + 20-30% headroom for safety.
- Pick hardware that fits: Ensure your VRAM capacity is greater than the number calculated in step one.
- Estimate speed: Use the memory bandwidth to compare different hardware options to see which will give you the tokens-per-second your application requires.

Questions people ask
Is compute or memory bandwidth more important for LLM inference?
For generating text (decoding), memory bandwidth is usually the bottleneck. For processing long prompts (prefill), compute is the limiting factor.
How much VRAM do I need for a 7B model?
At 4-bit quantization, a 7B model needs about 3.5 GB for weights, plus extra for the KV cache and runtime overhead.
Why is Apple Silicon good for large models?
Apple's unified memory allows the GPU to access a massive pool of system RAM, making it possible to load models that require much more VRAM than a single consumer GPU.
What is the KV cache?
The KV cache is a way for the model to store information about past tokens in a conversation so it doesn't have to re-calculate them every time a new word is generated.
Does adding more GPUs make inference faster?
For a single user, two GPUs do not make a model twice as fast; multi-GPU setups are primarily used to fit larger models or serve more users simultaneously.
