The Golden Rule: Memory First, Bandwidth Second
The best GPU for a local LLM is usually the one with the most VRAM you can afford. You must ensure you have enough fast memory to hold the model weights plus the KV cache for your context length, as memory bandwidth determines your actual token speed once the model fits.
When building an AI workstation or a server for your business, the most common mistake is focusing on raw compute power (TFLOPS) while ignoring memory. To run a large language model (LLM), you need enough fast memory to hold the model weights plus the KV cache for your context length. Once the model fits, memory bandwidth decides how many tokens per second you get.
This is why GPUs, with their very fast VRAM, beat CPUs for AI inference. A CPU has a few dozen powerful cores designed for varied logic, whereas a GPU has thousands of simpler cores designed to do the same operation on large blocks of numbers in parallel. That is exactly the shape of AI work.
Calculating Your VRAM Requirements
To determine which GPU for LLM tasks you need, you must use a simple arithmetic rule: parameters × bytes per parameter. This gives you the base memory needed for the model weights.
The number of bytes per parameter depends on the precision you use. For example, FP16 and BF16 use 2 bytes, 8-bit uses 1 byte, and 4-bit uses half a byte. However, in practice, you should budget roughly 4.5 to 5 bits per weight to account for scaling factors and metadata.
Consider these common scenarios for an 8B parameter model:
- FP16/BF16: Approximately 16 GB
- 8-bit: Approximately 8 GB
- 4-bit: Approximately 4 to 5 GB
Always leave headroom. You need to account for the KV cache and an additional 1 to 2 GB of runtime overhead for the framework and activations. A model that "just fits" will fail or slow down once a long conversation fills the cache.
| Model Size | FP16/BF16 Weights | 8-bit Weights | 4-bit Weights | Typical Single GPU (4-bit) |
|---|---|---|---|---|
| 3B | ≈ 6 GB | ≈ 3 GB | ≈ 1.5–2 GB | Any 6–8 GB card |
| 8B | ≈ 16 GB | ≈ 8 GB | ≈ 4–5 GB | 8–12 GB card |
| 14B | ≈ 28 GB | ≈ 14 GB | ≈ 7–9 GB | 12–16 GB card |
| 32B | ≈ 64 GB | ≈ 32 GB | ≈ 16–20 GB | 24–32 GB card |
| 70B | ≈ 140 GB | ≈ 70 GB | ≈ 35–42 GB | 48 GB or larger |
Why Long Context Costs More Memory
When generating text, a transformer stores a key and a value vector for every past token in every layer so it does not have to recompute them. This is known as the KV cache. Unlike model weights, which are a fixed cost, the KV cache grows linearly with the context length and the number of users you serve simultaneously.
For a Llama 3.1 8B style model (32 layers, 8 KV heads, head dimension 128, using FP16), the math works out as follows:
- Per token: 2 × 32 × 8 × 128 × 2 = 131,072 bytes (128 KiB).
- At 8,192 tokens: about 1 GiB.
- At 131,072 tokens (128K context): about 16 GiB.
At a 128K context window, the KV cache itself requires more memory than the 4-bit weights of the model. This is a critical consideration for businesses building long-context agents.
How Memory Bandwidth Affects Token Speed
For a single user, the speed of text generation (decoding) is limited by how fast the chip can read weights from memory. Because the arithmetic per weight is tiny, the chip spends most of its time waiting on the memory bus. A useful ceiling for estimating performance is:
Max tokens per second ≈ memory bandwidth ÷ bytes read per token
While real-world speeds are often 50 to 80 percent of this theoretical ceiling due to overheads, it remains the best way to compare hardware. For example, an RTX 4090 with 1,008 GB/s bandwidth can reach a ceiling of roughly 220 tokens per second when running an 8B model at 4-bit precision.
Note that "prefill" (reading your initial prompt) is different. Prefill is limited by compute power, which is why you often see a short pause before the model begins streaming text steadily.
Comparing GPU Tiers for AI Workloads
When selecting AI hardware, you must choose a tier based on your specific business use case. The following table compares the major categories of GPUs available in 2026.
| GPU Tier | Example Card | Memory Bandwidth | Best For |
|---|---|---|---|
| Consumer | GeForce RTX 5090 | 1,792 GB/s | Up to 32B at 4-bit; best bandwidth per rupee |
| Workstation | RTX PRO 6000 Blackwell | 1,792 GB/s | 70B models; large MoE; small-team serving |
| Datacenter | H200 | 4.8 TB/s | High-throughput serving; many concurrent users |
Consumer cards offer the best value for individual builders but are limited by VRAM (typically up to 32 GB). Workstation cards provide much more VRAM (up to 96 GB) and ECC memory for long-term stability in professional environments. Datacenter cards use HBM (High Bandwidth Memory) to provide massive throughput for high-scale applications.
Apple Silicon and Unified Memory
Apple’s M-series chips use a unique architecture called unified memory, where the CPU and GPU share a single pool of system RAM. This allows a Mac with a high memory configuration to load massive models that would otherwise require several consumer GPUs. For example, an M3 Ultra can reach up to 512 GB of memory.
However, there is a trade-off. While the capacity is high, the bandwidth is typically lower than top-tier NVIDIA cards. This means you can load a massive model, but it will generate tokens more slowly. This makes Apple Silicon excellent for local development and running large models that simply won't fit elsewhere, but less ideal for high-speed production serving.
The Speed Advantage of Mixture-of-Experts (MoE)
Mixture-of-Experts (MoE) models, such as the Gemma 4 26B model, offer a unique efficiency advantage for hardware owners. In an MoE model, only a fraction of the total parameters are activated for any given token. For instance, a model might have 25 billion total parameters but only use about 3.8 billion per token.
This creates a specific hardware requirement: you need enough VRAM to hold all the weights (the full 26B parameters), but you get the generation speed of a much smaller model. This makes MoE an excellent choice for hardware with large VRAM capacity, such as workstation or multi-GPU setups.
Frequently Asked Questions
- Does VRAM or compute power matter more for LLMs?
- VRAM capacity is the first limit you hit for fitting models, while memory bandwidth determines how fast the model generates text.
- Can I run an LLM on my CPU?
- Yes, using tools like llama.cpp, but it is much slower because CPU memory bandwidth is significantly lower than GPU VRAM.
- What is quantization?
- Quantization is storing model weights at lower precision (like 4-bit or 8-bit) to reduce memory requirements and increase speed.
- How much headroom should I leave for context?
- You should choose hardware with 20 to 30 percent more VRAM than the model weights alone to account for the KV cache.
- Should I rent or buy a GPU for AI?
- If the hardware will be busy most of the day for a year or more, owning it usually wins; if it will sit idle, renting is better.

Questions people ask
Does VRAM or compute power matter more for LLMs?
VRAM capacity is the first limit you hit for fitting models, while memory bandwidth determines how fast the model generates text.
Can I run an LLM on my CPU?
Yes, using tools like llama.cpp, but it is much slower because CPU memory bandwidth is significantly lower than GPU VRAM.
What is quantization?
Quantization is storing model weights at lower precision (like 4-bit or 8-bit) to reduce memory requirements and increase speed.
How much headroom should I leave for context?
You should choose hardware with 20 to 30 percent more VRAM than the model weights alone to account for the KV cache.
Should I rent or buy a GPU for AI?
If the hardware will be busy most of the day for a year or more, owning it usually wins; if it will sit idle, renting is better.
