The Golden Rule: Parameters × Bytes
To run a large language model (LLM), you need enough fast memory to hold the model weights plus the KV cache for your context length. A rough rule is: parameters multiplied by the bytes used per parameter.
When calculating your hardware needs, you must first determine the size of the model weights. This is a fixed cost that determines whether a model can even load onto your hardware. The number of bytes used per parameter depends on the precision of the model, often referred to as quantization.
- FP16 and BF16: These use 2 bytes per parameter.
- 8-bit quantization: This uses 1 byte per parameter.
| Model Size | FP16 / BF16 Weights | 8-bit Weights | 4-bit Weights | Typical Single GPU (4-bit) |
|---|---|---|---|---|
| 3B | ≈ 6 GB | ≈ 3 GB | ≈ 1.5–2 GB | Any 6–8 GB card, laptops, phones |
| 8B | ≈ 16 GB | ≈ 8 GB | ≈ 4–5 GB | 8–12 GB card |
| 14B | ≈ 28 GB | ≈ 14 GB | ≈ 7–9 GB | 12–16 GB card |
| 32B | ≈ 64 GB | ≈ 32 GB | ≈ 16–20 GB | 24–32 GB card |
| 70B | ≈ 140 GB | ≈ 70 GB | ≈ 35–42 GB | 48 GB or larger, or two GPUs |
VRAM vs. System RAM
Not all memory is created equal. For AI, the speed at which data moves is often more important than the total amount of data you can store. This is why VRAM is the gold standard for AI inference.
GPU memory (VRAM) is soldered next to the processor on a very wide bus, making it significantly faster than system RAM. In contrast, system RAM is much slower because it is located further from the processor and uses a narrower bus.
The Speed Gap
Once a model fits entirely in VRAM, your generation speed is primarily determined by memory bandwidth. If you attempt to run a model that is too large for your GPU, you can use tools like llama.cpp to split layers between the GPU and the CPU. However, because the CPU must use system RAM, the CPU half will slow down the entire process, making the experience much slower.
Apple Silicon takes a different approach using unified memory. This allows the GPU to access a large pool of system RAM, which can reach up to 512 GB on an M3 Ultra. While this offers massive capacity, the bandwidth is generally lower than a dedicated NVIDIA card, meaning you get more room but slower tokens per second.
The KV Cache: Why Context Costs Memory
A common mistake for builders is only calculating the size of the model weights. While weights are a fixed cost, the "memory" of the conversation—known as the KV cache—grows as you chat. The KV cache stores key and value vectors for past tokens so the model does not have to recompute them every time you send a new message.
The memory required for the KV cache grows linearly with the context length (how many tokens the model can "remember") and the number of users being served at once. If you are building an application for many users, the KV cache can quickly become your biggest memory consumer.
Example: The Cost of Long Context
Consider a Llama 3.1 8B style model with 32 layers and 8 KV heads. If you use FP16 precision for the cache, the memory usage per token is approximately 128 KiB. This leads to significant requirements as context grows:
- At 8,192 tokens: about 1 GiB of memory.
- At 131,072 tokens (128K context): about 16 GiB of memory.
In the 128K context example, the KV cache alone requires more memory than the 4-bit weights of the model itself. To manage this, modern architectures use Grouped-query attention (GQA), which allows several query heads to share one KV head, significantly shrinking the cache size. You can also quantize the KV cache to 8-bit to halve its footprint.
Why Bandwidth Decides Token Speed
When a model is generating text (the decoding phase), the chip spends most of its time waiting for the memory to read the weights. The actual math is relatively fast, but the "delivery" of the numbers is the bottleneck. This makes memory bandwidth the most critical factor for speed.
A useful rule of thumb for estimating performance is: max tokens per second ≈ memory bandwidth ÷ bytes read per token. While real-world speeds are often 50% to 80% of this theoretical ceiling due to overhead, it remains the best way to compare hardware.
| Hardware Example | Memory Bandwidth | Estimated Speed (4-bit Model) |
|---|---|---|
| RTX 4090 (8B model) | 1,008 GB/s | ~220 tokens/sec (ceiling) |
| High-end Card (70B model) | 1,792 GB/s | ~45 tokens/sec (ceiling) |
This is why high-end datacenter GPUs like the H100 SXM, which offer massive bandwidth (3.35 TB/s), are so much faster at serving models than consumer-grade hardware.
Choosing the Right Hardware Tier
Depending on whether you are a hobbyist, a startup, or an enterprise, your hardware needs will fall into one of three main categories. Understanding these tiers helps you avoid overspending on capacity you don't need or underspending on speed you require.
- Consumer (e.g., RTX 4090): These offer the best bandwidth per rupee. They are excellent for local development and running smaller models, but they are limited by their 24 GB of VRAM.
- Workstation (e.g., RTX PRO 6000 Blackwell): These are designed for professional use. They offer much more VRAM (up to 96 GB), include ECC (Error Correction Code) memory for stability, and are built to run 24/7.
- Datacenter (e.g., H100/H200): These use HBM (High Bandwidth Memory) and are built for high-throughput serving and massive capacity. They are intended for training or serving many users simultaneously.
For a single user running local LLMs, VRAM capacity is usually the first limit you will hit. Pick a card that fits your target model with enough headroom for context, then look at bandwidth to determine how fast it will feel.
Mixture-of-Experts (MoE) and Memory
Mixture-of-Experts (MoE) models, such as Google's Gemma 4 26B, use a clever architecture to balance capacity and speed. Instead of activating every parameter for every token, MoE models split layers into "expert" sub-networks and only activate a fraction of them per token.
This creates a unique memory requirement: all weights must sit in memory for the model to function, but each token only reads a small subset of them. For example, a model might have 25 billion total parameters but only activate about 3.8 billion per token. This allows MoE models to decode at speeds closer to much smaller models while maintaining the knowledge capacity of a much larger one. This makes them ideal for GPUs with large VRAM capacity.
How to Size Your AI Hardware
Sizing your hardware is a balancing act between the model size, the conversation length, and your budget. Follow this strategy to ensure your setup doesn't fail during a long session.
- Calculate the Total: Add the model weights + the expected KV cache + 1 to 2 GB for runtime overhead and activations.
- Add Headroom: Always choose hardware with 20% to 30% more memory than your calculated total. A model that "just fits" will crash or slow down significantly once the conversation history grows.
- Decide: Rent or Buy: Estimate your monthly GPU hours. If the hardware will be busy most of the day for a year or more, owning a dedicated machine usually wins. If it will sit idle, renting is more cost-effective.
If you are working with sensitive data, buying on-premise hardware gives you total control over where your data goes. However, if you need frontier-model quality that exceeds what you can host, using a paid API is still the best option for the hardest reasoning tasks.

Questions people ask
How much VRAM do I need for an 8B model?
At 4-bit quantization, you need about 4–5 GB for the weights, plus additional memory for the KV cache and runtime overhead.
Does more RAM make AI faster?
More RAM capacity allows you to run larger models, but memory bandwidth (GB/s) is what primarily determines how many tokens per second you generate.
What is the difference between VRAM and RAM in AI?
VRAM is high-bandwidth memory located next to the GPU, whereas system RAM is slower. Using system RAM for AI inference is significantly slower than using VRAM.
How does context length affect memory usage?
As context length increases, the KV cache grows linearly, requiring more memory to store the keys and values of previous tokens.
Can I run LLMs on a CPU?
Yes, using tools like llama.cpp, you can run quantized models on a CPU using system RAM, though it is much slower than a GPU.
