The Core VRAM Formula
To calculate the VRAM needed for an LLM, use this formula: Total VRAM = Model Weights + KV Cache + Runtime Overhead. You must account for the model's parameters, the memory used for conversation history (KV cache), and a safety buffer for system tasks.
- Model weights: Calculated as Parameters × Bytes per parameter.
- KV Cache: The memory required to store context and history.
- Runtime Overhead: Always leave 1–2 GB of headroom for framework activations and system overhead to prevent crashes.
Calculating Model Weights
The first step is determining how much memory the model weights themselves will occupy. This depends entirely on the precision (bit-depth) at which the model is running. Using VRAM efficiently often requires using quantization to reduce these requirements.
- FP16/BF16 (Half Precision): Uses 2 bytes per parameter.
- 8-bit Quantization: Uses 1 byte per parameter.
Example Weight Calculations
If you are looking at an 8B (8 billion parameter) model, your memory needs look like this:
- At FP16: ~16 GB
- At 8-bit: ~8 GB
- At 4-bit: ~4–5 GB
Accounting for the KV Cache
While weights are a fixed cost, the KV cache is a dynamic cost. The KV cache stores key and value vectors for past tokens so the model does not have to recompute the entire conversation every time it generates a new word. This cache grows linearly with the length of the conversation (context length) and the number of users you are serving simultaneously.
The KV Cache Formula
To get a precise estimate of the memory needed for the cache, use this formula:
KV cache bytes = 2 × layers × KV heads × head dimension × bytes per value × tokens
The "2" in the formula accounts for both keys and values. For a standard Llama 3.1 8B style model using FP16 (2 bytes per value) and a context of 8,192 tokens, the cache requires about 1 GiB.
The Role of Quantization
Quantization is the process of storing weights in lower precision (like 8-bit or 4-bit instead of 16-bit). This is the most effective way to run large models on consumer hardware. It reduces the memory footprint significantly, allowing a massive 70B model to fit on hardware that would otherwise only hold a tiny 7B model.
There is a trade-off: lower precision means less memory per number, but it can also mean more math is required per second to maintain accuracy. For most business applications, 4-bit quantization offers the best balance of memory savings and intelligence.
Why Bandwidth Matters for Speed
It is vital to distinguish between capacity and speed. VRAM capacity determines if a model can fit in your hardware; memory bandwidth determines how fast the model generates text (tokens per second).
During the "decoding" phase (when the model is streaming text to you), the chip must read every active weight from memory for every single token generated. Because the actual math is so simple, the chip spends most of its time waiting for data to arrive from the memory. This makes the process "memory-bound."
Estimating Token Speed
You can estimate the maximum speed of your hardware using this ceiling:
Max tokens per second ≈ memory bandwidth ÷ bytes read per token
While real-world speeds are often 50% to 80% of this theoretical ceiling due to overhead, it is an excellent way to compare different GPUs.
Choosing the Right Hardware
When building an AI workstation or choosing cloud instances, you will encounter three main tiers of hardware. Understanding these helps you decide whether to buy or rent GPUs.
| GPU Tier | Example Hardware | Memory (VRAM) | Best Use Case |
|---|---|---|---|
| Consumer | RTX 4090 / 5090 | 24 GB - 32 GB | Local development, single users, 32B models at 4-bit. |
| Workstation | RTX PRO 6000 Blackwell | 96 GB | Large models (70B+), small-team serving, MoE models. |
| Datacenter | NVIDIA H100 / H200 | 80 GB - 141 GB | High-throughput enterprise serving, massive context, training. |
For those looking for a different approach, Apple Silicon uses "unified memory." This allows the GPU to use the system's main RAM. While the bandwidth is lower than a dedicated NVIDIA card—meaning slower tokens per second—the capacity is much higher, allowing you to run massive models on a Mac that would require multiple expensive GPUs on a PC.
VRAM Estimation Cheat Sheet
Use this quick reference to plan your hardware requirements for common model sizes at 4-bit quantization.
| Model Size | Estimated 4-bit Weights | Recommended Hardware Target |
|---|---|---|
| 3B Model | ~1.5–2 GB | 6–8 GB card (Laptops, entry-level GPUs) |
| 8B Model | ~4–5 GB | 8–12 GB card (Standard consumer GPUs) |
| 14B Model | ~7–9 GB | 12–16 GB card |
| 32B Model | ~16–20 GB | 24–32 GB card |
| 70B Model | ~35–42 GB | 48 GB+ (Multiple GPUs or Workstation card) |
Frequently Asked Questions
- Does context length affect VRAM usage?
- Yes. The KV cache grows linearly with context length, meaning longer conversations require more VRAM beyond the initial model weights.
- What is the difference between VRAM and memory bandwidth?
- VRAM capacity determines if a model can fit in memory, while memory bandwidth determines the speed (tokens per second) at which the model generates text.
- Can I run an LLM on my CPU?
- Yes, using tools like llama.cpp, but it is limited by system RAM bandwidth, which is much slower than GPU VRAM. A 70B model on a typical desktop CPU might only reach 2 tokens per second.
- Why is 4-bit quantization so popular?
- It drastically reduces the memory footprint of a model, allowing large models to fit on consumer hardware with minimal loss in quality.
- How much headroom should I leave in my VRAM?
- You should choose hardware with 20 to 30 percent more VRAM than your calculated total to account for runtime overhead and context growth.

Questions people ask
Does context length affect VRAM usage?
Yes. The KV cache grows linearly with context length, meaning longer conversations require more VRAM beyond the initial model weights.
What is the difference between VRAM and memory bandwidth?
VRAM capacity determines if a model can fit in memory, while memory bandwidth determines the speed (tokens per second) at which the model generates text.
Can I run an LLM on my CPU?
Yes, using tools like llama.cpp, but it is limited by system RAM bandwidth, which is much slower than GPU VRAM.
Why is 4-bit quantization so popular?
It drastically reduces the memory footprint of a model, allowing large models to fit on consumer hardware with minimal loss in quality.
How much headroom should I leave in my VRAM?
You should choose hardware with 20 to 30 percent more VRAM than your calculated total to account for runtime overhead and context growth.
