Let's talk

Hardware Math

How to Calculate VRAM for an LLM

To run a large language model, you must account for more than just the model size. You need enough memory for weights, the KV cache, and runtime overhead.

By Divyaprakash JhaUpdated 4 min readTopic: AI hardware

Photo for How to Calculate VRAM for an LLM

The Core VRAM Formula

To calculate the VRAM needed for an LLM, use this formula: Total VRAM = Model Weights + KV Cache + Runtime Overhead. You must account for the model's parameters, the memory used for conversation history (KV cache), and a safety buffer for system tasks.

  • Model weights: Calculated as Parameters × Bytes per parameter.
  • KV Cache: The memory required to store context and history.
  • Runtime Overhead: Always leave 1–2 GB of headroom for framework activations and system overhead to prevent crashes.

Calculating Model Weights

The first step is determining how much memory the model weights themselves will occupy. This depends entirely on the precision (bit-depth) at which the model is running. Using VRAM efficiently often requires using quantization to reduce these requirements.

  • FP16/BF16 (Half Precision): Uses 2 bytes per parameter.
  • 8-bit Quantization: Uses 1 byte per parameter.

Example Weight Calculations

If you are looking at an 8B (8 billion parameter) model, your memory needs look like this:

  • At FP16: ~16 GB
  • At 8-bit: ~8 GB
  • At 4-bit: ~4–5 GB

Accounting for the KV Cache

While weights are a fixed cost, the KV cache is a dynamic cost. The KV cache stores key and value vectors for past tokens so the model does not have to recompute the entire conversation every time it generates a new word. This cache grows linearly with the length of the conversation (context length) and the number of users you are serving simultaneously.

The KV Cache Formula

To get a precise estimate of the memory needed for the cache, use this formula:

KV cache bytes = 2 × layers × KV heads × head dimension × bytes per value × tokens

The "2" in the formula accounts for both keys and values. For a standard Llama 3.1 8B style model using FP16 (2 bytes per value) and a context of 8,192 tokens, the cache requires about 1 GiB.

The Role of Quantization

Quantization is the process of storing weights in lower precision (like 8-bit or 4-bit instead of 16-bit). This is the most effective way to run large models on consumer hardware. It reduces the memory footprint significantly, allowing a massive 70B model to fit on hardware that would otherwise only hold a tiny 7B model.

There is a trade-off: lower precision means less memory per number, but it can also mean more math is required per second to maintain accuracy. For most business applications, 4-bit quantization offers the best balance of memory savings and intelligence.

Why Bandwidth Matters for Speed

It is vital to distinguish between capacity and speed. VRAM capacity determines if a model can fit in your hardware; memory bandwidth determines how fast the model generates text (tokens per second).

During the "decoding" phase (when the model is streaming text to you), the chip must read every active weight from memory for every single token generated. Because the actual math is so simple, the chip spends most of its time waiting for data to arrive from the memory. This makes the process "memory-bound."

Estimating Token Speed

You can estimate the maximum speed of your hardware using this ceiling:

Max tokens per second ≈ memory bandwidth ÷ bytes read per token

While real-world speeds are often 50% to 80% of this theoretical ceiling due to overhead, it is an excellent way to compare different GPUs.

Choosing the Right Hardware

When building an AI workstation or choosing cloud instances, you will encounter three main tiers of hardware. Understanding these helps you decide whether to buy or rent GPUs.

GPU Tier Example Hardware Memory (VRAM) Best Use Case
Consumer RTX 4090 / 5090 24 GB - 32 GB Local development, single users, 32B models at 4-bit.
Workstation RTX PRO 6000 Blackwell 96 GB Large models (70B+), small-team serving, MoE models.
Datacenter NVIDIA H100 / H200 80 GB - 141 GB High-throughput enterprise serving, massive context, training.

For those looking for a different approach, Apple Silicon uses "unified memory." This allows the GPU to use the system's main RAM. While the bandwidth is lower than a dedicated NVIDIA card—meaning slower tokens per second—the capacity is much higher, allowing you to run massive models on a Mac that would require multiple expensive GPUs on a PC.

VRAM Estimation Cheat Sheet

Use this quick reference to plan your hardware requirements for common model sizes at 4-bit quantization.

Model Size Estimated 4-bit Weights Recommended Hardware Target
3B Model ~1.5–2 GB 6–8 GB card (Laptops, entry-level GPUs)
8B Model ~4–5 GB 8–12 GB card (Standard consumer GPUs)
14B Model ~7–9 GB 12–16 GB card
32B Model ~16–20 GB 24–32 GB card
70B Model ~35–42 GB 48 GB+ (Multiple GPUs or Workstation card)

Frequently Asked Questions

Does context length affect VRAM usage?
Yes. The KV cache grows linearly with context length, meaning longer conversations require more VRAM beyond the initial model weights.
What is the difference between VRAM and memory bandwidth?
VRAM capacity determines if a model can fit in memory, while memory bandwidth determines the speed (tokens per second) at which the model generates text.
Can I run an LLM on my CPU?
Yes, using tools like llama.cpp, but it is limited by system RAM bandwidth, which is much slower than GPU VRAM. A 70B model on a typical desktop CPU might only reach 2 tokens per second.
Why is 4-bit quantization so popular?
It drastically reduces the memory footprint of a model, allowing large models to fit on consumer hardware with minimal loss in quality.
How much headroom should I leave in my VRAM?
You should choose hardware with 20 to 30 percent more VRAM than your calculated total to account for runtime overhead and context growth.
Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

Does context length affect VRAM usage?

Yes. The KV cache grows linearly with context length, meaning longer conversations require more VRAM beyond the initial model weights.

What is the difference between VRAM and memory bandwidth?

VRAM capacity determines if a model can fit in memory, while memory bandwidth determines the speed (tokens per second) at which the model generates text.

Can I run an LLM on my CPU?

Yes, using tools like llama.cpp, but it is limited by system RAM bandwidth, which is much slower than GPU VRAM.

Why is 4-bit quantization so popular?

It drastically reduces the memory footprint of a model, allowing large models to fit on consumer hardware with minimal loss in quality.

How much headroom should I leave in my VRAM?

You should choose hardware with 20 to 30 percent more VRAM than your calculated total to account for runtime overhead and context growth.