Let's talk

Hardware Essentials

VRAM for LLM: A Guide to Memory Requirements

To run a large language model, you need enough fast memory to hold the model weights plus the KV cache for your context length.

By Divyaprakash JhaUpdated 4 min readTopic: AI hardware

Photo for VRAM for LLM: A Guide to Memory Requirements

To run a large language model, you need enough fast memory to hold the model weights plus the KV cache for your context length. A rough rule of thumb is to multiply the number of parameters by the bytes per parameter.

The Golden Rule: Parameters × Bytes

When calculating how much VRAM you need, you must first account for the model weights. The memory needed for weights is calculated as: parameters × bytes per parameter.

The number of bytes per parameter depends on the precision or "quantization" you use. Using higher precision makes the model smarter but requires much more memory. Using lower precision (quantization) allows you to run large models on much cheaper, consumer-grade hardware.

  • FP16 and BF16: These use 2 bytes per parameter. This is standard high-precision format.
  • 8-bit quantization: This uses 1 byte per parameter.

VRAM Requirements by Model Size

Below is a breakdown of how much VRAM different model sizes require based on common formats. Remember to add 1 to 2 GB of runtime overhead for the framework and activations, plus extra for the KV cache.

Model Size FP16 / BF16 Weights 8-bit Weights 4-bit Weights Typical Single GPU (4-bit)
3B model ~6 GB ~3 GB ~1.5–2 GB Any 6–8 GB card, laptops, phones
8B model ~16 GB ~8 GB ~4–5 GB 8–12 GB card
14B model ~28 GB ~14 GB ~7–9 GB 12–16 GB card
32B model ~64 GB ~32 GB ~16–20 GB 24–32 GB card
70B model ~140 GB ~70 GB ~35–42 GB 48 GB or larger, or two GPUs

The Hidden Cost: KV Cache

The weights are a fixed cost, but the KV cache is a variable cost. While generating text, a transformer stores a key and a value vector for every past token in every layer so it does not have to recompute them. This is the KV cache.

The cache grows linearly with the context length (how many words the model "remembers") and the number of users being served at once. If you are building an application for many users, the cache can quickly become larger than the model weights themselves.

Example: The Cost of Long Context

Consider a Llama 3.1 8B style model (32 layers, 8 KV heads, head dimension 128, stored in FP16):

  • Per token: about 128 KiB.
  • At 8,192 tokens: about 1 GiB.
  • At 131,072 tokens (128K context): about 16 GiB.

If you want to serve ten users at 8K context each, you need about 10 GiB of VRAM just for the cache.

However, modern architectures use Grouped-query attention (GQA). This allows several query heads to share one KV head, making the cache significantly smaller. For example, if a model has 32 query heads but only 8 KV heads, the cache is four times smaller than it would otherwise be.

If you are new to how LLMs work, start with the LLM guide.

Memory Bandwidth vs. Compute

A common mistake is thinking that more "teraflops" (compute power) always means faster AI. Once the model fits in your VRAM, memory bandwidth is what decides your tokens per second.

For a single user, the chip spends most of its time waiting on memory during the decoding phase (generating one token at a time). Each new token requires reading every active weight from the memory once. Because the math for each weight is tiny, the bottleneck is how fast those weights can travel from the VRAM to the processor.

You can estimate the speed ceiling with this formula:

Max tokens per second ≈ memory bandwidth ÷ bytes read per token

In the real world, actual speed is often 50% to 80% of this theoretical ceiling due to software overhead. For example, an 8B model at 4-bit on an RTX 4090 (1,008 GB/s) has a ceiling of roughly 220 tokens per second. A 70B model at 4-bit on a high-end card with 1,792 GB/s has a ceiling of about 45 tokens per second.

Learn more about GPU for LLM.

Choosing the Right Hardware Tier

Hardware for AI is generally divided into three categories. Your choice depends on whether you are a hobbyist, a professional designer, or a large-scale service provider.

GPU Tier Example Memory/Bandwidth Traits Best Use Case
Consumer RTX 4090 / 5090 High bandwidth per rupee; limited VRAM (24GB–32GB) Local development, single-user testing, small models
Workstation RTX PRO 6000 Blackwell High VRAM (up to 96GB), ECC memory Large models (70B), MoE models, professional workflows
Datacenter H100 / H200 Massive HBM bandwidth (up to 4.8 TB/s) High-throughput serving, massive scale, training

Apple Silicon: Mac computers like the M3 Ultra use "unified memory," meaning the GPU and CPU share the same pool of RAM. This allows you to load massive models (up to 512 GB) that would require several expensive NVIDIA GPUs. While the capacity is incredible, the bandwidth is lower than top-tier NVIDIA cards, so tokens per second will be slower.

See our guide on GPU for AI.

Inference vs. Fine-Tuning Memory

The hardware you need depends heavily on whether you are just "using" the model (inference) or "teaching" it new things (fine-tuning).

Inference
The simplest task. You only need enough VRAM to hold the weights plus the KV cache for your users.
Fine-tuning (LoRA/QLoRA)
You need enough VRAM for the base weights, the small "adapter" layers being trained, and the "activations" (temporary data used during the math). QLoRA allows you to fine-tune large models on a single 24GB or 48GB GPU by keeping the base model in 4-bit.
Full Fine-tuning
This is extremely memory-intensive. You need enough space for weights, gradients, and optimizer states (like Adam). This can require roughly 16 bytes per parameter, meaning a 7B model would need over 100 GB of VRAM just to start training.
Pre-training
Building a model from scratch requires massive clusters of datacenter GPUs and trillions of tokens of data.

For most businesses, the most realistic path is a combination of inference and light fine-tuning or retrieval. Explore AI hardware for more details.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

How much VRAM do I need for an 8B model?

At 4-bit quantization, you need about 4–5 GB for the weights, plus additional VRAM for the KV cache and runtime overhead.

Does more VRAM make LLMs faster?

VRAM capacity determines if a model can fit; memory bandwidth determines how fast it generates tokens per second.

What is the KV cache?

The KV cache stores vectors for every past token in every layer to avoid recomputing them, growing with context length.

Can I run LLMs on a CPU?

Yes, using tools like llama.cpp, but performance is limited by system RAM bandwidth, which is much slower than GPU VRAM.

Why is 4-bit quantization common?

It reduces the memory required per parameter (from 2 bytes to 0.5 bytes), allowing larger models to fit on consumer hardware.