Let's talk

Hardware sizing guide

How Much VRAM Do I Need for LLM?

To run a large language model, you need enough fast memory to hold the model weights plus the KV cache for your context length.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for How Much VRAM Do I Need for LLM?

The Core Formula for VRAM

To run a large language model, you need enough fast memory to hold the model weights plus the KV cache for your context length. A good starting point is to multiply the number of parameters in the model by the bytes used per parameter.

When calculating your needs, you must account for how the model is stored. This is known as quantization. Using higher precision makes the model smarter but requires much more memory. Most builders use quantization to fit larger models onto smaller, more affordable hardware.

The rough rule is: parameters × bytes per parameter. Here is how that breaks down in practice:

  • FP16 and BF16: These use 2 bytes per parameter. This is the "full precision" standard for many models.
  • 8-bit quantization: Uses 1 byte per parameter.

VRAM Requirements by Model Size

Understanding how much memory a model needs depends on its size and the precision you choose. For example, a small 3B model is easy to run on a laptop, but a 70B model requires professional-grade hardware or multiple GPUs.

Model Size FP16 / BF16 weights 8-bit weights 4-bit weights Typical single GPU (4-bit)
3B ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB Any 6–8 GB card, laptops, phones
8B ≈ 16 GB ≈ 8 GB ≈ 4–5 GB 8–12 GB card
14B ≈ 28 GB ≈ 14 GB ≈ 7–9 GB 12–16 GB card
32B ≈ 64 GB ≈ 32 GB ≈ 16–20 GB 24–32 GB card
70B ≈ 140 GB ≈ 70 GB ≈ 35–42 GB 48 GB or larger, or two GPUs

Don't Forget the KV Cache

A common mistake for builders is thinking that if a model "fits" in VRAM, it is ready to use. This ignores the KV cache. While model weights are a fixed cost, the KV cache grows with every token of context you use.

The KV cache stores key and value vectors for past tokens so the model does not have to recompute them every time it generates a new word. This is what allows the model to "remember" the beginning of a long conversation.

If you choose a model that "just fits" the weights, the system will likely fail or slow down significantly once a long conversation fills the cache. For a Llama 3.1 8B style model, a context of 8,192 tokens requires about 1 GiB of cache. If you increase that to a 128K context, the cache alone can require 16 GiB—which is more than the 4-bit weights of the model itself!

If you are building a service to serve many users at once, you must account for the fact that each user needs their own cache space. Serving ten users at an 8K context each would require an additional 10 GiB of VRAM.

Choosing Your Hardware Tier

When selecting hardware for AI, you generally choose from three distinct tiers based on your budget and scale.

1. Consumer GPUs

Examples include the NVIDIA RTX 4090 or the RTX 5090. These offer the best bandwidth per rupee, making them excellent for individual developers. However, they have limited VRAM capacity, which limits the size of the models you can run.

2. Workstation GPUs

Cards like the RTX PRO 6000 Blackwell are designed for professional use. They offer much more VRAM (up to 96 GB), ECC memory for stability, and are built to run all day in a workstation or server environment.

3. Datacenter GPUs

High-end chips like the H100 or H200 use HBM (High Bandwidth Memory) to achieve massive throughput. These are designed for high-scale serving, training, and handling many users simultaneously.

Why Bandwidth Matters for Speed

If VRAM capacity determines if a model can run, memory bandwidth determines how fast it runs. For most LLM work, the chip spends more time waiting for data to move from memory to the processor than it does actually performing the math.

A useful rule of thumb for estimating speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token. For example, an 8B model at 4-bit quantization on an RTX 4090 (1,008 GB/s) has a theoretical ceiling of about 220 tokens per second.

It is also worth noting the rise of Mixture-of-Experts (MoE) models. These models trade memory capacity for speed. An MoE model might have many total parameters (requiring lots of VRAM), but it only activates a small subset of them for each token. This means it decodes at a speed much closer to a much smaller model.

Inference vs. Fine-Tuning

Your hardware requirements change drastically depending on whether you are just using the model (inference) or training it on your own data (fine-tuning).

Task Memory Need (Rough) Typical Hardware
Inference Weights + KV cache One GPU, a Mac, or a CPU
Fine-tuning (QLoRA) 4-bit weights + adapters + activations One 24–96 GB GPU
Full Fine-tuning Around 16 bytes per parameter Several datacenter GPUs

For most businesses, the most realistic path is using QLoRA. This method allows you to fine-tune large models on a single GPU by keeping the base weights in 4-bit precision while training only small "adapter" layers.

The Builder's Checklist

Before you buy hardware or rent a cloud instance, follow these steps to ensure your setup won't crash during a long conversation:

  1. Estimate weights: Calculate based on parameter count and your chosen quantization level.
  2. Add KV cache: Account for your target context length and the number of concurrent users.
  3. Add overhead: Budget an extra 1–2 GB for runtime overhead and activations.
  4. Add headroom: Choose hardware with 20% to 30% more VRAM than your total estimate.

Frequently Asked Questions

Does quantization reduce VRAM requirements?
Yes. Quantization stores weights in lower precision (like 4-bit or 8-bit), which significantly reduces the memory needed to load the model.
Why can't I just use my system RAM?
You can using tools like llama.cpp, but system RAM bandwidth is much lower than VRAM, which makes token generation much slower.
What is the KV cache?
The KV cache is memory used to store key and value vectors for past tokens so the model doesn't have to recompute them, allowing for faster text generation.
Will a 24GB GPU run a 70B model?
Only if the model is heavily quantized (e.g., 4-bit or lower) and you have very little context/headroom.
Is more VRAM better than faster memory?
For LLMs, VRAM capacity is usually the first limit you hit, but once the model fits, memory bandwidth determines your speed.
Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

Does quantization reduce VRAM requirements?

Yes. Quantization stores weights in lower precision (like 4-bit or 8-bit), which significantly reduces the memory needed to load the model.

Why can't I just use my system RAM?

You can using tools like llama.cpp, but system RAM bandwidth is much lower than VRAM, which makes token generation much slower.

What is the KV cache?

The KV cache is memory used to store key and value vectors for past tokens so the model doesn't have to recompute them, allowing for faster text generation.

Will a 24GB GPU run a 70B model?

Only if the model is heavily quantized (e.g., 4-bit or lower) and you have very little context/headroom.

Is more VRAM better than faster memory?

For LLMs, VRAM capacity is usually the first limit you hit, but once the model fits, memory bandwidth determines your speed.