Let's talk

Hardware sizing guide

VRAM Requirements for 7B, 13B, and 70B Models

To run large language models, you must account for model weights, the KV cache for context, and runtime overhead to avoid performance bottlenecks.

By Divyaprakash JhaUpdated 6 min readTopic: AI hardware

Photo for VRAM Requirements for 7B, 13B, and 70B Models

VRAM Estimation Table

To run large language models, you must account for model weights, the KV cache for context, and runtime overhead to avoid performance bottlenecks. A good rule of thumb is to choose hardware with 20% to 30% more VRAM than the model weights alone to ensure stability during long conversations.

3B ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB Any 6–8 GB card
8B ≈ 16 GB ≈ 8 GB ≈ 4–5 GB 8–12 GB card
14B ≈ 28 GB ≈ 14 GB ≈ 7–9 GB 12–16 GB card
32B ≈ 64 GB ≈ 32 GB ≈ 16–20 GB 24–32 GB card
70B ≈ 140 GB ≈ 70 GB ≈ 35–42 GB 48 GB+ or 2x GPUs

The Rule of Thumb for Memory Sizing

Estimating VRAM requirements does not require complex software; it is a matter of simple arithmetic. The fundamental formula for determining how much memory your model weights will occupy is: parameters × bytes per parameter.

The number of bytes used depends entirely on the precision of the model. If you are running a model in its original, uncompressed state, it typically uses FP16 or BF16, which requires 2 bytes per parameter. For example, an 8B model at FP16 requires roughly 16 GB of VRAM.

To save space, builders often use quantization. This process stores weights at lower precision to reduce the memory footprint:

  • 8-bit quantization: Uses 1 byte per parameter.

For a more accurate estimate, budget roughly 4.5 to 5 bits per weight.

Why Context Length Increases VRAM Needs

A common mistake for business owners is assuming that if a model "fits" in a GPU, it will run indefinitely. While model weights are a fixed cost, the KV cache is a dynamic cost that grows with every token of context.

While generating text, a transformer stores a key and a value vector for every past token in every layer so it does not have to recompute them. This is the KV cache. It grows linearly with the length of the conversation and the number of users being served simultaneously.

To understand the scale, consider a Llama 3.1 8B style model using 16-bit precision for the cache:

  • At 8,192 tokens: The cache requires about 1 GiB.

In the 128K context example, the KV cache actually requires more memory than the 4-bit weights of the model itself. If you are building an application that serves ten users at an 8K context each, you need to account for an additional 10 GiB of VRAM just for the cache.

Modern architectures like Grouped-query attention (GQA) help mitigate this by allowing several query heads to share one KV head, which can make the cache significantly smaller. Many runtimes also allow you to quantize the KV cache to 8-bit to halve its size again.

Capacity vs. Bandwidth: What Matters Most?

When selecting GPUs for LLM tasks, you must distinguish between capacity and bandwidth. They serve two very different purposes.

VRAM Capacity determines if a model can fit in memory at all. If your model requires 40 GB and your card only has 24 GB, the model simply will not run at full speed (or at all) without significant workarounds. Capacity is the "size of the bucket."

Memory Bandwidth determines how many tokens per second you get (the speed of generation). For most LLM inference work, bandwidth is more critical than raw compute power. During the "decoding" phase—when the model generates text one word at a time—the chip spends most of its time waiting for data to move from memory to the processor.

A useful theoretical ceiling for performance is:

max tokens per second ≈ memory bandwidth ÷ bytes read per token

For example, an 8B model at 4-bit (4.5 GB) on an RTX 4090 (1,008 GB/s) has a theoretical ceiling of about 220 tokens per second. A 70B model at 4-bit (40 GB) on a card with 1,792 GB/s has a ceiling of about 45 tokens per second. While real-world speeds are typically 50% to 80% of these numbers due to overhead, the ratio remains a vital comparison tool for hardware selection.

Choosing the Right Hardware Tier

Hardware for AI generally falls into three distinct tiers, depending on whether you are a hobbyist, a professional workstation user, or a high-scale enterprise.

GPU Tier Example Hardware Memory / Bandwidth Best Use Case
Consumer GeForce RTX 4090 24 GB GDDR6X / 1,008 GB/s Up to 32B models at 4-bit; best bandwidth per rupee.
Workstation RTX PRO 6000 Blackwell 96 GB GDDR7 / 1,792 GB/s 70B models at 4–8 bit; large MoE; small-team serving.
Datacenter H100 SXM 80 GB HBM3 / 3.35 TB/s High-throughput serving and massive training.
Datacenter H200 141 GB HBM3e / 4.8 TB/s Large models, long context, and many concurrent users.

Apple Silicon: The Unified Alternative

Apple's M-series chips use a unified memory architecture, meaning the CPU and GPU share the same pool of high-speed RAM. This is a massive advantage for running large models. A Mac with a high-capacity configuration (like the M3 Ultra with up to 512 GB) can load models that would require multiple expensive NVIDIA GPUs. While the bandwidth is lower than top-tier NVIDIA cards, the sheer capacity makes it a favorite for local LLM enthusiasts.

The CPU and System RAM Limitation

You can run models on a standard CPU using tools like llama.cpp, but you will face a massive performance penalty. System RAM (DDR5) has much lower bandwidth than VRAM. For example, dual-channel DDR5-5600 provides roughly 90 GB/s, which is about a tenth of a high-end GPU. A 70B model on a typical desktop might only reach 2 tokens per second.

The MoE Advantage

Mixture-of-Experts (MoE) models, such as Google's Gemma 4 26B, represent a clever way to get high performance without the massive memory requirements of traditional "dense" models. In an MoE architecture, the model's layers are split into many sub-networks, or "experts."

When the model processes a token, it only activates a small fraction of these experts. This provides a significant speed advantage: the model decodes at a speed closer to a much smaller model, even though it has a large number of total parameters. However, there is a catch: all weights must still reside in VRAM to ensure the model can switch between experts instantly. MoE trades memory capacity for speed, making it an ideal choice for GPUs with large VRAM buffers.

Frequently Asked Questions

Why can't I just use my system RAM for LLMs?
You can using tools like llama.cpp, but system RAM bandwidth is much lower than VRAM. A 70B model on standard DDR5 might only reach 2 tokens per second.
What is quantization?
Quantization is the process of storing model weights at lower precision (like 4-bit or 8-bit instead of 16-bit) to reduce VRAM requirements.
How much headroom should I leave for context?
You should choose hardware with 20% to 30% more VRAM than the model weights alone to account for the KV cache and runtime overhead.
Does more VRAM make the model faster?
Not necessarily. More VRAM allows you to run larger models or longer contexts, but memory bandwidth is what determines the generation speed.
Can I run a 70B model on a single consumer GPU?
A 70B model at 4-bit quantization requires roughly 35-42 GB. A single consumer card like the RTX 4090 (24 GB) is not enough; you would need two GPUs or a workstation/datacenter card.
Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

Why can't I just use my system RAM for LLMs?

You can using tools like llama.cpp, but system RAM bandwidth is much lower than VRAM. A 70B model on standard DDR5 might only reach 2 tokens per second.

What is quantization?

Quantization is the process of storing model weights at lower precision (like 4-bit or 8-bit instead of 16-bit) to reduce VRAM requirements.

How much headroom should I leave for context?

You should choose hardware with 20% to 30% more VRAM than the model weights alone to account for the KV cache and runtime overhead.

Does more VRAM make the model faster?

Not necessarily. More VRAM allows you to run larger models or longer contexts, but memory bandwidth is what determines the generation speed.

Can I run a 70B model on a single consumer GPU?

A 70B model at 4-bit quantization requires roughly 35-42 GB. A single consumer card like the RTX 4090 (24 GB) is not enough; you would need two GPUs or a workstation/datacenter card.