The Golden Rule: Parameters × Bytes
To run large language models, you need enough fast memory to hold the model weights plus the KV cache for your context length. VRAM needs are primarily determined by model size and precision, calculated by multiplying total parameters by the bytes used per parameter.
For builders and business owners, understanding this math is the difference between a smooth-running application and a system that crashes mid-sentence. You cannot simply look at the "parameter count" of a model to know if it will fit on your hardware; you must account for how those parameters are stored.
A rough rule of thumb is: total parameters × bytes per parameter. If you are using high-precision formats like FP16 or BF16, each parameter takes up 2 bytes. If you use 8-bit quantization, each parameter takes 1 byte. At 4-bit quantization, each parameter takes only half a byte.
| Model Size | FP16 / BF16 Weights | 8-bit Weights | 4-bit Weights | Typical Single GPU (4-bit) |
|---|---|---|---|---|
| 3B | ≈ 6 GB | ≈ 3 GB | ≈ 1.5–2 GB | Any 6–8 GB card, laptops, phones |
| 8B | ≈ 16 GB | ≈ 8 GB | ≈ 4–5 GB | 8–12 GB card |
| 14B | ≈ 28 GB | ≈ 14 GB | ≈ 7–9 GB | 12–16 GB card |
| 32B | ≈ 64 GB | ≈ 32 GB | ≈ 16–20 GB | 24–32 GB card |
| 70B | ≈ 140 GB | ≈ 70 GB | ≈ 35–42 GB | 48 GB or larger, or two GPUs |
Note that "real" 4-bit files are slightly larger than the theoretical minimum. This is because popular formats keep some layers at higher precision and store scaling factors to maintain quality.
Why Long Context Costs More VRAM
Weights are a fixed cost, but the KV cache is a variable cost. While generating text, a transformer stores a key and a value vector for every past token in every layer so it does not have to recompute them. This is the KV cache, and it is what allows the model to "remember" the beginning of a long conversation.
The KV cache grows linearly with the context length (the number of tokens in the conversation) and the number of users being served at once. If you are building an application for a small Indian boutique to handle customer queries, a short context might be fine. However, if you are building a legal analysis tool that reads 50-page documents, the KV cache will become a massive memory consumer.
The mathematical formula for the cache is: 2 × layers × KV heads × head dimension × bytes per value × tokens.
Consider a Llama 3.1 8B style model with 32 layers, 8 KV heads, a head dimension of 128, and the cache stored in FP16 (2 bytes). For a single token, the cost is about 128 KiB. At a context of 8,192 tokens, you need about 1 GiB. At a massive 128K context, you need about 16 GiB—which is more than the 4-bit weights of the model itself!
Grouped-Query Attention (GQA)
To combat this, many modern models use Grouped-query attention (GQA). Older designs gave every attention head its own keys and values. GQA lets several query heads share one KV head. In the example above, a model with 32 query heads but only 8 KV heads results in a cache that is four times smaller than it would otherwise be. Many runtimes can also quantize the KV cache to 8-bit, halving the footprint again with a small quality cost.
Using Quantization to Save Memory
Quantization is the process of lowering the precision of the model weights. By moving from 16-bit to 8-bit or 4-bit, you drastically reduce the amount of VRAM required to host the model. This is the primary reason why high-end AI can run on consumer-grade hardware rather than requiring a massive server rack.
For a business owner, this means you can run a much larger, more "intelligent" model on a cheaper GPU. A 70B model that would require 140 GB of VRAM at full precision can fit on a single 48 GB GPU if you use 4-bit quantization. This makes local deployment economically viable for many small-to-medium enterprises.
Mixture-of-Experts (MoE) and Memory
Mixture-of-Experts (MoE) models, such as Google's Gemma 4 26B, use a different architecture to balance intelligence and speed. Instead of activating every single parameter for every single token, MoE models split their layers into many expert sub-networks and route each token to only a few.
There is a catch: MoE models require enough VRAM to hold all weights. Even though Gemma 4 26B only activates about 3.8 billion parameters per token, you still need enough memory to store the full 25 billion total parameters. However, because it only reads the active parameters during decoding, it provides the speed of a much smaller model while maintaining the capacity and reasoning power of a larger one. This makes MoE models an excellent choice for GPUs with high VRAM capacity.
Choosing the Right Hardware Tier
Selecting hardware depends on whether you are experimenting, serving a company, or training a custom model. The market is divided into three main tiers:
- Consumer GPUs (e.g., RTX 4090/5090): These offer the best bandwidth per rupee. They are perfect for developers and small-scale testing. However, they have limited VRAM capacity (e.g., 24 GB on a 4090 or 32 GB on a 5090), which limits how large a model you can load.
- Workstation GPUs (e.g., RTX PRO 6000 Blackwell): These are designed for professional environments. They offer much higher VRAM (up to 96 GB) and ECC (Error Correction Code) memory, which is essential for running heavy workloads all day without crashes. They are ideal for serving small teams or running large MoE models.
- Datacenter GPUs (e.g., H100/H200): These are the heavy lifters. They use HBM (High Bandwidth Memory) to provide massive throughput. These are used for high-throughput serving, training, and massive-scale inference.
For those looking for a different approach, Apple Silicon offers a unique "unified memory" architecture. A Mac with a high memory configuration can load models that would require several consumer GPUs, though the memory bandwidth is generally lower than top-tier NVIDIA cards.
Bandwidth Decides Token Speed
A common mistake is thinking that more "TFLOPS" (compute power) always means faster AI. For most LLM tasks, the bottleneck is actually memory bandwidth. Once a model fits in VRAM, the speed at which it generates text (tokens per second) is determined by how fast the chip can read the weights from memory.
During the "decoding" phase (when the model is streaming text to you), the chip spends most of its time waiting for memory to read the weights. A useful ceiling for estimating speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token.
For example, an 8B model at 4-bit is about 4.5 GB. On an RTX 4090 with 1,008 GB/s of bandwidth, the theoretical ceiling is about 220 tokens per second. In the real world, you usually see 50 to 80 percent of that due to overheads. If you are building a real-time chatbot, bandwidth is your most important metric.
How to Size Your Hardware
To avoid expensive mistakes when buying hardware or renting cloud instances, follow this sizing strategy:
- Calculate weights: Determine your model size and the quantization level you plan to use (e.g., 4-bit or 8-bit).
- Add KV cache: Estimate the memory needed for your target context length and the number of concurrent users you expect to serve.
- Add headroom: Always add 20% to 30% extra VRAM above your total. This prevents the system from slowing down or crashing when a conversation gets long.
If you are unsure where to start, you can read more about next-generation AI chips to see how hardware is evolving to meet these memory demands.

Questions people ask
How much VRAM does a 70B model need?
At 4-bit quantization, it needs about 35–42 GB for weights, plus additional VRAM for the KV cache and overhead.
Does context length affect VRAM?
Yes, the KV cache grows linearly with context length; long conversations can require more memory than the model weights themselves.
What is the difference between VRAM capacity and bandwidth?
Capacity determines if a model can fit in memory, while bandwidth determines how many tokens per second you can generate.
Can I run large models on a CPU?
Yes, using tools like llama.cpp, but performance is limited by system RAM bandwidth, which is much slower than GPU VRAM.
Why are MoE models efficient for memory?
They require enough VRAM to hold all parameters, but they only read the active parameters per token, allowing for faster decoding.
