What is VRAM in AI?
To run a large language model, you need enough fast memory to hold the model weights and the KV cache. Once the model fits, memory bandwidth decides your generation speed.
VRAM (Video RAM) is the dedicated memory located next to a GPU processor. It is essential for holding model weights and the KV cache, which is the memory used for context length. For most LLM work, memory bandwidth is more important than raw compute power.
A neural network is mostly matrix multiplication. Generating one token means multiplying the current input by billions of weights. Because each multiplication is independent, the work can be split across thousands of small cores at once. This is why GPUs, with their very fast VRAM, beat CPUs for AI inference.
How much VRAM do you need?
The memory requirement for a model follows a simple rule of thumb: parameters × bytes per parameter. However, you cannot simply use the raw model size; you must account for quantization and runtime overhead.
When using quantization—storing weights in 4-bit or 8-bit instead of full precision—you significantly reduce the memory required. For example, an 8B model requires about 16 GB at FP16, 8 GB at 8-bit, and 4 to 5 GB at 4-bit. Note that real 4-bit files are slightly larger because popular formats keep some layers at higher precision and store scaling factors. You should budget roughly 4.5 to 5 bits per weight to be safe.
Always leave headroom for the KV cache and 1 to 2 GB of runtime overhead for the framework and activations. A model that "just fits" will slow down or fail as conversation length increases. For a comfortable experience, aim for the following targets:
- 8B models: Expect to need 4–5 GB at 4-bit or 8–12 GB for a comfortable experience.
- 14B models: Aim for a 12–16 GB card.
- 32B models: You will likely need a 24–32 GB card.
- 70B models: You will need 48 GB or larger, or a setup with two GPUs.
| Model Size | 4-bit Weights (Approx) | Typical Single GPU (4-bit) |
|---|---|---|
| 3B | 1.5–2 GB | Any 6–8 GB card, laptops, phones |
| 8B | 4–5 GB | 8–12 GB card |
| 14B | 7–9 GB | 12–16 GB card |
| 32B | 16–20 GB | 24–32 GB card |
| 70B | 35–42 GB | 48 GB or larger |
The Hidden Cost: KV Cache
While model weights are a fixed cost, the KV cache grows with every token of context. While generating text, a transformer stores a key and a value vector for every past token in every layer so it does not have to recompute them. This is the KV cache.
The cache grows linearly with your context length and the number of users being served. If you are building an application for many users, the memory requirements can scale quickly. For example, serving ten users at 8K context each requires about 10 GiB of cache alone.
To mitigate this, many modern runtimes use Grouped-query attention (GQA), which lets several query heads share one KV head, making the cache much smaller. You can also quantize the KV cache to 8-bit to halve its size with a small cost to quality.
Bandwidth vs. Compute
For a single user, generating each new token (the decoding phase) means reading every active weight from memory once. Because the math performed on each weight is tiny, the chip spends most of its time waiting on memory. This is why memory bandwidth, not raw compute power, determines your tokens per second.
A useful ceiling for speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token. Real-world speed is typically 50 to 80 percent of this ceiling due to various overheads.
Consider these two scenarios:
- An 8B model at 4-bit (4.5 GB) on an RTX 4090 (1,008 GB/s) has a ceiling of about 220 tokens per second.
- A 70B model at 4-bit (40 GB) on a high-end card with 1,792 GB/s has a ceiling of about 45 tokens per second.
Note that reading your initial prompt (the prefill phase) is different; it processes many tokens at once and is limited by compute, which is why you see a pause before the first word starts streaming.
GPU Tiers and Memory Types
Not all memory is created equal. The type of memory and the way it is connected to the processor determines the performance tier of your hardware. For more details on choosing specific parts, see our guide on GPUs for AI.
| GPU Tier | Example | Memory Type | Best For |
|---|---|---|---|
| Consumer | RTX 4090 | GDDR6X | Best bandwidth per rupee; limited capacity |
| Workstation | RTX PRO 6000 | GDDR7 with ECC | Large VRAM; long-term stability; large models |
| Datacenter | H100 SXM | HBM3 | High-throughput serving and training |
Consumer cards offer the best value for a single user, but their drivers and cooling are designed for desktops. Workstation cards trade some price efficiency for much more VRAM and ECC memory for stability. Datacenter cards use HBM (High Bandwidth Memory) and fast GPU-to-GPU links like NVLink to act as one massive accelerator.
Beyond Dedicated GPUs
While dedicated GPUs are the standard, there are other ways to run models depending on your budget and needs.
Apple Silicon Unified Memory
Apple's M-series chips use a unified memory architecture where the CPU and GPU share a single pool of RAM. A Mac with a large configuration, such as an M3 Ultra with up to 512 GB, can load massive models that would require several consumer GPUs. While the bandwidth is lower than a top NVIDIA card, the capacity is exceptionally high.
CPU and System RAM
You can run models on a standard CPU using tools like local LLM software like llama.cpp. However, system RAM bandwidth is much lower—often a tenth of a high-end GPU. A 40 GB model on a typical desktop might only reach 2 tokens per second. This is fine for background tasks or small models, but not for interactive use.
NPUs
Neural Processing Units (NPUs) in phones and laptops are small accelerators designed for low-power, on-device tasks. They are meant for small models (1B–4B parameters) for tasks like transcription or smart replies, rather than large-scale LLM reasoning.
Mixture-of-Experts (MoE) and VRAM
Mixture-of-Experts (MoE) models, such as Google's Gemma 4 26B, use a clever architectural trick to stay fast. These models split layers into many "expert" sub-networks and route each token to only a few of them. This means that while the model might have 26 billion total parameters, it only activates a fraction of them (for example, 3.8 billion) for any given token.
This creates a unique hardware requirement: you must have enough VRAM to hold all the parameters in memory, but you get a generation speed closer to a much smaller model. This makes MoE models an excellent choice for hardware with large VRAM capacity but moderate bandwidth.
Frequently Asked Questions
- Does more VRAM mean faster AI?
- Not necessarily. VRAM capacity determines if a model can fit; memory bandwidth determines how fast it generates text.
- What is the best way to save VRAM?
- Quantization (storing weights in 4-bit or 8-bit instead of FP16) significantly reduces the memory required to run a model.
- Why does a long conversation slow down my LLM?
- As the conversation grows, the KV cache expands, consuming more VRAM and increasing the amount of data that must be read from memory.
- Can I use my system RAM for AI?
- Yes, using tools like llama.cpp, but system RAM bandwidth is much lower than GPU VRAM, making inference much slower.
- What is the difference between GDDR and HBM?
- GDDR is used in consumer cards and is more affordable, while HBM is stacked memory used in datacenter GPUs to provide much higher bandwidth.

Questions people ask
Does more VRAM mean faster AI?
Not necessarily. VRAM capacity determines if a model can fit; memory bandwidth determines how fast it generates text.
What is the best way to save VRAM?
Quantization (storing weights in 4-bit or 8-bit instead of FP16) significantly reduces the memory required to run a model.
Why does a long conversation slow down my LLM?
As the conversation grows, the KV cache expands, consuming more VRAM and increasing the amount of data that must be read from memory.
Can I use my system RAM for AI?
Yes, using tools like llama.cpp, but system RAM bandwidth is much lower than GPU VRAM, making inference much slower.
What is the difference between GDDR and HBM?
GDDR is used in consumer cards and is more affordable, while HBM is stacked memory used in datacenter GPUs to provide much higher bandwidth.
