How Much VRAM for a 70B Model?
To run a 70B parameter model, you need 35–42 GB of VRAM for 4-bit weights. Always leave headroom for the KV cache and runtime overhead to ensure the model does not slow down or fail during long conversations.
Running a 70B parameter model requires significant memory capacity. The amount of VRAM you need depends heavily on your chosen quantization level.
The VRAM Calculation Rule
Calculating memory requirements is straightforward once you understand the relationship between parameters and precision. The fundamental rule is: parameters × bytes per parameter.
When you load a model, the weights take up a fixed amount of space. However, the precision (the number of bits used to represent each weight) determines how much VRAM is consumed. For example, at FP16/BF16 (2 bytes per parameter), a 70B model needs ~140 GB. If you use 8-bit quantization (1 byte per parameter), it needs ~70 GB. Most builders use 4-bit quantization to save space, which requires roughly 35–42 GB due to scaling factors.
A quick way to estimate your needs is to look at the following breakdown:
| 7B Model | ~14 GB | ~7 GB | ~3.5–5 GB |
| 32B Model | ~64 GB | ~32 GB | ~16–20 GB |
| 70B Model | ~140 GB | ~70 GB | ~35–42 GB |
Why You Need Headroom: The KV Cache
A common mistake for builders is choosing a GPU that "just fits" the model weights. While the model might load, it will likely fail or slow down as soon as the conversation gets longer. This is because of the KV cache.
While generating text, a transformer stores a key and a value vector for every past token in every layer so it does not have to recompute them. This cache grows linearly with the context length and the number of users being served at once. If you are building an application for many users, the memory required for the cache can become massive.
For example, in a model with a long context window (like 128K tokens), the KV cache can require more memory than the weights themselves. If you are serving ten users at an 8K context each, you might need an additional 10 GB of VRAM just for the cache.
Hardware Options for 70B Models
The hardware you choose depends on whether you are a hobbyist, a workstation user, or an enterprise provider. Understanding GPU tiers is essential for budgeting.
Consumer GPUs
Consumer cards like the RTX 4090 (24 GB) are excellent for speed but limited in capacity. A single 24GB card is insufficient for a 70B model in 4-bit quantization; you would need to use two or more GPUs or offload parts of the model to your system RAM.
Workstation GPUs
Workstation cards, such as the RTX PRO 6000 Blackwell (96 GB), are designed for these larger workloads. A single card of this caliber can hold a 70B model at 4-bit or 8-bit quantization, making it ideal for small-team serving or complex local workflows.
Apple Silicon
Mac models with high unified memory, such as the M3 Ultra (up to 512 GB), allow you to load massive models that would otherwise require multiple expensive NVIDIA GPUs. While the memory capacity is high, the bandwidth is often lower than high-end GPUs, meaning text generation might be slower.
Datacenter GPUs
For high-throughput commercial applications, datacenter cards like the H100 or H200 are the industry standard. These provide massive bandwidth and capacity to serve many users simultaneously with very low latency.
VRAM Capacity vs. Bandwidth
It is vital to distinguish between capacity and speed. VRAM capacity determines if the model can fit into your hardware at all. Memory bandwidth determines how fast the model actually generates text (tokens per second).
For a single user, the generation speed (decoding) is limited by memory bandwidth. Essentially, the chip spends most of its time waiting to read the model weights from the memory. A useful rule of thumb is that your max tokens per second is roughly your memory bandwidth divided by the bytes read per token.
For example, a 70B model at 4-bit on a high-bandwidth card (1,792 GB/s) has a theoretical ceiling of about 45 tokens per second. If you move to a slower memory setup, that speed will drop significantly, even if the model fits.
Mixture-of-Experts (MoE) Efficiency
Mixture-of-Experts (MoE) models offer a clever way to get high intelligence without the massive computational cost of traditional models. In an MoE model, the layers are split into many "expert" sub-networks. When a token is processed, the model only activates a small fraction of these experts.
This has a major implication for hardware: All weights must still sit in VRAM, so you still need the capacity to hold the full model. However, because only a fraction of the weights are active per token, the model can decode at speeds closer to much smaller models. This makes MoE models highly efficient for users with high-capacity, high-bandwidth hardware.
Running 70B on a CPU
If you do not have a dedicated GPU, you can run quantized models on a CPU using tools like llama.cpp and your system RAM. This is often called "offloading."
The bottleneck here is memory bandwidth. Standard system RAM, such as dual-channel DDR5-5600, has a theoretical peak of roughly 90 GB/s. This is roughly ten times slower than a high-end GPU. Consequently, a 40 GB model (a 70B model at 4-bit) might only reach about 2 tokens per second on a typical desktop CPU. This is usable for background tasks or simple text processing, but it is not suitable for interactive, real-time applications.
Quick Reference: 70B VRAM Estimates
When planning your hardware purchase or cloud rental, use these estimates as your baseline for a 70B parameter model.
| Quantization Level | Estimated VRAM Required | Best Use Case |
|---|---|---|
| 4-bit quantization | 35–42 GB | Local use, consumer/workstation GPUs |
| 8-bit quantization | ~70 GB | High-quality local inference |
| FP16 / BF16 | ~140 GB | Research, full-precision fine-tuning |
Frequently Asked Questions
- Can I run a 70B model on a 24GB GPU?
- A single 24GB GPU like an RTX 4090 cannot fit a 70B model in 4-bit quantization (which needs 35-42 GB). You would need to use multiple GPUs or offload layers to your CPU.
- What is the KV cache?
- The KV cache stores vectors for past tokens so the model doesn't have to recompute them, allowing for faster text generation. It grows as the conversation gets longer.
- Does quantization affect VRAM usage?
- Yes. Quantization reduces the number of bits used per parameter, which significantly lowers the total VRAM required to load the model.
- Why is memory bandwidth important for LLMs?
- During text generation, the chip spends most of its time waiting to read weights from memory. Higher bandwidth results in more tokens per second.
- Is it better to buy or rent GPUs for 70B models?
- If you use the hardware for bulk generation many hours every day, buying or leasing a dedicated server is often more cost-effective than paying per-token API fees.

Questions people ask
Can I run a 70B model on a 24GB GPU?
A single 24GB GPU like an RTX 4090 cannot fit a 70B model in 4-bit quantization (which needs 35-42 GB). You would need to use multiple GPUs or offload layers to your CPU.
What is the KV cache?
The KV cache stores vectors for past tokens so the model doesn't have to recompute them, allowing for faster text generation. It grows as the conversation gets longer.
Does quantization affect VRAM usage?
Yes. Quantization reduces the number of bits used per parameter (e.g., from 16-bit to 4-bit), which significantly lowers the total VRAM required to load the model.
Why is memory bandwidth important for LLMs?
During text generation, the chip spends most of its time waiting to read weights from memory. Higher bandwidth results in more tokens per second.
Is it better to buy or rent GPUs for 70B models?
If you use the hardware for bulk generation many hours every day, buying or leasing a dedicated server is often more cost-effective than paying per-token API fees.
