RTX 4090 VRAM and Bandwidth Specifications
The NVIDIA RTX 4090 has 24 GB of GDDR6X VRAM. This capacity allows you to run models up to approximately 32B parameters at 4-bit quantization.
The RTX 4090 features 24 GB of GDDR6X VRAM with 1,008 GB/s of bandwidth. This makes it a powerful consumer option for running local LLMs.
- The RTX 4090 comes with 24 GB of GDDR6X memory.
- It provides a memory bandwidth of 1,008 GB/s.
- This bandwidth determines the token generation speed for a single user.
What models can you run on 24 GB?
When deciding which model to run, you must calculate the memory required for the weights plus the extra space needed for conversation history. A common mistake for builders is picking a model that "just fits" the VRAM; such a model will fail or slow down once a long conversation fills the memory.
At 4-bit quantization, the 24 GB capacity can support models up to approximately 32B parameters. For an 8B model at 4-bit, the weights take about 4.5 GB, leaving significant room for context. Always leave headroom for the KV cache to prevent slowdowns or crashes during long conversations.
| Model Size | FP16 / BF16 weights | 8-bit weights | 4-bit weights | Typical single GPU (4-bit) |
|---|---|---|---|---|
| 3B | ≈ 6 GB | ≈ 3 GB | ≈ 1.5–2 GB | Any 6–8 GB card, laptops, phones |
| 8B | ≈ 16 GB | ≈ 8 GB | ≈ 4–5 GB | 8–12 GB card |
| 14B | ≈ 28 GB | ≈ 14 GB | ≈ 7–9 GB | 12–16 GB card |
| 32B | ≈ 64 GB | ≈ 32 GB | ≈ 16–20 GB | 24–32 GB card |
| 70B | ≈ 140 GB | ≈ 70 GB | ≈ 35–42 GB | 48 GB or larger, or two GPUs |
The Role of VRAM vs. Bandwidth in AI
To understand GPU for AI, you must distinguish between capacity and speed. VRAM capacity determines if a model fits on the card, while memory bandwidth determines how many tokens per second you get during decoding.
For most LLM work, the chip spends more time waiting for data from memory than performing math. This is why memory bandwidth often dictates speed more than raw compute. For an 8B model at 4-bit, the theoretical speed ceiling on a 4090 is roughly 220 tokens per second.
How Quantization Affects VRAM Usage
Quantization is the process of reducing the memory required per parameter to make large models fit on smaller hardware. This is why 4-bit or 8-bit versions of models are so common for inference.
- FP16/BF16 uses 2 bytes per parameter.
- 8-bit uses 1 byte per parameter.
Lower precision means less memory per number and more maths per second, which is why quantization is so common for inference.
Accounting for the KV Cache
The KV cache stores key and value vectors for past tokens to avoid recomputation. This is essential for generating text efficiently, but it comes at a memory cost. The cache grows linearly with context length and the number of users served.
A long context (e.g., 128K tokens) can require significantly more memory than the model weights themselves. For example, in a Llama 3.1 8B style model, a 131,072-token context can require about 16 GiB of memory, which is more than the 4-bit weights of the model itself.
Consumer vs. Workstation GPUs
Choosing the right AI hardware depends on whether you are building a single-user workstation or a multi-user server. Consumer cards like the 4090 offer the best bandwidth per rupee but have limited VRAM. They are designed for desktops and have specific cooling requirements.
Workstation cards (like the RTX PRO 6000 Blackwell) offer much higher VRAM (96 GB) and ECC memory. They trade some price efficiency for the ability to run all day in a server and support much larger models.
| GPU Tier | Example | Memory | Memory Bandwidth |
|---|---|---|---|
| Consumer | GeForce RTX 4090 | 24 GB GDDR6X | 1,008 GB/s |
| Workstation | RTX PRO 6000 Blackwell | 96 GB GDDR7 (w/ ECC) | 1,792 GB/s |
| Datacenter | H100 SXM | 80 GB HBM3 | 3.35 TB/s |
Mixture-of-Experts (MoE) and VRAM
Mixture-of-Experts (MoE) models split layers into expert sub-networks. This architecture allows MoE models to decode at speeds closer to much smaller models while maintaining high capacity. This is a highly efficient way to utilize hardware.
In an MoE model, all weights must reside in VRAM, but only a fraction are active per token. For example, a model might have 25 billion total parameters but only activate about 3.8 billion per token. This trades memory capacity for speed, which suits GPUs with large VRAM.
Frequently Asked Questions
- Can I run a 70B model on an RTX 4090?
- A 70B model at 4-bit requires roughly 35–42 GB of VRAM. Since the 4090 only has 24 GB, it will not fit entirely on a single 4090.
- Why is VRAM more important than compute for LLMs?
- For most LLM inference tasks, the chip spends more time waiting for data from memory than performing math. Therefore, memory bandwidth often dictates speed more than raw compute.
- How much VRAM is needed for an 8B model?
- An 8B model requires about 16 GB at FP16, 8 GB at 8-bit, and 4–5 GB at 4-bit, plus additional space for the KV cache and overhead.
- What is the benefit of the RTX 4090 for AI?
- It provides high memory bandwidth (1,008 GB/s) and enough VRAM (24 GB) to run medium-sized models (up to 32B at 4-bit) very quickly for a single user.
- What is the KV cache?
- The KV cache stores vectors for every past token in every layer so the model doesn't have to recompute them, allowing for faster text generation.

Questions people ask
Can I run a 70B model on an RTX 4090?
A 70B model at 4-bit requires roughly 35–42 GB of VRAM. Since the 4090 only has 24 GB, it will not fit entirely on a single 4090.
Why is VRAM more important than compute for LLMs?
For most LLM inference tasks, the chip spends more time waiting for data from memory than performing math. Therefore, memory bandwidth often dictates speed more than raw compute.
How much VRAM is needed for an 8B model?
An 8B model requires about 16 GB at FP16, 8 GB at 8-bit, and 4–5 GB at 4-bit, plus additional space for the KV cache and overhead.
What is the benefit of the RTX 4090 for AI?
It provides high memory bandwidth (1,008 GB/s) and enough VRAM (24 GB) to run medium-sized models (up to 32B at 4-bit) very quickly for a single user.
What is the KV cache?
The KV cache stores vectors for every past token in every layer so the model doesn't have to recompute them, allowing for faster text generation.
