Let's talk

Next-Gen AI Hardware

RTX 5090 for AI: Capacity and Speed for LLMs

The RTX 5090 brings significant upgrades in VRAM and bandwidth, making it a powerhouse for running large language models locally.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for RTX 5090 for AI: Capacity and Speed for LLMs

RTX 5090 AI Specifications

The RTX 5090 is a high-performance consumer GPU designed for heavy AI workloads, featuring 32 GB of GDDR7 memory and 1,792 GB/s of bandwidth. This setup allows it to run large 32B models at 4-bit quantization with significantly longer context than previous generations.

  • Features 32 GB of GDDR7 memory.
  • Offers 1,792 GB/s of memory bandwidth.
  • Capable of running 32B models at 4-bit quantization with longer context compared to previous generations.

Why VRAM Matters for AI

When you are building an AI workstation for your business, the most important number to watch is VRAM (Video RAM). To run a Large Language Model (LLM), you need enough fast memory to hold the model weights plus the KV cache for your context length.

The best GPU for a local LLM is usually the one with the most VRAM you can afford. If a model does not fit in VRAM, performance drops significantly as the system spills over to the CPU, which is much slower.

Calculating Your Memory Requirements

For builders and business owners, estimating memory needs is simple arithmetic. A rough rule for weights is: parameters × bytes per parameter.

At 4-bit quantization, a 7B model needs about 3.5 GB for weights, plus 1-2 GB for runtime overhead. However, you should always leave headroom for the KV cache. A model that "just fits" will slow down during long conversations because the KV cache grows as the conversation continues.

Model Size FP16 / BF16 Weights 8-bit Weights 4-bit Weights Typical Single GPU (4-bit)
3B ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB Any 6–8 GB card
8B ≈ 16 GB ≈ 8 GB ≈ 4–5 GB 8–12 GB card
14B ≈ 28 GB ≈ 14 GB ≈ 7–9 GB 12–16 GB card
32B ≈ 64 GB ≈ 32 GB ≈ 16–20 GB 24–32 GB card
70B ≈ 140 GB ≈ 70 GB ≈ 35–42 GB 48 GB or larger

How Bandwidth Affects Token Speed

While VRAM determines if a model can run, memory bandwidth determines how fast it responds. Memory bandwidth decides how many tokens per second you get during decoding (the part where the text streams onto your screen).

A useful ceiling for speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token. In reality, actual speed is often 50 to 80 percent of this ceiling due to technical overheads. High bandwidth is critical because the chip often spends more time waiting on memory than performing actual calculations.

For example, on an RTX 4090 with 1,008 GB/s, an 8B model at 4-bit might hit a ceiling of about 220 tokens per second. On an RTX 5090, the higher bandwidth will translate directly into faster, more fluid text generation.

RTX 5090 and Mixture-of-Experts (MoE)

Mixture-of-Experts (MoE) models are a special type of architecture that can take advantage of high VRAM. These models split layers into many "expert" sub-networks, activating only a few per token. This means that while the entire model must sit in your VRAM, the chip only reads the active weights for each token.

MoE trades memory capacity for speed. It requires a lot of VRAM to hold all the experts, but because it only reads a fraction of them during decoding, it is much faster than a standard model of the same size. This makes the RTX 5090, with its 32 GB of VRAM, an ideal candidate for running these efficient models.

Consumer vs. Workstation GPUs

When deciding on hardware for your office or lab, you will likely choose between three tiers of GPUs. Understanding these helps you avoid overspending or underspecifying your system.

GPU Tier Example Hardware Memory (VRAM) Best Use Case
Consumer GeForce RTX 5090 32 GB GDDR7 32B models at 4-bit with long context
Workstation RTX PRO 6000 Blackwell 96 GB GDDR7 (ECC) 70B models, large MoE, professional serving
Datacenter H100 SXM 80 GB HBM3 High-throughput serving and training

Consumer cards like the RTX 5090 offer the best bandwidth per rupee, but they have limited VRAM compared to professional gear. Workstation cards offer much more VRAM and Error Correction Code (ECC) memory for professional use. Datacenter GPUs use HBM (High Bandwidth Memory) for much higher throughput and are built for massive-scale serving.

Should You Buy an RTX 5090 for AI?

The decision to invest in hardware depends on your specific business needs and usage patterns. Here is a guide to help you decide between buying an RTX 5090 or renting cloud compute.

When to Buy (On-Premise)

  • Data Privacy: You have sensitive data that must stay on your own machines.
  • High Usage: You have high, consistent daily usage where the hardware will be busy most of the day.
  • Predictable Costs: You want a fixed cost rather than fluctuating monthly API or cloud bills.

When to Rent (Cloud)

  • Burst Workloads: You only need short bursts of training or fine-tuning.
  • Experimentation: You want to try different hardware configurations without upfront costs.
  • Low Frequency: Your estimated monthly GPU hours are low, making ownership less cost-effective.

Frequently Asked Questions

How much VRAM does the RTX 5090 have?
The RTX 5090 features 32 GB of GDDR7 memory.
Can the RTX 5090 run a 70B model?
A 70B model at 4-bit requires roughly 35-42 GB of VRAM. Since the RTX 5090 has 32 GB, you would likely need to use a smaller quantization or a second GPU to fit the full model and KV cache.
What is the benefit of the RTX 5090's bandwidth?
The 1,792 GB/s bandwidth allows for faster token generation (decoding) because the chip spends less time waiting for data to move from memory to the processor.
Is the RTX 5090 better than a CPU for AI?
Yes, GPUs are much faster for AI inference because they have thousands of cores designed for parallel matrix multiplication, whereas CPUs are designed for branching and logic.
What is the KV cache in AI memory usage?
The KV cache stores key and value vectors for past tokens so the model doesn't have to recompute them, allowing for faster generation during long conversations.
Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

How much VRAM does the RTX 5090 have?

The RTX 5090 features 32 GB of GDDR7 memory.

Can the RTX 5090 run a 70B model?

A 70B model at 4-bit requires roughly 35-42 GB of VRAM. Since the RTX 5090 has 32 GB, you would likely need to use a smaller quantization or a second GPU to fit the full model and KV cache.

What is the benefit of the RTX 5090's bandwidth?

The 1,792 GB/s bandwidth allows for faster token generation (decoding) because the chip spends less time waiting for data to move from memory to the processor.

Is the RTX 5090 better than a CPU for AI?

Yes, GPUs are much faster for AI inference because they have thousands of cores designed for parallel matrix multiplication, whereas CPUs are designed for branching and logic.

What is the KV cache in AI memory usage?

The KV cache stores key and value vectors for past tokens so the model doesn't have to recompute them, allowing for faster generation during long conversations.