Let's talk

Hardware Selection Guide

The Best GPU for LLM Inference

Choosing the right hardware for large language models requires balancing VRAM capacity for model weights and memory bandwidth for generation speed.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for The Best GPU for LLM Inference

The Golden Rule: Memory First, Bandwidth Second

To find the best GPU for LLM inference, prioritize VRAM capacity to fit model weights and memory bandwidth to increase token generation speed. The best GPU for a local LLM is usually the one with the most VRAM you can afford.

Choosing the right hardware for large language models requires balancing VRAM capacity for model weights and memory bandwidth for generation speed.

To run an LLM, you must have enough fast memory to hold the model weights plus the KV cache for your context length. If the model does not fit in the available memory, it will not run efficiently, or it will fail entirely.

Once the model fits in memory, memory bandwidth determines how many tokens per second you get. This is why GPUs, with their very fast VRAM, beat CPUs for AI inference. While a CPU is a generalist, a GPU is a specialist designed for the massive parallel math required by neural networks.

How Much VRAM Do You Actually Need?

A rough rule for weights is: parameters × bytes per parameter. For example, at FP16 (full precision), a model uses 2 bytes per parameter. At 8-bit quantization, it uses 1 byte, and at 4-bit, it uses half a byte.

At 4-bit quantization, an 8B model needs about 4–5 GB; a 70B model needs 35–42 GB. However, you should never buy a card that exactly matches your model size. You must always leave headroom for the KV cache (the memory used for context) and 1–2 GB of runtime overhead for the framework and activations.

A model that "just fits" will slow down or fail once a long conversation fills the KV cache. The KV cache grows linearly with context length and the number of users served at once. If you are building an application for many users, you need significantly more memory than you would for a single-user chatbot.

Model Size FP16 / BF16 weights 8-bit weights 4-bit weights Typical single GPU (4-bit)
3B ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB Any 6–8 GB card
8B ≈ 16 GB ≈ 8 GB ≈ 4–5 GB 8–12 GB card
14B ≈ 28 GB ≈ 14 GB ≈ 7–9 GB 12–16 GB card
32B ≈ 64 GB ≈ 32 GB ≈ 16–20 GB 24–32 GB card
70B ≈ 140 GB ≈ 70 GB ≈ 35–42 GB 48 GB or larger

Why Bandwidth Dictates Token Speed

During decoding (the process of generating text one word at a time), the chip spends most of its time waiting on memory, not compute. The arithmetic performed per weight is tiny, so the bottleneck is how fast the weights can be moved from memory to the processor.

A useful ceiling for performance is: max tokens per second ≈ memory bandwidth ÷ bytes read per token. While real-world speed is often 50 to 80 percent of this ceiling due to overhead, it is the best way to compare hardware.

For example, an 8B model at 4-bit is about 4.5 GB. On an RTX 4090 with 1,008 GB/s of bandwidth, the ceiling is roughly 220 tokens per second. On a 70B model at 4-bit (40 GB), a card with 1,792 GB/s of bandwidth would have a ceiling of about 45 tokens per second.

Comparing GPU Tiers: Consumer, Workstation, and Datacenter

GPUs for AI fall into three distinct categories. Choosing the right tier depends on whether you are a hobbyist, a professional designer, or a large-scale service provider.

GPU Tier Example Memory Bandwidth Best Use Case
Consumer RTX 4090 / 5090 Up to 1,792 GB/s High bandwidth per rupee; limited VRAM (24–32 GB)
Workstation RTX PRO 6000 Blackwell 1,792 GB/s High VRAM (up to 96 GB) and ECC memory for professional use
Datacenter H100 / H200 3.35–4.8 TB/s Massive bandwidth (HBM) for high-throughput serving

Consumer cards like the RTX 4090 offer the best bandwidth per rupee but have limited VRAM. They are designed for desktops and have specific cooling requirements. Workstation cards trade some price efficiency for much more VRAM and ECC (Error Correction Code) memory, making them suitable for running all day in a server. Datacenter cards use HBM (High Bandwidth Memory) and fast GPU-to-GPU links like NVLink, allowing many cards to act as one massive accelerator for high-throughput serving.

Beyond Discrete GPUs: Apple Silicon and CPUs

While dedicated GPUs are the gold standard, there are other ways to run models depending on your budget and needs.

Apple Silicon

Apple's M-series chips use unified memory, which allows the CPU and GPU to share a single pool of memory. A Mac with a large memory configuration can load models that would require several consumer GPUs. For example, the M3 Ultra can have up to 512 GB of memory. While the bandwidth is lower than a top NVIDIA card, the capacity is exceptionally high and the system is very efficient.

CPUs and System RAM

You can run LLMs on a standard CPU using tools like llama.cpp. However, the limit is memory bandwidth. Standard dual-channel DDR5 memory is roughly a tenth of the speed of a high-end GPU. A 40 GB model might only top out at 2 tokens per second on a typical desktop. CPU inference is best suited for small models, background batch jobs, or embeddings.

NPUs and Phones

Neural Processing Units (NPUs) in laptops and phones are small accelerators designed for low-power, small-scale tasks. They are meant for models with 1B to 4B parameters, such as summarizing notifications or smart replies, rather than large-scale reasoning.

The Speed Advantage of Mixture-of-Experts (MoE)

Mixture-of-Experts (MoE) models, such as Google's Gemma 4 26B, offer a unique way to get high performance. Instead of using every parameter for every token, an MoE model splits its layers into many "expert" sub-networks and routes each token to only a fraction of them.

This means that while the entire model must still sit in your VRAM, each token only needs to read the active parameters. This allows MoE models to decode at speeds closer to much smaller models while maintaining the knowledge capacity of a much larger one. MoE models benefit greatly from GPUs with large VRAM capacity.

Should You Rent or Buy Your Hardware?

For business owners, the decision to own hardware or use the cloud is a matter of economics and data privacy.

Situation Better Choice Reasoning
Exploring models or short bursts Rent No upfront cost; easy to switch GPUs
Bulk generation (many hours daily) Buy or Lease Fixed costs beat per-token/hour bills at high use
Sensitive data requirements Buy (On-premise) You maintain total control over data privacy
Spiky public traffic Rent or API Scale up only when demand actually arrives

A simple rule of thumb: estimate your monthly GPU hours. If your hardware would be busy most of the day for a year or more, owning a machine usually wins. If it would sit idle, renting is the smarter investment. For more on hardware selection, see our AI hardware guide.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

Does more VRAM make LLM inference faster?

Not necessarily. VRAM capacity determines if a model can fit; memory bandwidth determines how fast it generates text.

Can I run an LLM on my CPU?

Yes, using tools like llama.cpp, but it is much slower because system RAM has much lower bandwidth than GPU VRAM.

What is the benefit of quantization for inference?

Quantization (e.g., 4-bit or 8-bit) reduces the memory required for weights, allowing larger models to fit on smaller GPUs.

Why does a long conversation slow down the model?

As conversation length increases, the KV cache grows, consuming more memory and increasing the computational load.

What is the best way to serve multiple users?

For high-throughput serving of many users, frameworks like vLLM or SGLang are preferred over simple local runners.