Let's talk

Hardware Guide

Choosing the Right GPU for AI

To run large language models effectively, you need to balance VRAM capacity for model weights and memory bandwidth for generation speed.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for Choosing the Right GPU for AI

Why AI runs on GPUs

To run large language models effectively, you need to balance VRAM capacity for model weights and memory bandwidth for generation speed. For most builders, the best GPU is the one with the most VRAM you can afford that still provides enough bandwidth for your required speed.

Neural networks rely heavily on matrix multiplication. Generating a single token means multiplying the current input by billions of weights. Because each of these multiplications is independent, the work can be split across thousands of small cores simultaneously.

A CPU is designed for complex, branching logic and typically has only a few dozen powerful cores. In contrast, a GPU features thousands of simpler cores designed for parallel operations, making it the ideal shape for AI work. Modern NVIDIA GPUs also include tensor cores, which are specialized units designed to multiply small matrices in a single step at reduced precision, such as FP16 or FP8.

The VRAM Rule of Thumb

When selecting hardware, memory capacity is your first hurdle. You must have enough VRAM to hold the model weights plus the KV cache. A simple rule for calculating weight requirements is: parameters × bytes per parameter.

The number of bytes used depends on the precision of the model. For example, at 16-bit precision (FP16), a 7B model requires about 14 GB. At 8-bit, it needs about 7 GB, and at 4-bit quantization, it needs roughly 3.5 GB. In practice, you should budget roughly 4.5 to 5 bits per weight to account for scaling factors.

However, you cannot simply match the model size to your VRAM. You must leave headroom for the KV cache, which grows as the conversation gets longer or as you serve more users. A good rule for builders is to choose hardware with 20% to 30% more VRAM than the model weights alone require.

Model Size FP16 Weights 8-bit Weights 4-bit Weights Typical GPU (4-bit)
3B ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB Any 6–8 GB card
8B ≈ 16 GB ≈ 8 GB ≈ 4–5 GB 8–12 GB card
70B ≈ 140 GB ≈ 70 GB ≈ 35–42 GB 48 GB or larger

Memory Bandwidth and Token Speed

Once you have enough VRAM to fit the model, the next factor is speed. During the decoding phase (when the model is generating text), the chip spends most of its time waiting on memory to read the weights. Therefore, memory bandwidth determines how many tokens per second you get.

A useful ceiling to estimate performance is: max tokens per second ≈ memory bandwidth ÷ bytes read per token. Actual speeds are typically 50% to 80% of this theoretical ceiling due to software overhead.

For example, an 8B model at 4-bit quantization on an RTX 4090 (which has 1,008 GB/s bandwidth) has a theoretical ceiling of about 220 tokens per second. A larger 70B model at 4-bit on a card with 1,792 GB/s bandwidth would have a ceiling of roughly 45 tokens per second.

GPU Tiers: Consumer, Workstation, and Datacenter

Not all GPUs are built for the same purpose. Choosing the right tier depends on whether you are building a prototype, running a small office server, or training massive models.

GPU Tier Example Memory Bandwidth Best Use Case
Consumer GeForce RTX 4090 ~1,008 GB/s Up to 32B models at 4-bit
Workstation RTX PRO 6000 Blackwell 1,792 GB/s 70B models, large MoE, serving
Datacenter H100 SXM 3.35 TB/s High-throughput serving, training
  • Consumer cards (like the RTX 4090) offer the best bandwidth per rupee, but they have limited VRAM and are designed for desktops.
  • Workstation cards (like the RTX PRO 6000) provide much higher VRAM and ECC memory for long-term reliability and running all day in a server.
  • Datacenter cards (like the H100) use HBM memory and high-speed links like NVLink. They are built for racks with massive power and cooling requirements.

Beyond Dedicated GPUs: Apple and CPUs

You don't always need a massive NVIDIA card to run AI. There are several alternative paths depending on your budget and scale.

Apple Silicon Unified Memory

Apple's M-series chips use a unified memory architecture where the CPU and GPU share a single pool of RAM. A Mac with a high memory configuration can load models that would require several consumer GPUs. While the bandwidth is lower than high-end NVIDIA cards, the capacity is incredibly high and the system is very efficient. Tools like llama.cpp work exceptionally well on Apple hardware.

Running on CPUs

It is possible to run quantized models on a CPU using system RAM. However, because system RAM bandwidth is much lower than GPU VRAM, speed is significantly limited. For example, dual-channel DDR5-5600 provides roughly 90 GB/s, which is about a tenth of a high-end GPU. A 40 GB model on a typical desktop might only top out at 2 tokens per second.

NPUs in Laptops and Phones

Neural Processing Units (NPUs) are small, low-power accelerators found in modern laptops and phones. These are designed for small-scale, on-device tasks like summarizing notifications or transcription, typically handling models between 1B and 4B parameters.

Should You Rent or Buy AI Hardware?

For business owners, the decision to buy hardware or rent it from a cloud provider is a matter of economics and data privacy. Use the following guide to help decide.

Situation Better Choice
Exploring models for a few hours a week Rent (No upfront cost)
Short training or fine-tuning bursts Rent (Datacenter GPUs)
Bulk generation many hours every day Buy/Lease (Dedicated server)
Sensitive data must stay on-premise Buy (On-premise control)
Spiky public traffic Rent/API (Scale as needed)

A simple way to decide is to estimate your monthly GPU hours. If the hardware will be busy most of the day for a year or more, owning or leasing a dedicated machine usually wins. If the hardware would sit idle, renting is much more cost-effective.

Keep in mind that high-end GPUs are essentially heaters. A single workstation can draw close to a kilowatt under load. If you are setting up hardware in an office in India, ensure you have a robust UPS or inverter sized for that load and adequate cooling for the room.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

How much VRAM do I need for an LLM?

You need enough to hold the model weights (parameters × bytes per parameter) plus extra for the KV cache and runtime overhead. For example, an 8B model at 4-bit needs about 4-5 GB of VRAM.

What is the difference between VRAM and memory bandwidth?

VRAM is the total capacity to hold the model; memory bandwidth is the speed at which that data is read. Capacity determines if a model can run; bandwidth determines how fast it generates text.

Can I run LLMs on my CPU?

Yes, using tools like llama.cpp, you can run quantized models on a CPU. However, because system RAM bandwidth is much lower than GPU VRAM, the generation speed will be significantly slower.

What is the KV cache?

The KV cache stores key and value vectors for past tokens so they don't need to be recomputed. It grows linearly with the length of the conversation and the number of users served.

Why are GPUs better than CPUs for AI?

AI work consists of massive parallel matrix multiplications. GPUs have thousands of simple cores to handle these in parallel, whereas CPUs are designed for complex, branching logic.