Let's talk

Hardware Guide

GPU vs CPU for AI Inference

Choosing between a GPU and a CPU for AI inference depends on your model size, required speed, and available memory bandwidth.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for GPU vs CPU for AI Inference

The Short Answer: Speed vs. Capacity

GPUs are significantly faster for LLM inference because their high memory bandwidth allows for much higher tokens per second. CPUs are suitable for small models, background batch jobs, or embeddings, but they are limited by much lower memory bandwidth. The primary bottleneck for inference is usually memory bandwidth, not raw compute power.

Why GPUs Excel at AI Workloads

Neural networks are essentially massive collections of math problems. Most of the work involves matrix multiplication, which is a task that can be split across thousands of small, parallel cores. While a CPU is designed to handle complex logic and branching, a GPU is built to do the same operation on massive blocks of numbers all at once.

Modern NVIDIA GPUs include specialized hardware called Tensor Cores. These are designed to multiply small matrices in a single step at reduced precision, such as FP16 or FP8. This specialized math, combined with massive parallelization, is why GPUs dominate the field.

Furthermore, the physical design of a GPU is optimized for speed. GPU memory (VRAM) is soldered directly next to the processor on a very wide bus. Consumer cards typically use GDDR memory, while high-end datacenter cards use HBM (High Bandwidth Memory), which is stacked and offers even more massive bandwidth.

Understanding CPU Inference Limits

CPUs are the "brains" of your computer, designed to handle a few dozen powerful cores that manage complex logic. However, AI inference is not about complex logic; it is about moving massive amounts of data very quickly. This is where CPUs struggle.

The main bottleneck for a CPU is system RAM bandwidth. For example, dual-channel DDR5-5600 memory has a theoretical peak bandwidth of roughly 90 GB/s. This means a CPU is roughly a tenth the speed of a high-end GPU when it comes to moving data.

To put this in perspective: if you try to run a 70B model (at 4-bit quantization) on a typical desktop CPU, you might find it tops out at only about 2 tokens per second. This is far too slow for a smooth, conversational experience, making CPUs better suited for small models or background tasks like embeddings.

The Golden Rule: Memory First, Bandwidth Second

If you are a builder or a business owner looking to invest in hardware, you must understand the relationship between memory and speed. To run a Large Language Model (LLM), you must first satisfy the capacity requirement: you must have enough VRAM to hold the model weights plus the KV cache for your context length.

A simple way to calculate the memory you need is: parameters × bytes per parameter. For example, an 8B model requires about 16 GB at FP16 precision, but only 4 to 5 GB if you use 4-bit quantization.

Once the model actually fits into your memory, the memory bandwidth becomes the deciding factor for speed. This is why the best GPU for a local LLM is usually the one with the most VRAM you can afford. If the model doesn't fit, it won't run efficiently, regardless of how many "teraflops" your chip claims to have.

How Much VRAM Do You Need?

Calculating VRAM is more complex than just looking at the model file size. You have to account for two main things: the weights and the KV cache.

Model weights are a fixed cost. However, the KV cache—which stores the "memory" of the conversation so the model doesn't have to recompute every previous word—grows linearly with your context length. If you are chatting alone, the cache might only take 1 GB. But if you are serving ten users at an 8K context each, you might need an additional 10 GB of VRAM just for the cache.

To manage this, many developers use quantization. By using 4-bit or 8-bit weights instead of full 16-bit precision, you can significantly reduce the memory footprint. This makes it possible to run large models on consumer-grade hardware. However, you should always leave 1–2 GB of headroom for runtime overhead and the growing KV cache to avoid system slowdowns or crashes.

Comparing Hardware Tiers

Not all GPUs are created equal. Depending on whether you are a hobbyist or an enterprise, your hardware needs will fall into one of three tiers:

GPU Tier Example Hardware Memory Type/Capacity Best Use Case
Consumer RTX 4090 / 5090 24–32 GB GDDR6X/GDDR7 Best bandwidth per rupee; great for individual developers.
Workstation RTX PRO 6000 Blackwell 96 GB GDDR7 (with ECC) Large models (70B+), MoE models, and small-team serving.
Datacenter H100 / H200 80–141 GB HBM3/HBM3e High-throughput serving, massive scale, and training.

For most businesses, the choice depends on scale. Consumer cards give you the best "bang for your buck" but have limited capacity. Workstation cards offer much more VRAM and error-correcting code (ECC) memory for professional stability. Datacenter cards are built for racks, requiring specialized cooling and high-speed links like NVLink to act as a single massive accelerator.

The Apple Silicon Alternative

Apple has taken a unique approach with its M-series chips. Instead of having separate memory for the CPU and the GPU, Apple uses unified memory. This allows the GPU to access a single, large pool of RAM that is shared with the CPU.

This architecture is a massive advantage for running large models. A Mac with a high-end memory configuration can load models that would require multiple expensive NVIDIA GPUs to hold. For example, an M3 Ultra can support up to 512 GB of memory. While the bandwidth (speed) of Apple's memory is lower than a top-tier NVIDIA card, the massive capacity makes it an incredibly efficient and quiet option for running local models via tools like Ollama or MLX.

Frequently Asked Questions

Why is memory bandwidth more important than compute for LLMs?
During the "decoding" phase (when the model is generating text word-by-word), the chip spends most of its time waiting to read the model weights from memory. Because the math required for each token is relatively small, the speed at which you can move those weights from memory to the processor determines your actual speed.
Can I run a large model on my CPU?
Yes, using tools like llama.cpp, you can run quantized models on an ordinary CPU. However, it will be significantly slower than a GPU because your system RAM cannot feed the processor data nearly as fast as VRAM can.
What is the KV cache and why does it matter?
The KV cache stores the "key" and "value" vectors for every past token in a conversation. This prevents the model from having to re-calculate everything every time it generates a new word. Because it stores this data, the cache grows as your conversation gets longer, consuming more VRAM.
When should I rent a GPU instead of buying one?
If you only need high-end hardware for occasional bursts or for a specific training job, renting is much more cost-effective. If you plan to run bulk generation for many hours every single day, owning or leasing dedicated hardware usually becomes the cheaper option in the long run.
Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

Why is memory bandwidth more important than compute for LLMs?

During decoding, the chip spends most of its time waiting to read weights from memory, making bandwidth the primary driver of token speed.

Can I run a large model on my CPU?

Yes, using tools like llama.cpp, you can run quantized models on a CPU, but it will be much slower than a GPU due to lower memory bandwidth.

What is the KV cache and why does it matter?

The KV cache stores key and value vectors for past tokens so they don't need to be recomputed; it grows with your conversation length and consumes VRAM.

Does quantization affect performance?

Quantization (e.g., 4-bit vs FP16) reduces the memory footprint and increases math speed, which is essential for running large models on limited hardware.

When should I rent a GPU instead of buying one?

If you only need high-end hardware for occasional bursts or training, renting is more cost-effective; if you use it daily for bulk generation, buying may win.