Let's talk

Guide · AI hardware

GPU, VRAM, RAM and CPU for AI: what you need to run LLMs

The short answer: memory decides which model you can run, and memory bandwidth decides how fast it answers. This guide shows the arithmetic, the hardware tiers and when to rent instead of buy.

By Divyaprakash JhaUpdated 13 min read

The short answer: memory first, bandwidth second

To run a large language model (LLM), you need enough fast memory to hold the model weights plus the KV cache for your context length. A rough rule: parameters × bytes per parameter. An 8B model needs about 16 GB at FP16, 8 GB at 8-bit and 4 to 5 GB at 4-bit. Once the model fits, memory bandwidth decides how many tokens per second you get. That is why GPUs, with their very fast VRAM, beat CPUs for AI inference, and why the best GPU for a local LLM is usually the one with the most VRAM you can afford.

This guide is written for builders and business owners who want to run open models on their own hardware, or who want to understand what they are paying for when they rent it. Every number here comes from simple arithmetic or a published specification, so you can redo the maths for any model.

If you are new to how LLMs work, start with the LLM guide, then come back here.

Why AI runs on GPUs

A neural network is mostly matrix multiplication. Generating one token means multiplying the current input by billions of weights. Each multiplication is independent of the others, so the work can be split across thousands of small cores at once.

A CPU has a few dozen powerful cores designed for branching, varied logic. A GPU has thousands of simpler cores designed to do the same operation on large blocks of numbers in parallel. That is exactly the shape of AI work.

Tensor cores and low precision

Modern NVIDIA GPUs add tensor cores: units that multiply small matrices in a single step, at reduced precision such as FP16, BF16, FP8 and, on recent generations, FP4. AMD and Apple have their own equivalents. Lower precision means less memory per number and more maths per second, which is why quantization (storing weights in 8 or 4 bits) is so common for inference.

Memory sits next to the chip

The second reason is memory. GPU memory (VRAM) is soldered next to the processor on a very wide bus. Consumer cards use GDDR memory; datacenter cards use HBM, stacked memory with even more bandwidth. This matters more than raw compute for most LLM work, as the bandwidth section shows.

How much VRAM do I need? The memory rule of thumb

Model weights take parameters × bytes per parameter. FP16 and BF16 use 2 bytes, 8-bit uses 1 byte and 4-bit uses half a byte. So a 7B model is about 14 GB at FP16, about 7 GB at 8-bit and about 3.5 GB at 4-bit.

Real 4-bit files are a little larger, because popular formats keep some layers at higher precision and store scaling factors. Budget roughly 4.5 to 5 bits per weight. Then add the KV cache (next section) and 1 to 2 GB of runtime overhead for the framework and activations.

Model sizeFP16 / BF16 weights8-bit weights4-bit weightsTypical single GPU at 4-bit
3B≈ 6 GB≈ 3 GB≈ 1.5–2 GBAny 6–8 GB card, laptops, phones
8B≈ 16 GB≈ 8 GB≈ 4–5 GB8–12 GB card
14B≈ 28 GB≈ 14 GB≈ 7–9 GB12–16 GB card
32B≈ 64 GB≈ 32 GB≈ 16–20 GB24–32 GB card
70B≈ 140 GB≈ 70 GB≈ 35–42 GB48 GB or larger, or two GPUs

Weights only. Leave headroom for context: a model that "just fits" will fail or slow down once a long conversation fills the KV cache.

A 24 GB VRAM budget split into model weights, KV cache, runtime overhead and free headroom Example VRAM budget: 32B model at 4-bit on a 24 GB card Weights ≈ 18 GB KV ≈ 3 GB Gold: weights (fixed) Blue: KV cache (grows with context) Grey: runtime overhead ≈ 1.5 GB · Dark: free headroom ≈ 1.5 GB Illustrative arithmetic, not a benchmark. Longer context eats the headroom first.
Weights are a fixed cost. The KV cache grows with every token of context.

The KV cache: why long context costs memory

While generating text, a transformer stores a key and a value vector for every past token in every layer, so it does not recompute them. This is the KV cache. It grows linearly with context length and with the number of users served at once.

The formula:

KV cache bytes = 2 × layers × KV heads × head dimension × bytes per value × tokens

The 2 is for keys and values.

Worked example

Take a Llama 3.1 8B style model: 32 layers, 8 KV heads, head dimension 128, cache stored in FP16 (2 bytes).

  • Per token: 2 × 32 × 8 × 128 × 2 = 131,072 bytes, which is 128 KiB.
  • At 8,192 tokens: about 1 GiB.
  • At 131,072 tokens (128K context): about 16 GiB, which is more than the 4-bit weights themselves.

Serve ten users at 8K context each and you need about 10 GiB of cache. That is why serving many users needs much more memory than chatting alone.

Grouped-query attention shrinks it

Older designs gave every attention head its own keys and values. Grouped-query attention (GQA) lets several query heads share one KV head. In the example above, the model has 32 query heads but only 8 KV heads, so the cache is four times smaller than it would otherwise be. Many runtimes can also quantize the KV cache to 8-bit, halving it again with a small quality cost.

Memory bandwidth decides token speed

For one user, generating each new token (decoding) means reading every active weight from memory once. The arithmetic per weight is tiny, so the chip spends most of its time waiting on memory. A useful ceiling:

max tokens per second ≈ memory bandwidth ÷ bytes read per token

Real speed is lower, often 50 to 80 percent of this ceiling, because of overheads. But the estimate is good for comparing hardware.

Two examples

  • An 8B model at 4-bit is about 4.5 GB. On an RTX 4090 (1,008 GB/s), the ceiling is about 1,008 ÷ 4.5 ≈ 220 tokens per second.
  • A 70B model at 4-bit is about 40 GB. On a card with 1,792 GB/s, the ceiling is about 45 tokens per second.

Reading your prompt (prefill) is different: it processes many tokens at once, so it is limited by compute, not bandwidth. That is why a long prompt has a pause before the first word, then words stream steadily.

Why mixture-of-experts models are fast

A mixture-of-experts (MoE) model splits its layers into many expert sub-networks and routes each token to only a few. All weights must sit in memory, but each token reads only the active ones. Google's Gemma 4 26B A4B, released in April 2026, has about 25 billion total parameters but activates about 3.8 billion per token (model card). It needs memory like a 26B model but decodes at a speed closer to a 4B model. MoE trades memory capacity for speed, which suits GPUs with large VRAM.

Consumer, workstation and datacenter GPUs

GPUs for AI fall into three tiers. The table lists published memory specifications; it leaves out prices because they move quickly and vary by country.

GPUTierMemoryBandwidthGood for
GeForce RTX 4090Consumer24 GB GDDR6X1,008 GB/sUp to about 32B at 4-bit
GeForce RTX 5090Consumer32 GB GDDR71,792 GB/s32B at 4-bit with longer context
RTX PRO 6000 BlackwellWorkstation96 GB GDDR7 with ECC1,792 GB/s70B at 4–8-bit, large MoE, small-team serving
H100 SXMDatacenter80 GB HBM33.35 TB/sHigh-throughput serving, training
H200Datacenter141 GB HBM3e4.8 TB/sLarge models, long context, many users

Sources: NVIDIA RTX PRO 6000 Blackwell, RTX 5090 vs 4090 comparison. Datacenter figures are for the SXM form factor.

What the tiers really differ on

  • Consumer cards give the best bandwidth per rupee but limited VRAM. Their drivers and cooling are designed for desktops.
  • Workstation cards trade some price efficiency for much more VRAM, ECC memory and support for running all day in a server.
  • Datacenter cards use HBM and fast GPU-to-GPU links (NVLink), so many of them can act as one large accelerator. They are built for racks with serious power and cooling.

For a single user running local models, VRAM capacity is usually the first limit you hit. Pick the card that fits your target model with context headroom, then compare bandwidth.

Apple Silicon, CPUs with system RAM, and NPUs

Apple Silicon unified memory

Apple's M-series chips put CPU, GPU and memory in one package and share a single pool of unified memory. A Mac with a large memory configuration can load models that would need several consumer GPUs. Apple lists the M3 Ultra with up to 512 GB at 819 GB/s and the M4 Max with up to 128 GB at 546 GB/s. Bandwidth is below a top NVIDIA card, so tokens per second are lower for the same model, but capacity is very high and the machine is quiet and efficient. Tools such as llama.cpp, Ollama and Apple's MLX run well on it.

Running on a CPU with llama.cpp and system RAM

llama.cpp runs quantized models on ordinary CPUs. The limit is again bandwidth. Dual-channel DDR5-5600 has a theoretical peak of 5,600 MT/s × 8 bytes × 2 channels ≈ 90 GB/s, roughly a tenth of a high-end GPU. So a 40 GB model (70B at 4-bit) tops out at about 2 tokens per second on a typical desktop.

Servers with many memory channels do much better, and CPU inference is perfectly fine for small models, background batch jobs and embeddings. For RAM sizing: have at least as much system RAM as the model file, plus room for the operating system. When a model does not fit in VRAM, llama.cpp can split layers between GPU and CPU; it works, but the CPU half sets the pace.

NPUs and phones

Neural processing units (NPUs) in laptops and phones are small accelerators designed for low-power AI tasks. They run compact models of roughly 1B to 4B parameters for on-device features such as summarising notifications, transcription and smart replies. Phone memory is shared with the rest of the system, so on-device models stay small and heavily quantized.

Training vs fine-tuning vs inference

The hardware need depends heavily on what you do with the model.

TaskMemory need (rough)Typical hardware
InferenceWeights + KV cache; 4-bit cuts weights to about a quarter of FP16One consumer or workstation GPU, a Mac, or a CPU for small models
Fine-tuning with LoRA / QLoRAFrozen (often 4-bit) base weights + small adapters + activationsOne 24–96 GB GPU for models up to about 70B with QLoRA
Full fine-tuningAround 16 bytes per parameter with mixed precision and the Adam optimizer, before activationsSeveral datacenter GPUs even for a 7B model
Pre-training from scratchSame per-parameter cost, across trillions of tokensClusters of thousands of datacenter GPUs

Full training needs memory for weights, gradients and optimizer state, so it costs many times more than inference. The QLoRA method (2023) showed a 65B model could be fine-tuned on a single 48 GB GPU by keeping the base model in 4-bit and training only small adapter layers. For most businesses, inference plus light fine-tuning or retrieval is the realistic path.

Multi-GPU setups and serving software

Splitting a model across GPUs

  • Tensor parallelism splits each layer's matrices across GPUs. Every GPU works on every token, so they must talk constantly; it needs fast links such as NVLink and works best inside one server.
  • Pipeline parallelism gives each GPU a block of whole layers. Tokens flow from one GPU to the next. It needs less communication but leaves GPUs idle unless many requests are in flight.

Two cards do not make one model twice as fast for a single user. Multi-GPU mainly lets you fit bigger models or serve more users.

Serving software, one line each

Ollama
The easiest way to download and run open models locally, with a simple API; built on llama.cpp.
llama.cpp
Lean C/C++ engine with GGUF quantized models that runs on CPUs, NVIDIA, AMD and Apple hardware.
vLLM
High-throughput server with paged KV-cache management and continuous batching for many concurrent users.
SGLang
Fast serving framework that reuses shared prompt prefixes, useful for agents and structured output.
TensorRT-LLM
NVIDIA's optimised inference library for getting the most out of NVIDIA datacenter GPUs.

Rule of thumb: Ollama or llama.cpp for one person or a few; vLLM or SGLang when a team or an application sends many requests at once.

Power, cooling and renting vs buying

A GPU is a heater. The RTX PRO 6000 Blackwell Workstation Edition is rated up to 600 W, and a high-end consumer card is in the same range. Add the CPU and the rest of the system, and one AI workstation can draw close to a kilowatt under load. Plan for a suitable power supply, a cool room and, in India, a UPS or inverter sized for that load. Datacenter GPUs need rack power and cooling that most offices do not have.

SituationBetter choiceWhy
Exploring models, a few hours a weekRentNo upfront cost; try different GPUs
Short training or fine-tuning burstRentDatacenter GPUs for days, not years
Bulk generation many hours every dayBuy or lease a dedicated serverFixed cost beats per-token or per-hour bills at high use
Sensitive data must stay on your machineBuy (on-premise)You control where data goes
Spiky public trafficRent or API, plus a small owned baseScale up only when demand arrives
Frontier-model quality neededPaid APIThe largest models are not available to self-host

A simple test: estimate your monthly GPU hours. If the hardware would be busy most of the day for a year or more, owning or leasing a dedicated machine usually wins. If it would sit idle, rent.

A real example: one 96 GB GPU for daily AI work

I run open models on a dedicated server with an NVIDIA RTX PRO 6000 Blackwell (96 GB). The main writing model is a 26B mixture-of-experts model with about 4B active parameters, used for drafting in English and Hindi. A second, larger model handles vision and quality checks.

Why this setup works:

  • Capacity: 96 GB holds both models plus generous KV cache, so they stay loaded and respond immediately.
  • Speed: because the MoE model reads only its active parameters per token, decoding is fast even though the full model is large.
  • Independence: bulk jobs, such as drafting hundreds of pages or checking content, run on owned hardware at a fixed monthly cost, so they do not depend on paid API limits or pricing.

Paid frontier APIs still have a place for the hardest reasoning tasks. The point is to route bulk, repeatable work to hardware you control. This is the same thinking behind the agentic architecture approach: pick the right model and the right machine for each step.

  1. Pick the model you actually need and its quantization.
  2. Add the KV cache for your context length and number of users.
  3. Choose hardware with 20 to 30 percent headroom above that total.
  4. Estimate speed from bandwidth, then test with your real prompts.
  5. Decide rent or buy from your expected monthly hours.

For where this hardware is heading next, read the future of AI, and for how we got here, the evolution of AI.

Questions people ask

How much VRAM do I need to run a 7B or 8B model?

At 4-bit quantization an 8B model needs about 4 to 5 GB for weights plus room for the KV cache, so an 8 GB card works for short contexts and 12 to 16 GB is comfortable. At FP16 it needs about 16 GB for weights alone.

How much VRAM do I need for a 70B model?

About 140 GB at FP16, 70 GB at 8-bit and roughly 35 to 42 GB at 4-bit, before the KV cache. In practice that means a 48 GB card or larger at 4-bit, or several GPUs.

Can I run an LLM on a CPU without a GPU?

Yes. llama.cpp runs quantized models on a CPU using system RAM. It works well for small models and batch jobs, but it is several times slower than a GPU because ordinary desktop RAM has much lower bandwidth than GPU memory.

Is RAM or VRAM more important for AI?

For GPU inference, VRAM matters most because the whole model and its KV cache should fit in it. System RAM matters when you run on the CPU, offload layers, or load models from disk; a good rule is at least as much RAM as the model file size, plus headroom.

Why does memory bandwidth matter more than TFLOPS for LLMs?

When one user generates text, every new token requires reading all active weights from memory once. The chip usually waits on memory, not on maths, so tokens per second tracks bandwidth closely.

Is a Mac good for running local LLMs?

Apple Silicon Macs share one pool of unified memory between CPU and GPU, so a high-memory Mac can hold models that do not fit on a single consumer GPU. They are slower than a top NVIDIA card at the same model size but quiet and power-efficient.

Should I rent cloud GPUs or buy my own?

Rent when usage is bursty, experimental or needs datacenter GPUs for a short time. Buy when you run models many hours every day, need data to stay on your own machine, or want predictable monthly cost.