The short answer: memory first, bandwidth second
To run a large language model (LLM), you need enough fast memory to hold the model weights plus the KV cache for your context length. A rough rule: parameters × bytes per parameter. An 8B model needs about 16 GB at FP16, 8 GB at 8-bit and 4 to 5 GB at 4-bit. Once the model fits, memory bandwidth decides how many tokens per second you get. That is why GPUs, with their very fast VRAM, beat CPUs for AI inference, and why the best GPU for a local LLM is usually the one with the most VRAM you can afford.
This guide is written for builders and business owners who want to run open models on their own hardware, or who want to understand what they are paying for when they rent it. Every number here comes from simple arithmetic or a published specification, so you can redo the maths for any model.
If you are new to how LLMs work, start with the LLM guide, then come back here.
Why AI runs on GPUs
A neural network is mostly matrix multiplication. Generating one token means multiplying the current input by billions of weights. Each multiplication is independent of the others, so the work can be split across thousands of small cores at once.
A CPU has a few dozen powerful cores designed for branching, varied logic. A GPU has thousands of simpler cores designed to do the same operation on large blocks of numbers in parallel. That is exactly the shape of AI work.
Tensor cores and low precision
Modern NVIDIA GPUs add tensor cores: units that multiply small matrices in a single step, at reduced precision such as FP16, BF16, FP8 and, on recent generations, FP4. AMD and Apple have their own equivalents. Lower precision means less memory per number and more maths per second, which is why quantization (storing weights in 8 or 4 bits) is so common for inference.
Memory sits next to the chip
The second reason is memory. GPU memory (VRAM) is soldered next to the processor on a very wide bus. Consumer cards use GDDR memory; datacenter cards use HBM, stacked memory with even more bandwidth. This matters more than raw compute for most LLM work, as the bandwidth section shows.
How much VRAM do I need? The memory rule of thumb
Model weights take parameters × bytes per parameter. FP16 and BF16 use 2 bytes, 8-bit uses 1 byte and 4-bit uses half a byte. So a 7B model is about 14 GB at FP16, about 7 GB at 8-bit and about 3.5 GB at 4-bit.
Real 4-bit files are a little larger, because popular formats keep some layers at higher precision and store scaling factors. Budget roughly 4.5 to 5 bits per weight. Then add the KV cache (next section) and 1 to 2 GB of runtime overhead for the framework and activations.
| Model size | FP16 / BF16 weights | 8-bit weights | 4-bit weights | Typical single GPU at 4-bit |
|---|---|---|---|---|
| 3B | ≈ 6 GB | ≈ 3 GB | ≈ 1.5–2 GB | Any 6–8 GB card, laptops, phones |
| 8B | ≈ 16 GB | ≈ 8 GB | ≈ 4–5 GB | 8–12 GB card |
| 14B | ≈ 28 GB | ≈ 14 GB | ≈ 7–9 GB | 12–16 GB card |
| 32B | ≈ 64 GB | ≈ 32 GB | ≈ 16–20 GB | 24–32 GB card |
| 70B | ≈ 140 GB | ≈ 70 GB | ≈ 35–42 GB | 48 GB or larger, or two GPUs |
Weights only. Leave headroom for context: a model that "just fits" will fail or slow down once a long conversation fills the KV cache.
The KV cache: why long context costs memory
While generating text, a transformer stores a key and a value vector for every past token in every layer, so it does not recompute them. This is the KV cache. It grows linearly with context length and with the number of users served at once.
The formula:
KV cache bytes = 2 × layers × KV heads × head dimension × bytes per value × tokens
The 2 is for keys and values.
Worked example
Take a Llama 3.1 8B style model: 32 layers, 8 KV heads, head dimension 128, cache stored in FP16 (2 bytes).
- Per token: 2 × 32 × 8 × 128 × 2 = 131,072 bytes, which is 128 KiB.
- At 8,192 tokens: about 1 GiB.
- At 131,072 tokens (128K context): about 16 GiB, which is more than the 4-bit weights themselves.
Serve ten users at 8K context each and you need about 10 GiB of cache. That is why serving many users needs much more memory than chatting alone.
Grouped-query attention shrinks it
Older designs gave every attention head its own keys and values. Grouped-query attention (GQA) lets several query heads share one KV head. In the example above, the model has 32 query heads but only 8 KV heads, so the cache is four times smaller than it would otherwise be. Many runtimes can also quantize the KV cache to 8-bit, halving it again with a small quality cost.
Memory bandwidth decides token speed
For one user, generating each new token (decoding) means reading every active weight from memory once. The arithmetic per weight is tiny, so the chip spends most of its time waiting on memory. A useful ceiling:
max tokens per second ≈ memory bandwidth ÷ bytes read per token
Real speed is lower, often 50 to 80 percent of this ceiling, because of overheads. But the estimate is good for comparing hardware.
Two examples
- An 8B model at 4-bit is about 4.5 GB. On an RTX 4090 (1,008 GB/s), the ceiling is about 1,008 ÷ 4.5 ≈ 220 tokens per second.
- A 70B model at 4-bit is about 40 GB. On a card with 1,792 GB/s, the ceiling is about 45 tokens per second.
Reading your prompt (prefill) is different: it processes many tokens at once, so it is limited by compute, not bandwidth. That is why a long prompt has a pause before the first word, then words stream steadily.
Why mixture-of-experts models are fast
A mixture-of-experts (MoE) model splits its layers into many expert sub-networks and routes each token to only a few. All weights must sit in memory, but each token reads only the active ones. Google's Gemma 4 26B A4B, released in April 2026, has about 25 billion total parameters but activates about 3.8 billion per token (model card). It needs memory like a 26B model but decodes at a speed closer to a 4B model. MoE trades memory capacity for speed, which suits GPUs with large VRAM.
Consumer, workstation and datacenter GPUs
GPUs for AI fall into three tiers. The table lists published memory specifications; it leaves out prices because they move quickly and vary by country.
| GPU | Tier | Memory | Bandwidth | Good for |
|---|---|---|---|---|
| GeForce RTX 4090 | Consumer | 24 GB GDDR6X | 1,008 GB/s | Up to about 32B at 4-bit |
| GeForce RTX 5090 | Consumer | 32 GB GDDR7 | 1,792 GB/s | 32B at 4-bit with longer context |
| RTX PRO 6000 Blackwell | Workstation | 96 GB GDDR7 with ECC | 1,792 GB/s | 70B at 4–8-bit, large MoE, small-team serving |
| H100 SXM | Datacenter | 80 GB HBM3 | 3.35 TB/s | High-throughput serving, training |
| H200 | Datacenter | 141 GB HBM3e | 4.8 TB/s | Large models, long context, many users |
Sources: NVIDIA RTX PRO 6000 Blackwell, RTX 5090 vs 4090 comparison. Datacenter figures are for the SXM form factor.
What the tiers really differ on
- Consumer cards give the best bandwidth per rupee but limited VRAM. Their drivers and cooling are designed for desktops.
- Workstation cards trade some price efficiency for much more VRAM, ECC memory and support for running all day in a server.
- Datacenter cards use HBM and fast GPU-to-GPU links (NVLink), so many of them can act as one large accelerator. They are built for racks with serious power and cooling.
For a single user running local models, VRAM capacity is usually the first limit you hit. Pick the card that fits your target model with context headroom, then compare bandwidth.
Apple Silicon, CPUs with system RAM, and NPUs
Apple Silicon unified memory
Apple's M-series chips put CPU, GPU and memory in one package and share a single pool of unified memory. A Mac with a large memory configuration can load models that would need several consumer GPUs. Apple lists the M3 Ultra with up to 512 GB at 819 GB/s and the M4 Max with up to 128 GB at 546 GB/s. Bandwidth is below a top NVIDIA card, so tokens per second are lower for the same model, but capacity is very high and the machine is quiet and efficient. Tools such as llama.cpp, Ollama and Apple's MLX run well on it.
Running on a CPU with llama.cpp and system RAM
llama.cpp runs quantized models on ordinary CPUs. The limit is again bandwidth. Dual-channel DDR5-5600 has a theoretical peak of 5,600 MT/s × 8 bytes × 2 channels ≈ 90 GB/s, roughly a tenth of a high-end GPU. So a 40 GB model (70B at 4-bit) tops out at about 2 tokens per second on a typical desktop.
Servers with many memory channels do much better, and CPU inference is perfectly fine for small models, background batch jobs and embeddings. For RAM sizing: have at least as much system RAM as the model file, plus room for the operating system. When a model does not fit in VRAM, llama.cpp can split layers between GPU and CPU; it works, but the CPU half sets the pace.
NPUs and phones
Neural processing units (NPUs) in laptops and phones are small accelerators designed for low-power AI tasks. They run compact models of roughly 1B to 4B parameters for on-device features such as summarising notifications, transcription and smart replies. Phone memory is shared with the rest of the system, so on-device models stay small and heavily quantized.
Training vs fine-tuning vs inference
The hardware need depends heavily on what you do with the model.
| Task | Memory need (rough) | Typical hardware |
|---|---|---|
| Inference | Weights + KV cache; 4-bit cuts weights to about a quarter of FP16 | One consumer or workstation GPU, a Mac, or a CPU for small models |
| Fine-tuning with LoRA / QLoRA | Frozen (often 4-bit) base weights + small adapters + activations | One 24–96 GB GPU for models up to about 70B with QLoRA |
| Full fine-tuning | Around 16 bytes per parameter with mixed precision and the Adam optimizer, before activations | Several datacenter GPUs even for a 7B model |
| Pre-training from scratch | Same per-parameter cost, across trillions of tokens | Clusters of thousands of datacenter GPUs |
Full training needs memory for weights, gradients and optimizer state, so it costs many times more than inference. The QLoRA method (2023) showed a 65B model could be fine-tuned on a single 48 GB GPU by keeping the base model in 4-bit and training only small adapter layers. For most businesses, inference plus light fine-tuning or retrieval is the realistic path.
Multi-GPU setups and serving software
Splitting a model across GPUs
- Tensor parallelism splits each layer's matrices across GPUs. Every GPU works on every token, so they must talk constantly; it needs fast links such as NVLink and works best inside one server.
- Pipeline parallelism gives each GPU a block of whole layers. Tokens flow from one GPU to the next. It needs less communication but leaves GPUs idle unless many requests are in flight.
Two cards do not make one model twice as fast for a single user. Multi-GPU mainly lets you fit bigger models or serve more users.
Serving software, one line each
- Ollama
- The easiest way to download and run open models locally, with a simple API; built on llama.cpp.
- llama.cpp
- Lean C/C++ engine with GGUF quantized models that runs on CPUs, NVIDIA, AMD and Apple hardware.
- vLLM
- High-throughput server with paged KV-cache management and continuous batching for many concurrent users.
- SGLang
- Fast serving framework that reuses shared prompt prefixes, useful for agents and structured output.
- TensorRT-LLM
- NVIDIA's optimised inference library for getting the most out of NVIDIA datacenter GPUs.
Rule of thumb: Ollama or llama.cpp for one person or a few; vLLM or SGLang when a team or an application sends many requests at once.
Power, cooling and renting vs buying
A GPU is a heater. The RTX PRO 6000 Blackwell Workstation Edition is rated up to 600 W, and a high-end consumer card is in the same range. Add the CPU and the rest of the system, and one AI workstation can draw close to a kilowatt under load. Plan for a suitable power supply, a cool room and, in India, a UPS or inverter sized for that load. Datacenter GPUs need rack power and cooling that most offices do not have.
| Situation | Better choice | Why |
|---|---|---|
| Exploring models, a few hours a week | Rent | No upfront cost; try different GPUs |
| Short training or fine-tuning burst | Rent | Datacenter GPUs for days, not years |
| Bulk generation many hours every day | Buy or lease a dedicated server | Fixed cost beats per-token or per-hour bills at high use |
| Sensitive data must stay on your machine | Buy (on-premise) | You control where data goes |
| Spiky public traffic | Rent or API, plus a small owned base | Scale up only when demand arrives |
| Frontier-model quality needed | Paid API | The largest models are not available to self-host |
A simple test: estimate your monthly GPU hours. If the hardware would be busy most of the day for a year or more, owning or leasing a dedicated machine usually wins. If it would sit idle, rent.
A real example: one 96 GB GPU for daily AI work
I run open models on a dedicated server with an NVIDIA RTX PRO 6000 Blackwell (96 GB). The main writing model is a 26B mixture-of-experts model with about 4B active parameters, used for drafting in English and Hindi. A second, larger model handles vision and quality checks.
Why this setup works:
- Capacity: 96 GB holds both models plus generous KV cache, so they stay loaded and respond immediately.
- Speed: because the MoE model reads only its active parameters per token, decoding is fast even though the full model is large.
- Independence: bulk jobs, such as drafting hundreds of pages or checking content, run on owned hardware at a fixed monthly cost, so they do not depend on paid API limits or pricing.
Paid frontier APIs still have a place for the hardest reasoning tasks. The point is to route bulk, repeatable work to hardware you control. This is the same thinking behind the agentic architecture approach: pick the right model and the right machine for each step.
- Pick the model you actually need and its quantization.
- Add the KV cache for your context length and number of users.
- Choose hardware with 20 to 30 percent headroom above that total.
- Estimate speed from bandwidth, then test with your real prompts.
- Decide rent or buy from your expected monthly hours.
For where this hardware is heading next, read the future of AI, and for how we got here, the evolution of AI.
Questions people ask
How much VRAM do I need to run a 7B or 8B model?
At 4-bit quantization an 8B model needs about 4 to 5 GB for weights plus room for the KV cache, so an 8 GB card works for short contexts and 12 to 16 GB is comfortable. At FP16 it needs about 16 GB for weights alone.
How much VRAM do I need for a 70B model?
About 140 GB at FP16, 70 GB at 8-bit and roughly 35 to 42 GB at 4-bit, before the KV cache. In practice that means a 48 GB card or larger at 4-bit, or several GPUs.
Can I run an LLM on a CPU without a GPU?
Yes. llama.cpp runs quantized models on a CPU using system RAM. It works well for small models and batch jobs, but it is several times slower than a GPU because ordinary desktop RAM has much lower bandwidth than GPU memory.
Is RAM or VRAM more important for AI?
For GPU inference, VRAM matters most because the whole model and its KV cache should fit in it. System RAM matters when you run on the CPU, offload layers, or load models from disk; a good rule is at least as much RAM as the model file size, plus headroom.
Why does memory bandwidth matter more than TFLOPS for LLMs?
When one user generates text, every new token requires reading all active weights from memory once. The chip usually waits on memory, not on maths, so tokens per second tracks bandwidth closely.
Is a Mac good for running local LLMs?
Apple Silicon Macs share one pool of unified memory between CPU and GPU, so a high-memory Mac can hold models that do not fit on a single consumer GPU. They are slower than a top NVIDIA card at the same model size but quiet and power-efficient.
Should I rent cloud GPUs or buy my own?
Rent when usage is bursty, experimental or needs datacenter GPUs for a short time. Buy when you run models many hours every day, need data to stay on your own machine, or want predictable monthly cost.