Memory First, Bandwidth Second
To run a large language model (LLM), you need enough fast memory to hold the model weights plus the KV cache for your context length. Once the model fits, memory bandwidth determines your tokens per second.
Running an LLM is a heavy task for hardware. A neural network is mostly matrix multiplication. Generating one token means multiplying the current input by billions of weights. Because each multiplication is independent, the work can be split across thousands of small cores at once.
This is why GPUs beat CPUs for AI inference. A CPU has a few dozen powerful cores designed for branching and varied logic. A GPU has thousands of simpler cores designed to do the same operation on large blocks of numbers in parallel. This is exactly the shape of AI work.
However, for most LLM work, the chip spends most of its time waiting on memory, not performing calculations. This is why the best GPU for a local LLM is usually the one with the most VRAM you can afford.
Calculating VRAM Requirements
To choose the right hardware, you must first understand the "memory rule of thumb." Model weights are a fixed cost based on the number of parameters and the precision (bits) used to store them. A rough rule is: parameters × bytes per parameter.
- FP16 / BF16: 2 bytes per parameter
- 8-bit: 1 byte per parameter
When calculating your needs, do not just look at the model file size. You must budget for the KV cache. The KV cache stores key and value vectors for every past token in every layer so the model does not have to recompute them. This grows linearly with your context length and the number of users you serve at once.
You should also leave 1 to 2 GB of runtime overhead for the framework and activations. If a model "just fits" in your memory, it will likely fail or slow down once a long conversation fills the KV cache.
| Model Size | FP16 / BF16 Weights | 8-bit Weights | 4-bit Weights | Typical Single GPU (4-bit) |
|---|---|---|---|---|
| 3B | ≈ 6 GB | ≈ 3 GB | ≈ 1.5–2 GB | Any 6–8 GB card, laptops, phones |
| 8B | ≈ 16 GB | ≈ 8 GB | ≈ 4–5 GB | 8–12 GB card |
| 14B | ≈ 28 GB | ≈ 14 GB | ≈ 7–9 GB | 12–16 GB card |
| 32B | ≈ 64 GB | ≈ 32 GB | ≈ 16–20 GB | 24–32 GB card |
| 70B | ≈ 140 GB | ≈ 70 GB | ≈ 35–42 GB | 48 GB or larger, or two GPUs |
Consumer, Workstation, and Datacenter GPUs
Hardware for AI generally falls into three distinct tiers. Choosing the wrong tier can lead to massive wasted spend or a system that simply cannot run the model you want.
| GPU Tier | Example Card | Memory Capacity | Memory Bandwidth | Best Use Case |
|---|---|---|---|---|
| Consumer | GeForce RTX 4090 | 24 GB GDDR6X | 1,008 GB/s | Up to ~32B models at 4-bit |
| Consumer | GeForce RTX 5090 | 32 GB GDDR7 | 1,792 GB/s | 32B models with longer context |
| Workstation | RTX PRO 6000 Blackwell | 96 GB GDDR7 (ECC) | 1,792 GB/s | 70B models, large MoE, serving |
| Datacenter | H100 SXM | 80 GB HBM3 | 3.35 TB/s | High-throughput serving, training |
| Datacenter | H200 | 141 GB HBM3e | 4.8 TB/s | Large models, many users |
Consumer cards offer the best bandwidth per rupee but have limited VRAM. They are designed for desktops and have specific cooling requirements. Workstation cards trade some price efficiency for much higher VRAM and ECC (Error Correction Code) memory, making them suitable for professional, all-day use. Datacenter cards use HBM (High Bandwidth Memory) and fast GPU-to-GPU links like NVLink, making them ideal for high-throughput serving and massive training tasks.
How Bandwidth Dictates Token Speed
When you are using an LLM, there are two phases: Prefill and Decoding. During prefill (reading your prompt), the chip is limited by compute power. However, during decoding (generating the response word by word), the chip spends most of its time waiting on memory. This is why memory bandwidth is the most important factor for generation speed.
A useful ceiling for your calculations is: max tokens per second ≈ memory bandwidth ÷ bytes read per token. While real-world speeds are often 50 to 80 percent of this ceiling due to overhead, it is an excellent way to compare hardware.
For example, an 8B model at 4-bit is about 4.5 GB. On an RTX 4090 (1,008 GB/s), the theoretical ceiling is about 220 tokens per second. A 70B model at 4-bit is about 40 GB; on a card with 1,792 GB/s, the ceiling is about 45 tokens per second.
Mixture-of-Experts (MoE) models change this math. A model like Google's Gemma 4 26B has about 25 billion total parameters but only activates about 3.8 billion per token. This means you need enough VRAM to hold the full 26B model, but you get decoding speeds closer to a much smaller 4B model. MoE trades memory capacity for speed.
Apple Silicon and CPU Inference
Not all AI hardware is a dedicated NVIDIA card. There are several alternatives depending on your budget and performance needs.
Apple Silicon Unified Memory
Apple's M-series chips use a unified memory architecture. This allows the CPU and GPU to share a single pool of RAM. A Mac with a large memory configuration, like the M3 Ultra (up to 512 GB), can load massive models that would require several expensive consumer GPUs. While the bandwidth is lower than a top NVIDIA card, the capacity is unmatched for a single machine.
CPU Inference
You can run models on a standard CPU using tools like llama.cpp. However, this is limited by the much lower memory bandwidth of system RAM. For example, dual-channel DDR5-5600 has a peak of roughly 90 GB/s. A 40 GB model would top out at about 2 tokens per second on a typical desktop. This is fine for background batch jobs or small models, but not for interactive use.
NPUs and Low-Power AI
Neural Processing Units (NPUs) found in modern laptops and phones are specialized for low-power, small-scale tasks. They are designed for models with roughly 1B to 4B parameters, such as summarizing notifications or transcription, rather than running large-scale LLMs.
Should You Rent or Buy Hardware?
For business owners, the decision to buy or rent depends on your expected usage. If you are just exploring models for a few hours a week, renting is the best choice because there is no upfront cost. If you need a short burst of power for fine-tuning, renting a datacenter GPU for a few days is often more cost-effective than buying one.
If you have high-use, bulk generation tasks where you are running models many hours every day, buying or leasing a dedicated server is usually the winner. The fixed cost eventually beats the hourly rental fees. Additionally, if you handle sensitive data that must remain strictly on-premise, buying your own hardware is the only way to ensure total control.
| Situation | Better Choice | Why? |
|---|---|---|
| Exploring models, a few hours a week | Rent | No upfront cost; try different GPUs |
| Short training or fine-tuning burst | Rent Datacenter GPUs | Avoid massive upfront hardware costs |
| Bulk generation many hours every day | Buy or lease a dedicated server | Fixed cost beats per-token/hour bills |
| Sensitive data must stay on your machine | Buy (on-premise) | You control where data goes |

Questions people ask
How much VRAM do I need for an 8B model?
An 8B model needs about 16 GB at FP16, 8 GB at 8-bit, and 4–5 GB at 4-bit, plus extra for the KV cache and overhead.
Why is VRAM more important than raw compute for LLMs?
For most LLM inference, the chip spends most of its time waiting on memory. Therefore, memory bandwidth often determines the generation speed (tokens per second).
What is the KV cache?
The KV cache stores key and value vectors for past tokens so the model doesn't have to recompute them, but it grows with context length.
Can I run an LLM on a CPU?
Yes, using tools like llama.cpp, but performance is limited by the much lower memory bandwidth of system RAM compared to a GPU.
What is the benefit of Mixture-of-Experts (MoE) models?
MoE models can decode at a speed closer to a much smaller model because they only activate a fraction of their total parameters per token.
