Memory First, Bandwidth Second
To run a large language model (LLM), you need enough fast memory to hold the model weights plus the KV cache for your context length. Once the model fits, memory bandwidth determines how many tokens per second you get.
For builders and business owners, the most important takeaway is this: Memory comes first, bandwidth second. If the model weights do not fit into your available VRAM, the system will either fail to run or will default to much slower system RAM, making the AI feel sluggish and unusable for real-time tasks.
A rough rule of thumb for calculating memory needs is: parameters × bytes per parameter. For example, if you are running a model with 8 billion parameters at 16-bit precision (2 bytes per parameter), you need at least 16 GB of memory just for the weights.
Once you have ensured the model fits, your speed is dictated by memory bandwidth. This is why GPUs are the standard for AI; their VRAM provides significantly higher bandwidth than the standard RAM found in a typical desktop or server CPU. While a CPU is great for logic and branching, a GPU is designed for the massive, parallel matrix multiplications that drive neural networks.
Calculating VRAM for Different Models
When planning your hardware, do not just aim for the exact size of the model. Model weights are a fixed cost, but you must also leave "headroom" for the KV cache and runtime overhead. If you run out of memory during a long conversation, the system will slow down significantly or crash.
The amount of memory required depends heavily on the precision (quantization) of the model. Using 4-bit or 8-bit quantization allows you to run much larger models on smaller, more affordable hardware by reducing the number of bits used to represent each weight.
| Model Size | FP16 / BF16 Weights | 8-bit Weights | 4-bit Weights |
|---|---|---|---|
| 3B Model | ≈ 6 GB | ≈ 3 GB | ≈ 1.5–2 GB |
| 8B Model | ≈ 16 GB | ≈ 8 GB | ≈ 4–5 GB |
| 14B Model | ≈ 28 GB | ≈ 14 GB | ≈ 7–9 GB |
| 32B Model | ≈ 64 GB | ≈ 32 GB | ≈ 16–20 GB |
| 70B Model | ≈ 140 GB | ≈ 70 GB | ≈ 35–42 GB |
Pro Tip: When using 4-bit weights, budget roughly 4.5 to 5 bits per weight. This extra cushion accounts for scaling factors and the precision needed to maintain model quality.
The Role of the KV Cache
While model weights are static, the KV (Key-Value) cache is dynamic. As the AI generates text, it stores vectors for every past token in the conversation so it doesn't have to recompute the entire history every time it generates a new word. This is essential for "long context" capabilities.
The KV cache grows linearly based on two factors: the length of the conversation (context length) and the number of users you are serving simultaneously. For a single user, this is manageable, but for a business serving hundreds of customers, the memory requirements for the cache can eventually exceed the memory required for the model itself.
There are two main ways to manage this memory pressure:
- Grouped-Query Attention (GQA): This architectural feature allows several "query" heads to share a single "KV" head. This can make the cache up to four times smaller without a major loss in intelligence.
- Quantization: Many modern software tools allow you to quantize the KV cache itself (e.g., to 8-bit), which can halve its size with minimal impact on the quality of the AI's responses.
Understanding GPU Tiers
Not all GPUs are created equal. Depending on whether you are a developer experimenting with code or a company deploying a service in a data center, your hardware needs will differ significantly.
| GPU Tier | Example Model | Memory Type/Spec | Best Use Case |
|---|---|---|---|
| Consumer | RTX 4090 | 24 GB GDDR6X | Individual developers, prototyping, small models (up to 32B at 4-bit) |
| Workstation | RTX PRO 6000 Blackwell | 96 GB GDDR7 (with ECC) | Small teams, large MoE models, 70B models, 24/7 server use |
| Datacenter | H100 / H200 | 80–141 GB HBM3/e | High-throughput enterprise serving, large-scale training |
Consumer cards offer the best "bandwidth per rupee" (or dollar), making them excellent for individual builders. However, they lack the Error Correction Code (ECC) memory and the cooling designed for the constant, heavy workloads found in professional environments. Datacenter GPUs use HBM (High Bandwidth Memory) and technologies like NVLink to allow multiple cards to act as one massive, unified accelerator.
Apple Silicon and Unified Memory
Apple has taken a unique approach with its M-series chips. Instead of having separate memory for the CPU and the GPU, Apple uses a unified memory architecture. This means the CPU and GPU share a single, large pool of high-speed memory.
This is a massive advantage for running very large models. For example, a Mac with a high-end memory configuration can load massive models that would require several expensive NVIDIA consumer GPUs to hold in memory. While the memory bandwidth on a Mac is typically lower than a top-tier NVIDIA card—meaning you will see fewer tokens per second—the sheer capacity makes it a highly efficient and quiet option for local development.
If you are interested in running AI locally on your laptop, you might want to explore on-device AI capabilities.
Running AI on CPUs
Can you run AI on a standard computer CPU? Yes, but there are significant trade-offs. CPU inference is best suited for small models, background batch jobs (like processing a large pile of documents overnight), or generating text embeddings.
The primary bottleneck for CPUs is memory bandwidth. A typical dual-channel DDR5 system provides roughly 90 GB/s of bandwidth, which is about a tenth of the speed of a high-end GPU. Because the chip spends most of its time waiting for data to arrive from the RAM, the text generation will feel very slow—often only a few words per second for medium-sized models.
When a model is too large for your GPU's VRAM, tools like llama.cpp allow you to "split" the model. The layers that fit on the GPU run fast, while the remaining layers run on the CPU. This allows you to run large models on modest hardware, but the CPU portion will dictate the overall speed of the system.
Renting vs. Buying Hardware
For business owners, the ultimate question is: Should you buy your own hardware or rent it from a cloud provider?
| Situation | Better Choice | Why? |
|---|---|---|
| Exploring models for a few hours a week | Rent | No upfront cost; you can try different GPUs without commitment. |
| Short training or fine-tuning bursts | Rent | Rent datacenter GPUs for days, rather than buying expensive hardware for occasional use. |
| Bulk generation (many hours every day) | Buy/Lease | A dedicated server becomes more cost-effective than paying hourly cloud rates. |
| Sensitive data requirements | Buy (On-premise) | You maintain complete control over where your data resides. |
A simple way to decide is to estimate your monthly GPU hours. If your hardware would be busy for a significant portion of the year, owning or leasing a dedicated machine is usually the winner. If the machine would sit idle most of the time, renting is the smarter financial move.

Questions people ask
How much VRAM do I need for an LLM?
You need enough to hold the model weights (parameters × bytes per parameter) plus additional space for the KV cache and runtime overhead.
Why is memory bandwidth important for AI?
During decoding, the chip spends most of its time waiting on memory; higher bandwidth directly translates to more tokens per second.
What is the difference between a GPU and a CPU for AI?
A CPU has a few powerful cores for logic, while a GPU has thousands of simpler cores designed for the parallel matrix multiplication required by AI.
Yes, using tools like llama.cpp, but performance is limited by the much lower memory bandwidth of system RAM compared to GPU VRAM.
What are the benefits of Mixture-of-Experts (MoE) models?
MoE models trade memory capacity for speed; they require enough VRAM to hold all weights, but only activate a fraction of them per token.
