Memory First, Bandwidth Second
To run a large language model (LLM) locally, you must prioritize memory capacity to fit the model weights and the KV cache. Once the model fits in your hardware, memory bandwidth becomes the primary factor that determines how fast the model generates text.
Running an LLM is essentially a massive exercise in matrix multiplication. When you generate a single token, the chip must multiply the current input by billions of weights. Because these operations are independent, they can be split across thousands of small cores simultaneously. This is why GPUs, with their high-speed VRAM, significantly outperform CPUs for AI inference.
A useful rule of thumb for calculating memory needs is: parameters × bytes per parameter.
While capacity determines if a model can run at all, bandwidth determines your tokens per second (speed). A high-end GPU has a much wider "bus" or pathway for data to travel compared to a CPU, allowing it to feed the processor data much faster. If you are building a system for a business, always prioritize the card with the most VRAM you can afford to ensure you can run larger, more capable models.
How Much VRAM Do You Need?
When selecting hardware, you must account for two distinct memory costs: the fixed cost of the model weights and the variable cost of the KV cache. If you only buy enough memory for the weights, your system will likely fail or slow down as soon as you start a conversation.
The Weight Cost
Model weights are a fixed requirement. For a 4-bit quantized model, a rough estimation of memory needed for weights is:
- 7B Model: ~3.5 GB
- 14B Model: ~7–9 GB
- 32B Model: ~16–20 GB
- 70B Model: ~35–42 GB
In practice, you should budget roughly 4.5 to 5 bits per weight for 4-bit quantization to account for scaling factors and runtime overhead. You can find more detailed breakdown on memory requirements in our VRAM guide.
The KV Cache: The Hidden Cost
The KV cache stores key and value vectors for every past token in every layer so the model does not have to recompute them. This allows for faster generation during long conversations. However, the cache grows linearly with the length of the conversation (context length) and the number of users being served.
For a standard Llama 3.1 8B style model, a context length of 8,192 tokens requires about 1 GiB of memory. If you increase that to a 128K context length, the cache alone requires about 16 GiB—which is more than the 4-bit weights themselves. For businesses serving multiple users simultaneously, this memory requirement can scale very quickly.
| Model Size | 4-bit Weights (Approx) | Recommended VRAM (with headroom) |
|---|---|---|
| 3B | 1.5–2 GB | 6–8 GB Card |
| 8B | 4–5 GB | 8–12 GB Card |
| 32B | 16–20 GB | 24–32 GB Card |
| 70B | 35–42 GB | 48 GB or larger |
Choosing the Right GPU Tier
Not all GPUs are created equal. Depending on whether you are a hobbyist builder or a business owner setting up a server, you will fall into one of three main tiers of hardware.
- Consumer GPUs (e.g., NVIDIA RTX 4090 / 5090): These offer the best bandwidth per rupee. They are excellent for individual developers or small-scale testing. However, they have limited VRAM (24 GB to 32 GB), meaning they struggle with very large models like 70B unless you use heavy quantization.
- Workstation GPUs (e.g., RTX PRO 6000 Blackwell): These are designed for professional environments. They offer much higher VRAM (up to 96 GB) and ECC (Error Correction Code) memory support. These are ideal for running 70B models or serving small teams.
- Datacenter GPUs (e.g., H100 / H200): These use HBM (High Bandwidth Memory) and are built for massive throughput. They are intended for large-scale training or high-traffic API serving in a data center environment.
For a detailed comparison of these hardware categories, see our GPU for AI guide.
Apple Silicon and Unified Memory
Apple's M-series chips (M3, M4, etc.) use a "unified memory" architecture. This means the CPU and GPU share a single pool of high-speed memory. This provides a unique advantage for running large models on a single machine.
A high-capacity Mac, such as an M3 Ultra, can be configured with massive amounts of memory (up to 512 GB). This allows it to load massive models that would otherwise require multiple expensive NVIDIA GPUs. This is a popular choice for researchers who need to run large models locally without building a massive server rack.
However, there is a trade-off. While the capacity is high, the bandwidth is typically lower than top-tier NVIDIA cards. This means that while a Mac can load a massive model, it may generate text (tokens per second) more slowly than a dedicated workstation GPU.
NPUs, CPUs, and Small Models
In recent years, many laptops and phones have included an NPU (Neural Processing Unit). It is important to understand where these fit into your AI strategy.
The Role of the NPU
NPUs are small, highly efficient accelerators designed for low-power AI tasks. They are perfect for "on-device" features like transcribing audio, summarizing notifications, or smart replies in a chat app. They are not meant for running massive LLMs; they are optimized for compact models, typically between 1B and 4B parameters.
Running on a CPU
You can run LLMs on a standard CPU using tools like llama.cpp. While this is possible, you are limited by system RAM bandwidth. For example, standard dual-channel DDR5 memory is significantly slower than GPU VRAM. A 40 GB model running on a typical desktop CPU might only produce about 2 tokens per second, which is too slow for fluid conversation but fine for background batch jobs or simple embeddings.
Speeding Up with Mixture-of-Experts (MoE)
Mixture-of-Experts (MoE) is a specialized architecture that allows models to be both large and fast. Instead of activating every single parameter for every word generated, MoE models split their layers into sub-networks (experts) and only use a fraction of them for each token.
Take Gemma 4 26B as an example. While it has about 25 billion total parameters, it only activates about 3.8 billion parameters per token. This means the model requires enough memory to hold all 25 billion parameters, but it generates text at a speed much closer to a much smaller 4B model. This architecture is highly efficient for hardware with large VRAM capacities.
Should You Rent or Buy Hardware?
For business owners, the decision to invest in hardware or use cloud services is a matter of economics and data security. There is no single right answer, but there are clear guidelines.
| Situation | Better Choice | Reasoning |
|---|---|---|
| Exploring models for a few hours a week | Rent | No upfront cost; allows you to try different GPUs. |
| Short training or fine-tuning bursts | Rent Datacenter GPUs | Cheaper to rent for a few days than to buy a server. |
| Bulk generation (many hours every day) | Buy or Lease | Fixed costs eventually beat per-token or per-hour fees. |
| Sensitive data must stay on-premise | Buy (On-premise) | You maintain total control over your data. |
A simple way to decide is to estimate your monthly GPU hours. If your hardware would be busy most of the day for a year or more, owning a dedicated machine usually wins. If the machine would sit idle most of the time, renting is the smarter financial move.

Questions people ask
How much memory does an 8B model need?
At 4-bit quantization, an 8B model needs about 4-5 GB of memory, plus additional space for the KV cache and runtime overhead.
Why is VRAM important for AI?
VRAM capacity determines if a model can fit on your hardware, while VRAM bandwidth determines how fast the model generates text.
What is the KV cache?
The KV cache stores key and value vectors for past tokens so the model doesn't have to recompute them, allowing for faster generation during long conversations.
Can I run LLMs on my CPU?
Yes, using tools like llama.cpp, but performance is limited by your system RAM bandwidth, which is much slower than a GPU.
What is an NPU?
A Neural Processing Unit (NPU) is a small accelerator in phones and laptops designed for low-power, on-device AI tasks like summarization.
