The Golden Rule: Memory First
To run a large language model locally, you need enough fast memory to hold the model weights plus the KV cache for your context length. Once the model fits in your memory, the speed at which it generates text is determined by your memory bandwidth.
When building a local AI setup, many people make the mistake of focusing on raw processing power. However, for LLMs, memory is the primary bottleneck. You must calculate your requirements using a simple formula: Total memory needed = model parameters × bytes per parameter.
This calculation only covers the model weights themselves. To actually use the model for conversation, you must add 1–2 GB of headroom to account for runtime overhead and activations. Crucially, you must also leave extra space for the KV cache. The KV cache is the memory used to store past tokens so the model does not have to recompute them; it grows as your conversation gets longer.
Calculating VRAM for Different Quantizations
Quantization is the process of storing model weights in lower precision to reduce memory requirements and increase speed. This is why you can run a massive model on a consumer laptop that would otherwise require a server. For builders, understanding the "bits per parameter" is essential for budgeting your VRAM needs.
- FP16/BF16: Uses 2 bytes per parameter. This is the highest precision for most local use cases.
- 8-bit quantization: Uses 1 byte per parameter.
| Model Size | FP16 / BF16 Weights | 8-bit Weights | 4-bit Weights (Typical) | Recommended Hardware (4-bit) |
|---|---|---|---|---|
| 3B | ≈ 6 GB | ≈ 3 GB | ≈ 1.5–2 GB | Any 6–8 GB card, laptops, phones |
| 8B | ≈ 16 GB | ≈ 8 GB | ≈ 4–5 GB | 8–12 GB card |
| 14B | ≈ 28 GB | ≈ 14 GB | ≈ 7–9 GB | 12–16 GB card |
| 32B | ≈ 64 GB | ≈ 32 GB | ≈ 16–20 GB | 24–32 GB card |
| 70B | ≈ 140 GB | ≈ 70 GB | ≈ 35–42 GB | 48 GB or larger, or two GPUs |
Why GPUs Outperform CPUs
If you try to run a model on a standard CPU, you will notice it is significantly slower than a dedicated graphics card. This is due to the fundamental difference in how these chips are built.
A CPU has a few dozen powerful cores designed for branching and complex logic. It is the "brain" of your computer, meant to handle many different tasks. A GPU, however, has thousands of simpler cores designed to do the same operation on large blocks of numbers in parallel. Since neural networks are mostly massive matrix multiplications, the GPU's architecture is a much better fit for the job.
Furthermore, memory bandwidth is the primary driver of token speed. GPUs use specialized memory (like GDDR or HBM) that is soldered close to the processor on a very wide bus. Standard system RAM used by CPUs has much lower bandwidth, meaning the CPU spends most of its time waiting for data to arrive.
For a better understanding of why this matters for your hardware builds, see our guide on AI hardware.
Choosing Your Hardware Tier
Depending on whether you are a hobbyist, a developer, or a business owner, your hardware needs will vary significantly. We categorize GPUs into three main tiers:
- Consumer GPUs: (e.g., NVIDIA RTX 4090) These offer the best bandwidth per rupee, making them excellent for individual developers. However, they have limited VRAM capacity.
- Workstation GPUs: (e.g., RTX PRO 6000 Blackwell) These provide much higher VRAM and ECC (Error Correction Code) memory support. They are designed for running large models or serving small teams all day.
- Datacenter GPUs: (e.g., H100, H200) These use HBM (High Bandwidth Memory) to provide massive throughput. They are built for high-throughput serving and training for many concurrent users.
The Apple Silicon Alternative
Apple's M-series chips use a "unified memory" architecture. This means the CPU and GPU share a single pool of memory. A Mac with a large memory configuration can load massive models that would require several expensive consumer GPUs to run. While the bandwidth is lower than a top-tier NVIDIA card, the capacity is much higher and the system is much more efficient.
Understanding Token Speed
When you see an LLM "streaming" text, you are watching the decoding process. For a single user, generating each new token means reading every active weight from memory once. Because the actual math per weight is tiny, the chip spends most of its time waiting for the memory to deliver the data.
A useful ceiling to estimate your speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token. While real-world speed is often 50% to 80% of this ceiling due to overhead, it is the best way to compare two different pieces of hardware.
For example, an 8B model at 4-bit quantization on an RTX 4090 (1,008 GB/s) has a ceiling of roughly 220 tokens per second. On a 70B model at 4-bit on a high-end card with 1,792 GB/s bandwidth, the ceiling drops to about 45 tokens per second.
Mixture-of-Experts (MoE) Advantages
Models like Google's Gemma 4 26B use a technique called Mixture-of-Experts. These models have many parameters, but for every token generated, they only "activate" a fraction of them. This means they require the memory capacity of a large model, but they decode at a speed much closer to a much smaller model.
Best Software to Get Started
Once you have the hardware, you need the right software to run the models. The choice depends on whether you are an individual researcher or a business operator.
- Ollama
- The easiest way to download and run models locally. It provides a simple API and is built on top of llama.cpp. Great for beginners.
- llama.cpp
- A lean, high-performance engine that runs quantized models on almost anything: CPUs, NVIDIA GPUs, AMD GPUs, and Apple Silicon.
- vLLM or SGLang
- High-throughput serving frameworks. If you are building an application that serves many concurrent users, these are the professional choice.
Should You Rent or Buy Hardware?
For business owners, the decision to invest in hardware is a financial one. A good rule of thumb is to estimate your monthly GPU usage. If your hardware will be busy most of the day for a year or more, buying or leasing a dedicated machine usually wins. If you only need it for occasional bursts, renting is better.
| Situation | Better Choice | Why? |
|---|---|---|
| Exploring models a few hours a week | Rent | No upfront cost; easy to try different GPUs. |
| Short training or fine-tuning bursts | Rent Datacenter GPUs | Cheaper to rent for days than to buy for years. |
| Bulk generation many hours every day | Buy or lease a dedicated server | Fixed cost beats high per-token/per-hour bills. |
| Sensitive data requirements | Buy (on-premise) | You maintain total control over where data goes. |
For more specific guidance on choosing between consumer and professional gear, refer to our GPU for LLM guide.

Questions people ask
How much VRAM do I need for a 7B model?
At 4-bit quantization, a 7B model needs about 3.5 GB for weights, but you should budget 4.5 to 5 GB to account for quantization overhead and context headroom.
Why is my LLM slow when using my CPU?
LLM inference is limited by memory bandwidth. Standard system RAM (like DDR5) has much lower bandwidth than GPU VRAM, making CPU inference significantly slower.
What is the KV cache and why does it matter?
The KV cache stores past tokens so the model doesn't have to recompute them. It grows with your context length, meaning longer conversations require more memory.
Can I run a model if it doesn't fit in my GPU VRAM?
Yes, tools like llama.cpp can split layers between your GPU and CPU, but the parts running on the CPU will set the speed of the entire process.
What is quantization?
Quantization is the process of storing model weights in lower precision (like 4-bit or 8-bit) to reduce memory requirements and increase speed.
