The Golden Rule: Memory First
To run a local LLM on Windows, you must prioritize having enough fast memory to hold the model weights and the KV cache. Once you have enough capacity, your memory bandwidth determines how fast the model actually generates text.
To run a large language model (LLM) locally on Windows, you need enough fast memory to hold the model weights plus the KV cache for your context length. A rough rule for sizing is: parameters × bytes per parameter.
For example, an 8B model requires approximately 16 GB of memory at FP16 precision, about 8 GB at 8-bit, and between 4 to 5 GB at 4-bit. When planning your hardware, always leave 1-2 GB of headroom for runtime overhead and activations to prevent crashes.
GPU vs. CPU Inference
When choosing hardware, you must decide between using a Graphics Processing Unit (GPU) or a Central Processing Unit (CPU). The difference in performance comes down to how they are built to handle math.
GPUs use thousands of simple cores designed for parallel matrix multiplication, which is the core of AI work. A CPU, by contrast, has fewer, powerful cores designed for branching and logic, making them much slower for AI inference.
GPU memory, known as VRAM (using GDDR or HBM), is soldered next to the processor on a wide bus for high speed. This is critical because AI work is often limited by how fast data can move, not just how fast the chip can calculate.
If a model is too large to fit entirely in your VRAM, tools like llama.cpp can split layers between your GPU and CPU. However, because the CPU is significantly slower, the CPU half will slow down the entire process, bottlenecking your speed.
Why Bandwidth Dictates Speed
Once a model fits in your memory, the factor that determines your "tokens per second" (the speed of text generation) is memory bandwidth. During the decoding phase—when the model is generating text one word at a time—the chip must read every active weight from memory for every single new token.
Because the math required for each weight is tiny, the chip spends most of its time waiting for data to arrive from the memory. A useful ceiling for estimating speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token.
High-end consumer cards like the RTX 4090 offer much higher bandwidth than standard system RAM, leading to much faster responses. For example, an 8B model at 4-bit on an RTX 4090 (1,008 GB/s) has a theoretical ceiling of about 220 tokens per second, whereas a typical desktop's DDR5 RAM would be significantly slower.
Using Quantization to Save Space
Quantization is a technique used to reduce the memory requirements of a model by storing weights at lower precision (such as 8-bit or 4-bit) instead of the standard 16-bit. Using lower precision means there is less memory required per number, which often results in more maths performed per second.
In the industry, 4-bit quantization is very common for inference because it significantly reduces the VRAM required for large models without a massive loss in intelligence. You can also quantize the KV cache (for example, to 8-bit) to halve its memory footprint, allowing for longer conversations on the same hardware.
Choosing Your Hardware Tier
Hardware for AI generally falls into three distinct categories. Depending on whether you are a solo developer in a small office or a company running a service, your needs will change.
| GPU Tier | Example Hardware | Memory Type/Capacity | Best Use Case |
|---|---|---|---|
| Consumer | GeForce RTX 4090 | 24 GB GDDR6X | Up to 32B models at 4-bit; best bandwidth per rupee. |
| Workstation | RTX PRO 6000 Blackwell | 96 GB GDDR7 (with ECC) | 70B models or small-team serving; high VRAM capacity. |
| Datacenter | H100 SXM | 80 GB HBM3 | High-throughput serving and large-scale training. |
For a single user running local models, VRAM capacity is usually the first limit you hit. Pick the card that fits your target model with enough headroom for context, then look at bandwidth to ensure it is fast enough for your needs.
The Speed Advantage of MoE Models
Mixture-of-Experts (MoE) models offer a unique way to get high performance. Instead of activating every part of the model for every word, MoE models split layers into sub-networks and only activate a few "experts" per token.
This creates a specific hardware requirement: MoE models require enough memory to hold all the weights, but they decode at a speed closer to a much smaller model. This makes MoE an excellent choice for GPUs with large VRAM capacity, as you get the intelligence of a large model with the speed of a small one.
The Hidden Cost: KV Cache
While many builders focus on the size of the model weights, they often forget the "KV cache." The KV cache stores key and value vectors for past tokens so the model does not have to recompute them every time a new word is generated.
The cache grows linearly with the context length (the number of words in the conversation) and the number of users being served. For very long conversations—such as a 128K context window—the KV cache can require more memory than the model weights themselves.
This is why a model that "just fits" in your VRAM might work for short questions but fail or slow down significantly once you start having long, complex discussions. Always budget extra VRAM for your context window.

Questions people ask
How much VRAM do I need for a local LLM?
You need enough to hold the model weights plus the KV cache for your context length. A safe rule of thumb is to budget 4.5 to 5 bits per weight for 4-bit quantized models, plus 1-2 GB of overhead.
Is it better to use a CPU or a GPU for AI?
GPUs are significantly faster for LLM inference because their thousands of simple cores are designed for the parallel matrix multiplication required by neural networks.
What is the benefit of quantization?
Quantization reduces the number of bits used to store weights (e.g., from 16-bit to 4-bit), which allows larger models to fit into smaller amounts of VRAM.
Does more VRAM mean a faster model?
VRAM capacity determines if a model can fit; memory bandwidth determines how fast it generates tokens. You need enough capacity to fit the model, then high bandwidth for speed.
Can I run LLMs on my laptop?
Yes, using tools like llama.cpp, you can run quantized models on your CPU and system RAM, though it will be much slower than a dedicated GPU.
