Let's talk

Hardware Essentials

Running a Local LLM: A Guide for Builders and Owners

To run a large language model (LLM) locally, you need enough fast memory to hold the model weights and the KV cache for your context length.

By Divyaprakash JhaUpdated 7 min readTopic: AI hardware

Photo for Running a Local LLM: A Guide for Builders and Owners

The Golden Rule: Memory First

To run a local LLM, you need enough fast memory to hold the model weights plus the KV cache for your context length. Memory capacity determines if a model will fit, while memory bandwidth determines how fast it generates text.

The most important thing to understand is that an LLM is essentially a massive collection of numbers called weights. To perform any calculation, those weights must be loaded from your memory into the processor. If you don't have enough memory to hold all the weights, the model simply will not run.

A rough rule for calculating the memory needed for weights is: parameters × bytes per parameter. For example, if you are using a model in FP16 (which uses 2 bytes per parameter), an 8B model requires 16 GB of memory just for the weights. If you use 8-bit quantization (1 byte per parameter), that same 8B model only needs 8 GB.

However, you cannot use 100% of your VRAM for weights. You must always leave headroom for the KV cache (the memory used to handle the conversation history) and a small amount of runtime overhead for the software. A model that "just fits" your total memory will become incredibly slow or crash as your conversation grows longer.

Model Size FP16 Weights 8-bit Weights 4-bit Weights Typical GPU (at 4-bit)
3B ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB Any 6–8 GB card
8B ≈ 16 GB ≈ 8 GB ≈ 4–5 GB 8–12 GB card
14B ≈ 28 GB ≈ 14 GB ≈ 7–9 GB 12–16 GB card
32B ≈ 64 GB ≈ 32 GB ≈ 16–20 GB 24–32 GB card
70B ≈ 140 GB ≈ 70 GB ≈ 35–42 GB 48 GB or larger

Capacity vs. Bandwidth

It is easy to confuse capacity with speed. For AI work, these are two very different metrics. VRAM capacity determines if a model will fit on your hardware. Memory bandwidth determines how many tokens per second you get during generation.

Once you have chosen a model that fits comfortably in your memory, bandwidth becomes the primary factor for speed. This is why GPUs are so much faster than standard computer CPUs for AI. In a GPU, the memory (VRAM) is soldered right next to the processor on a very wide bus, allowing data to move back and forth at incredible speeds.

If you are building a system for a business, remember that high-throughput serving requires massive bandwidth. This is why datacenter GPUs use specialized memory called HBM (High Bandwidth Memory), which offers significantly more speed than the GDDR memory found in consumer gaming cards.

To understand the hardware landscape better, you can explore more about AI hardware or specific GPUs for AI.

Understanding Quantization

Quantization is a technique used to compress large models so they can run on smaller, more affordable hardware. It works by storing the model weights at a lower precision. Instead of using high-precision numbers that take up a lot of space, we use lower-precision numbers.

Common formats you will encounter include:

  • FP16 / BF16: High precision, uses 2 bytes per parameter. This is the standard for high-quality models.
  • 8-bit: A middle ground that uses 1 byte per parameter, roughly halving the memory requirement.
  • 4-bit: The most common format for running models locally. It significantly reduces hardware requirements with a relatively small impact on intelligence.

The trade-off is simple: lower precision means less memory per number, which allows you to fit larger models on your hardware and increases the number of math operations you can perform per second. However, if you go too low (like 2-bit), the model may start to lose its ability to reason correctly.

The KV Cache: The Hidden Memory Cost

When you chat with an LLM, the model doesn't just look at your latest message; it needs to remember everything you said previously in the conversation. To avoid re-calculating every single word every time you press enter, the model stores "key" and "value" vectors for every past token. This is called the KV cache.

The KV cache is a "hidden" memory cost because it isn't part of the model file itself, but it grows as you use it. The cache grows linearly with two things: the context length (how many tokens the model can "remember" at once) and the number of users being served simultaneously.

To see how this impacts your hardware, consider a Llama 3.1 8B style model using FP16 precision. For every token generated, the cache requires about 128 KiB of memory. If you are running a long conversation with 8,192 tokens, you will need about 1 GiB of memory just for the cache. If you try to run a massive 128K context window, that cache alone will require about 16 GiB—which is more than the memory required for the 4-bit weights of the model itself!

Modern techniques like Grouped-query attention (GQA) help manage this. GQA allows several parts of the model to share the same KV cache data, making the cache significantly smaller and more efficient for serving multiple users.

Choosing Your Hardware Tier

Depending on whether you are an individual developer or a business owner, your hardware needs will vary wildly. We generally categorize GPUs into three tiers:

GPU Tier Memory Type/Capacity Best Use Case
Consumer (e.g., RTX 4090) 24 GB GDDR6X Individual developers, testing models up to 32B at 4-bit.
Workstation (e.g., RTX PRO 6000) 96 GB GDDR7 (with ECC) Small teams, running 70B models, or large MoE models.
Datacenter (e.g., H100) 80 GB+ HBM3 High-throughput enterprise serving and training.

For a single user, VRAM capacity is usually the first limit you will hit. If you want to run the most powerful open-source models (like 70B models), you will need a workstation-class card or multiple consumer GPUs linked together.

If you are interested in more specific hardware details, see our guide on VRAM and how it relates to VRAM for LLMs.

Apple Silicon and Unified Memory

Apple's M-series chips (M1, M2, M3, M4) use a unique architecture called unified memory. Unlike a PC where the CPU has system RAM and the GPU has its own VRAM, Apple's chips allow the GPU to access the entire pool of system memory.

This is a massive advantage for running large models. A Mac with a high-memory configuration (like an M3 Ultra with up to 512 GB of memory) can load massive models that would require several expensive NVIDIA GPUs to run on a traditional PC. This makes Macs a favorite for researchers who need high capacity on a single, quiet, and energy-efficient machine.

The trade-off is bandwidth. While the capacity is huge, the speed at which data moves in a Mac is often lower than a top-tier NVIDIA card. This means you will get fewer "tokens per second" (slower text generation) than a high-end PC, even if the Mac can fit a much larger model.

Running on CPUs and System RAM

If you don't have a powerful GPU, you can still run LLMs using your computer's standard CPU and system RAM. Tools like llama.cpp make this possible by using highly optimized code to run "quantized" models on ordinary hardware.

However, there is a significant performance penalty. Standard system memory (like DDR5) is much slower than GPU memory. For example, dual-channel DDR5-5600 is roughly a tenth the speed of a high-end GPU. This means a model that flies on a GPU might only generate 2 tokens per second on a CPU, making it feel very sluggish.

CPU inference is best suited for:

  • Running very small models (under 3B parameters).
  • Background batch jobs where speed isn't critical.
  • Generating "embeddings" for search tasks.

If you find that a model is too big for your GPU, you can split the work: some layers run on the GPU and the rest on the CPU. It works, but the CPU will act as a bottleneck, and the entire process will slow down to the speed of the CPU.

Why Mixture-of-Experts (MoE) is Fast

You may have heard of "Mixture-of-Experts" (MoE) models. These are a clever way to get the intelligence of a massive model without the massive computational cost. Instead of activating every single parameter for every single word, an MoE model only activates a small subset of "expert" sub-networks for each token.

Take Google's Gemma 4 26B A4B as an example. It has about 25 billion total parameters, but it only activates about 3.8 billion parameters per token. This means:

  • The Memory Requirement: You still need enough VRAM to hold all 26 billion parameters.
  • The Speed: Because it only does math on 3.8 billion parameters per token, it decodes at a speed much closer to a much smaller model.

MoE models trade memory capacity for speed. This is a perfect fit for modern GPUs that have plenty of VRAM but want to maintain high generation speeds.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

How much VRAM do I need for a local LLM?

It depends on the model size and precision. A 7B model needs about 14 GB at FP16, 7 GB at 8-bit, and 3.5 GB at 4-bit, plus extra for the KV cache and overhead.

Why is my LLM generation slow?

Generation speed is largely determined by memory bandwidth. If your hardware has low bandwidth, the chip spends most of its time waiting for data from memory.

Can I run an LLM on my CPU?

Yes, using tools like llama.cpp, you can run quantized models on your CPU using system RAM, though it will be significantly slower than a GPU.

What is the benefit of quantization?

Quantization reduces the number of bits used to represent weights, which lowers the memory required to run the model and increases inference speed.

Does context length affect memory usage?

Yes. The KV cache grows linearly with the context length, meaning longer conversations require more memory.