Let's talk

Hardware Sizing Guide

How Much RAM Do I Need for Local LLM?

To run a large language model locally, you need enough fast memory to hold the model weights plus the KV cache for your context length.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for How Much RAM Do I Need for Local LLM?

To run a large language model locally, you need enough fast memory to hold the model weights plus the KV cache for your context length. A rough rule of thumb is to multiply the number of parameters by the bytes per parameter.

The Golden Rule: Parameters × Bytes

The base memory requirement for any model is determined by its size (parameters) and the precision at which those weights are stored. If you are a builder looking to deploy a model, you must first decide on the precision, as this dictates your hardware costs.

  • FP16 and BF16 precision: Uses 2 bytes per parameter. This is high precision but requires significant memory.
  • 8-bit quantization: Uses 1 byte per parameter. This is a common middle ground for balancing quality and size.

Quick Reference: Model Size vs. Memory

For business owners planning a budget, the following table provides a quick estimate of how much VRAM or RAM you will need based on common model sizes and quantization levels.

Model Size FP16 / BF16 Weights 8-bit Weights 4-bit Weights Typical Single GPU (4-bit)
3B Model ~6 GB ~3 GB ~1.5–2 GB Any 6–8 GB card, laptops, phones
8B Model ~16 GB ~8 GB ~4–5 GB 8–12 GB card
14B Model ~28 GB ~14 GB ~7–9 GB 12–16 GB card
32B Model ~64 GB ~32 GB ~16–20 GB 24–32 GB card
70B Model ~140 GB ~70 GB ~35–42 GB 48 GB or larger, or two GPUs

Don't Forget the KV Cache

A common mistake for builders is calculating memory based only on the model weights. A model that "just fits" its weights will fail or slow down once a long conversation fills the KV cache.

The KV cache stores key and value vectors for past tokens so the model does not have to recompute them every time you send a new message. This cache grows linearly with your context length (the amount of text the model "remembers" in a single session).

The Cost of Long Context

To understand the scale, consider a Llama 3.1 8B style model using FP16 precision. At a standard context of 8,192 tokens, the cache requires about 1 GiB. However, if you increase that to a 128K context, the cache requires about 16 GiB—which is more than the 4-bit weights of the model itself.

  • Grouped-query attention (GQA): This technology makes the cache significantly smaller by allowing multiple query heads to share a single KV head.
  • Quantization: Many runtimes allow you to quantize the KV cache to 8-bit, halving its memory footprint with only a small cost to quality.
  • Overhead: Always leave an extra 1 to 2 GB of runtime overhead for the framework and activations to ensure stability.

VRAM vs. System RAM

When choosing hardware, you must distinguish between VRAM and standard system RAM. This distinction determines whether your model runs at usable speeds or becomes painfully slow.

GPUs use VRAM (such as GDDR or HBM), which is soldered close to the processor to provide extremely high bandwidth. CPUs use system RAM (DDR), which is much slower for AI tasks.

In AI inference, memory bandwidth—the speed at which data moves—decides how many tokens per second you get. For example, dual-channel DDR5-5600 is roughly a tenth of the bandwidth of a high-end GPU. If you run a large model on a CPU using system RAM, the experience will be significantly slower.

Apple Silicon offers a unique middle ground through unified memory. A Mac can allow the GPU to access a large pool of system RAM, providing high capacity, though the bandwidth is generally lower than top-tier NVIDIA cards.

Choosing Your Hardware Tier

Depending on your business use case—whether you are a developer experimenting or a company serving customers—you will fall into one of these hardware categories:

  • Consumer GPUs (e.g., RTX 4090): These offer the best bandwidth per rupee/dollar, but they are limited by their VRAM capacity (e.g., 24 GB).
  • Workstation GPUs (e.g., RTX PRO 6000): These provide much more VRAM and ECC (Error Correction Code) support, making them suitable for running heavy workloads in a server all day.
  • Datacenter GPUs (e.g., H100): These use high-throughput HBM memory designed for massive models and serving many users simultaneously.
  • NPUs (Neural Processing Units): Found in phones and modern laptops, these are small accelerators meant for low-power, compact models (roughly 1B to 4B parameters).

If you are unsure which GPU for AI fits your needs, start with the capacity required by your target model and then look at the bandwidth.

The MoE Advantage

Mixture-of-Experts (MoE) models change the math of memory requirements. Instead of activating every part of the model for every word, they split layers into sub-networks and only activate a few per token.

This creates a unique hardware requirement: you still need enough VRAM to hold all the weights for the entire model, but the model decodes at a speed closer to a much smaller model. This makes MoE models ideal for GPUs with large VRAM capacity, as they offer high speed without requiring the massive compute power of a much larger dense model.

Summary: How to Size Your Hardware

To ensure your hardware setup is sufficient for your local LLM deployment, follow this checklist:

  1. Estimate model weight size: Based on parameters and your chosen quantization (e.g., 4-bit or 8-bit).
  2. Add KV cache: Calculate the memory needed for your desired context length and number of concurrent users.
  3. Add overhead: Include 1–2 GB for the software framework and activations.
  4. Apply a buffer: Choose hardware with 20% to 30% headroom above your total calculation to prevent crashes.
  5. Prioritize bandwidth: If your goal is high speed (tokens per second), prioritize memory bandwidth over raw compute.

Frequently Asked Questions

Will a model run if it's larger than my VRAM?
You can use tools like llama.cpp to split layers between the GPU and CPU, but the CPU portion will significantly slow down performance.
Does quantization reduce RAM requirements?
Yes, quantization (e.g., 4-bit vs FP16) significantly reduces the memory needed to store model weights, allowing larger models to fit on smaller hardware.
Why does long context use more memory?
Longer conversations increase the size of the KV cache, which grows linearly with the number of tokens in the context.
What is the difference between VRAM and RAM for AI?
VRAM is high-bandwidth memory located on the GPU for fast processing, while system RAM is slower and used when running models on a CPU.
How much headroom should I leave in my memory?
You should aim for 20% to 30% headroom above your calculated total to account for overhead and context growth.
Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

Will a model run if it's larger than my VRAM?

You can use tools like llama.cpp to split layers between the GPU and CPU, but the CPU portion will significantly slow down performance.

Does quantization reduce RAM requirements?

Yes, quantization (e.g., 4-bit vs FP16) significantly reduces the memory needed to store model weights, allowing larger models to fit on smaller hardware.

Why does long context use more memory?

Longer conversations increase the size of the KV cache, which grows linearly with the number of tokens in the context.

What is the difference between VRAM and RAM for AI?

VRAM is high-bandwidth memory located on the GPU for fast processing, while system RAM is slower and used when running models on a CPU.

How much headroom should I leave in my memory?

You should aim for 20% to 30% headroom above your calculated total to account for overhead and context growth.