Let's talk

Build your own

The Best Local LLM for Coding: Hardware Requirements

Running a local LLM for coding requires balancing model size with memory bandwidth to ensure fast, responsive code generation. This guide helps you select the right hardware for your development workflow.

By Divyaprakash JhaUpdated 6 min readTopic: AI hardware

Photo for The Best Local LLM for Coding: Hardware Requirements

How much VRAM do you need for coding models?

To run a local LLM for coding, you must prioritize memory capacity (VRAM) first to fit the model, and memory bandwidth second to ensure the code generates at a readable speed. A good rule of thumb is to choose hardware that provides 20 to 30 percent more memory than your model requires to account for long code files and conversation history.

When selecting hardware, you need to understand how much memory a model actually occupies. The memory requirement is calculated as: parameters × bytes per parameter. For example, an 8B parameter model requires about 16 GB of memory if using FP16 precision, 8 GB if using 8-bit quantization, and roughly 4 to 5 GB if using 4-bit quantization.

However, you cannot simply match your VRAM to the model size. You must always budget an extra 1-2 GB of headroom for the framework and activations. Furthermore, for coding tasks, you will likely work with long files or entire repositories. This requires a "KV cache," which stores the history of the conversation so the model doesn't have to recompute everything every time you ask a question. This cache grows linearly with your context length.

Model Size FP16 / BF16 weights 8-bit weights 4-bit weights Typical GPU (4-bit)
3B ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB Any 6–8 GB card
8B ≈ 16 GB ≈ 8 GB ≈ 4–5 GB 8–12 GB card
14B ≈ 28 GB ≈ 14 GB ≈ 7–9 GB 12–16 GB card
32B ≈ 64 GB ≈ 32 GB ≈ 16–20 GB 24–32 GB card
70B ≈ 140 GB ≈ 70 GB ≈ 35–42 GB 48 GB or larger

Why GPUs are essential for coding speed

Neural networks are essentially massive series of matrix multiplications. When you ask an LLM to write a function, the hardware must multiply your input by billions of weights. A CPU is designed for complex logic and branching; it has a few dozen powerful cores. A GPU, however, has thousands of simpler cores designed to do the same operation on large blocks of numbers in parallel. This makes GPUs significantly faster for AI inference.

The location of the memory is just as important as the processor. In a GPU, the VRAM is soldered right next to the processor on a very wide bus, providing the high bandwidth needed for fast token generation. If you try to run a large model on a CPU using system RAM, the data has to travel much further through much narrower paths. If a model is too large for your GPU's VRAM, it "spills" over to the system RAM (CPU), which is significantly slower.

For developers, this speed difference is the difference between a helpful assistant that types in real-time and a system that hangs for several seconds between every word.

Understanding Token Speed and Bandwidth

Many builders mistakenly focus on "TFLOPS" or raw compute power, but for LLM inference, memory bandwidth is the true bottleneck. When generating text (decoding), the chip must read every active weight from memory for every single token it produces. Because the math per weight is relatively small, the chip spends most of its time waiting for data to arrive from the memory.

You can estimate your maximum speed using this formula: max tokens per second ≈ memory bandwidth ÷ bytes read per token. While real-world speeds are usually 50 to 80 percent of this theoretical ceiling due to overhead, it is the best way to compare hardware.

For example, an 8B model at 4-bit quantization on an RTX 4090 (which has roughly 1,008 GB/s of bandwidth) has a theoretical ceiling of about 220 tokens per second. A much larger 70B model on a high-end card with 1,792 GB/s of bandwidth might only reach about 45 tokens per second.

You may also encounter Mixture-of-Experts (MoE) models. These are clever because they only activate a fraction of their total parameters for each token. This allows them to decode at speeds much closer to a much smaller model, even though they require enough VRAM to hold all their parameters.

Choosing your hardware tier

Depending on whether you are a solo developer or a business owner building a team tool, your hardware needs will vary significantly. Here is how the market breaks down:

  • Consumer GPUs (e.g., RTX 4090/5090): These offer the best bandwidth per rupee. They are ideal for individual developers who need fast, real-time coding assistance on their local machines.
  • Workstation GPUs (e.g., RTX PRO 6000): These trade some price efficiency for much higher VRAM capacity (up to 96 GB) and ECC memory. They are built for running all day and are ideal for serving models to a small team or running massive MoE models.
  • Apple Silicon (M-series): These use a "unified memory" architecture. This allows the GPU to access a massive pool of system RAM. While the bandwidth is lower than a top NVIDIA card, the capacity is much higher, allowing you to load models that would require multiple expensive GPUs.
  • Datacenter GPUs (e.g., H100): These are designed for high-throughput serving and training in server environments. They use specialized memory (HBM) and high-speed links to act as massive accelerators for many users at once.

If you are interested in how these components fit into a broader system, you can read more about AI hardware or specific GPU requirements for AI.

The role of quantization in local development

Quantization is the process of storing model weights in lower precision, such as 8-bit or 4-bit, instead of the standard 16-bit. This is the "secret sauce" that makes local LLMs practical for most people.

By using quantization, you reduce the memory footprint of a model significantly. Using 4-bit weights is a common way to run powerful models on standard hardware with 8, 12, or 24 GB of VRAM. While there is a small quality cost to using lower precision, the trade-off is almost always worth it to get a larger, smarter model running on your local machine.

Should you buy hardware or rent it?

For business owners, the decision to buy local hardware or use the cloud is a matter of economics and data security. Here is a simple way to decide:

Situation Better Choice Why?
Exploring models or short bursts of work Rent (Cloud/API) No upfront cost; try different GPUs easily.
Bulk generation or daily heavy use Buy (On-premise) Fixed costs beat per-token or per-hour bills at high use.
Sensitive code must stay private Buy (On-premise) You have total control over where your data goes.
Frontier-model quality needed Paid API The largest, most capable models are not available to self-host.

A good rule of thumb: if the hardware would be busy most of the day for a year or more, buying is usually more cost-effective. If it would sit idle most of the time, renting is the better path.

Best software for running local models

Once you have the hardware, you need the software to talk to it. The "stack" depends on your use case:

  1. Ollama: The easiest way to download and run open models locally. It provides a simple API and is great for getting started quickly.
  2. llama.cpp: A lean, high-performance engine that can run quantized models on almost anything—CPUs, NVIDIA, AMD, or Apple hardware.
  3. vLLM or SGLang: If you are building a production application or serving a team, these high-throughput options are designed to handle many concurrent requests efficiently.

For more information on optimizing your setup, see our guide on VRAM for LLMs.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

Can I run a coding LLM on my CPU?

Yes, using tools like llama.cpp, you can run quantized models on a CPU. However, it is significantly slower than a GPU because system RAM has much lower bandwidth.

How does context length affect my hardware needs?

The KV cache grows with context length. Long conversations or large codebases require more VRAM to store the key and value vectors for every token.

What is the benefit of Apple Silicon for LLMs?

Apple's unified memory allows the GPU to access a large pool of system RAM, enabling you to load much larger models than most consumer GPUs can handle.

Why is VRAM more important than compute for LLMs?

For most LLM work, the chip spends more time waiting for data from memory than performing calculations. High memory bandwidth is the key to fast token generation.

What is a Mixture-of-Experts (MoE) model?

MoE models split layers into sub-networks and only activate a few per token. This allows them to decode at speeds closer to much smaller models while maintaining large capacity.