Let's talk

Hardware for Privacy

Local First AI: A Hardware Guide for Builders

Running AI locally requires prioritizing memory capacity and bandwidth over raw compute. This guide helps you select the right hardware for private, high-performance inference.

By Divyaprakash JhaUpdated 6 min readTopic: AI hardware

Photo for Local First AI: A Hardware Guide for Builders

What is Local First AI?

Local first AI refers to running large language models (LLMs) on your own hardware rather than relying solely on external APIs. This approach provides total control over data residency and eliminates dependency on third-party API limits or pricing, allowing you to route bulk, repeatable work to hardware you control.

For builders and business owners, moving away from a purely cloud-based model is about more than just privacy; it is about predictability. When you run models on your own silicon, you are no longer subject to the sudden price hikes or rate limits of API providers. Instead, you are managing your own compute resources, much like managing a fleet of delivery vehicles or a local server room.

By adopting a private AI strategy, organizations can ensure that sensitive proprietary data never leaves their local network. This is particularly critical for industries dealing with legal, medical, or highly confidential corporate information where data residency is a non-negotiable requirement.

The Golden Rule: Memory First, Bandwidth Second

To run an LLM, you need enough fast memory to hold the model weights plus the KV cache for your context length. If the model is too large for your memory, it simply will not run, or it will fall back to your system RAM, which is significantly slower.

A rough rule for calculating memory needs is: parameters × bytes per parameter. For example, if you are running a model with 7 billion parameters in 16-bit precision (2 bytes per parameter), you need roughly 14 GB of memory just to hold the weights.

Once the model fits into your memory, the speed at which it generates text is determined by memory bandwidth. This is why GPUs, which use specialized VRAM with extremely high bandwidth, vastly outperform CPUs for AI inference. A CPU might have a lot of raw compute power, but its connection to the system RAM is a narrow straw compared to the wide highway of a GPU's memory bus.

How Much VRAM Do You Need?

When planning your hardware, you must account for two distinct memory costs. First, there are the model weights, which are a fixed cost. Second, there is the KV cache, which stores past tokens so the model does not have to recompute them every time you ask a follow-up question. The KV cache grows as the conversation gets longer.

Always leave headroom for the KV cache. A model that "just fits" the weights will become incredibly slow or fail entirely once you start a long conversation and the context fills up the remaining memory.

Memory Estimates by Model Size

The following table provides a guide for memory requirements based on common quantization levels (the precision of the model weights).

Model Size FP16 / BF16 weights 8-bit weights 4-bit weights Typical Single GPU (4-bit)
3B ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB Any 6–8 GB card, laptops, phones
8B ≈ 16 GB ≈ 8 GB ≈ 4–5 GB 8–12 GB card
14B ≈ 28 GB ≈ 14 GB ≈ 7–9 GB 12–16 GB card
32B ≈ 64 GB ≈ 32 GB ≈ 16–20 GB 24–32 GB card
70B ≈ 140 GB ≈ 70 GB ≈ 35–42 GB 48 GB or larger, or two GPUs

Note that real-world 4-bit files are often slightly larger than these theoretical minimums because popular formats keep some layers at higher precision to maintain quality. It is wise to budget roughly 4.5 to 5 bits per weight for your calculations.

Why Bandwidth Decides Token Speed

During the decoding phase—when the model is actually typing out the answer word by word—the hardware spends most of its time waiting on memory to read the weights. The actual math is relatively fast; the bottleneck is moving the data from the memory to the processor.

A useful ceiling for estimating your speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token. While real-world speed is typically 50% to 80% of this theoretical ceiling due to software overhead, it remains the best way to compare two different pieces of hardware.

For example, an 8B model at 4-bit precision is about 4.5 GB. On an RTX 4090 with 1,008 GB/s of bandwidth, the theoretical ceiling is roughly 220 tokens per second. On a 70B model at 4-bit (40 GB), a card with 1,792 GB/s bandwidth would have a ceiling of about 45 tokens per second.

Interestingly, Mixture-of-Experts (MoE) models can decode faster than traditional models of the same total size. Even though the whole model must sit in memory, each token only requires the chip to read a fraction of the total weights. This makes MoE models a great fit for hardware with high VRAM capacity but moderate bandwidth.

Comparing GPU Tiers for AI

Not all hardware is created equal. Depending on whether you are a solo developer or a company serving hundreds of customers, you will look at different tiers of hardware.

GPU Tier Example Hardware Memory Type/Bandwidth Best Use Case
Consumer GeForce RTX 4090 24 GB GDDR6X (1,008 GB/s) Best bandwidth per rupee; up to 32B models at 4-bit.
Workstation RTX PRO 6000 Blackwell 96 GB GDDR7 (1,792 GB/s) 70B models; large MoE; small-team serving.
Datacenter H100 SXM 80 GB HBM3 (3.35 TB/s) High-throughput serving and training.
Datacenter H200 141 GB HBM3e (4.8 TB/s) Large models, long context, many concurrent users.

Consumer cards offer the best "bang for your buck" regarding bandwidth, but they are limited by their VRAM capacity. If you need to run a massive 70B model at high precision, a single consumer card won't cut it. Workstation cards offer much more VRAM and ECC (Error Correction Code) memory support, making them more reliable for running 24/7 in a professional environment. Datacenter cards use HBM (High Bandwidth Memory) and are designed for massive scale, but they require specialized cooling and power infrastructure.

Apple Silicon offers a unique middle ground. Because it uses unified memory, the CPU and GPU share the same pool of RAM. A Mac with a large memory configuration can load massive models that would require several expensive NVIDIA GPUs, though the bandwidth is generally lower than a top-tier NVIDIA card.

Should You Rent or Buy Hardware?

The decision to invest in local hardware or stick to the cloud is a classic business trade-off. There is no one-size-fits-all answer, but there is a simple way to test your situation.

Rent (Cloud/API) if:

  • You are still exploring different models and don't know what you need yet.
  • You only need AI for short bursts of work or occasional fine-tuning.
  • You require the absolute "frontier" quality of models like GPT-4 that cannot be run locally.

Buy (On-premise) if:

  • You have sensitive data that must stay on your own machines for compliance.
  • Your hardware would be busy running tasks most of the day for a year or more.
  • You want to avoid the "variable cost" trap of API usage as you scale.

A simple test: Estimate your monthly GPU hours. If your hardware would be busy most of the day for a year or more, owning or leasing a dedicated machine usually wins. If it would sit idle, stick to renting. For those building on-device AI or edge solutions, the math shifts toward specialized, low-power hardware, but the principle of "utilization vs. cost" remains the same.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

Why is VRAM more important than raw compute for LLMs?

Most LLM work is memory-bound. The chip spends most of its time waiting for weights to be read from memory, so bandwidth and capacity matter more than raw FLOPS.

What is the KV cache and why does it matter?

The KV cache stores key and value vectors for past tokens so they don't have to be recomputed. It grows linearly with context length, requiring extra memory headroom.

Can I run LLMs on my CPU?

Yes, using tools like llama.cpp, but you are limited by system RAM bandwidth. A typical desktop CPU is roughly ten times slower than a high-end GPU.

What is quantization?

Quantization is storing weights in lower precision (like 4-bit or 8-bit) to reduce the memory footprint and increase speed with a small cost to quality.

Does Apple Silicon work for local AI?

Yes. Apple's unified memory allows models to access large pools of system RAM, making it possible to run large models that would require multiple consumer GPUs.