Let's talk

Hardware Guide

Building a Personal AI Server

To run large language models (LLMs) locally, you need to prioritize memory capacity and bandwidth over raw compute. This guide helps builders and business owners select the right hardware for local inference.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for Building a Personal AI Server

Memory First, Bandwidth Second

To run a personal AI server, you must prioritize memory capacity to fit the model and memory bandwidth to ensure fast response speeds. While compute power is important, the bottleneck for most local LLM tasks is how quickly data can move from memory to the processor.

To run a large language model (LLM) locally, you need enough fast memory to hold the model weights plus the KV cache for your context length. A rough rule for memory is: parameters × bytes per parameter.

Once the model fits in your memory, memory bandwidth determines your tokens per second (speed). This is why GPUs beat CPUs for inference; they possess much higher memory bandwidth, allowing them to stream the necessary data to the cores much faster than a standard system RAM setup.

How Much VRAM Do You Need?

Model weights represent a fixed cost in terms of memory. For example, an 8B model requires approximately 16 GB of memory at FP16 precision, about 8 GB at 8-bit precision, and roughly 4 to 5 GB at 4-bit precision. When planning your hardware, you should always budget roughly 4.5 to 5 bits per weight to account for scaling factors and higher precision layers.

However, memory is not just for the model weights. You must leave headroom for the KV cache (the model's "short-term memory" for the conversation) and 1 to 2 GB of runtime overhead for the framework and activations. A model that "just fits" your available VRAM will likely slow down significantly or fail entirely once a long conversation fills the KV cache.

Model Size FP16 / BF16 Weights 8-bit Weights 4-bit Weights Typical Single GPU (4-bit)
3B ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB Any 6–8 GB card, laptops, phones
8B ≈ 16 GB ≈ 8 GB ≈ 4–5 GB 8–12 GB card
14B ≈ 28 GB ≈ 14 GB ≈ 7–9 GB 12–16 GB card
32B ≈ 64 GB ≈ 32 GB ≈ 16–20 GB 24–32 GB card
70B ≈ 140 GB ≈ 70 GB ≈ 35–42 GB 48 GB or larger, or two GPUs

The Role of the KV Cache

While generating text, a transformer model stores a key and a value vector for every past token in every layer. This is known as the KV cache. It prevents the model from having to recompute the entire conversation history every time it generates a new word.

The KV cache grows linearly with the context length (how many tokens the model can "see" at once) and the number of users being served simultaneously. For example, if you are using a Llama 3.1 8B style model with a 128K context window, the KV cache alone can require about 16 GiB of memory—which is actually more than the 4-bit model weights themselves.

Modern models often use Grouped-Query Attention (GQA) to manage this. GQA allows several query heads to share one KV head, which can make the cache significantly smaller. Additionally, many runtimes allow you to quantize the KV cache to 8-bit, which can halve its memory footprint with only a small cost to quality.

Choosing Your GPU Tier

When building a server, you will likely choose from one of three hardware tiers depending on whether you are a hobbyist, a professional designer, or a high-scale service provider.

  • Consumer Tier (e.g., GeForce RTX 4090): These cards offer the best bandwidth per rupee (or dollar). They are excellent for individual builders, but they are limited by VRAM capacity, typically topping out at 24 GB.
  • Workstation Tier (e.g., RTX PRO 6000 Blackwell): These are built for 24/7 server use. They offer much more VRAM (up to 96 GB), include Error Correction Code (ECC) memory for stability, and are designed for professional workflows.
  • Datacenter Tier (e.g., H100): These use HBM (High Bandwidth Memory) to provide massive throughput. They are designed for high-throughput serving and training in large-scale environments.
GPU Tier Memory Bandwidth Good for...
GeForce RTX 4090 (Consumer) 1,008 GB/s Up to about 32B at 4-bit
GeForce RTX 5090 (Consumer) 1,792 GB/s 32B at 4-bit with longer context
RTX PRO 6000 Blackwell (Workstation) 1,792 GB/s 70B at 4–8-bit, large MoE, small-team serving
H100 SXM (Datacenter) 3.35 TB/s High-throughput serving, training
H200 (Datacenter) 4.8 TB/s Large models, long context, many users

Apple Silicon and Unified Memory

Apple's M-series chips use a unique architecture called unified memory. This allows the GPU to access a large pool of system RAM directly, rather than having its own separate, limited memory chip. This is a significant advantage for running very large models on a single machine.

For instance, a Mac with a high memory configuration can load massive models that would otherwise require several expensive consumer GPUs. However, there is a trade-off: while the capacity is very high, the memory bandwidth is generally lower than high-end NVIDIA cards. This means you can run a large model, but it will generate text at a lower "tokens per second" rate.

If you are looking into on-device AI, Apple Silicon is a highly efficient and quiet option for developers.

Should You Rent or Buy?

For business owners, the decision to invest in local hardware versus using cloud APIs or rented GPUs is a financial one. There is no single right answer, but there are clear indicators for each path.

Rent if: You are just exploring different models, you only need AI for short bursts of time (like a weekend of fine-tuning), or if your hardware would sit idle most of the time. Renting is also better if you need the massive scale of a datacenter GPU for a single day.

Buy if: You have high-use bulk generation (like an Indian content agency drafting hundreds of pages daily), you have sensitive data that must stay on-premise for privacy, or you have predictable, heavy daily workloads. Owning the hardware provides a fixed cost that eventually becomes cheaper than paying hourly rental fees.

A simple test for your business: Estimate your monthly GPU hours. If the hardware would be busy most of the day for a year or more, buying is usually the more economical choice.

Serving Software for Local AI

Once you have the hardware, you need the software to actually run the models. The choice depends on whether you are a single user or a team serving many requests.

For Individuals and Prototyping

  • Ollama: The easiest way to download and run open models locally. It provides a simple API and is perfect for a single person experimenting with local-first AI.
  • llama.cpp: A lean, highly efficient engine that can run on almost anything—CPUs, NVIDIA GPUs, AMD GPUs, or Apple hardware. It is the foundation for many other tools.

For Teams and Applications

  • vLLM: A high-throughput server designed for many concurrent users. It uses advanced techniques like paged KV-cache management to handle many requests at once.
  • SGLang: A fast serving framework that is particularly useful for agents or applications that need to reuse shared prompt prefixes to save time.

In summary: use Ollama or llama.cpp if you are one person; use vLLM or SGLang if you are building an application that many people will use simultaneously.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

What is the most important spec for a personal AI server?

Memory capacity (VRAM) is the first limit you will hit to fit a model, while memory bandwidth determines how fast the model generates text.

Can I run LLMs on my computer's CPU?

Yes, using tools like llama.cpp, but it is much slower because system RAM bandwidth is significantly lower than GPU VRAM.

Why does long context use so much memory?

Longer conversations require a larger KV cache to store the history of tokens, which grows linearly with the context length.

What is quantization?

Quantization is storing model weights in lower precision (like 4-bit or 8-bit) to reduce memory requirements with minimal quality loss.

Do I need a special GPU for AI?

While many GPUs work, NVIDIA GPUs are preferred for their dedicated Tensor Cores which accelerate the matrix multiplications used in AI.