Let's talk

Hardware for AI

Inference Server: Choosing Hardware for LLM Deployment

An inference server provides the compute and memory required to run large language models (LLMs) and serve tokens to users. Success depends on balancing VRAM capacity with memory bandwidth.

By Divyaprakash JhaUpdated 6 min readTopic: AI hardware

Photo for Inference Server: Choosing Hardware for LLM Deployment

What is an AI Inference Server?

An inference server is the hardware and software stack used to run trained models and generate outputs. It provides the compute and memory required to load large language models (LLMs) and serve tokens to users, where success depends on balancing VRAM capacity with memory bandwidth.

When you interact with an LLM, the server is performing a task called inference. This is essentially a massive series of matrix multiplications: the system multiplies your input by billions of weights to predict and generate the next token in a sequence.

To do this effectively, an inference server must accomplish two main goals:

  • Load the Model Weights: The server must have enough memory to hold the entire set of parameters that make up the model.
  • Manage the KV Cache: The server needs extra memory to store the "context history" (the conversation so far) so it doesn't have to recompute everything every time you ask a question.

For builders, the primary goal of an inference server is to provide enough memory for the weights and the KV cache while maintaining high enough bandwidth to ensure the text streams out at a human-readable speed.

The Memory Rule: Weights and KV Cache

If you are building an AI application, the most important number to calculate is your total memory requirement. You cannot simply look at the model size; you must account for the "overhead" required to actually run the model in a real-world conversation.

The total memory needed follows this basic formula:

Total Memory = Model Weights + KV Cache + Runtime Overhead

Calculating Model Weights

Model weights are calculated by multiplying the number of parameters by the number of bytes used per parameter. This is where quantization comes in. Storing a model in 4-bit precision (using half a byte per parameter) makes it much smaller and faster than storing it in 16-bit precision (2 bytes per parameter).

3B Model ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB
8B Model ≈ 16 GB ≈ 8 GB ≈ 4–5 GB
14B Model ≈ 28 GB ≈ 14 GB ≈ 7–9 GB
32B Model ≈ 64 GB ≈ 32 GB ≈ 16–20 GB
70B Model ≈ 140 GB ≈ 70 GB ≈ 35–42 GB

Note: Real 4-bit files are slightly larger because they often keep some layers at higher precision to maintain quality. A safe rule is to budget roughly 4.5 to 5 bits per weight.

The KV Cache: The Hidden Memory Consumer

The KV cache stores key and value vectors for every past token in every layer. This allows the model to "remember" the context of a long conversation without recomputing every word every time a new word is generated. Unlike model weights, which are a fixed cost, the KV cache grows linearly with the length of the conversation and the number of users you are serving at once.

For example, in a model like Llama 3.1 8B, the KV cache for a single user at an 8,192-token context might take up about 1 GiB. If you are serving 10 concurrent users at that same context length, you need an additional 10 GiB of VRAM just for the conversation history.

Why Bandwidth Matters More Than Compute

A common misconception is that the "speed" of an AI server depends entirely on how many TFLOPS (teraflops) the chip has. While compute is vital for the prefill stage (reading a long prompt), it is rarely the bottleneck for decoding (generating tokens one by one).

When generating text for a single user, the chip spends most of its time waiting to read the model weights from the memory. This is known as being "memory-bandwidth limited." The chip is so fast at math that it spends most of its time sitting idle, waiting for the next set of weights to arrive from the VRAM.

A useful way to estimate your maximum speed is the following ceiling:

Max tokens per second ≈ Memory Bandwidth ÷ Bytes read per token

For example, an 8B model at 4-bit precision is about 4.5 GB. On a high-end consumer card like an RTX 4090 with 1,008 GB/s of bandwidth, the theoretical ceiling is roughly 220 tokens per second. In reality, you will see 50% to 80% of that due to software overheads.

Choosing Your Hardware Tier

Hardware for AI is generally categorized into three tiers based on the intended use case. Choosing the wrong tier can lead to either wasted money or a system that cannot run the models you need.

GPU Tier Example Hardware Memory Spec Best For...
Consumer GeForce RTX 4090 24 GB GDDR6X Best bandwidth per rupee; up to ~32B models at 4-bit.
Workstation RTX PRO 6000 Blackwell 96 GB GDDR7 (with ECC) High VRAM for 70B models, large MoE models, and all-day serving.
Datacenter H100 / H200 (SXM) 80–141 GB HBM3/e High-throughput serving for many users or massive models.

Consumer cards are excellent for enthusiasts and developers due to their high bandwidth-to-cost ratio, but they are limited by their VRAM capacity. Workstation cards offer much more memory and Error Correction Code (ECC) memory for stability. Datacenter cards use HBM (High Bandwidth Memory) to provide massive throughput, often used in clusters to serve hundreds of users simultaneously.

Apple Silicon, CPUs, and NPUs

Not all AI hardware is a dedicated NVIDIA GPU. Depending on your budget and your specific needs, other options might be more appropriate.

Apple Silicon (Unified Memory)

Apple's M-series chips (like the M3 Ultra or M4 Max) use a "unified memory" architecture. This means the CPU and GPU share the same pool of fast memory. A Mac with a large memory configuration can load massive models that would require several expensive NVIDIA GPUs. While the bandwidth is lower than top-tier NVIDIA cards, the capacity is incredibly high, making it a favorite for running large models locally and quietly.

CPUs and System RAM

You can run LLMs on a standard computer CPU using tools like llama.cpp. However, the speed is significantly lower because system RAM (like DDR5) has much lower bandwidth than GPU VRAM. For example, dual-channel DDR5 is roughly 10 times slower than a high-end GPU. CPU inference is best suited for small models, background batch jobs, or embeddings.

NPUs (Neural Processing Units)

NPUs are small, specialized accelerators found in modern laptops and phones. They are designed for low-power, compact AI tasks like summarizing notifications or transcription. They are generally not powerful enough for running large, general-purpose LLMs but are excellent for on-device, small-scale AI features.

Rent vs. Buy for Inference

Deciding whether to rent cloud GPU time or buy your own hardware is a critical business decision. The best way to decide is to estimate your monthly GPU hours.

Situation Better Choice Reasoning
Exploring models/testing Rent (Cloud/API) No upfront cost; easy to switch hardware.
Spiky/Unpredictable traffic Rent (Cloud/API) Scale up only when demand arrives.
Bulk generation (high use) Buy (On-premise) Fixed hardware cost beats high hourly rental bills.
Sensitive/Private data Buy (On-premise) You maintain total control over data privacy.

If your hardware would be busy most of the day for a year or more, owning or leasing a dedicated machine usually wins. If the hardware would sit idle, renting is the smarter financial move.

For those looking for the best performance on a budget, you should explore GPU for AI options or look into VRAM for LLM requirements to ensure your hardware can handle the specific models you intend to deploy.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

How much VRAM do I need for an LLM?

You need enough to hold the model weights (parameters × bytes per parameter) plus extra for the KV cache and runtime overhead. Always leave 20-30% headroom.

Why is my LLM generation slow?

Inference speed is often limited by memory bandwidth rather than raw compute. The chip must read every weight from memory for every token generated.

What is the difference between consumer and datacenter GPUs?

Consumer cards offer great bandwidth per rupee but limited VRAM. Datacenter cards use HBM (High Bandwidth Memory) and NVLink for massive throughput and high-capacity serving.

Can I run an LLM on a CPU?

Yes, using tools like llama.cpp, but it is significantly slower because system RAM bandwidth is much lower than GPU VRAM bandwidth.

What is the KV cache?

The KV cache stores key and value vectors for past tokens so the model doesn't have to recompute them, allowing for faster generation during long conversations.