Let's talk

Hardware Essentials

AI Hardware: A Guide for Builders and Business Owners

To run large language models, you need enough fast memory to hold model weights and the KV cache. This guide explains how to select the right hardware for your AI workloads.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for AI Hardware: A Guide for Builders and Business Owners

Memory First, Bandwidth Second

To run a large language model (LLM), you need enough fast memory to hold the model weights plus the KV cache for your context length. Once the model fits, memory bandwidth determines your tokens per second (speed).

To run an LLM, you need enough fast memory to hold the model weights plus the KV cache for your context length. A rough rule for memory is: parameters × bytes per parameter.

Once the model fits in memory, memory bandwidth decides your tokens per second (speed). This is why GPUs beat CPUs for inference; their VRAM provides much higher bandwidth than standard system RAM.

A neural network is mostly matrix multiplication. Generating one token means multiplying the current input by billions of weights. Because each multiplication is independent, the work can be split across thousands of small cores at once. A CPU has a few dozen powerful cores designed for branching and varied logic, while a GPU has thousands of simpler cores designed to do the same operation on large blocks of numbers in parallel. This is exactly the shape of AI work.

How Much VRAM Do You Need?

Model weights are a fixed cost. For example, an 8B model needs about 16 GB at FP16, 8 GB at 8-bit, and 4 to 5 GB at 4-bit. Note that real 4-bit files are a little larger because popular formats keep some layers at higher precision and store scaling factors. A good rule of thumb is to budget roughly 4.5 to 5 bits per weight to account for this overhead.

The KV cache (context window) grows linearly with context length and the number of users served at once. While weights are a fixed cost, the KV cache grows with every token of context. If you are building a service for many users, you must account for this growth.

Always leave headroom for the KV cache; a model that "just fits" will fail or slow down once a long conversation fills the cache. For a Llama 3.1 8B style model, a 128K context window requires about 16 GiB of cache—which is more than the 4-bit weights themselves!

Model Size FP16 / BF16 weights 8-bit weights 4-bit weights Typical single GPU (4-bit)
3B ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB Any 6–8 GB card, laptops, phones
8B ≈ 16 GB ≈ 8 GB ≈ 4–5 GB 8–12 GB card
14B ≈ 28 GB ≈ 14 GB ≈ 7–9 GB 12–16 GB card
32B ≈ 64 GB ≈ 32 GB ≈ 16–20 GB 24–32 GB card
70B ≈ 140 GB ≈ 70 GB ≈ 35–42 GB 48 GB or larger, or two GPUs

Why Bandwidth Decides Token Speed

During decoding (generating text), the chip spends most of its time waiting on memory rather than performing math. The arithmetic per weight is tiny, so the bottleneck is how fast you can move those weights from memory to the processor.

A useful ceiling for speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token. Real speed is usually 50 to 80 percent of this ceiling due to overheads, but it is an excellent estimate for comparing hardware.

For example, an 8B model at 4-bit is about 4.5 GB. On an RTX 4090 (1,008 GB/s), the ceiling is about 220 tokens per second. A 70B model at 4-bit is about 40 GB; on a card with 1,792 GB/s, the ceiling is about 45 tokens per second.

Note that reading your prompt (prefill) is different. It processes many tokens at once, so it is limited by compute, not bandwidth. This is why a long prompt has a pause before the first word, followed by a steady stream of text.

Understanding GPU Tiers

GPUs for AI generally fall into three categories based on their intended use and memory technology. Choosing the right tier depends on whether you need raw speed, massive capacity, or high reliability.

GPU Tier Example Memory Bandwidth Best For
Consumer GeForce RTX 4090 1,008 GB/s Up to ~32B at 4-bit; best bandwidth per rupee
Consumer (Next Gen) GeForce RTX 5090 1,792 GB/s 32B at 4-bit with longer context
Workstation RTX PRO 6000 Blackwell 1,792 GB/s 70B at 4–8-bit; large MoE; small-team serving
Datacenter H100 SXM 3.35 TB/s High-throughput serving and training
Datacenter H200 4.8 TB/s Large models, long context, many users

Consumer cards give the best bandwidth per rupee but have limited VRAM and are designed for desktops. Workstation cards trade some price efficiency for much more VRAM, ECC memory, and support for continuous server use. Datacenter cards use HBM (High Bandwidth Memory) and fast GPU-to-GPU links like NVLink to act as a single large accelerator for massive workloads.

Apple Silicon, CPUs, and NPUs

Apple Silicon Unified Memory

Apple's M-series chips use a unified memory architecture where the CPU and GPU share a single pool of RAM. A Mac with a large memory configuration can load models that would require several consumer GPUs. While bandwidth is lower than top NVIDIA cards—the M3 Ultra offers up to 819 GB/s—the capacity is exceptionally high, making it great for running large models quietly and efficiently. Tools like Ollama and MLX run well on this architecture.

CPUs and System RAM

You can run quantized models on ordinary CPUs using tools like llama.cpp, but you are limited by memory bandwidth. For example, dual-channel DDR5-5600 has a theoretical peak of roughly 90 GB/s, which is about a tenth of a high-end GPU. A 40 GB model (70B at 4-bit) might top out at about 2 tokens per second on a typical desktop. CPU inference is fine for small models, background batch jobs, or embeddings.

NPUs and Phones

Neural Processing Units (NPUs) in laptops and phones are small accelerators designed for low-power AI tasks. They are built for compact, on-device models (roughly 1B to 4B parameters) to handle tasks like transcription or smart replies without draining the battery.

Should You Rent or Buy Hardware?

Deciding whether to rent cloud compute or buy your own hardware depends on your expected usage and data sensitivity. A simple way to decide is to estimate your monthly GPU hours. If the hardware would be busy most of the day for a year or more, owning a machine usually wins. If it would sit idle, renting is better.

Situation Better Choice Reasoning
Exploring models or short bursts Rent No upfront cost; try different GPUs
Bulk generation (many hours daily) Buy or lease Fixed cost beats per-token or per-hour bills
Sensitive data requirements Buy (on-premise) You control where data goes
Spiky public traffic Rent or API Scale up only when demand arrives
Frontier-model quality needed Paid API The largest models are not available to self-host

For businesses in India, remember that high-end GPUs are heaters. An RTX PRO 6000 Blackwell can draw up to 600 W. When adding the CPU and other components, a single AI workstation can draw close to a kilowatt. Plan for a suitable power supply, a cool room, and a UPS or inverter sized for that load.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

How much memory does an 8B model need?

An 8B model needs about 16 GB at FP16, 8 GB at 8-bit, and 4 to 5 GB at 4-bit. You must also add extra for the KV cache and runtime overhead.

Why is VRAM more important than raw compute for LLMs?

For most LLM inference, the chip spends most of its time waiting on memory. Therefore, memory bandwidth often determines the tokens per second more than raw compute power.

What is the KV cache?

The KV cache stores key and value vectors for past tokens so the model doesn't have to recompute them, allowing for faster text generation.

Can I run LLMs on a CPU?

Yes, using tools like llama.cpp, you can run quantized models on a CPU, but the speed is limited by your system's memory bandwidth.

What are the benefits of Mixture-of-Experts (MoE) models?

MoE models trade memory capacity for speed; they require memory for all weights, but only activate a fraction of them per token, allowing for faster decoding.