The Golden Rule: Memory First, Bandwidth Second
To run a large language model (LLM), you need enough fast memory to hold the model weights plus the KV cache for your context length. Once the model fits in your hardware, memory bandwidth determines your tokens per second (speed).
Building an AI server requires a clear understanding of how data moves. Many builders make the mistake of focusing on raw compute power (TFLOPS) when they should be focusing on how much data the chip can access at once. For most LLM inference tasks, the chip spends most of its time waiting on memory.
There is a simple mathematical rule for sizing your hardware: parameters × bytes per parameter. For example, if you are running a model with 8 billion parameters in FP16 (which uses 2 bytes per parameter), you need at least 16 GB of VRAM just to hold the weights. If you use 8-bit quantization (1 byte per parameter), that same model only needs 8 GB.
Once you have ensured the model fits, your speed is dictated by memory bandwidth. This is why high-end GPUs, which use specialized memory like HBM or fast GDDR, significantly outperform standard computer RAM for AI tasks. If you are just starting your journey, you might want to read our guide on AI hardware to understand the components better.
GPU Tiers: Consumer, Workstation, and Datacenter
Not all GPUs are created equal. Depending on whether you are a hobbyist in a home office or a business owner running a high-traffic API, your hardware needs will vary significantly. We categorize them into three main tiers:
| GPU Tier | Example Card | Memory Type/Capacity | Best Use Case |
|---|---|---|---|
| Consumer | GeForce RTX 4090 | 24 GB GDDR6X | Up to 32B models at 4-bit; best bandwidth per rupee. |
| Workstation | RTX PRO 6000 Blackwell | 96 GB GDDR7 (with ECC) | 70B models; large MoE models; 24/7 server operation. |
| Datacenter | H100 / H200 | 80GB - 141GB HBM3/e | High-throughput serving and massive training tasks. |
Consumer cards like the RTX 4090 offer the best performance for every rupee spent, but they have limited VRAM and are designed for desktop use. They are excellent for individual developers or small-scale testing.
Workstation cards provide much more VRAM and include ECC (Error Correction Code) memory, which is vital for stability during long-running tasks. They are designed to run 24/7 in a professional environment. If you are looking for the right GPU for AI, consider your VRAM needs first.
Datacenter cards (like the H100) use HBM (High Bandwidth Memory) and high-speed links like NVLink. These allow multiple GPUs to act as one massive accelerator, which is necessary for high-throughput serving or training models from scratch.
Calculating Required VRAM
Calculating VRAM is not as simple as looking at the model size. You must account for the model weights, the overhead, and the "context" (the conversation history).
The Weights Cost
Model weights are a fixed cost. A 7B (7 billion parameter) model has different requirements depending on its precision:
- FP16 (High precision): ~14 GB
- 8-bit (Standard quantization): ~7 GB
- 4-bit (Highly compressed): ~3.5 GB
A useful rule of thumb for builders is to budget roughly 4.5 to 5 bits per weight. This provides a safety margin for scaling factors used in popular 4-bit formats.
The KV Cache: The Hidden Cost
When you chat with an AI, it doesn't re-read the entire conversation every time it generates a new word. Instead, it stores previous "keys" and "values" in a temporary space called the KV cache. This cache grows linearly with the length of the conversation and the number of users you are serving.
If you are building an application for many users, you must leave significant headroom in your VRAM to prevent the system from crashing when conversations get long. For more on memory limits, see our guide on VRAM basics.
How Bandwidth Dictates Token Speed
In the world of LLMs, "speed" is measured in tokens per second. For a single user, the bottleneck is almost always memory bandwidth. During the "decoding" phase (when the model is spitting out words one by one), the chip must read every single weight from the memory for every single token generated. Because the actual math is relatively simple, the chip spends most of its time waiting for the data to arrive from the VRAM.
You can estimate the maximum speed using this formula:
Max tokens per second ≈ Memory Bandwidth ÷ Bytes read per token
For example, an RTX 4090 has a bandwidth of roughly 1,008 GB/s. If you are running a 4-bit 8B model (which is about 4.5 GB), the theoretical ceiling is about 220 tokens per second. Real-world speeds are typically 50% to 80% of this ceiling due to software overhead.
The Mixture-of-Experts (MoE) Advantage
Some newer models use a "Mixture-of-Experts" architecture. These models, such as Gemma 4 26B, have a high total parameter count but only "activate" a small fraction of those parameters for any given token. This means they require the memory capacity of a large model (to hold all the experts) but can decode at a speed closer to a much smaller model. This makes MoE models an excellent choice for hardware with large VRAM but moderate bandwidth.
Beyond Dedicated GPUs: Apple and CPUs
While NVIDIA GPUs are the industry standard, there are other ways to run AI models depending on your budget and goals.
Apple Silicon and Unified Memory
Apple's M-series chips (like the M3 Ultra or M4 Max) use a "unified memory" architecture. This means the CPU and GPU share the same pool of high-speed RAM. A Mac with 128 GB of RAM can load massive models that would require multiple expensive NVIDIA cards. While the bandwidth is lower than a top-tier datacenter GPU, the high capacity makes it a favorite for local development.
CPUs and System RAM
You can run models on a standard computer CPU using tools like llama.cpp. This uses your system's DDR RAM. However, because system RAM bandwidth is much lower than GPU VRAM (often around 50-90 GB/s), the speed is very slow. A large 70B model might only generate 2 tokens per second on a standard desktop. This is fine for background tasks or small models, but not for interactive use.
NPUs (Neural Processing Units)
Most modern laptops and phones now include an NPU. These are tiny, highly efficient accelerators designed for small, low-power tasks like transcribing audio or summarizing notifications. They are not intended for running large, complex LLMs.
Should You Rent or Buy Your AI Server?
For business owners, the decision to rent cloud compute or buy physical hardware is a matter of economics and data privacy.
| Situation | Better Choice | Reasoning |
|---|---|---|
| Exploring models or testing | Rent | No upfront cost; you can switch GPUs easily. |
| Short bursts of training | Rent | Datacenter GPUs are expensive to own but cheap to rent for a few days. |
| High, consistent usage | Buy | If the hardware is busy most of the day, owning it is cheaper over time. |
| Sensitive data / Privacy | Buy (On-premise) | You maintain total control over where your data lives. |
A good rule of thumb for Indian small businesses and developers: calculate your monthly cloud bill. If you would be paying for a dedicated server for more than 12 months anyway, it is usually more cost-effective to buy the hardware and host it locally. Just remember to plan for the extra power and cooling requirements—an AI workstation can draw close to a kilowatt of power under full load.

Questions people ask
How much VRAM do I need for an LLM?
You need enough to hold the model weights (parameters × bytes per parameter) plus extra for the KV cache and runtime overhead. Always leave 20-30% headroom.
Why are GPUs better than CPUs for AI?
AI work involves massive matrix multiplication that can be split across thousands of simple GPU cores in parallel, whereas CPUs have fewer, more complex cores designed for logic.
What is the KV cache?
The KV cache stores key and value vectors for past tokens so the model doesn't have to recompute them, allowing for faster text generation.
What is the difference between HBM and GDDR?
GDDR is common in consumer cards, while HBM (High Bandwidth Memory) is used in datacenter cards and offers significantly higher bandwidth.
Does more VRAM make AI faster?
VRAM capacity determines if a model can fit; memory bandwidth determines how fast the model generates tokens.
