Understanding GPU Tiers for AI Workloads
Building AI infrastructure in India requires balancing memory capacity, bandwidth, and power. Learn how to size your hardware for LLM inference and training by understanding the differences between consumer, workstation, and datacenter GPUs.
AI hardware is not a one-size-fits-all category. Depending on whether you are a startup in Bengaluru running local tests or a large enterprise in Mumbai serving millions of users, your hardware requirements will differ significantly. The primary difference between these tiers lies in their memory capacity, how they handle heat, and how they connect to other chips.
| GPU Tier | Memory Bandwidth | Best Use Case |
|---|---|---|
| Consumer (e.g., RTX 4090/5090) | High (up to 1,792 GB/s) | Best bandwidth per rupee; limited VRAM; designed for desktops. |
| Workstation (e.g., RTX PRO 6000 Blackwell) | High (1,792 GB/s) | More VRAM, ECC memory, and support for 24/7 server operation. |
| Datacenter (e.g., H100, H200) | Ultra-High (up to 4.8 TB/s) | High-throughput serving and large-scale training using HBM. |
For a single user running local models, VRAM capacity is usually the first limit you hit. While consumer cards offer great value, they are designed for desktops and lack the specialized cooling and error correction found in professional-grade hardware.
The Golden Rule: Memory First, Bandwidth Second
When building an AI system, many business owners make the mistake of focusing solely on raw compute power (TFLOPS). However, for Large Language Models (LLMs), the real bottleneck is often moving data from the memory to the processor.
To run an LLM, you must have enough fast memory to hold the model weights plus the KV cache for your context length. If the model does not fit in your memory, it will either fail to run or become incredibly slow as it spills over to much slower system RAM.
A rough rule for sizing your memory is: parameters × bytes per parameter. For example, an 8B model needs about 16 GB at FP16 precision, or about 4 to 5 GB if you use 4-bit quantization. Once the model fits in your memory, the memory bandwidth determines your tokens per second—essentially, your inference speed.
This is why GPUs, with their very fast VRAM, beat CPUs for AI inference. If you are looking at the future of this hardware, you may want to explore next-generation AI chips that aim to solve these data movement challenges.
How Much VRAM Do You Actually Need?
Estimating VRAM is a two-step process: you must account for the fixed cost of the model weights and the variable cost of the conversation history.
1. The Model Weights
Model weights are a fixed cost. For instance, a 70B model at 4-bit quantization requires roughly 35–42 GB of VRAM. If you try to run this on a 24 GB consumer card, it simply won't work without significant workarounds that sacrifice speed.
2. The KV Cache and Headroom
The KV cache stores key and value vectors for every past token in every layer so the model does not have to recompute them. This cache grows linearly with the context length (how much text the model "remembers") and the number of users served simultaneously. If you are building an application for many users, the KV cache can eventually consume more memory than the model itself.
Why Memory Bandwidth Dictates Speed
During the "decoding" phase—when the model is actually generating text word by word—the chip spends most of its time waiting on memory rather than performing complex math. This is a fundamental characteristic of LLM inference.
A useful way to estimate your maximum speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token. This is why high-bandwidth memory, such as HBM (High Bandwidth Memory), is so critical for datacenter-grade performance.
Interestingly, Mixture-of-Experts (MoE) models can decode faster than traditional models of a similar size. Even though an MoE model might have massive total weights, it only activates a subset of parameters per token. This means it needs the memory capacity of a large model but can achieve the speed of a much smaller one.
Renting vs. Buying: The Business Decision
For an Indian business owner, the decision to own hardware or use the cloud is a matter of predictable costs versus flexibility. There is no single "correct" answer, but there are clear patterns based on usage.
| Scenario | Recommended Approach | Reasoning |
|---|---|---|
| Spiky traffic or testing new models | Rent (Cloud/API) | No upfront cost; easy to scale up or down. |
| High, consistent daily usage | Buy or Lease Dedicated Server | Fixed costs eventually beat hourly rental rates. |
| Sensitive or proprietary data | Buy (On-premise) | You maintain total control over data privacy. |
For most businesses, the most realistic path is a combination: use paid APIs for high-reasoning tasks, and use your own hardware for high-volume, repeatable tasks like drafting or summarizing.
Power and Cooling Requirements in India
AI hardware is essentially a high-powered heater. A single high-end workstation can draw close to a kilowatt of power under a heavy load. This has significant implications for your office setup.
Datacenter-grade GPUs require specialized rack power and cooling that standard office buildings in India may not provide. A typical office AC unit might struggle to maintain temperature if several AI workstations are running in a small room.
If you are building an on-premise AI lab, ensure you have a UPS (Uninterruptible Power Supply) or an inverter sized specifically to handle the sudden power draws of your AI hardware. A sudden voltage drop during a heavy training run can lead to corrupted data or hardware damage.
Optimizing Your Software Stack
Hardware is only half the battle; the software you use determines how efficiently you use that hardware. The "best" software depends entirely on your deployment goal.
- For Local Development: Use Ollama or llama.cpp. These are excellent for single-user applications or developers testing models on a laptop or a single workstation.
- For Production Environments: Use vLLM or SGLang. These frameworks are designed for high-throughput, meaning they can handle many concurrent users by managing memory more efficiently.
- For Maximum Performance: If you are using NVIDIA datacenter GPUs, TensorRT-LLM is the preferred choice for squeezing every bit of performance out of your hardware.

Questions people ask
What is the main bottleneck for AI inference?
Memory bandwidth is typically the primary bottleneck. While compute is important for the initial prompt (prefill), the actual generation of tokens is limited by how fast weights can be read from memory.
How does quantization affect memory requirements?
Quantization reduces the number of bits used per weight. For example, an 8B model requires 16 GB at FP16 (2 bytes per parameter) but only about 4-5 GB at 4-bit (0.5 bytes per parameter).
What is the KV cache and why does it matter?
The KV cache stores key and value vectors for past tokens so they don't need to be recomputed. It grows with context length and the number of users, consuming significant VRAM during long conversations.
Can I run large models on a CPU?
Yes, using tools like llama.cpp, you can run quantized models on CPUs. However, because CPU memory bandwidth is much lower than GPU VRAM, the speed is significantly slower.
When should I use HBM instead of GDDR?
HBM (High Bandwidth Memory) is used in datacenter GPUs to provide the massive bandwidth required for high-throughput serving and training, whereas GDDR is standard in consumer and workstation cards.
