HBM vs. GDDR: Understanding the Difference
To run large language models efficiently, the speed of your memory often matters more than the raw compute power of your chip. High Bandwidth Memory (HBM) provides the massive data throughput required for high-speed AI inference, whereas consumer-grade GDDR is better suited for lighter tasks or smaller models.
When building AI infrastructure, you will encounter two main types of video memory (VRAM). Consumer-grade graphics cards, such as the NVIDIA RTX 4090, use GDDR (Graphics Double Data Rate) memory. This is standard for gaming and general-purpose computing.
In contrast, datacenter-grade accelerators use HBM (High Bandwidth Memory). HBM is a "stacked" memory technology, meaning the memory chips are placed vertically on top of each other and connected directly to the processor. This architecture allows for a much wider bus, providing significantly more bandwidth than GDDR. This extra bandwidth is critical for high-throughput AI serving, where the goal is to move massive amounts of data to the chip as fast as possible.
| Memory Type | Common Use Case | Key Characteristic |
|---|---|---|
| GDDR (e.g., RTX 4090) | Consumer PCs, Gaming, Local LLMs | High speed, lower cost, limited bandwidth |
| HBM (e.g., H100) | Datacenter AI, Large-scale Training | Extreme bandwidth, stacked architecture |
Why Bandwidth Decides Token Speed
A common misconception is that a chip with more "TFLOPS" (teraflops) will always run an AI model faster. For Large Language Models (LLMs), this is often untrue during the text generation phase, known as decoding.
During decoding, the chip must read every single active weight from the memory exactly once to generate each new token. Because the mathematical operations performed on these weights are relatively simple, the processor often spends most of its time sitting idle, waiting for the next batch of weights to arrive from the memory. This is a classic example of the memory wall in AI computing.
You can estimate the maximum speed of your hardware using a simple formula. A useful ceiling for speed is:
max tokens per second ≈ memory bandwidth ÷ bytes read per token
While real-world speeds are typically 50% to 80% of this theoretical ceiling due to software overhead, the principle remains: higher bandwidth directly translates to faster text generation (tokens per second). If you are building a service for customers, bandwidth is your primary bottleneck for speed.
Calculating VRAM Needs for AI Models
Before you worry about speed, you must ensure the model can actually fit into your hardware. If a model is too large for your available VRAM, it will either fail to load or fall back to system RAM, which is significantly slower.
The total memory needed for an AI application is calculated as follows:
Total memory needed = Model Weights + KV Cache + Runtime Overhead
The model weights are determined by the number of parameters and the precision (bit-depth) you use. For example, a 7B (7 billion parameter) model has different requirements depending on how it is stored:
- FP16 (Full Precision): 2 bytes per parameter (approx. 14 GB)
- 8-bit (Quantized): 1 byte per parameter (approx. 7 GB)
Always leave headroom for the KV cache and 1 to 2 GB of runtime overhead for the framework and activations. A model that "just fits" will likely crash or slow down once a conversation grows long.
The Role of the KV Cache in Memory Usage
When you chat with an AI, it doesn't re-read the entire conversation from scratch every time it generates a new word. Instead, it stores the "keys" and "values" of previous tokens in a temporary storage area called the KV cache.
The KV cache is essential for performance, but it comes at a cost: it grows linearly with the length of the conversation (the context length). If you want to support very long documents or long-running chats, the KV cache can eventually require more memory than the model weights themselves.
For example, a standard 8B model might only need 5 GB for its weights at 4-bit precision, but if you ask it to process a massive 128K context window, the KV cache could require an additional 16 GB of memory.
To combat this, modern AI architectures use techniques like Grouped-query attention (GQA), which allows several "query" heads to share a single "key/value" head. This can shrink the cache size significantly, allowing for longer conversations on the same hardware.
GPU Tiers: Consumer, Workstation, and Datacenter
Hardware choices depend on whether you are a hobbyist, a developer, or an enterprise provider. The following tiers represent the current landscape of AI hardware:
| Tier | Example Hardware | Best For |
|---|---|---|
| Consumer | NVIDIA RTX 4090 | Best bandwidth per rupee; limited VRAM capacity. |
| Workstation | NVIDIA RTX PRO 6000 | More VRAM and ECC support for long-term reliability. |
| Datacenter | NVIDIA H100 / H200 | Massive HBM throughput and high-speed GPU-to-GPU links (NVLink). |
For a single user running local models, VRAM capacity is usually the first limit you hit. For a business serving hundreds of users, memory bandwidth and the ability to link multiple GPUs together become the priority.
How Mixture-of-Experts (MoE) Uses Memory
Mixture-of-Experts (MoE) is a specialized architecture designed to provide the intelligence of a massive model with the speed of a much smaller one. Instead of activating every single parameter for every token, an MoE model splits its layers into sub-networks and only activates a few "experts" per token.
This creates a unique memory profile: all weights must reside in memory, but only a small fraction are actually read per token. This allows a model to have massive capacity (high parameter count) while maintaining the fast decoding speeds of a smaller model. This is why MoE models are highly efficient for high-end hardware with large VRAM capacities.
Apple Silicon and Unified Memory
Apple’s M-series chips offer a unique alternative to traditional PC architectures through Unified Memory. In a Mac, the CPU and GPU share a single pool of memory.
This is a massive advantage for running very large models. A Mac with a high-end memory configuration can load models that would require multiple expensive consumer GPUs to hold. For example, an M3 Ultra can access up to 512 GB of memory. While the bandwidth of Apple's chips is typically lower than top-tier NVIDIA datacenter cards, the sheer capacity makes them excellent for researchers and developers who need to test large models locally without a massive server rack.

Questions people ask
What is the difference between HBM and GDDR?
GDDR is used in consumer graphics cards, while HBM is stacked memory used in datacenter GPUs to provide much higher bandwidth for AI workloads.
Does model size or memory bandwidth matter more for inference?
Model size (VRAM capacity) determines if a model can fit, but memory bandwidth determines how many tokens per second you actually get.
How much VRAM do I need for a 7B model?
A 7B model requires about 14 GB at FP16, 7 GB at 8-bit, and roughly 3.5 GB at 4-bit, plus extra for the KV cache and overhead.
Why does long context use so much memory?
Longer conversations increase the size of the KV cache, which stores vectors for every past token in every layer to avoid recomputation.
Can I run LLMs on a CPU?
Yes, using tools like llama.cpp, but you are limited by system RAM bandwidth, which is much slower than a GPU.
