The VRAM Rule: Fitting the Model
To run an LLM on an AMD GPU, you must first ensure the card has enough VRAM to hold the model weights and the KV cache. For builders, the most important rule is: memory capacity determines if a model can run, while memory bandwidth determines how fast it responds.
To run an LLM, you need enough VRAM to hold model weights plus the KV cache for your context length. A rough rule for memory is: parameters × bytes per parameter.
When choosing an AMD card, remember that "just fitting" the weights is often not enough. You must always leave 1-2 GB of headroom for runtime overhead and activations. If you run out of VRAM, the system will offload to your system RAM (CPU), which is significantly slower.
| Model Size | FP16 / BF16 weights | 8-bit weights | 4-bit weights | Typical AMD GPU Target (4-bit) |
|---|---|---|---|---|
| 3B | ~6 GB | ~3 GB | ~1.5–2 GB | Any 6–8 GB card |
| 8B | ~16 GB | ~8 GB | ~4–5 GB | 8–12 GB card |
| 14B | ~28 GB | ~14 GB | ~7–9 GB | 12–16 GB card |
| 32B | ~64 GB | ~32 GB | ~16–20 GB | 24–32 GB card |
| 70B | ~140 GB | ~70 GB | ~35–42 GB | 48 GB or larger (or dual GPUs) |
Memory Bandwidth and Token Speed
Once a model fits in VRAM, memory bandwidth determines your tokens per second. During the decoding phase (when the model generates text one word at a time), the chip spends most of its time waiting on memory, not on raw compute. This makes bandwidth the primary driver of "snappiness" in an AI application.
A useful ceiling for speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token. While real-world speed is often 50 to 80 percent of this ceiling due to overhead, it remains the best way to compare different hardware setups.
For example, if you are comparing an AMD consumer card with GDDR memory against a datacenter card with HBM, the HBM card will provide significantly more speed because it can move data to the processor much faster.
AMD vs. NVIDIA for AI Inference
NVIDIA GPUs are the industry standard because they feature specialized Tensor Cores designed to multiply small matrices at reduced precision (such as FP16, BF16, FP8, and FP4). These cores allow for much faster math than standard processing units.
AMD and Apple have their own hardware equivalents designed to handle these specialized operations. While NVIDIA has a massive software lead, AMD hardware is a very viable alternative for running open models. Using runtimes like llama.cpp, you can achieve excellent results on AMD hardware.
A key concept for both manufacturers is quantization. This technique reduces the precision of the numbers (e.g., from 16-bit to 4-bit), which reduces the memory required per weight. This allows you to fit much larger models on the same amount of hardware.
Why Context Length Demands More Memory
When you chat with an LLM, the model needs to "remember" the previous parts of the conversation to provide coherent answers. It does this using the KV cache, which stores key and value vectors for every past token so they do not need to be recomputed every time a new word is generated.
The KV cache grows linearly with the context length and the number of users you are serving. For example, in a Llama 3.1 8B style model, a 128K context length can require about 16 GiB of cache—which is actually more than the 4-bit weights of the model itself!
If you are building a service for many users, you must account for this growth. If you don't leave enough VRAM for the KV cache, the model will run out of memory during long conversations. Modern models often use Grouped-query attention (GQA) to help shrink this cache size, making long conversations more efficient.
Mixture-of-Experts (MoE) and Hardware
Mixture-of-Experts (MoE) models are a specific architecture designed for efficiency. Instead of activating every single parameter for every word generated, an MoE model splits its layers into sub-networks and only activates a few "experts" per token.
This creates a unique hardware requirement: All weights must sit in VRAM, but each token only reads the active ones. This means MoE models trade memory capacity for speed. They are highly efficient on GPUs with large VRAM, as they provide the intelligence of a large model with the generation speed of a much smaller one.
Understanding GPU Tiers
Hardware for AI generally falls into three distinct categories. Choosing the right tier depends on whether you are a hobbyist, a professional, or a large-scale service provider.
- Consumer cards: These offer the best bandwidth per rupee/dollar. They are great for local experimentation, but they have limited VRAM.
- Workstation cards: These trade price efficiency for much more VRAM and include ECC (Error Correction Code) memory support. They are designed to run for long periods in professional environments.
- Datacenter cards: These use HBM (High-Bandwidth Memory) for massive throughput. They are ideal for high-throughput serving and training where speed and scale are the priority.
Should You Rent or Buy AI Hardware?
For business owners, the decision to own hardware or use the cloud is a financial one. The best way to decide is to estimate your monthly GPU hours.
| Situation | Better choice | Why? |
|---|---|---|
| Exploring models or short bursts | Rent | No upfront cost; try different GPUs. |
| Bulk generation many hours every day | Buy or Lease | Fixed cost beats per-hour bills at high use. |
| Sensitive data must stay on-premise | Buy (On-premise) | You have total control over your data. |
A simple rule of thumb: If your hardware would be busy most of the day for a year or more, owning a machine usually wins financially. If it would sit idle most of the time, renting is the smarter move.

Questions people ask
How much VRAM do I need for a 7B model?
A 7B model requires about 14 GB at FP16, 7 GB at 8-bit, and 3.5 GB at 4-bit, plus extra for the KV cache and overhead.
Does memory bandwidth matter for LLMs?
Yes. While VRAM capacity determines if a model can run, memory bandwidth determines how many tokens per second you generate.
What is the benefit of quantization?
Quantization (storing weights in 4 or 8 bits) uses less memory per number, allowing larger models to fit on your GPU.
What is the KV cache?
The KV cache stores vectors for past tokens so the model doesn't have to recompute them, but it grows as your conversation gets longer.
Can I run LLMs on a CPU?
Yes, using tools like llama.cpp, but CPU inference is much slower because system RAM has much lower bandwidth than GPU VRAM.
