Let's talk

AI Hardware Guide

AMD GPU for LLM: Hardware Selection Guide

Choosing the right AMD GPU for LLM workloads requires balancing VRAM capacity for model weights and memory bandwidth for token speed.

By Divyaprakash JhaUpdated 4 min readTopic: AI hardware

Photo for AMD GPU for LLM: Hardware Selection Guide

The VRAM Rule: Fitting the Model

To run an LLM on an AMD GPU, you must first ensure the card has enough VRAM to hold the model weights and the KV cache. For builders, the most important rule is: memory capacity determines if a model can run, while memory bandwidth determines how fast it responds.

To run an LLM, you need enough VRAM to hold model weights plus the KV cache for your context length. A rough rule for memory is: parameters × bytes per parameter.

When choosing an AMD card, remember that "just fitting" the weights is often not enough. You must always leave 1-2 GB of headroom for runtime overhead and activations. If you run out of VRAM, the system will offload to your system RAM (CPU), which is significantly slower.

Model Size FP16 / BF16 weights 8-bit weights 4-bit weights Typical AMD GPU Target (4-bit)
3B ~6 GB ~3 GB ~1.5–2 GB Any 6–8 GB card
8B ~16 GB ~8 GB ~4–5 GB 8–12 GB card
14B ~28 GB ~14 GB ~7–9 GB 12–16 GB card
32B ~64 GB ~32 GB ~16–20 GB 24–32 GB card
70B ~140 GB ~70 GB ~35–42 GB 48 GB or larger (or dual GPUs)

Memory Bandwidth and Token Speed

Once a model fits in VRAM, memory bandwidth determines your tokens per second. During the decoding phase (when the model generates text one word at a time), the chip spends most of its time waiting on memory, not on raw compute. This makes bandwidth the primary driver of "snappiness" in an AI application.

A useful ceiling for speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token. While real-world speed is often 50 to 80 percent of this ceiling due to overhead, it remains the best way to compare different hardware setups.

For example, if you are comparing an AMD consumer card with GDDR memory against a datacenter card with HBM, the HBM card will provide significantly more speed because it can move data to the processor much faster.

AMD vs. NVIDIA for AI Inference

NVIDIA GPUs are the industry standard because they feature specialized Tensor Cores designed to multiply small matrices at reduced precision (such as FP16, BF16, FP8, and FP4). These cores allow for much faster math than standard processing units.

AMD and Apple have their own hardware equivalents designed to handle these specialized operations. While NVIDIA has a massive software lead, AMD hardware is a very viable alternative for running open models. Using runtimes like llama.cpp, you can achieve excellent results on AMD hardware.

A key concept for both manufacturers is quantization. This technique reduces the precision of the numbers (e.g., from 16-bit to 4-bit), which reduces the memory required per weight. This allows you to fit much larger models on the same amount of hardware.

Why Context Length Demands More Memory

When you chat with an LLM, the model needs to "remember" the previous parts of the conversation to provide coherent answers. It does this using the KV cache, which stores key and value vectors for every past token so they do not need to be recomputed every time a new word is generated.

The KV cache grows linearly with the context length and the number of users you are serving. For example, in a Llama 3.1 8B style model, a 128K context length can require about 16 GiB of cache—which is actually more than the 4-bit weights of the model itself!

If you are building a service for many users, you must account for this growth. If you don't leave enough VRAM for the KV cache, the model will run out of memory during long conversations. Modern models often use Grouped-query attention (GQA) to help shrink this cache size, making long conversations more efficient.

Mixture-of-Experts (MoE) and Hardware

Mixture-of-Experts (MoE) models are a specific architecture designed for efficiency. Instead of activating every single parameter for every word generated, an MoE model splits its layers into sub-networks and only activates a few "experts" per token.

This creates a unique hardware requirement: All weights must sit in VRAM, but each token only reads the active ones. This means MoE models trade memory capacity for speed. They are highly efficient on GPUs with large VRAM, as they provide the intelligence of a large model with the generation speed of a much smaller one.

Understanding GPU Tiers

Hardware for AI generally falls into three distinct categories. Choosing the right tier depends on whether you are a hobbyist, a professional, or a large-scale service provider.

  • Consumer cards: These offer the best bandwidth per rupee/dollar. They are great for local experimentation, but they have limited VRAM.
  • Workstation cards: These trade price efficiency for much more VRAM and include ECC (Error Correction Code) memory support. They are designed to run for long periods in professional environments.
  • Datacenter cards: These use HBM (High-Bandwidth Memory) for massive throughput. They are ideal for high-throughput serving and training where speed and scale are the priority.

Should You Rent or Buy AI Hardware?

For business owners, the decision to own hardware or use the cloud is a financial one. The best way to decide is to estimate your monthly GPU hours.

Situation Better choice Why?
Exploring models or short bursts Rent No upfront cost; try different GPUs.
Bulk generation many hours every day Buy or Lease Fixed cost beats per-hour bills at high use.
Sensitive data must stay on-premise Buy (On-premise) You have total control over your data.

A simple rule of thumb: If your hardware would be busy most of the day for a year or more, owning a machine usually wins financially. If it would sit idle most of the time, renting is the smarter move.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

How much VRAM do I need for a 7B model?

A 7B model requires about 14 GB at FP16, 7 GB at 8-bit, and 3.5 GB at 4-bit, plus extra for the KV cache and overhead.

Does memory bandwidth matter for LLMs?

Yes. While VRAM capacity determines if a model can run, memory bandwidth determines how many tokens per second you generate.

What is the benefit of quantization?

Quantization (storing weights in 4 or 8 bits) uses less memory per number, allowing larger models to fit on your GPU.

What is the KV cache?

The KV cache stores vectors for past tokens so the model doesn't have to recompute them, but it grows as your conversation gets longer.

Can I run LLMs on a CPU?

Yes, using tools like llama.cpp, but CPU inference is much slower because system RAM has much lower bandwidth than GPU VRAM.