How AI Accelerators Work
An AI accelerator is specialized hardware designed to handle the massive matrix multiplications required by neural networks. For LLMs, performance is driven by memory capacity and bandwidth rather than just raw compute.
To understand why we need specialized hardware, you have to look at the nature of the math involved. Neural networks rely on matrix multiplication, where billions of weights are multiplied by inputs to generate a single response.
A standard CPU is like a highly skilled mathematician who can solve any complex logic problem but can only do one or two at a time. CPUs are designed for branching and logic with a few powerful cores. They are great for running an operating system or a web server, but they struggle with the sheer volume of repetitive math required by AI.
In contrast, GPUs and accelerators are like a stadium full of thousands of students all doing simple multiplication at the exact same time. They use thousands of simpler cores to perform the same operation on large blocks of numbers in parallel. This is exactly the shape of AI work.
Modern NVIDIA GPUs also include tensor cores. These are specialized units designed to multiply small matrices in a single step at reduced precision (such as FP16, BF16, FP8, or even FP4). By using lower precision, the hardware can perform more maths per second and use less memory, which is why techniques like quantization are so common in the industry.
The Memory Rule: Capacity vs. Bandwidth
When building an AI system, there is a fundamental distinction between how much data you can hold and how fast you can move it. To run a Large Language Model (LLM), you need enough fast memory to hold the model weights plus the KV cache for your context length.
The math for capacity is straightforward: parameters × bytes per parameter. For example, if you are running a model in 8-bit precision, each parameter takes up 1 byte of space. If you use 16-bit precision (FP16), each parameter takes 2 bytes.
However, capacity is only half the story. Once the model fits in your memory, memory bandwidth determines your tokens per second (speed). If the memory is slow, the chip sits idle waiting for data, no matter how many cores the processor has.
How Much VRAM Do You Need?
Choosing the right VRAM is the most critical decision for a builder. You must account for the model weights, the KV cache (the "memory" of the conversation), and runtime overhead for the software framework.
Weights take up a fixed amount of space based on their precision. An 8B model needs approximately 16 GB at FP16, about 8 GB at 8-bit, and roughly 4–5 GB at 4-bit. Because real-world files often keep some layers at higher precision to maintain quality, you should budget roughly 4.5 to 5 bits per weight to be safe.
| Model Size | FP16 / BF16 Weights | 8-bit Weights | 4-bit Weights | Typical GPU (4-bit) |
|---|---|---|---|---|
| 3B | ≈ 6 GB | ≈ 3 GB | ≈ 1.5–2 GB | Any 6–8 GB card |
| 8B | ≈ 16 GB | ≈ 8 GB | ≈ 4–5 GB | 8–12 GB card |
| 14B | ≈ 28 GB | ≈ 14 GB | ≈ 7–9 GB | 12–16 GB card |
| 32B | ≈ 64 GB | ≈ 32 GB | ≈ 16–20 GB | 24–32 GB card |
| 70B | ≈ 140 GB | ≈ 70 GB | ≈ 35–42 GB | 48 GB or larger |
Keep in mind that the KV cache grows linearly with your context length. If you want to use a massive 128K context window, the cache itself can require more memory than the weights of the model itself. This is why the best GPU for a local LLM is usually the one with the most VRAM you can afford.
Why Bandwidth Decides Token Speed
In the world of AI inference (running a model), the bottleneck is rarely the "compute" power of the chip. Instead, it is the speed of the connection between the chip and the memory. During decoding—the process of generating text one word at a time—the chip spends most of its time waiting on memory, not performing calculations.
A useful rule of thumb for estimating speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token. While real-world speeds are often 50 to 80 percent of this theoretical ceiling due to software overhead, it is the best way to compare hardware.
For example, an 8B model at 4-bit precision is about 4.5 GB. On a high-end consumer card like an RTX 4090, which has a bandwidth of 1,008 GB/s, the ceiling is roughly 220 tokens per second. On a massive 70B model, the speed will be much lower because there is simply more data to move for every single token generated.
Comparing Hardware Tiers
Not all accelerators are created equal. Depending on whether you are a hobbyist, a professional designer, or a data center operator, your needs will vary wildly.
| GPU Tier | Example | Memory Type/Capacity | Best Use Case |
|---|---|---|---|
| Consumer | RTX 4090 | 24 GB GDDR6X | Best bandwidth per rupee; desktop use. |
| Workstation | RTX PRO 6000 | 96 GB GDDR7 (ECC) | Large models, small-team serving, all-day use. |
| Datacenter | H100 SXM | 80 GB HBM3 | High-throughput serving and training. |
Apple Silicon offers a unique middle ground. Using unified memory, a Mac with high memory configuration can load enormous models that would require multiple expensive NVIDIA GPUs. While the bandwidth is lower than a top-tier NVIDIA card, the sheer capacity makes it a favorite for running large models locally.
NPUs and Small Accelerators
You may have heard of NPUs (Neural Processing Units) in your phone or new "AI PCs." These are small, highly specialized accelerators designed for low-power, on-device tasks. They aren't meant to run a massive 70B model; instead, they handle compact models (roughly 1B to 4B parameters) for features like transcription, smart replies, or summarizing notifications.
Because they share memory with the rest of the system, these models must be heavily quantized and kept small to remain efficient without draining your battery.
Should You Rent or Buy AI Hardware?
For business owners, the decision to invest in hardware or use the cloud is a financial one. The rule of thumb is simple: If the hardware would be busy most of the day for a year, owning it usually wins.
Rent if:
- You are just exploring different models.
- You only need short bursts of power for fine-tuning.
- You have spiky public traffic that changes every week.
Buy if:
- You have bulk generation tasks (like processing thousands of documents) running many hours every day.
- You need to keep sensitive data on-premise for privacy or compliance.
- You want fixed costs rather than variable API bills.
Ultimately, the goal is to route bulk, repeatable work to hardware you control, while using paid APIs for the most difficult, high-reasoning tasks.

Questions people ask
What is the main bottleneck in AI inference?
For most LLM work, memory bandwidth is the primary bottleneck. The chip spends most of its time waiting to read weights from memory rather than performing calculations.
Does more VRAM make AI faster?
VRAM capacity determines if a model can fit; VRAM bandwidth determines how fast it generates text. More VRAM allows for larger models and longer context, but bandwidth dictates the tokens per second.
What is the KV cache?
The KV cache stores key and value vectors for past tokens so the model doesn't have to recompute them. It grows with the length of the conversation and the number of users served.
Can I run LLMs on my CPU?
Yes, using tools like llama.cpp, you can run quantized models on a CPU. However, because system RAM bandwidth is much lower than GPU VRAM, it is significantly slower.
What are Tensor Cores?
Tensor cores are specialized units in modern GPUs designed to multiply small matrices in a single step, often at reduced precision to increase speed and efficiency.
