Let's talk

Hardware for Builders

Next Generation AI Chips: Beyond Raw Compute

As AI models grow, the focus is shifting from raw processing power to memory bandwidth and capacity to handle massive context windows.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for Next Generation AI Chips: Beyond Raw Compute

Why Memory Matters More Than Compute

Next generation AI chips prioritize memory bandwidth and VRAM capacity to drive high-speed LLM inference and agentic workflows. While compute power is vital for processing prompts, the speed of generating text is primarily determined by how fast the chip can access model weights in memory.

To run a Large Language Model (LLM), you need enough fast memory to hold the model weights plus the KV cache for your context length. If the model does not fit entirely in your available memory, performance will drop significantly as the system struggles to swap data.

Once the model fits in memory, memory bandwidth—not just raw compute—decides how many tokens per second you get. For a single user, generating each new token (decoding) means reading every active weight from memory once. Because the arithmetic per weight is tiny, the chip often spends most of its time waiting on memory. This is a classic example of the memory wall in modern computing.

Tensor Cores and Low-Precision Math

Modern GPUs include specialized tensor cores designed to multiply small matrices in a single step. These units are essential for the heavy matrix multiplication that forms the backbone of neural networks.

Next generation efficiency comes from reduced precision like FP16, BF16, FP8, and even FP4. Lower precision reduces the memory required per number, allowing for more maths per second through quantization. By storing weights in 8-bit or 4-bit formats, you can fit much larger models into the same amount of hardware.

Quantization is a standard practice for inference. For example, storing weights in 4-bit instead of 16-bit can reduce the memory footprint by nearly 4x, allowing a business to run a much more capable model on cheaper, consumer-grade hardware.

Calculating VRAM for Modern Models

For builders and business owners, calculating the required VRAM is a critical step in hardware procurement. A rough rule for memory is: parameters × bytes per parameter.

To account for real-world file sizes, it is safer to budget roughly 4.5 to 5 bits per weight. You must also always leave headroom for the KV cache and 1 to 2 GB of runtime overhead for the framework and activations. A model that "just fits" will often fail or slow down once a long conversation fills the cache.

Model Size FP16 / BF16 weights 8-bit weights 4-bit weights Typical single GPU (4-bit)
3B ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB Any 6–8 GB card, laptops, phones
8B ≈ 16 GB ≈ 8 GB ≈ 4–5 GB 8–12 GB card
14B ≈ 28 GB ≈ 14 GB ≈ 7–9 GB 12–16 GB card
32B ≈ 64 GB ≈ 32 GB ≈ 16–20 GB 24–32 GB card
70B ≈ 140 GB ≈ 70 GB ≈ 35–42 GB 48 GB or larger, or two GPUs

The Growing Cost of Long Context

While model weights are a fixed cost, the KV cache is a dynamic cost. While generating text, a transformer stores a key and a value vector for every past token in every layer so it does not need to recompute them. This is known as the KV cache.

The cache grows linearly with context length and the number of users served simultaneously. For example, in a Llama 3.1 8B style model, the cache for 8,192 tokens is about 1 GiB. However, at a 128K context, the KV cache can reach 16 GiB, which is more than the 4-bit weights themselves.

If you are building an application for many users, the memory requirements for the cache can quickly exceed the memory required for the model weights. For instance, serving ten users at 8K context each requires about 10 GiB of cache space.

Consumer vs. Workstation vs. Datacenter

Hardware falls into three distinct tiers, each serving different business needs. Choosing the wrong tier can lead to either wasted budget or insufficient performance.

GPU Tier Example Memory Bandwidth Best Use Case
Consumer RTX 4090 / 5090 Up to 1,792 GB/s Exploring models, high bandwidth per rupee, personal use
Workstation RTX PRO 6000 Blackwell 1,792 GB/s 70B models, large MoE, small-team serving, ECC memory
Datacenter H100 / H200 Up to 4.8 TB/s High-throughput serving, training, many users, long context

Consumer GPUs offer great bandwidth per rupee but have limited VRAM capacity. Workstation cards trade some price efficiency for much higher VRAM and ECC memory, making them suitable for running all day in a server. Datacenter cards use HBM (High Bandwidth Memory) to achieve massive throughput for high-scale serving.

Apple Silicon and NPU Integration

Not all AI hardware follows the traditional discrete GPU model. Apple Silicon uses unified memory, allowing the CPU, GPU, and other components to share a single large pool of RAM. A Mac with a large memory configuration can load models that would need several consumer GPUs, making it a popular choice for developers who need high capacity in a quiet machine.

NPUs (Neural Processing Units) are small, low-power accelerators found in phones and laptops. These are designed for compact models, typically between 1B and 4B parameters, to handle on-device tasks like transcription or smart replies.

Standard CPUs can run quantized models using tools like llama.cpp, but they are limited by much lower memory bandwidth. For example, a typical desktop with DDR5 memory might only achieve about 90 GB/s, which is roughly a tenth of a high-end GPU, limiting the speed of text generation.

Mixture-of-Experts (MoE) Hardware Dynamics

Mixture-of-Experts (MoE) models are a specialized architecture that can offer significant speed advantages if you have the right hardware. An MoE model splits its layers into many expert sub-networks and routes each token to only a fraction of those weights.

This allows a model to have massive total capacity while decoding at the speed of a much smaller model. For example, a model might have 25 billion total parameters but only activate about 4 billion per token. While this makes the model very fast, it creates a specific hardware requirement: all weights must still sit in memory. Therefore, MoE models trade memory capacity for speed, making high VRAM capacity essential to keep all weights resident.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

What determines the speed of AI inference?

While compute is important for the initial prompt (prefill), the generation speed (decoding) is primarily limited by memory bandwidth.

How much VRAM do I need for a 70B model?

At 4-bit quantization, a 70B model needs roughly 35–42 GB of weights, plus additional space for the KV cache and overhead.

What is the difference between GDDR and HBM?

Consumer cards typically use GDDR memory, while datacenter cards use HBM (High Bandwidth Memory) which offers significantly higher bandwidth through stacked memory.

Why is quantization used in AI hardware?

Quantization stores weights in lower precision (like 4-bit or 8-bit), which reduces memory requirements and increases the number of maths performed per second.

Can I run LLMs on my CPU?

Yes, using tools like llama.cpp, but you are limited by the much lower memory bandwidth of standard system RAM compared to a GPU.