Let's talk

The Hardware Evolution

AI PC 2030: Navigating the Hardware Landscape

As AI models grow, the definition of a high-performance PC shifts from raw compute to memory capacity and bandwidth.

By Divyaprakash JhaUpdated 4 min readTopic: AI hardware

Photo for AI PC 2030: Navigating the Hardware Landscape

The Golden Rule: Memory First, Bandwidth Second

To run a large language model (LLM) effectively, you must prioritize memory capacity to hold the model weights and the KV cache. Once the model fits in your hardware, memory bandwidth becomes the primary factor determining how fast the model generates text.

For builders and business owners, sizing hardware is a matter of simple arithmetic. To run a model, you need enough fast memory to hold the model weights plus the KV cache for your specific context length. A rough rule for sizing is: parameters × bytes per parameter.

Once the model fits in your available memory, bandwidth determines your tokens per second (speed). This is why GPUs, with their very fast VRAM, outperform CPUs for AI inference. If you are looking to understand the underlying limitations of this architecture, you should explore the memory wall in AI.

Sizing Your VRAM for Local Models

When planning your hardware, it is vital to distinguish between fixed costs and variable costs. Model weights are a fixed cost—they take up a set amount of space. However, the KV cache grows linearly with your context length (the length of the conversation or document being processed).

If you choose a model that "just fits" the available VRAM, the system will slow down or fail once a long conversation fills the KV cache. You must always leave 1–2 GB of headroom for runtime overhead and activations.

Model Size FP16 / BF16 weights 8-bit weights 4-bit weights
3B ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB
8B ≈ 16 GB ≈ 8 GB ≈ 4–5 GB
14B ≈ 28 GB ≈ 14 GB ≈ 7–9 GB
32B ≈ 64 GB ≈ 32 GB ≈ 16–20 GB
70B ≈ 140 GB ≈ 70 GB ≈ 35–42 GB

Why Bandwidth Dictates Token Speed

During the decoding phase (when the model is generating text one word at a time), the chip spends most of its time waiting for memory, not performing math. The arithmetic per weight is tiny, so the bottleneck is how fast that data can move from the memory to the processor.

A useful ceiling for your calculations is: max tokens per second ≈ memory bandwidth ÷ bytes read per token. In real-world scenarios, actual speed is often 50 to 80 percent of this theoretical ceiling due to various overheads.

For example, an 8B model at 4-bit quantization on an RTX 4090 (1,008 GB/s) has a theoretical ceiling of about 220 tokens per second. In contrast, a 70B model at 4-bit on a card with 1,792 GB/s bandwidth has a ceiling of about 45 tokens per second.

If you are interested in how specialized hardware addresses these bottlenecks, see our guide on AI inference chips.

Three Tiers of AI Hardware

Hardware generally falls into three distinct categories based on use case, memory type, and bandwidth. Choosing the right tier is essential for balancing cost and performance.

GPU Tier Memory Type/Spec Typical Use Case
Workstation (e.g., RTX PRO 6000) 96 GB GDDR7 w/ ECC | ~1,792 GB/s Trades price efficiency for high VRAM; ideal for 70B models or large MoE.

Beyond Discrete GPUs: Apple and NPUs

Not all AI hardware follows the discrete GPU model. Different architectures offer different trade-offs between capacity and speed.

Apple Silicon and Unified Memory

Apple's M-series chips use a unified memory architecture, meaning the CPU and GPU share a single pool of RAM. This allows a Mac to load massive models that would require multiple consumer GPUs. For instance, an M3 Ultra can support up to 512 GB of memory. While the bandwidth is lower than a top NVIDIA card, the capacity is a massive advantage for running large models locally.

NPUs and Mobile AI

Neural Processing Units (NPUs) in laptops and phones are specialized, low-power accelerators. They are designed for small, compact models (roughly 1B to 4B parameters) to handle tasks like transcription or smart replies without draining your battery.

Running on a CPU

It is possible to run models on a standard CPU using tools like llama.cpp, but you are limited by memory bandwidth. Dual-channel DDR5-5600 memory offers roughly 90 GB/s, which is about a tenth of a high-end GPU. This means a large 70B model might only produce 2 tokens per second on a typical desktop.

The Speed Advantage of Mixture-of-Experts (MoE)

Mixture-of-Experts (MoE) models offer a clever way to bypass the "large model = slow speed" problem. Instead of activating every parameter for every token, an MoE model splits layers into sub-networks and only activates a few "experts" per token.

This allows a model to have a massive memory footprint (to hold all the weights) while decoding at the speed of a much smaller model. For example, a model like Gemma 4 26B might have 25 billion total parameters but only activates about 3.8 billion per token. It requires the memory of a 26B model but delivers the speed of a 4B model.

Strategic Decision: Renting vs. Buying

For business owners, the decision to own hardware or rent cloud compute is a financial one. The right choice depends on your workload's predictability and volume.

Situation Better Choice Why?
Exploring models or occasional testing Rent No upfront cost; allows you to try different GPU types quickly.
Short bursts of training or fine-tuning Rent Datacenter GPUs are expensive to own; rent them for days, not years.
Bulk, repeatable work (many hours/day) Buy Fixed costs eventually beat per-token or per-hour cloud bills.
Sensitive data requirements Buy (On-premise) Total control over where your data resides and how it is processed.
Spiky, unpredictable public traffic Rent/API Scale up only when demand arrives to avoid paying for idle hardware.

A simple rule of thumb: Estimate your monthly GPU hours. If the hardware would be busy most of the day for a year or more, owning a dedicated machine usually wins. If it would sit idle, renting is the smarter move.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

How much VRAM do I need for a 70B model?

At 4-bit quantization, a 70B model requires roughly 35–42 GB of VRAM, plus additional space for the KV cache and overhead.

Why is my AI generation slow even with a fast GPU?

The bottleneck is often memory bandwidth rather than raw compute. If the model is larger than your VRAM, it may spill to slower system RAM, significantly reducing speed.

What is the purpose of the KV cache?

The KV cache stores key and value vectors for past tokens so the model doesn't have to recompute them, allowing for faster text generation.

Does quantization affect AI performance?

Yes, quantization (like 4-bit or 8-bit) reduces the memory required per parameter, allowing larger models to fit on smaller hardware with minimal quality loss.

What is the difference between a GPU and an NPU?

GPUs are general-purpose parallel processors designed for heavy matrix multiplication, while NPUs are specialized, low-power accelerators for small, on-device AI tasks.