Let's talk

Hardware Guide

Best CPU for AI Inference

While GPUs dominate high-speed AI tasks, understanding the role of the CPU is essential for running local LLMs and managing memory-intensive workloads.

By Divyaprakash JhaUpdated 4 min readTopic: AI hardware

Photo for Best CPU for AI Inference

CPU vs. GPU: Why Speed Varies

For high-speed AI inference, a GPU is significantly faster than a CPU because it is designed for massive parallel math. While a CPU is better for complex logic, a GPU's thousands of cores are built specifically for the matrix multiplication that drives AI.

To understand why your hardware choice matters, you have to look at how these chips are built. A neural network is essentially a massive series of matrix multiplications. When you ask an LLM to generate a word, the chip has to multiply your input by billions of weights.

A CPU is like a highly skilled specialist. It has a few dozen powerful cores designed for branching and varied logic—the kind of "if this, then that" tasks a computer performs constantly. A GPU is more like a massive factory of thousands of simple workers. Each worker isn't very smart, but they can all perform the same simple mathematical operation at the exact same time. Because AI work is mostly these repetitive, parallel math operations, the GPU wins on speed.

Feature CPU GPU
Core Count A few dozen powerful cores Thousands of simpler cores
Best Use Case Complex logic and branching Parallel matrix multiplication
AI Performance Lower (Sequential) Higher (Parallel)

The Memory Bandwidth Bottleneck

When running AI, the biggest problem isn't usually how fast the chip can "think"—it is how fast the data can move. This is known as the memory bandwidth bottleneck. For a single user, the chip spends most of its time waiting on memory rather than actually computing.

Think of it like a chef in a kitchen. The chef (the chip) can chop vegetables incredibly fast, but if the ingredients (the model weights) are being delivered one by one by a slow assistant, the chef spends most of the time standing idle. In AI inference, the "assistant" is your memory bandwidth.

A useful way to estimate your speed is to look at the theoretical ceiling. Your maximum tokens per second is roughly limited by your memory bandwidth divided by the bytes read per token.

Running LLMs on a CPU

You do not strictly need a massive GPU to start playing with AI. You can run quantized models—versions of models that have been compressed to take up less space—on ordinary CPUs using tools like llama.cpp. This is a great way for builders to test ideas without a heavy hardware investment.

CPU inference is suitable for several specific use cases:

  • Small models that don't require massive intelligence.
  • Background batch jobs where speed isn't the priority.
  • Generating embeddings for a search system.

If you find that a model is too large for your GPU's dedicated video memory (VRAM), you can use llama.cpp to split the work. It can place some layers on your GPU and the rest on your CPU. However, be warned: the CPU half sets the pace. Your entire system will only move as fast as the slowest component, which is usually the CPU.

For more on how to optimize your hardware setup, see our guide on local LLM setup.

How Much RAM Do You Need?

If you are building a system for an office in Mumbai or a startup in Bangalore, you need to know exactly how much RAM to buy. If you buy too little, the model won't run; if you buy just enough, it will be painfully slow.

You need enough system RAM to hold two things: the model weights and the KV cache. The weights are the "knowledge" of the model, and the KV cache is the "short-term memory" used to keep track of the current conversation.

The Weight Rule

A rough rule for calculating the memory needed for weights is: parameters × bytes per parameter. For example, an 8B model (8 billion parameters) needs:

  • 16 GB at FP16 (2 bytes per parameter)
  • 8 GB at 8-bit (1 byte per parameter)

Always leave headroom for the KV cache. A model that "just fits" in your RAM will slow down significantly as the conversation grows and the context fills up. For a 128K context length, the KV cache can actually become larger than the model weights themselves.

To understand the relationship between memory and performance, read our deep dive into VRAM requirements.

The Apple Silicon Exception

Apple's M-series chips (like the M3 or M4) change the rules of the game through something called unified memory. In a traditional PC, the CPU has system RAM and the GPU has its own separate VRAM. In a Mac, the CPU and GPU share a single, massive pool of memory.

This architecture allows Macs to load massive models that would require several expensive consumer GPUs to run on a PC. For example, an M3 Ultra can have up to 512 GB of memory available to the GPU. While the bandwidth on a Mac is lower than a top-tier NVIDIA card, the sheer capacity is a massive advantage for running large models.

If you are a developer looking for a quiet, efficient machine that can handle large models, Apple Silicon is a very strong contender. Tools like Ollama and MLX are specifically designed to take advantage of this unique setup.

NPUs and On-Device AI

You may have seen "AI PCs" or smartphones advertised with "NPUs" (Neural Processing Units). These are small, specialized accelerators built into modern chips designed for low-power, compact AI tasks.

NPUs are not meant to run massive LLMs like GPT-4. Instead, they are designed for "on-device" AI—tasks that need to happen instantly without sending data to the cloud. Common examples include:

  • Transcribing audio in real-time.
  • Summarizing incoming notifications.
  • Smart replies in messaging apps.

These models are typically very small (around 1B to 4B parameters) and are heavily quantized to save power. They share memory with the rest of the system, making them highly efficient for mobile devices and laptops.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

Is a CPU good for AI inference?

Yes, CPUs are perfectly fine for small models, background batch jobs, and embeddings. However, they are significantly slower than GPUs because they lack the massive memory bandwidth required for high-speed token generation.

How much RAM do I need for an LLM?

You need enough to fit the model weights plus the KV cache for your context length. For example, an 8B model needs about 4-5 GB at 4-bit quantization, plus 1-2 GB of overhead and additional space for the KV cache.

What is the role of memory bandwidth in AI?

Memory bandwidth determines how many tokens per second you get during decoding. The chip spends most of its time waiting to read weights from memory, so higher bandwidth equals faster inference.

Can I use my CPU if I run out of VRAM?

Yes, tools like llama.cpp allow you to split model layers between your GPU and CPU. However, the CPU portion will be much slower and will dictate the overall speed of the model.

What is the difference between CPU and GPU cores for AI?

CPUs have a few powerful cores designed for complex logic and branching, whereas GPUs have thousands of simpler cores designed to perform the same mathematical operation on large blocks of numbers in parallel.