Let's talk

Hardware Architecture

Unified Memory: The Key to Running Large Language Models

In AI hardware, memory architecture determines whether a model fits on your device and how fast it generates text.

By Divyaprakash JhaUpdated 6 min readTopic: AI hardware

Photo for Unified Memory: The Key to Running Large Language Models

What is Unified Memory?

Unified memory is a shared pool of high-speed memory that allows the CPU and GPU to access the same data directly without copying it between different chips. This architecture is critical for running large language models locally because it allows the GPU to utilize much larger amounts of memory than standard dedicated video cards.

In traditional computer architectures, you have two separate pools of memory: the system RAM used by the CPU and the dedicated VRAM used by the GPU. When you want to run an AI model, the data often has to be moved back and forth between these two pools, creating a bottleneck.

Unified memory changes this dynamic. In architectures like Apple Silicon, the CPU, GPU, and other components are integrated into a single package. They all share one single pool of resources. This means the GPU can access the same memory used by the CPU instantly. For a business owner looking to run heavy AI workloads, this means you can potentially load a massive model that would otherwise require multiple expensive, power-hungry professional graphics cards.

Capacity: Why It Matters for LLMs

When you are choosing hardware for AI, the most important question is: "Will the model fit?" To run a Large Language Model (LLM), you must fit two main things into your available memory: the model weights and the KV cache (which is the memory used for the conversation context).

The model weights are the "brain" of the AI. If you are using a 4-bit quantized model, the memory requirement is roughly half a byte per parameter. If you are using a 70B model, you are looking at roughly 35 to 42 GB just for the weights themselves. If your hardware doesn't have enough memory to hold those weights, the model simply won't run, or it will run incredibly slowly by offloading to your system RAM.

This is where unified memory shines for local setups. While a standard consumer graphics card might only have 8 GB or 12 GB of VRAM, a high-end Mac with an M3 Ultra can have up to 512 GB of unified memory. This massive capacity allows you to load massive models that would require a server rack of specialized hardware to run on traditional PC setups.

Model Size 4-bit Weights (Approx) Suggested Hardware (VRAM/Unified)
3B Model 1.5–2 GB Any 6–8 GB card or laptop
8B Model 4–5 GB 8–12 GB card
14B Model 7–9 GB 12–16 GB card
32B Model 16–20 GB 24–32 GB card
70B Model 35–42 GB 48 GB or larger (or two GPUs)

The Bandwidth vs. Capacity Tradeoff

In the world of AI hardware, there is a constant tug-of-war between capacity (how much you can hold) and bandwidth (how fast you can move it). Unified memory is the king of capacity, but it often loses the race on bandwidth when compared to dedicated high-end GPUs.

Memory bandwidth determines your "tokens per second"—essentially how fast the AI types its response. A high-end NVIDIA card, like the RTX 4090, has incredibly fast dedicated VRAM. This allows it to generate text very quickly because it can read the model weights at lightning speeds.

Apple Silicon's unified memory offers a different value proposition. While its bandwidth is generally lower than a top-tier NVIDIA card, it provides a massive amount of memory that is much more efficient and quiet. For a developer building an application, this means you can run a very large, highly intelligent model locally on a laptop or a workstation without the massive power draw and heat of a dedicated AI server.

If your goal is raw speed for a single user, a dedicated GPU with high bandwidth is the winner. If your goal is to run a massive model that simply won't fit on a standard card, unified memory is your best friend.

The Role of the KV Cache

A common mistake builders make is calculating memory needs based only on the model size. If you buy a machine that has exactly enough memory for a 70B model, you will likely find that the model crashes or slows to a crawl as soon as you start a long conversation.

This happens because of the KV cache. As the AI generates text, it stores "key" and "value" vectors for every past token in the conversation so it doesn't have to recompute the entire history every time you ask a new question. This cache grows linearly with the length of the conversation (the context length).

In this scenario, the "context" actually requires more memory than the "brain" itself. This is why choosing hardware with extra headroom is vital for any real-world application.

Unified Memory vs. Standard System RAM

It is important to distinguish between "Unified Memory" (like in Apple's M-series) and "Standard System RAM" (like DDR5 in a typical Windows PC). While both can be used to run AI, they perform very differently.

When you run a model on a CPU using standard system RAM, you are limited by the speed at which the CPU can talk to that RAM. For example, dual-channel DDR5-5600 memory provides roughly 90 GB/s of bandwidth. Compare that to a high-end GPU, which can be ten times faster. Because of this, running a large model on a CPU is often painfully slow—you might only get 2 tokens per second.

Modern software like llama.cpp allows you to "split" a model. If a model is too big for your GPU's VRAM, the software can put some layers on the GPU and the rest on the CPU. This allows you to run larger models, but the CPU portion acts as a bottleneck, slowing down the entire process to the speed of the slowest component.

NPUs and Mobile Architectures

In the mobile world—your smartphone or a lightweight laptop—you will encounter NPUs (Neural Processing Units). These are specialized, low-power chips designed specifically for AI tasks like face recognition or voice transcription.

NPUs typically use the same shared system memory as the rest of the device. Because mobile devices have limited total memory and must save power to preserve battery life, the models running on NPUs are usually very small. You will typically see models with 1 billion to 4 billion parameters. These are "highly quantized," meaning they have been compressed significantly to fit into the tiny memory footprint required for on-device AI features.

Frequently Asked Questions

Does unified memory make AI faster?
Not necessarily. While it allows you to run much larger models by providing more capacity, dedicated GPU VRAM usually offers higher bandwidth, which results in faster token generation speeds.
Can I run a 70B model on a Mac?
Yes, if the Mac has enough unified memory. A 70B model at 4-bit requires roughly 35-42 GB of memory, so a Mac with 64 GB or more would be suitable.
Why does context length increase memory needs?
As you chat, the system stores 'key' and 'value' vectors for every past token in the KV cache so it doesn't have to recompute them, which consumes more memory as the conversation grows.
What is the difference between VRAM and Unified Memory?
VRAM is dedicated memory physically located next to a GPU, while unified memory is a single pool shared between the CPU and GPU, allowing for much larger model capacities on certain devices.
How much headroom should I leave for memory?
You should choose hardware with 20 to 30 percent more memory than the total required for the model weights and the expected KV cache.
Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

Does unified memory make AI faster?

Not necessarily. While it allows you to run much larger models by providing more capacity, dedicated GPU VRAM usually offers higher bandwidth, which results in faster token generation speeds.

Can I run a 70B model on a Mac?

Yes, if the Mac has enough unified memory. A 70B model at 4-bit requires roughly 35-42 GB of memory, so a Mac with 64 GB or more would be suitable.

Why does context length increase memory needs?

As you chat, the system stores 'key' and 'value' vectors for every past token in the KV cache so it doesn't have to recompute them, which consumes more memory as the conversation grows.

What is the difference between VRAM and Unified Memory?

VRAM is dedicated memory physically located next to a GPU, while unified memory is a single pool shared between the CPU and GPU, allowing for much larger model capacities on certain devices.

How much headroom should I leave for memory?

You should choose hardware with 20 to 30 percent more memory than the total required for the model weights and the expected KV cache.