The Core Difference: Capacity vs. Speed
To run large language models (LLMs), you need enough fast memory to hold model weights and the KV cache. While RAM is for your system, VRAM is the engine for AI performance.
When building AI systems, the distinction between memory types is the difference between a model that works and a model that is actually useful. For most business owners, the goal is to find the right balance between how much a model "knows" (capacity) and how fast it "speaks" (speed).
VRAM (Video RAM) is high-bandwidth memory soldered next to a GPU, designed for massive parallel math. It is the specialized workspace for the graphics processor. System RAM, on the other hand, is used by the CPU and is significantly slower for AI tasks because it is designed for general-purpose computing rather than the intense, repetitive math required by neural networks.
For LLM inference, VRAM capacity determines if a model fits, while bandwidth determines how many tokens per second you get. If you are building a customer service bot for a small business in India, you need enough VRAM to hold the model, but you need high bandwidth so the customer isn't waiting seconds for every single word to appear on the screen.
| Feature | System RAM | VRAM (GPU Memory) |
|---|---|---|
| Primary User | CPU (Processor) | GPU (Graphics Processor) |
| Primary Role | General system tasks & OS | Massive parallel math (AI/Graphics) |
| Speed (Bandwidth) | Significantly slower | Extremely high speed |
| AI Impact | Determines if you can run a model at all | Determines if the model is fast or slow |
How Much VRAM Do You Need?
Calculating your hardware needs is not guesswork; it is simple arithmetic. The basic rule is: Model size = parameters × bytes per parameter.
The "bytes per parameter" depends on how much you compress the model, a process known as quantization. At FP16 (full precision), each parameter uses 2 bytes. However, most people use 4-bit quantization to save space. At 4-bit quantization, a 7B model needs ~3.5 GB for weights, but you should budget 4-5 GB to account for overhead and scaling factors.
If you are looking at larger models, the requirements grow quickly:
- 3B Model: ~1.5–2 GB (at 4-bit)
- 8B Model: ~4–5 GB (at 4-bit)
- 14B Model: ~7–9 GB (at 4-bit)
- 32B Model: ~16–20 GB (at 4-bit)
- 70B Model: ~35–42 GB (at 4-bit)
Always leave headroom for the KV cache and runtime overhead. A model that "just fits" in your VRAM might work for a single short question, but it will fail or slow down significantly once a long conversation fills the memory.
Why Long Context Costs More Memory
Many beginners make the mistake of only calculating the size of the model weights. They forget that the model also needs "workspace" memory to remember what you just said. This is called the KV cache.
The KV cache stores key and value vectors for every past token so the model doesn't recompute them every time it generates a new word. As context length increases, the KV cache grows linearly. If you are building an application that reads long legal documents or entire chat histories, your memory needs will skyrocket.
For a Llama 3.1 8B model, a 128K context window can require ~16 GB of memory, which is more than the 4-bit weights themselves. If you are serving ten users simultaneously, each with an 8K context, you need to add roughly 10 GiB of extra memory just for the cache.
Why Bandwidth Decides Token Speed
If VRAM capacity is about "can it run?", then bandwidth is about "how fast is it?". During decoding (the process of generating text), the chip spends most of its time waiting on memory to read weights. The math is actually very simple for the chip, so the bottleneck is almost always how fast the data can travel from the memory to the processor.
A useful ceiling for speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token. In the real world, you usually get about 50 to 80 percent of this theoretical ceiling due to various software overheads.
Consider these two examples to see the impact of hardware choice:
- Consumer Speed: An 8B model at 4-bit on an RTX 4090 (1,008 GB/s bandwidth) has a ceiling of roughly 220 tokens per second.
- Large Model Speed: A 70B model at 4-bit on a high-end card with 1,792 GB/s bandwidth has a ceiling of about 45 tokens per second.
This is why high-end consumer cards like the RTX 4090 offer much higher bandwidth than standard system DDR5 RAM, making them much better for local AI development.
The Apple Silicon Exception: Unified Memory
Apple's M-series chips (like the M3 or M4) change the rules slightly through Unified Memory. In a standard PC, you have separate pools of RAM and VRAM. In a Mac, the CPU and GPU share a single pool of memory.
This allows Macs to load massive models that would require multiple expensive NVIDIA GPUs. For example, an M3 Ultra can have up to 512 GB of memory. This makes it a powerhouse for loading huge models that simply won't fit on a single consumer graphics card.
However, there is a trade-off. While capacity is high, the bandwidth is often lower than top-tier NVIDIA cards, resulting in lower tokens per second. A Mac is great for running a massive model that is a bit slow, whereas an NVIDIA setup is better for running a medium model extremely fast.
Comparing Hardware Tiers
When deciding what to buy for your business, you must choose the right tier for your specific use case. Hardware costs vary wildly, so understand what you are paying for.
| GPU Tier | Example Hardware | Best For... |
|---|---|---|
| Consumer | RTX 4090 / 5090 | Best bandwidth per rupee; great for single users or developers. |
| Workstation | RTX PRO 6000 Blackwell | Large VRAM (up to 96GB), ECC memory, and stability for long tasks. |
| Datacenter | NVIDIA H100 / H200 | High-throughput serving, training, and massive multi-user loads. |
Consumer cards give the best value for your money but have limited VRAM. Workstation cards are built for reliability and much higher capacity. Datacenter cards use specialized HBM (High Bandwidth Memory) and are designed to be packed into racks in massive data centers.
Running AI on CPU and System RAM
If you don't have a high-end GPU, don't worry—you can still run AI. Using tools like llama.cpp, you can run quantized models on a CPU using your system's standard RAM.
However, there is a massive performance penalty. System RAM is much slower; dual-channel DDR5 RAM is roughly a tenth the speed of a high-end GPU. If you try to run a 40 GB model (like a 70B model at 4-bit) on a typical desktop, you might only see 2 tokens per second. It is usable for background batch jobs or testing, but it is not suitable for real-time interaction.
When a model is too big for your GPU, software can split the layers between the GPU and the CPU. This allows you to run larger models, but the CPU half will act as a bottleneck, slowing down the entire process to the speed of your system RAM.

Questions people ask
Does more VRAM make AI faster?
Not necessarily. VRAM capacity determines if a model can fit in memory, while VRAM bandwidth determines how fast the model generates text (tokens per second).
Can I use my system RAM for AI?
Yes, using tools like llama.cpp, you can run models on your CPU and system RAM, but it will be significantly slower than using a GPU.
What is the KV cache?
The KV cache is a way for transformers to store information from previous tokens in a conversation so they don't have to be recomputed, but it requires extra memory.
Why is 4-bit quantization popular?
Quantization reduces the number of bits used per parameter, which significantly lowers the VRAM required to run a model with minimal loss in quality.
What is the best way to run a large model?
For a single user, a GPU with the most VRAM you can afford is best. For many users, high-bandwidth datacenter GPUs are required.
