What is Ollama?
Ollama is a streamlined tool built on llama.cpp designed to make downloading and running open-source models on your own machine simple. It provides an easy way to run large language models locally with minimal setup.
- The easiest way to download and run open models locally.
- Provides a simple API for developers.
- Built on the llama.cpp engine for efficient execution.
Hardware Requirements for Ollama
To run a large language model (LLM), you need enough fast memory to hold the model weights plus the KV cache for your context length. Because of this, your choice of hardware is critical for performance.
Ollama is versatile and can run on various hardware configurations, including CPUs, NVIDIA GPUs, AMD GPUs, and Apple Silicon. However, the specific type of memory you use matters immensely.
VRAM capacity is the primary limit for which models you can load. For best performance, choose hardware with the most VRAM you can afford. If you are looking to build a dedicated AI system, you should understand the different tiers of AI hardware.
| GPU Tier | Memory Bandwidth | Best Use Case |
|---|---|---|
| Consumer (e.g., RTX 4090) | High (e.g., 1,008 GB/s) | Up to 32B models at 4-bit |
| Workstation (e.g., RTX PRO 6000) | High (e.g., 1,792 GB/s) | 70B models, large MoE, small teams |
| Datacenter (e.g., H100) | Very High (e.g., 3.35 TB/s) | High-throughput serving and training |
Calculating Required Memory
Estimating how much memory you need is a matter of simple arithmetic. A rough rule for model weights is: parameters × bytes per parameter. The number of bytes depends on the precision of the model.
Note that real 4-bit files are often slightly larger because they store scaling factors or keep some layers at higher precision.
When planning your budget, remember to account for 1 to 2 GB of runtime overhead for the framework and activations, in addition to the weights and the KV cache.
VRAM vs. System RAM
Not all memory is created equal. GPUs use VRAM (GDDR or HBM) which offers much higher bandwidth than system RAM. This high bandwidth is critical because the chip spends most of its time waiting on memory during inference.
If a model doesn't fit in VRAM, llama.cpp can split layers between the GPU and CPU. While this allows you to run larger models on cheaper hardware, the CPU half will slow down the process significantly. This is because dual-channel DDR5-5600 system RAM has a theoretical peak of roughly 90 GB/s, which is about a tenth of a high-end GPU's speed.
For a business owner in India setting up a local workstation, if you want to run a 70B model at 4-bit, you would need a card with 48 GB or larger, or a setup with two GPUs. Relying on system RAM will lead to very slow performance, often topping out at just a few tokens per second.
Running Ollama on Apple Silicon
Apple's M-series chips use unified memory, which shares a single pool between the CPU and GPU. This architecture is a significant advantage for running large models on a desktop.
This allows Macs to load massive models that would require several consumer GPUs in a traditional PC setup. For instance, an M3 Ultra can have up to 512 GB of memory. While the bandwidth is lower than a top-tier NVIDIA card, the sheer capacity makes it a powerhouse for local LLMs. Ollama and MLX run very well on Apple hardware.
What Determines Inference Speed?
When you are running a model, you are likely interested in "tokens per second"—how fast the text appears on your screen. Memory bandwidth determines how many tokens per second you get. A useful ceiling for speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token.
For example, an 8B model at 4-bit is about 4.5 GB. On an RTX 4090 with 1,008 GB/s bandwidth, the theoretical ceiling is about 220 tokens per second. In real-world use, you can expect 50 to 80 percent of that ceiling due to various overheads.
Mixture-of-Experts (MoE) models can offer faster decoding. Even though they might require a lot of memory to hold all their weights, they only activate a subset of parameters per token. This means they can decode at a speed closer to a much smaller model.
When to Use Ollama vs. Other Tools
Choosing the right software depends on your specific use case and how many people will be using the model at once.
| Use Case | Recommended Tool |
|---|---|
| Single user or a small group of developers | Ollama or llama.cpp |
| Team or application needing many concurrent requests | vLLM or SGLang |
| High-throughput NVIDIA datacenter serving | TensorRT-LLM |
Ollama is ideal for simplicity and ease of use. If you are just exploring models or need a quick way to get an API running on your laptop, Ollama is the best starting point. However, if you are building a production-grade service for a large team, you will likely need a high-throughput server like vLLM.
Frequently Asked Questions
- Can I run Ollama on a CPU?
- Yes, Ollama can run on CPUs using llama.cpp, but performance is limited by the much lower bandwidth of system RAM compared to GPU VRAM.
- Why is VRAM more important than raw compute for LLMs?
- During inference, the chip spends most of its time waiting on memory to read weights. Therefore, memory bandwidth often dictates token speed more than raw compute power.
- What is the KV cache and why does it matter?
- The KV cache stores past tokens so they don't need to be recomputed. It grows with your context length, meaning longer conversations require more memory.
- Will a model run if it "just fits" in my VRAM?
- It may fail or slow down significantly once the conversation grows and the KV cache requires additional memory headroom.
- What is quantization?
- Quantization is the process of storing weights in lower precision (like 4-bit or 8-bit) to reduce the memory required for a model.

Questions people ask
Can I run Ollama on a CPU?
Yes, Ollama can run on CPUs using llama.cpp, but performance is limited by the much lower bandwidth of system RAM compared to GPU VRAM.
Why is VRAM more important than raw compute for LLMs?
During inference, the chip spends most of its time waiting on memory to read weights. Therefore, memory bandwidth often dictates token speed more than raw compute power.
What is the KV cache and why does it matter?
The KV cache stores past tokens so they don't need to be recomputed. It grows with your context length, meaning longer conversations require more memory.
Will a model run if it 'just fits' in my VRAM?
It may fail or slow down significantly once the conversation grows and the KV cache requires additional memory headroom.
What is quantization?
Quantization is the process of storing weights in lower precision (like 4-bit or 8-bit) to reduce the memory required for a model.
