The Quick Answer
Choosing between Ollama, llama.cpp, and vLLM depends on whether you are a single user running models locally or a business serving many concurrent requests. Use Ollama or llama.cpp for local experimentation, and vLLM for high-throughput production environments.
- Use Ollama or llama.cpp if you are one person or a small team running models locally.
- Use vLLM or SGLang when you have an application or a team sending many concurrent requests at once.
- Ollama is built on top of llama.cpp and is designed for ease of use.
llama.cpp: The Versatile Engine
At its core, llama.cpp is a lean C/C++ engine designed specifically to run quantized GGUF models. It is the foundational technology that makes running large language models on consumer hardware possible.
It is highly flexible and can run on a wide variety of hardware, including CPUs, NVIDIA GPUs, AMD GPUs, and Apple Silicon. This versatility makes it a go-to tool for developers who need to squeeze performance out of whatever hardware is available.
One of its most powerful features is its ability to split layers between a GPU and a CPU. If you are trying to run a large model that is too big for your available VRAM, llama.cpp can offload parts of the model to your system RAM. While the CPU half will set the pace and slow down generation, it allows you to run models that would otherwise be impossible to load.
Ollama: The Easiest Local Setup
If you are a developer who wants to get up and running quickly without getting bogged down in complex configuration files, Ollama is the solution. It is widely considered the simplest way to download and run open models locally.
Ollama provides a built-in API that makes it incredibly easy to integrate local models into your existing software workflows. It handles the heavy lifting of downloading models and setting up the environment, so you can focus on building your application.
Under the hood, Ollama leverages llama.cpp for its inference capabilities. This means you get the ease of use of a modern tool with the proven, efficient engine of llama.cpp powering the actual math.
vLLM: High-Throughput Serving
When you move from local experimentation to a production environment, your requirements change. You no longer care just about how fast one person can chat; you care about how many users your server can handle at once. This is where vLLM shines.
vLLM is designed specifically for high-performance production serving. It uses advanced techniques like paged KV-cache management and continuous batching to handle many concurrent requests efficiently. Instead of making users wait in a queue, vLLM manages memory dynamically to keep the "pipeline" of text generation full.
While it is more complex to set up than Ollama, it is optimized for throughput. For a business owner building a customer-facing AI agent, vLLM is often the better choice for a scalable backend.
How Hardware Affects Your Choice
The hardware you choose will dictate the speed and scale of your AI implementation. When it comes to LLM inference, memory bandwidth is often the most critical factor. It determines your "tokens per second"—the actual speed at which text appears on the screen.
High-end GPUs like the NVIDIA RTX 4090 or the H100 provide the best speed because they have massive bandwidth. For example, an 8B model at 4-bit quantization on an RTX 4090 can reach a theoretical ceiling of about 220 tokens per second, whereas a CPU will be significantly slower.
For local users, Apple Silicon offers a unique advantage through its unified memory architecture. Because the CPU and GPU share the same pool of memory, a Mac with a large memory configuration can load massive models that would require multiple expensive consumer GPUs to run. However, keep in mind that while the capacity is high, the bandwidth is lower than a top-tier NVIDIA card.
If you are forced to run on a CPU using system RAM, llama.cpp is the industry standard. However, expect much slower speeds. A typical desktop with dual-channel DDR5-5600 memory has a peak of roughly 90 GB/s, which is about a tenth of a high-end GPU.
| GPU Tier | Memory Bandwidth | Best Use Case |
|---|---|---|
| Consumer (e.g., RTX 4090) | High (e.g., 1,008 GB/s) | Local development, single-user speed |
| Workstation (e.g., RTX PRO 6000) | Very High (e.g., 1,792 GB/s) | Large models (70B+), small-team serving |
| Datacenter (e.g., H100) | Extreme (e.g., 3.35 TB/s) | High-throughput production, many users |
Calculating Your Memory Needs
Before buying hardware or renting cloud GPUs, you must understand the math of LLM memory. A common mistake is only looking at the model size and forgetting the "hidden" costs of running the model.
The total memory you need is: Model Weights + KV Cache + Runtime Overhead.
First, calculate the weights. A general rule is: parameters × bytes per parameter. For an 8B parameter model:
- At FP16 (2 bytes per param): ~16 GB
- At 8-bit (1 byte per param): ~8 GB
Second, you must account for the KV cache. As you chat, the model stores past tokens in a "key and value" cache so it doesn't have to recompute the whole conversation every time you send a new message. This cache grows linearly with the context length. For a Llama 3.1 8B model, a conversation with 8,192 tokens requires about 1 GiB of extra memory. If you increase that to a 128K context, the cache alone could require 16 GiB—more than the model weights themselves!
Finally, add 1 to 2 GB for runtime overhead. If you are building a service for many users, you must reserve significant extra VRAM to handle their simultaneous conversation histories.
Frequently Asked Questions
- When should I use vLLM instead of Ollama?
- Use vLLM when you are building a production application that needs to serve many concurrent users efficiently. Ollama is better for individual experimentation and local development.
- Can I run LLMs on my CPU?
- Yes, llama.cpp allows you to run quantized models on ordinary CPUs using system RAM, though it is significantly slower than using a GPU due to lower memory bandwidth.
- What is the difference between Ollama and llama.cpp?
- Ollama is a user-friendly wrapper built on top of llama.cpp. It simplifies the process of downloading and running models through a simple API and interface.
- Does quantization affect performance?
- Yes, quantization (like 4-bit or 8-bit) reduces the memory required to hold model weights, which allows larger models to fit on smaller hardware, but it changes the precision of the math.
- Why does context length matter for VRAM?
- As you chat, the KV cache stores past tokens to avoid recomputing them. This cache grows with every token, meaning long conversations require more memory than just the model weights themselves.

Questions people ask
When should I use vLLM instead of Ollama?
Use vLLM when you are building a production application that needs to serve many concurrent users efficiently. Ollama is better for individual experimentation and local development.
Can I run LLMs on my CPU?
Yes, llama.cpp allows you to run quantized models on ordinary CPUs using system RAM, though it is significantly slower than using a GPU due to lower memory bandwidth.
What is the difference between Ollama and llama.cpp?
Ollama is a user-friendly wrapper built on top of llama.cpp. It simplifies the process of downloading and running models through a simple API and interface.
Does quantization affect performance?
Yes, quantization (like 4-bit or 8-bit) reduces the memory required to hold model weights, which allows larger models to fit on smaller hardware, but it changes the precision of the math.
Why does context length matter for VRAM?
As you chat, the KV cache stores past tokens to avoid recomputing them. This cache grows with every token, meaning long conversations require more memory than just the model weights themselves.
