Let's talk

Hardware Guide

How to Set Up Ollama on a GPU Server

Running large language models requires balancing model weights with memory headroom. This guide helps you configure Ollama on GPU hardware for maximum efficiency.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for How to Set Up Ollama on a GPU Server

Quick Setup for Ollama

To set up Ollama on a GPU server, install the Ollama binary, ensure your NVIDIA or AMD drivers are correctly configured, and run the service to offload computations to your GPU's tensor cores. This allows you to run large language models locally via a simple API.

Ollama is a simple way to run open models locally via an API, built on the llama.cpp engine. For a single user or a small team, Ollama is the easiest choice for running models on NVIDIA, AMD, or Apple hardware.

To maximize performance, ensure your GPU drivers are installed so Ollama can offload computations from the CPU to the GPU's tensor cores. Without proper driver configuration, the system may fall back to the CPU, which is significantly slower.

Installation Steps

  1. Install Drivers: Ensure your server has the latest proprietary drivers for your GPU (e.g., NVIDIA CUDA drivers).
  2. Install Ollama: Use the official installation script provided by Ollama for your specific operating system (Linux, macOS, or Windows).
  3. Verify GPU Access: Run a model like ollama run llama3 and monitor your system to ensure the GPU is handling the workload.

Calculating VRAM for Your Model

The primary constraint when setting up a GPU server is VRAM: you must fit the model weights plus the KV cache (context memory) into the available video memory. If the model is too large, the system will offload parts of it to the system RAM, which drastically reduces performance.

A rough rule for weights is: parameters × bytes per parameter. For example, an 8B model needs approximately 16 GB at FP16 or about 4–5 GB at 4-bit quantization. Because real 4-bit files are slightly larger due to scaling factors, it is best to budget roughly 4.5 to 5 bits per weight.

Model Size FP16/BF16 Weights 8-bit Weights 4-bit Weights Typical Single GPU (4-bit)
3B ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB Any 6–8 GB card
8B ≈ 16 GB ≈ 8 GB ≈ 4–5 GB 8–12 GB card
14B ≈ 28 GB ≈ 14 GB ≈ 7–9 GB 12–16 GB card
32B ≈ 64 GB ≈ 32 GB ≈ 16–20 GB 24–32 GB card
70B ≈ 140 GB ≈ 70 GB ≈ 35–42 GB 48 GB or larger

The Role of the KV Cache

While model weights are a fixed cost, the KV cache is a dynamic cost. While generating text, a transformer stores a key and a value vector for every past token in every layer so it does not recompute them. This allows for much faster generation.

The cache grows linearly with context length and the number of users served. For a model like Llama 3.1 8B, a context window of 8,192 tokens requires about 1 GiB of memory. However, a 128K context window can require about 16 GiB, which is more than the 4-bit weights themselves.

Reducing the Memory Footprint

If you are running a server for multiple users, the KV cache can quickly consume your VRAM. There are two main ways to manage this:

  • Grouped-query attention (GQA): This architectural feature lets several query heads share one KV head, making the cache significantly smaller.
  • Quantization: Many runtimes allow you to quantize the KV cache to 8-bit, which can halve the memory footprint with a small cost to quality.

Memory Bandwidth and Token Speed

Once a model fits in VRAM, memory bandwidth determines your tokens per second. For a single user, generating each new token (decoding) means reading every active weight from memory once. The chip spends most of its time waiting on memory, not performing calculations.

A useful ceiling for speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token. Because of software overhead, real-world speed is often 50 to 80 percent of this theoretical ceiling.

For example, an 8B model at 4-bit is about 4.5 GB. On an RTX 4090 with 1,008 GB/s of bandwidth, the ceiling is approximately 220 tokens per second. In contrast, a 70B model at 4-bit is about 40 GB; on a card with 1,792 GB/s, the ceiling is about 45 tokens per second.

It is important to note that reading your initial prompt (prefill) is different. This is a compute-limited task, which is why a long prompt often causes a brief pause before the text begins streaming steadily.

Choosing the Right GPU Tier

Not all GPUs are created equal for AI workloads. When selecting hardware for your server, you must decide whether you need raw speed, massive capacity, or a balance of both. Understanding these tiers is essential for choosing the right GPU for LLMs.

GPU Tier Example Memory Bandwidth Best Use Case
Consumer RTX 4090 1,008 GB/s Up to 32B models at 4-bit; great bandwidth per rupee.
Workstation RTX PRO 6000 1,792 GB/s 70B models; large MoE models; small-team serving.
Datacenter H100 SXM 3.35 TB/s High-throughput serving and training.

Consumer cards like the RTX 4090 offer great bandwidth but have limited VRAM (e.g., 24 GB). Workstation cards provide much higher VRAM and ECC memory, making them suitable for running all day in a server environment. Datacenter cards use HBM (High Bandwidth Memory) to provide the massive bandwidth needed for high-throughput serving and many concurrent users.

Multi-GPU and Scaling

If a model is too large for a single card, you can use multiple GPUs. However, how you connect them matters. There are two primary ways to split a model:

  • Tensor Parallelism: This splits each layer's matrices across GPUs. Every GPU works on every token simultaneously. This requires very fast links like NVLink to be effective.
  • Pipeline Parallelism: This gives each GPU a block of whole layers. Tokens flow from one GPU to the next. This is easier to implement but can leave GPUs idle if you aren't processing many requests at once.

A critical distinction for builders: two cards do not make a single user's experience twice as fast. Multi-GPU setups are primarily used to increase capacity (fitting larger models) or to serve many users at once, rather than making a single chat response faster.

Should You Rent or Buy Hardware?

Deciding whether to invest in your own hardware or use the cloud depends on your expected usage patterns. If you are a startup in India testing different models, renting might be more cost-effective than a large upfront capital expenditure.

Situation Best Choice Why?
Exploring models for a few hours a week Rent No upfront cost; easy to try different GPUs.
Short training or fine-tuning burst Rent Datacenter GPUs Cheaper than buying a high-end GPU for a one-time task.
Bulk generation many hours every day Buy or Lease Fixed costs beat per-hour bills at high usage.
Sensitive data must stay on-premise Buy (On-premise) You maintain full control over data privacy.

A useful rule of thumb: estimate your monthly GPU hours. If the hardware will be busy most of the day for a year or more, owning or leasing a dedicated machine usually wins. If the machine would sit idle, renting is the smarter financial move.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

Will Ollama run faster on a GPU than a CPU?

Yes, because GPUs have thousands of simple cores designed for parallel matrix multiplication, whereas CPUs are designed for varied logic.

What happens if my model is larger than my VRAM?

Runtimes like llama.cpp can split layers between the GPU and CPU, but the CPU portion will significantly slow down the generation speed.

Do I need more RAM or more VRAM for LLMs?

VRAM (on the GPU) is critical for speed due to high bandwidth. System RAM is used if you are running on a CPU or if the model exceeds VRAM capacity.

What is the benefit of quantization for Ollama?

Quantization (e.g., 4-bit) reduces the memory required for weights, allowing larger models to fit into smaller, more affordable GPUs.

Why does a long prompt cause a pause before the text starts streaming?

Reading the initial prompt (prefill) is a compute-limited task, unlike token generation (decoding), which is memory-bandwidth limited.