Let's talk

Scaling AI Hardware

Multi GPU Setup for LLM: Scaling Capacity and Speed

Running large language models often requires more memory than a single card can provide. Learn how multi-GPU configurations help you fit larger models and serve more users.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for Multi GPU Setup for LLM: Scaling Capacity and Speed

Why Use a Multi-GPU Setup?

Multi-GPU setups are used to increase the total VRAM available to a system, allowing you to run massive models that cannot fit on a single card. While they significantly increase capacity and the number of users you can serve, they do not necessarily double the speed for a single user.

For many builders, the primary limitation in running Large Language Models (LLMs) is not raw computing power, but memory capacity. If you want to run a 70B parameter model, you need roughly 35–42 GB of VRAM just for the weights at 4-bit precision. Since most high-end consumer cards only offer 24 GB, a single card simply cannot hold the model.

Adding a second or third GPU solves this by pooling the memory. This allows you to:

  • Fit larger models: You can load models that exceed the capacity of a single workstation card.
  • Increase throughput: By having more memory, you can store a larger KV cache, which allows you to serve more concurrent users or handle much longer conversations.
  • Handle complex workflows: As seen in professional setups, you might run one model for drafting and a second, larger model for quality checks simultaneously.

Splitting Models: Parallelism Methods

When you use multiple GPUs, the software must decide how to split the model's workload across them. There are two main ways to do this: Tensor Parallelism and Pipeline Parallelism.

Tensor Parallelism

In tensor parallelism, each individual layer's mathematical matrices are split across the GPUs. Because every GPU needs to work together on every single token generated, they must communicate constantly. This requires extremely fast interconnects, such as NVIDIA's NVLink, to prevent the communication overhead from slowing everything down. This method works best inside a single, high-performance server.

Pipeline Parallelism

Pipeline parallelism is a different approach where you give each GPU a whole block of layers. This requires much less communication between cards, making it easier to implement on standard hardware. However, it can lead to "idle" time where one GPU is waiting for the other to finish its work, unless you are managing many different requests at once to keep the "pipeline" full.

Calculating Required VRAM for Multiple GPUs

To build a reliable system, you cannot just match the model size to your total VRAM. You must account for several different types of memory usage. If you design a system that "just fits" the model, it will likely fail or become incredibly slow as soon as you start a conversation.

The total VRAM you need is calculated as follows:

  1. Model Weights: This is the base size of the model. A simple rule of thumb is parameters × bytes per parameter. For a 70B model at 4-bit precision, this is roughly 35–42 GB.
  2. KV Cache: This is the memory used to store the "context" (the history of the conversation). This grows linearly with every new token and every new user. For a Llama 3.1 8B model, a 128K context window can require about 16 GiB of memory.
  3. Runtime Overhead: Always leave 1–2 GB of extra space for the operating system, the AI framework, and activations.
  4. For a professional deployment, you should always aim for 20% to 30% more VRAM than your minimum requirement to ensure stability during long interactions.

    Model Size 4-bit Weights (Approx) Recommended VRAM (with headroom)
    8B Model 4–5 GB 8–12 GB Card
    32B Model 16–20 GB 24–32 GB Card
    70B Model 35–42 GB 48 GB or larger (or 2x GPUs)

How Bandwidth Affects Multi-GPU Performance

While VRAM capacity determines if a model can *run*, memory bandwidth determines how *fast* it generates text. For a single user, the bottleneck during the "decoding" phase (when the model is streaming words to you) is almost always memory bandwidth, not the raw compute power of the chip.

Every time the model generates a new token, it must read every single active weight from the VRAM. Therefore, the theoretical limit of your speed is roughly: Memory Bandwidth ÷ Bytes read per token.

For example, an 8B model at 4-bit precision on a high-end consumer card might reach a ceiling of 220 tokens per second. A much larger 70B model will be significantly slower because it has more data to read for every single word it produces.

It is also important to distinguish between Prefill and Decoding. Reading your initial long prompt (prefill) is a compute-heavy task. Generating the subsequent words (decoding) is a bandwidth-heavy task. This is why you often see a pause before a long prompt is processed, followed by a steady stream of text.

Choosing the Right Hardware Tier

When building your setup, you must decide which tier of hardware fits your business model. Your choice depends on whether you prioritize cost-per-token, memory capacity, or reliability.

GPU Tier Example Hardware Primary Advantage
Consumer RTX 4090 / 5090 Best bandwidth per rupee; high speed for small models.
Workstation RTX PRO 6000 Blackwell Massive VRAM (up to 96 GB) and ECC memory for stability.
Datacenter H100 / H200 Extreme throughput using HBM memory and NVLink.
  • Consumer GPUs: Excellent for developers and small-scale testing. They offer the best speed for the price but are limited by their 24 GB or 32 GB VRAM.
  • Workstation GPUs: Ideal for businesses running dedicated local servers. They provide much more memory (e.g., 96 GB) and Error Correction Code (ECC) memory, which is vital for long-running tasks.
  • Datacenter GPUs: These are designed for massive scale and high-throughput serving. They use HBM (High Bandwidth Memory) and are built for rack-mounted environments.

If you are running out of VRAM on a standard PC, you can use tools like llama.cpp to split the model between your GPU and your system RAM (CPU). However, because system RAM is much slower, the CPU portion will significantly slow down your generation speed.

Power and Cooling Requirements

Running multiple high-end GPUs is a significant physical undertaking. A single high-end card can draw up to 600 W of power.

For business owners, this means you cannot simply plug an AI server into a standard wall outlet without checking your circuit capacity. You must plan for:

  • Robust Power Supplies: You will need high-wattage, high-efficiency power supply units (PSUs).
  • Thermal Management: High-end GPUs are significant heaters. You need a dedicated, cool room or advanced server-grade cooling to prevent "thermal throttling," which happens when the hardware slows itself down to avoid melting.
  • Power Stability: In regions with inconsistent power, a large Uninterruptible Power Supply (UPS) or inverter is essential to protect your expensive hardware from sudden outages.

If you are unsure whether to buy this hardware or rent it, a simple rule is to estimate your monthly usage. If your hardware will be busy most of the day for a year or more, owning the machine is usually more cost-effective. If it sits idle frequently, renting cloud GPUs is the smarter choice.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

Do two GPUs make LLM inference twice as fast?

No. For a single user, two GPUs primarily allow you to fit larger models or provide more memory for more users, rather than doubling token speed.

What is the difference between Tensor and Pipeline parallelism?

Tensor parallelism splits matrix multiplications across GPUs and requires fast links like NVLink. Pipeline parallelism assigns whole layers to different GPUs, requiring less communication.

How does the KV cache affect my GPU memory needs?

The KV cache grows linearly with context length and number of users. For a Llama 3.1 8B model, a 128K context can require about 16 GiB of memory.

Can I use a CPU if I run out of VRAM?

Yes, tools like llama.cpp can split layers between the GPU and CPU, but the CPU portion will significantly slow down the inference speed.

What is the best way to serve many users at once?

For high-throughput serving with many concurrent users, use frameworks like vLLM or SGLang which manage KV caches and batching efficiently.