Let's talk

Architecting AI

Hybrid Cloud and Local AI: Balancing Compute and Control

Optimizing AI workflows requires knowing when to use local hardware for privacy and speed, and when to scale to the cloud. This guide explains the hardware math behind that decision.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for Hybrid Cloud and Local AI: Balancing Compute and Control

What is Hybrid AI Architecture?

Hybrid AI involves routing workloads between local hardware and cloud services. By using local hardware for sensitive data and bulk tasks, and the cloud for high-end reasoning, you can optimize both cost and performance.

Optimizing AI workflows requires knowing when to use local hardware for privacy and speed, and when to scale to the cloud. This guide explains the hardware math behind that decision.

  • Hybrid AI involves routing workloads between local hardware and cloud services.
  • Local AI is best for sensitive data, bulk repeatable tasks, and low-latency single-user needs.
  • Cloud/API is best for frontier-model quality and handling spiky public traffic.
  • The goal is to route bulk work to hardware you control to avoid unpredictable API costs.

The Golden Rule: Memory First, Bandwidth Second

To run a large language model (LLM), you need enough fast memory to hold the model weights plus the KV cache for your context length. A rough rule of thumb is: parameters × bytes per parameter.

Once the model fits in memory, memory bandwidth determines your tokens per second. This is why GPUs are so effective; GPU VRAM is superior to CPU RAM because it is soldered next to the processor on a wide bus, providing much higher bandwidth.

Calculating Your Local VRAM Requirements

Model weights represent a fixed cost in your memory budget. However, you cannot simply match your VRAM to the model size; you must account for overhead and context.

The Weight Math

The amount of memory required depends on the precision (quantization) you use:

  • FP16/BF16: 2 bytes per parameter.
  • 8-bit: 1 byte per parameter.
Model Size FP16 Weights 8-bit Weights 4-bit Weights Typical GPU (4-bit)
3B ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB Any 6–8 GB card
8B ≈ 16 GB ≈ 8 GB ≈ 4–5 GB 8–12 GB card
14B ≈ 28 GB ≈ 14 GB ≈ 7–9 GB 12–16 GB card
32B ≈ 64 GB ≈ 32 GB ≈ 16–20 GB 24–32 GB card
70B ≈ 140 GB ≈ 70 GB ≈ 35–42 GB 48 GB or larger

Important: Always budget 1-2 GB of overhead for framework activations and runtime. Additionally, you must leave headroom for the KV cache. The KV cache stores past tokens so the model doesn't have to recompute them, and it grows linearly with context length and the number of users served.

A model that "just fits" the weights will fail or slow down significantly once a long conversation fills the KV cache. For example, at a 128K context length, the KV cache for an 8B model can require roughly 16 GiB of memory—more than the weights themselves.

Choosing the Right Hardware Tier

Hardware selection depends on whether you need massive capacity, extreme speed, or a balance of both. Understanding AI hardware tiers is essential for builders.

GPU Tier Example Memory (Approx) Best For
Consumer RTX 4090 24 GB GDDR6X Up to 32B models at 4-bit; best bandwidth per rupee.
Workstation RTX PRO 6000 96 GB GDDR7 70B models; large MoE; small-team serving.
Datacenter H100 SXM 80 GB HBM3 High-throughput serving and training.

Specialized Hardware

  • Apple Silicon: Uses unified memory, allowing large models (up to 512 GB on M3 Ultra) to load. While capacity is high, bandwidth is lower than top NVIDIA cards.
  • CPUs: Viable for small models or background batch jobs, but bandwidth is often 1/10th of a GPU, making them much slower.
  • NPUs: Found in phones and laptops; designed for low-power, compact models (1B to 4B parameters) for tasks like transcription.

Estimating Inference Speed

For a single user, generating text (decoding) is a memory-bound task. A useful ceiling for speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token.

For example, an 8B model at 4-bit on an RTX 4090 (1,008 GB/s) has a theoretical ceiling of about 220 tokens per second. A 70B model on a high-end card might reach about 45 tokens per second.

Mixture-of-Experts (MoE)

MoE models, like Google's Gemma 4 26B, offer a unique advantage. They require enough memory to hold all weights, but they only activate a subset of parameters per token. This allows them to decode at a speed closer to a much smaller model while maintaining the capability of a larger one.

If you are building applications for many concurrent users, you should move beyond simple setups and use high-throughput serving software like vLLM or SGLang.

The Decision: Renting vs. Buying Hardware

Deciding whether to invest in on-device AI or use the cloud is a matter of economics and data privacy.

Situation Better Choice Reasoning
Exploring models / Short bursts Rent No upfront cost; easy to try different GPUs.
Bulk generation / High daily use Buy or Lease Fixed costs beat per-token bills at high use.
Sensitive data requirements Buy (On-premise) You maintain total control over where data goes.
Spiky public traffic Rent or API Scale up only when demand arrives.

The Simple Test: Estimate your monthly GPU hours. If the hardware would be busy most of the day for a year or more, owning or leasing a dedicated machine usually wins. If it would sit idle, rent.

Practical Implementation for Builders

To successfully implement a hybrid architecture, follow these practical rules of thumb:

  1. Choose the right software: Use Ollama or llama.cpp for a single user or simple local testing. Use vLLM or SGLang when building an application for a team or many concurrent requests.
  2. Plan for headroom: Always choose hardware with 20% to 30% more VRAM than your calculated needs (weights + KV cache + overhead).
  3. Route intelligently: Use frontier-model APIs for complex reasoning tasks, but route bulk, repeatable work (like drafting hundreds of pages) to hardware you control.

By combining local-first AI strategies with cloud scaling, you can build systems that are both cost-effective and highly capable.

Frequently Asked Questions

How much VRAM do I need for a 7B model?
At 4-bit quantization, a 7B model needs about 3.5 GB for weights, plus 1-2 GB for overhead and additional space for the KV cache.
Why is memory bandwidth more important than raw compute for LLMs?
During decoding, the chip spends most of its time waiting to read weights from memory, making bandwidth the primary bottleneck for token speed.
What is the KV cache and why does it matter?
The KV cache stores past tokens so the model doesn't have to recompute them. It grows with context length, meaning longer conversations require more memory.
Can I run large models on a CPU?
Yes, using tools like llama.cpp, but speed is limited by system RAM bandwidth, which is significantly slower than GPU VRAM.
When should I use a Mixture-of-Experts (MoE) model?
MoE models are useful when you want the speed of a small model but the capability of a larger one, as they only activate a subset of parameters per token.
Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

How much VRAM do I need for a 7B model?

At 4-bit quantization, a 7B model needs about 3.5 GB for weights, plus 1-2 GB for overhead and additional space for the KV cache.

Why is memory bandwidth more important than raw compute for LLMs?

During decoding, the chip spends most of its time waiting to read weights from memory, making bandwidth the primary bottleneck for token speed.

What is the KV cache and why does it matter?

The KV cache stores past tokens so the model doesn't have to recompute them. It grows with context length, meaning longer conversations require more memory.

Can I run large models on a CPU?

Yes, using tools like llama.cpp, but speed is limited by system RAM bandwidth, which is significantly slower than GPU VRAM.

When should I use a Mixture-of-Experts (MoE) model?

MoE models are useful when you want the speed of a small model but the capability of a larger one, as they only activate a subset of parameters per token.