Let's talk

Hardware Strategy

Rent GPU vs Buy GPU for AI: A Guide for Builders

Choosing between cloud rental and on-premise hardware depends on your model size, context needs, and how many hours a day you run workloads.

By Divyaprakash JhaUpdated 4 min readTopic: AI hardware

Photo for Rent GPU vs Buy GPU for AI: A Guide for Builders

The Quick Decision: When to Rent vs. Buy

Deciding between renting or buying AI hardware depends on your model size, context needs, and how many hours a day you run workloads. If your hardware sits idle most of the time, renting is more cost-effective; if you run bulk generation for many hours every day, owning a dedicated server is usually the winner.

Situation Better Choice
Exploring models for a few hours a week or needing a short burst for training/fine-tuning Rent
Performing bulk generation many hours every day Buy or lease a dedicated server
Spiky public traffic where you want to scale only when demand arrives Rent or use APIs
Sensitive data must stay on your machine Buy (on-premise)

Calculating Your VRAM Needs

When building AI systems, memory is your first priority; bandwidth is second. To run a large language model (LLM), you need enough fast memory to hold the model weights plus the KV cache for your context length. A rough rule of thumb is: parameters × bytes per parameter.

For example, an 8B model requires approximately 16 GB at FP16, 8 GB at 8-bit, and 4 to 5 GB at 4-bit. Because real 4-bit files are slightly larger to accommodate scaling factors and higher precision layers, you should budget roughly 4.5 to 5 bits per weight.

The Importance of Headroom

You cannot simply match your VRAM to the model size. You must always leave headroom for the KV cache, which grows with every token of context and the number of users served at once. A model that "just fits" the weights will slow down or fail entirely once a long conversation fills the KV cache.

Model Size FP16 / BF16 Weights 8-bit Weights 4-bit Weights Typical Single GPU (4-bit)
3B ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB Any 6–8 GB card
8B ≈ 16 GB ≈ 8 GB ≈ 4–5 GB 8–12 GB card
32B ≈ 64 GB ≈ 32 GB ≈ 16–20 GB 24–32 GB card
70B ≈ 140 GB ≈ 70 GB ≈ 35–42 GB 48 GB or larger

Understanding Speed and Bandwidth

Once the model fits in memory, memory bandwidth determines your tokens per second. During the generation (decoding) phase, the chip often spends most of its time waiting on memory. A useful ceiling to estimate speed is: max tokens per second ≈ memory bandwidth ÷ bytes read per token.

For instance, an 8B model at 4-bit quantization on an RTX 4090 (1,008 GB/s) has a theoretical ceiling of about 220 tokens per second. In real-world use, you should expect 50 to 80 percent of that ceiling due to various overheads.

Mixture-of-Experts (MoE) and Speed

Models like Gemma 4 26B use a Mixture-of-Experts (MoE) architecture. These models trade memory capacity for speed. While the entire model must sit in VRAM, each token only activates a subset of parameters. This means a model like Gemma 4 26B can have the memory footprint of a 26B model but decode at a speed closer to a much smaller 4B model.

Comparing GPU Tiers

Hardware for AI generally falls into three distinct tiers based on your budget and scale. Understanding these helps you decide whether to invest in local ai-hardware or rent high-end compute.

  • Consumer (e.g., NVIDIA RTX 4090 / 5090): These offer the best bandwidth per rupee/dollar but have limited VRAM. They are excellent for up to 32B models at 4-bit quantization.
  • Workstation (e.g., RTX PRO 6000 Blackwell): These provide much more VRAM and ECC memory. They are designed for 70B models, large MoE models, and serving small teams.
  • Datacenter (e.g., H100 / H200): These use HBM (High Bandwidth Memory) to provide massive throughput. They are built for high-scale serving, massive context lengths, and training.

Apple Silicon, CPUs, and NPUs

Not all AI work happens on dedicated NVIDIA cards. Depending on your use case, other hardware options might be more accessible or efficient.

Apple Silicon and Unified Memory

Apple's M-series chips use unified memory, meaning the CPU and GPU share the same pool. A Mac with a large memory configuration can load massive models that would require multiple consumer GPUs. While the bandwidth is lower than a top-tier NVIDIA card, the capacity is exceptionally high and the system is very efficient. Tools like local-llm runtimes work very well on this architecture.

CPUs and NPUs

You can run quantized models on a standard CPU using tools like llama.cpp, but be prepared for slow speeds. A typical desktop with DDR5 memory has a bandwidth roughly 1/10th that of a high-end GPU. This makes CPU inference suitable for small models, background batch jobs, or embeddings, but not for interactive high-speed chat.

NPUs (Neural Processing Units) are small accelerators found in phones and modern laptops. They are designed for low-power, small-model tasks (roughly 1B to 4B parameters) like transcription or smart replies.

Power, Cooling, and Infrastructure

If you choose to buy hardware, remember that a high-end GPU is essentially a heater. A single workstation under full load can draw close to 1 kilowatt of power. For businesses in India, this means you must plan for robust power supplies, cooling solutions, and a UPS or inverter capable of handling that significant load.

Datacenter-grade hardware requires specialized rack power and cooling that standard office environments simply cannot provide. If your workload is intermittent, the infrastructure costs of "buying" might outweigh the simple monthly cost of "renting."

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

How much VRAM do I need for an 8B parameter model?

At 4-bit quantization, you need about 4–5 GB for weights, plus 1–2 GB of overhead and additional space for the KV cache.

What is the difference between VRAM and memory bandwidth?

VRAM is the capacity to hold model weights and KV cache; memory bandwidth determines how fast tokens are generated.

Can I run a large model on a CPU?

Yes, using tools like llama.cpp, but it is significantly slower than a GPU because CPU memory bandwidth is much lower.

Why is the KV cache important for long conversations?

The KV cache stores past tokens so they don't have to be recomputed; it grows linearly with context length and user count.

Is it better to buy a GPU for fine-tuning?

For short bursts of fine-tuning, renting datacenter GPUs is often more efficient than buying hardware you won't use daily.