Let's talk

Hardware for builders

GPU Server for AI Startup

Selecting the right hardware depends on your model size, context length, and whether you are running inference or fine-tuning. This guide helps you calculate your specific needs.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for GPU Server for AI Startup

How to Choose a GPU Server for Your AI Startup

To choose the right GPU server, prioritize VRAM capacity to ensure your model and its context fit in memory, then prioritize memory bandwidth to drive inference speed. For most startups, the best strategy is to rent cloud GPUs for experimentation and bursty workloads, while buying or leasing dedicated hardware for high-utilization, bulk generation tasks.

  • Prioritize VRAM capacity: You must have enough memory to hold the model weights plus the KV cache required for your specific context length.
  • Prioritize memory bandwidth: This dictates your tokens per second; higher bandwidth means faster text generation.
  • Calculate total memory: Use the formula: (Parameters × Bytes per parameter) + KV cache + 1-2 GB overhead.
  • Choose based on workload: Rent for bursts and testing; buy for high-utilization bulk work where the hardware is busy most of the day.

Calculating VRAM Requirements

Selecting hardware starts with understanding that model weights are a fixed cost. The amount of memory required depends heavily on the precision (bit-depth) you use. For example, an 8B parameter model requires approximately 16 GB at FP16, about 8 GB at 8-bit, and roughly 4 to 5 GB at 4-bit precision.

Model Size FP16 / BF16 weights 8-bit weights 4-bit weights Typical single GPU (4-bit)
3B ≈ 6 GB ≈ 3 GB ≈ 1.5–2 GB Any 6–8 GB card
8B ≈ 16 GB ≈ 8 GB ≈ 4–5 GB 8–12 GB card
14B ≈ 28 GB ≈ 14 GB ≈ 7–9 GB 12–16 GB card
32B ≈ 64 GB ≈ 32 GB ≈ 16–20 GB 24–32 GB card
70B ≈ 140 GB ≈ 70 GB ≈ 35–42 GB 48 GB or larger

The Hidden Cost: The KV Cache

Weights are only half the story. While generating text, a transformer stores a "key and value" vector for every past token in every layer so it does not have to recompute them. This is the KV cache. It grows linearly with your context length and the number of users you serve simultaneously.

To prevent slowdowns or crashes, always leave 20-30% headroom above the total memory needed for weights and cache. If you are tight on memory, quantization (e.g., using 4-bit instead of 16-bit) significantly reduces memory needs, allowing larger models to run on smaller hardware.

Why Bandwidth Dictates Inference Speed

For a single user, generating each new token (decoding) is a memory-bound task. This means the chip spends most of its time waiting on memory to deliver data, rather than performing complex math. The actual compute power of the chip often sits idle while waiting for the next batch of weights.

The Speed Ceiling

A useful way to estimate your maximum possible speed is: Max tokens per second ≈ Memory Bandwidth ÷ Bytes read per token.

Consider these two scenarios for an 8B model at 4-bit (approx. 4.5 GB):

  • Consumer GPU (RTX 4090): With 1,008 GB/s bandwidth, the ceiling is roughly 220 tokens per second.
  • High-end Card: On a card with 1,792 GB/s bandwidth, the ceiling is roughly 45 tokens per second for a much larger 70B model.

This is why GPU architecture matters. High-end datacenter GPUs use HBM (High Bandwidth Memory) to achieve massive speeds, whereas consumer cards use GDDR memory. Additionally, Mixture-of-Experts (MoE) models can offer a speed advantage; they require the memory capacity to hold all weights, but only activate a fraction of them per token, allowing for faster decoding.

Comparing GPU Tiers

Not all GPUs are created equal. Your choice depends on whether you need massive capacity, raw speed, or cost-effective local testing.

GPU Tier Example Memory Type/Capacity Best Use Case
Consumer RTX 4090 24 GB GDDR6X Best bandwidth per rupee; testing and small models.
Workstation RTX PRO 6000 Blackwell 96 GB GDDR7 (ECC) Large models (70B), MoE, and 24/7 server use.
Datacenter H100 / H200 80-141 GB HBM3/e High-throughput serving and large-scale training.
Apple Silicon M3/M4 Ultra Up to 512 GB (Unified) Massive capacity for large models; quiet and efficient.

For developers working on a budget, Apple Silicon offers a unique advantage: Unified Memory. Because the CPU and GPU share the same pool of RAM, a Mac with high memory can load models that would require multiple expensive NVIDIA GPUs. While the bandwidth is lower than top-tier NVIDIA cards—meaning slower tokens per second—the capacity is unmatched for the price.

Renting vs. Buying Hardware

Deciding whether to rent cloud compute or buy your own server is a critical business decision for an AI startup. The decision usually comes down to utilization: how many hours a day will this hardware be working?

When to Rent

  • Exploring Models: If you are testing different architectures or quantization levels.
  • Short Bursts: If you only need to fine-tune a model for a few days.
  • Spiky Traffic: If your application has unpredictable user demand that requires scaling up and down quickly.

When to Buy or Lease

  • Bulk Generation: If your business model involves running massive, continuous batch jobs (e.g., processing thousands of documents daily).
  • High Utilization: If your hardware is busy most of the day, the fixed cost of ownership will eventually beat the hourly rental rate.
  • Data Sensitivity: If you have strict requirements that data must stay on-premise.

Hardware for Different AI Tasks

Your specific task determines whether you need more VRAM (capacity) or more bandwidth (speed).

Inference
The goal is speed and stability. You need enough VRAM for the weights plus the KV cache for your expected context length. One or more GPUs are typically sufficient for most startup use cases.
Fine-tuning (LoRA / QLoRA)
This requires more memory than inference because you must store the base weights, the new adapter weights, and the "activations" used during the math. Using QLoRA (4-bit fine-tuning) allows you to fine-tune a 65B model on a single 48 GB GPU.
Full Fine-tuning / Pre-training
This is the most demanding task. You need massive memory for gradients and optimizer states (often 16 bytes per parameter). This typically requires clusters of datacenter GPUs.

Frequently Asked Questions

How much VRAM do I need for an 8B parameter model?
You need about 16 GB at FP16, 8 GB at 8-bit, and 4-5 GB at 4-bit, plus extra for the KV cache and runtime overhead.
What is the difference between VRAM capacity and bandwidth?
VRAM capacity determines if a model fits in memory; bandwidth determines how fast the model generates tokens (tokens per second).
When should I use a CPU for AI instead of a GPU?
CPUs are suitable for small models, background batch jobs, or embeddings, but they are significantly slower due to lower memory bandwidth.
Do I need multiple GPUs to run large models?
Multiple GPUs allow you to fit larger models (via tensor parallelism) or serve more users, but they do not necessarily make a single user's inference faster.
Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

How much VRAM do I need for an 8B parameter model?

You need about 16 GB at FP16, 8 GB at 8-bit, and 4-5 GB at 4-bit, plus extra for the KV cache and runtime overhead.

What is the difference between VRAM capacity and bandwidth?

VRAM capacity determines if a model fits in memory; bandwidth determines how fast the model generates tokens (tokens per second).

When should I use a CPU for AI instead of a GPU?

CPUs are suitable for small models, background batch jobs, or embeddings, but they are significantly slower due to lower memory bandwidth.

The KV cache stores past tokens to avoid recomputation; it grows with context length and the number of concurrent users, consuming significant VRAM.

Do I need multiple GPUs to run large models?

Multiple GPUs allow you to fit larger models (via tensor parallelism) or serve more users, but they do not necessarily make a single user's inference faster.