What is Tokens Per Second (TPS)?
Tokens per second (TPS) is the primary metric for measuring how fast an AI model generates text. For builders, understanding what drives this speed is the key to choosing the right hardware.
- TPS measures the generation speed of a Large Language Model (LLM).
- It represents how many tokens (words or parts of words) the model outputs every second.
- For a single user, the speed is primarily limited by memory bandwidth, not raw compute power.
The Math: Why Bandwidth is King
During the decoding phase—the part where the model is actually streaming text to your screen—the chip must read every active weight from memory to generate a single token. Because the actual mathematical calculation (the arithmetic) performed on each weight is tiny, the chip spends most of its time simply waiting for the data to arrive from the memory.
This makes memory bandwidth the deciding factor for speed. If you are building an application, you should look at the memory bandwidth of your GPU rather than just the TFLOPS (teraflops) or raw compute power. A useful ceiling for speed is calculated as follows:
max tokens per second ≈ memory bandwidth ÷ bytes read per token
Keep in mind that real-world speed is often 50 to 80 percent of this theoretical ceiling due to various system overheads.
How Model Size and Quantization Affect Speed
The size of the model you choose has a direct, inverse relationship with your speed. Larger models require more memory bandwidth to move all those parameters into the processor, which naturally lowers your tokens per second. For a business owner in India setting up a local server, this means a massive 70B model will feel significantly "slower" than a small 8B model, even on the same hardware.
However, you can combat this using quantization. Quantization is the process of storing model weights in lower precision (like 4-bit or 8-bit instead of the standard FP16). This reduces the number of bytes that must be read from memory for each token, effectively increasing your speed.
Consider these two scenarios on a high-end consumer card:
- Small Model: An 8B model at 4-bit quantization on an RTX 4090 can reach a ceiling of approximately 220 tokens per second.
- Large Model: A 70B model at 4-bit quantization on a high-end card might only reach approximately 45 tokens per second.
To understand how much memory you need to hold these models, you should review our guide on VRAM.
Comparing Hardware Performance
Not all hardware is created equal. The hardware you choose depends on whether you need raw speed for a single user or high throughput for many users.
| GPU Tier | Memory Bandwidth | Best Use Case |
|---|---|---|
| Consumer (e.g., RTX 4090) | ~1,008 GB/s | Up to ~32B models at 4-bit; best bandwidth per rupee. |
| Workstation (e.g., RTX PRO 6000) | ~1,792 GB/s | 70B models at 4–8-bit; large MoE models; small-team serving. |
| Datacenter (e.g., H100 SXM) | ~3.35 TB/s | High-throughput serving and training. |
For those using non-GPU hardware, the performance gap is massive. Apple Silicon uses unified memory, which allows a Mac to load massive models that would require multiple GPUs, though the bandwidth is lower than top-tier NVIDIA cards. CPUs are limited by system RAM bandwidth; a typical desktop might only manage about 2 tokens per second for a 70B model, making them unsuitable for interactive chat but fine for background tasks.
The Mixture-of-Experts (MoE) Speed Boost
Mixture-of-Experts (MoE) is a specialized architecture that offers a clever way to get high speed without sacrificing intelligence. Instead of activating every single parameter for every token, an MoE model splits its layers into many "expert" sub-networks and only activates a small fraction of them at a time.
This provides a significant advantage: a model can have a huge total parameter count (requiring lots of VRAM), but it decodes at a speed closer to a much smaller model. For example, Google's Gemma 4 26B A4B has about 25 billion total parameters but only activates about 3.8 billion per token. It requires the memory capacity of a 26B model but delivers the speed of a 4B model. This makes MoE models ideal for GPUs with large VRAM.
The Role of the KV Cache
When you are having a long conversation with an AI, the model doesn't re-read the entire history every time you send a new message. Instead, it stores the "keys" and "values" of past tokens in a temporary workspace called the KV cache.
While the KV cache is essential for continuous, fast generation, it comes with a cost: it consumes VRAM. The cache grows linearly as the conversation gets longer. If you are running a model that "just fits" in your VRAM, the moment the conversation reaches a certain length, the KV cache will exceed your available memory. This will cause the model to slow down significantly or fail entirely.
Frequently Asked Questions
- Does more compute power mean more tokens per second?
- Not necessarily. For most LLM inference tasks, speed is limited by memory bandwidth rather than raw compute power.
- Why is my LLM slow when I use a long prompt?
- Reading a long prompt (prefill) is compute-limited, while generating text (decoding) is bandwidth-limited. A long prompt causes a pause before the steady stream of tokens begins.
- Will quantization make my AI faster?
- Yes. Quantization (like 4-bit or 8-bit) reduces the number of bytes that must be read from memory for each token, increasing tokens per second.
- How does VRAM affect AI speed?
- VRAM capacity determines if a model can fit on your hardware, while VRAM bandwidth determines how many tokens per second you can generate.
- Is a CPU good for running LLMs?
- CPUs can run quantized models using tools like llama.cpp, but they are significantly slower than GPUs because system RAM has much lower bandwidth.

Questions people ask
Does more compute power mean more tokens per second?
Not necessarily. For most LLM inference tasks, speed is limited by memory bandwidth rather than raw compute power.
Why is my LLM slow when I use a long prompt?
Reading a long prompt (prefill) is compute-limited, while generating text (decoding) is bandwidth-limited. A long prompt causes a pause before the steady stream of tokens begins.
Will quantization make my AI faster?
Yes. Quantization (like 4-bit or 8-bit) reduces the number of bytes that must be read from memory for each token, increasing tokens per second.
How does VRAM affect AI speed?
VRAM capacity determines if a model can fit on your hardware, while VRAM bandwidth determines how many tokens per second you can generate.
Is a CPU good for running LLMs?
CPUs can run quantized models using tools like llama.cpp, but they are significantly slower than GPUs because system RAM has much lower bandwidth.
