Let's talk

AI Hardware Optimization

What is Quantization in LLM?

Quantization is the process of reducing the precision of model weights to decrease memory usage and increase inference speed.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for What is Quantization in LLM?

Understanding Quantization

Quantization is the process of reducing the precision of model weights to decrease memory usage and increase inference speed. By storing numbers in lower precision formats, you can run massive AI models on much smaller, more affordable hardware.

At its core, quantization involves storing model weights in lower precision formats like 8-bit or 4-bit instead of the standard FP16 or BF16. In a standard high-precision model, each parameter is represented by a large, detailed number. Quantization simplifies these numbers, essentially "rounding" them to a lower precision.

This simplification has two major benefits for builders and business owners:

  • Reduced Memory Footprint: Lower precision means less memory is required per number, allowing larger models to fit on smaller hardware.
  • Increased Speed: Modern GPUs use specialized units like tensor cores to perform math on these reduced-precision matrices much faster.

How Quantization Saves VRAM

For any business owner looking to deploy AI, the most important constraint is often Video RAM (VRAM). To understand how much memory you need, follow a simple rule: parameters × bytes per parameter. Memory usage is a direct result of how much space each individual weight takes up in your hardware's memory.

When we move from high precision (FP16) to lower precision (4-bit), the savings are massive. For example, a 7B (7 billion parameter) model has the following requirements:

7B Model ~14 GB ~7 GB ~3.5 GB
8B Model ~16 GB ~8 GB ~4–5 GB
70B Model ~140 GB ~70 GB ~35–42 GB

Note that real-world 4-bit files are often slightly larger than the theoretical minimum. This is because developers often keep some layers at higher precision and store "scaling factors" to maintain accuracy. You should budget roughly 4.5 to 5 bits per weight for real-world applications.

Quantization is essential for running large models on consumer GPUs, laptops, or even mobile phones. Without it, a 70B model would require a massive datacenter setup; with it, you can run it on a single high-end workstation GPU.

The Trade-off: Speed vs. Precision

In engineering, there is rarely a free lunch. Quantization is a trade-off between mathematical precision and computational efficiency. While reducing precision introduces a small quality cost compared to full-precision models, the benefits usually outweigh the drawbacks for most business use cases.

The primary advantage is speed. Reduced precision leads to more math operations per second, which can increase token generation speed. Because the numbers are smaller, the hardware can process them much more quickly, leading to a snappier user experience.

However, builders must be aware of the "quality gap." While a 4-bit model is incredibly efficient, it might be slightly less "smart" or accurate than the original version. For many applications—like drafting emails or summarizing text—this difference is negligible. But for highly complex reasoning tasks, you might prefer higher precision.

Quantization in Inference

Inference is the process of running a model to generate text. This is where quantization is most commonly applied to optimize performance and reduce costs.

Beyond just the model weights, you can also quantize the KV cache. The KV cache is the memory used to store conversation history so the model doesn't have to recompute everything every time you send a new message. As conversations get longer, this cache grows. By quantizing the KV cache to 8-bit, you can halve its size, allowing for much longer conversations on the same hardware.

If you are looking to run models locally, tools like llama.cpp are designed specifically to handle these quantized formats (such as GGUF). These tools allow you to run efficient models on a wide variety of hardware, including standard CPUs and Apple Silicon, by leveraging these optimized, lower-precision weights.

For those building high-traffic applications, understanding VRAM requirements is critical for scaling your inference servers.

Impact on AI Hardware

Quantization changes the hardware landscape entirely. It allows a 70B model—which normally requires 140 GB of VRAM at FP16—to fit on a 48 GB GPU when quantized to 4-bit. This makes high-end AI accessible to smaller teams and individual developers.

When running these models, the performance bottleneck often shifts. On consumer hardware, the bottleneck for quantized models is often memory bandwidth rather than raw compute power. This means the speed at which your hardware can move data from the memory to the processor is often more important than how many calculations the processor can do per second.

Here is how different hardware tiers typically handle these models:

  • Consumer GPUs (e.g., RTX 4090): Great bandwidth per rupee, but limited VRAM. Good for models up to about 32B when quantized to 4-bit.
  • Workstation GPUs (e.g., RTX PRO 6000): Offers much more VRAM (up to 96 GB), allowing you to run massive 70B models at 4-bit or 8-bit precision.
  • Apple Silicon (M-series): Uses unified memory, meaning the GPU can access the large pool of system RAM. This allows a Mac to load models that would require multiple expensive NVIDIA cards.
  • CPUs: Using llama.cpp, you can run quantized models on a standard computer. However, speed is limited by your system's DDR memory bandwidth, which is significantly slower than a GPU's VRAM.

For a deeper dive into choosing the right hardware, see our guide on GPUs for LLMs.

Frequently Asked Questions

Does quantization make an LLM less smart?
Quantization can result in a small quality cost, but it is often a necessary trade-off to make large models run on accessible hardware.
How much VRAM does a 7B model need at 4-bit?
A 7B model requires approximately 3.5 GB of memory for the weights at 4-bit, though you should budget roughly 4.5 to 5 GB for real-world files to account for overhead.
Can I quantize the KV cache?
Yes, many runtimes allow you to quantize the KV cache to 8-bit, which can halve its memory footprint with a small cost to quality.
Why is 4-bit quantization so popular?
It significantly reduces the memory footprint—roughly to a quarter of the original FP16 size—allowing massive models to fit on consumer-grade GPUs.
What is the difference between FP16 and 4-bit?
FP16 uses 2 bytes per parameter (16-bit), while 4-bit uses half a byte per parameter, drastically reducing the total memory required.
Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

Does quantization make an LLM less smart?

Quantization can result in a small quality cost, but it is often a necessary trade-off to make large models run on accessible hardware.

How much VRAM does a 7B model need at 4-bit?

A 7B model requires approximately 3.5 GB of memory for the weights at 4-bit, though you should budget roughly 4.5 to 5 bits per weight for real-world files.

Can I quantize the KV cache?

Yes, many runtimes allow you to quantize the KV cache to 8-bit, which can halve its memory footprint with a small cost to quality.

Why is 4-bit quantization so popular?

It significantly reduces the memory footprint—roughly to a quarter of the original FP16 size—allowing massive models to fit on consumer-grade GPUs.

What is the difference between FP16 and 4-bit?

FP16 uses 2 bytes per parameter (16-bit), while 4-bit uses half a byte per parameter, drastically reducing the total memory required.