Let's talk

Hardware Reality Check

Can I Run an LLM on 8GB VRAM?

While 8GB of VRAM is not enough for large models in full precision, quantization allows you to run highly capable smaller models effectively.

By Divyaprakash JhaUpdated 5 min readTopic: AI hardware

Photo for Can I Run an LLM on 8GB VRAM?

The Short Answer: Yes, with Quantization

Yes, you can run LLMs on 8GB VRAM if you use quantization to compress the model. While a large model might require 16 GB at full precision, quantization can shrink that requirement down to 4 or 5 GB, making it fit comfortably on an 8GB card.

  • Quantization is key: You can run models on 8GB VRAM if you use quantization (compressing the model).
  • The 8B Example: An 8B parameter model requires ~16 GB at FP16, but only ~4 to 5 GB at 4-bit quantization.
  • What fits: 8GB VRAM is sufficient for 3B models at 4-bit or 8B models at 4-bit with limited context.

The Math: How Much VRAM Do You Need?

For builders and business owners, understanding the math is the difference between a model that runs smoothly and one that crashes your system. The fundamental rule of thumb for calculating memory needs is: Parameters × Bytes per parameter.

The number of bytes used depends on the precision of the model. If you are using high-precision formats, the memory requirements are much higher than if you use compressed (quantized) versions:

Precision Format Bytes per Parameter Notes
FP16 / BF16 2 bytes Full precision; highest quality.
8-bit 1 byte Good balance of quality and size.

When planning your hardware, do not just calculate the model weight size. You must always budget extra for the KV cache (the memory used for conversation context) and 1–2 GB of runtime overhead for the framework and activations. If you "just fit" the model into your VRAM, the system will likely slow down or fail once the conversation gets long.

Why Context Length Matters

A common frustration for users is when an LLM runs perfectly at the start of a chat but becomes incredibly slow or crashes after several exchanges. This is usually due to the KV cache.

While generating text, a transformer stores a key and a value vector for every past token in every layer so it does not have to recompute them. This cache grows linearly with the context length. For example, a Llama 3.1 8B style model might use about 1 GiB of VRAM for 8,192 tokens of context. However, if you increase that context to 131,072 tokens, the cache alone could require 16 GiB—which is more than the weights of the model itself!

If you are building an application for multiple users, remember that the cache grows with the number of users served at once. Serving ten users at 8K context each requires roughly 10 GiB of cache memory.

Memory Bandwidth vs. VRAM Capacity

It is vital to distinguish between capacity and bandwidth. They serve two different purposes in AI inference:

  • VRAM Capacity: This determines if a model can load. If the model is 10 GB and you only have 8 GB, it simply won't fit in the GPU.
  • Memory Bandwidth: This determines how fast the model generates text.

For a single user, the speed of text generation (decoding) is limited by how fast the chip can read the model weights from memory. This is why high-end GPUs like the RTX 4090, which offer much higher bandwidth than standard system RAM, feel so much faster even if the model size is similar. For example, an 8B model at 4-bit might reach a ceiling of roughly 220 tokens per second on an RTX 4090, whereas a system with much lower bandwidth would struggle to keep up.

Choosing the Right Hardware

Depending on your budget and use case, hardware generally falls into three tiers. When deciding whether to buy or rent, consider your expected monthly usage.

GPU Tier Example Hardware Typical VRAM / Bandwidth Best Use Case
Consumer GeForce RTX 4090 24 GB GDDR6X / 1,008 GB/s Up to 32B models at 4-bit; great bandwidth per rupee.
Workstation RTX PRO 6000 Blackwell 96 GB GDDR7 / 1,792 GB/s 70B models; large MoE models; small-team serving.
Datacenter H100 SXM 80 GB HBM3 / 3.35 TB/s High-throughput serving and training.

If you are a developer working with Apple hardware, the Apple Silicon approach is different. Macs use unified memory, meaning the CPU and GPU share a single pool of memory. A Mac with a large memory configuration can load massive models that would require several expensive consumer GPUs. While the bandwidth (speed) might be lower than a top-tier NVIDIA card, the capacity is much higher and the system is more efficient for local development.

Running LLMs on CPU and System RAM

If you do not have a powerful GPU, you can still run models using your computer's ordinary CPU and system RAM. Tools like llama.cpp make this possible through quantization.

However, there is a significant performance trade-off. System RAM bandwidth is much lower than dedicated VRAM. For instance, dual-channel DDR5-5600 memory has a theoretical peak of roughly 90 GB/s, which is about a tenth of the speed of a high-end GPU. This means a large model running on a CPU will be significantly slower, often producing only a few tokens per second.

You can choose to split layers between the GPU and the CPU. This allows you to use the GPU for the parts of the model that fit and the CPU for the rest, but the overall speed will be limited by the slower CPU/system RAM component.

For those interested in more hardware details, see our guides on AI hardware and GPUs for AI.

Frequently Asked Questions

What is quantization in LLMs?
Quantization is the process of storing model weights in lower precision (like 4-bit or 8-bit instead of 16-bit) to reduce memory requirements.
Why does my LLM run slow when the context gets long?
As the conversation grows, the KV cache consumes more VRAM, which can lead to memory exhaustion or slower processing.
Can I run a 70B model on 8GB VRAM?
Not entirely on the GPU. You would need to use llama.cpp to offload most of the model to your system RAM, resulting in very slow speeds.
Is more VRAM better than faster speed?
Capacity determines if the model fits; bandwidth determines how fast it generates tokens. You need enough capacity first, then speed follows.
What is the best way to run a local LLM?
Ollama and llama.cpp are excellent for individuals, while vLLM or SGLang are better for high-throughput serving.
Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

What is quantization in LLMs?

Quantization is the process of storing model weights in lower precision (like 4-bit or 8-bit instead of 16-bit) to reduce memory requirements.

Why does my LLM run slow when the context gets long?

As the conversation grows, the KV cache consumes more VRAM, which can lead to memory exhaustion or slower processing.

Can I run a 70B model on 8GB VRAM?

Not entirely on the GPU. You would need to use llama.cpp to offload most of the model to your system RAM, resulting in very slow speeds.

Is more VRAM better than faster speed?

Capacity determines if the model fits; bandwidth determines how fast it generates tokens. You need enough capacity first, then speed follows.

What is the best way to run a local LLM?

Ollama and llama.cpp are excellent for individuals, while vLLM or SGLang are better for high-throughput serving.