The Two Factors That Determine GB Size
The size of a Large Language Model in gigabytes is determined by its number of parameters and how those parameters are stored (precision). While a model's "size" is often discussed in terms of billions of parameters, the actual memory required to run it depends heavily on how much data is used to represent each parameter.
For builders and business owners, understanding this distinction is critical for choosing the right hardware. If you are setting up a local server for a clinic or a retail office, you need to know if your hardware can actually hold the model you want to use.
There are two main drivers for the file size and memory footprint:
- The number of parameters: "Large" refers to the count of adjustable numbers the model learns during training. These range from a few hundred million in small models to hundreds of billions in the largest ones.
- Precision (Quantization): This refers to how many bits are used to store each parameter. This determines the total file size and how much RAM or VRAM you need.
To understand the math, you must understand how a single parameter is stored. In a standard 16-bit floating point (FP16) format, a single parameter takes up 2 bytes of memory.
Calculating Size: A Quick Reference
Because each parameter takes up a specific amount of space, you can estimate the memory requirements of a model based on its parameter count and its precision. This is vital for deciding whether you need an expensive high-end GPU or if a standard computer will suffice.
The following table illustrates how precision changes the memory footprint of a model. We will use a 7B (7 billion parameter) model as our baseline example.
| Precision Format | Bytes per Parameter | Approx. Memory for 7B Model |
|---|---|---|
| 8-bit | 1 byte | ~7 GB |
As you can see, lowering the precision significantly reduces the hardware requirements. For a small business owner in India looking to run a model locally on a mid-range laptop, 4-bit quantization makes the difference between the model being able to run or crashing the system entirely.
What is Quantization?
Quantization is the process of storing weights in fewer bits to save space. It is essentially a way of "compressing" the model's knowledge so it fits into smaller memory pools.
When we talk about how LLM models work, we are talking about these massive arrays of numbers. Quantization reduces the precision of these numbers. For example, 8-bit precision roughly halves the memory required compared to the standard 16-bit (FP16) format. Moving down to 4-bit quantization roughly quarters the memory required compared to FP16.
This reduction allows large models to fit on smaller, cheaper hardware. There are specific file formats designed for this, such as GGUF. GGUF is a format that allows for efficient inference on CPUs, Apple silicon, and GPUs. It is widely used by tools like Ollama to make running AI accessible on consumer-grade hardware.
However, there is a trade-off. While lowering the bit-depth (e.g., to 4-bit) makes models run faster and fit on cheaper hardware, the quality loss can be more noticeable in smaller models. For very large models, the loss is usually modest, but for a tiny model, you might notice a drop in how "smart" it feels.
Why Memory Usage Goes Beyond Model Size
A common mistake for builders is assuming that if a model is 5GB, they only need 5GB of VRAM. In reality, you need more than just the model size to actually run the model effectively. This is because of something called the KV cache.
The KV cache is a way for the model to stay fast. While generating text, the model would waste effort recomputing the attention keys and values for every earlier token at every step. The KV cache stores them so the model can simply look them up.
This cache consumes memory that grows based on two things:
- Context length: The longer the conversation or the longer the document you provide, the more memory the cache needs.
- Simultaneous users: If you are running a service for many customers at once, each user's conversation history takes up space in the memory.
This is why long context windows require significantly more GPU memory to maintain performance. If you are building a tool to summarize long legal documents or entire threads of WhatsApp messages, you must account for this extra memory overhead beyond the base model size.
The MoE Exception: Total vs. Active Size
There is a specific type of architecture called Mixture-of-Experts (MoE) that changes how we think about size. You may see models described as having a massive parameter count, but they might actually run very quickly.
In an MoE model, the architecture replaces each feed-forward block with many smaller "experts" and a router. When a token is processed, the router sends it to only a few of those experts at a time. This is known as the "active" count.
This creates a unique hardware requirement:
- Total Size: You must have enough memory to hold the *entire* model (all the experts) so they can be accessed instantly.
- Active Size: Because only a few experts are used for any single token, the actual computation (the math being done) is much smaller.
The practical upshot is that an MoE model runs at roughly the speed of a model its active size, while needing memory for its total size. This is a powerful way to get high-quality reasoning without the massive computational cost of a traditional "dense" model of the same total size.
Choosing Hardware Based on Model Size
When deciding on your infrastructure, you should work backward from your task. A helpful rule of thumb is to "start large, then shrink." Get your workflow working with a highly capable, large model first to ensure the task is possible, then test whether a smaller or quantized model can handle it just as well.
Hardware needs vary wildly based on your deployment goals:
- Mobile/Edge: Small models (a few billion parameters) quantized to around 4-bit can run fully offline on recent mobile phones. This is excellent for users in areas with patchy connectivity or for applications requiring high data privacy.
- Server/Cloud: Larger models require significant GPU memory to accommodate both the weights and the KV cache for multiple users.
Remember that inference speed is often limited by how fast weights can be read from memory. This is why quantization is so popular; by making the files smaller, you make them easier and faster to move through your hardware's memory channels.
For a deeper dive into the underlying mechanics of these systems, you can read our guide on how large language models work.
Frequently Asked Questions
- Does a larger model always mean more GB?
- Generally, yes, because "large" refers to the number of parameters. However, a large model can be shrunk in GB using quantization.
- What is a 7B model?
- It is a model with 7 billion parameters. The "B" stands for billion.
- Can I run an LLM on my phone?
- Yes, models with a few billion parameters quantized to 4-bit can run fully offline on recent mobile devices.
- What is the difference between 16-bit and 4-bit quantization?
- 16-bit precision uses 2 bytes per parameter, while 4-bit uses significantly less, roughly quartering the memory requirement.
- Why does context length affect memory?
- The KV cache stores information for every token in the context, so longer conversations require more memory.

Questions people ask
Does a larger model always mean more GB?
Generally, yes, because 'large' refers to the number of parameters. However, a large model can be shrunk in GB using quantization.
What is a 7B model?
It is a model with 7 billion parameters. The 'B' stands for billion.
Can I run an LLM on my phone?
Yes, models with a few billion parameters quantized to 4-bit can run fully offline on recent mobile devices.
What is the difference between 16-bit and 4-bit quantization?
16-bit precision uses 2 bytes per parameter, while 4-bit uses significantly less, roughly quartering the memory requirement.
Why does context length affect memory?
The KV cache stores information for every token in the context, so longer conversations require more memory.
