What are Small Language Models on Device?
Running AI locally requires balancing model size, memory capacity, and bandwidth. This guide explains how to choose the right hardware for on-device inference.
On-device AI refers to running models locally on hardware like phones, laptops, or dedicated workstations. Instead of sending your data to a massive server in the cloud, the "thinking" happens right on your desk or in your pocket.
Neural Processing Units (NPUs) found in modern phones and laptops are specifically designed for these compact models, typically ranging from 1B to 4B parameters. These specialized chips are built for low-power, efficient operation.
These models are highly effective at handling specific, high-frequency tasks such as transcription, summarization, and smart replies without needing a constant internet connection.
The Memory Rule for Local Models
The most critical factor in running a model is ensuring it fits into your hardware's memory. To run a model, you must fit the weights plus the KV cache into memory. If the model is larger than your available VRAM, the system will slow down significantly as it swaps data to much slower system RAM or disk storage.
A rough rule for calculating weight requirements is: parameters × bytes per parameter. For example, if you are using a model in FP16 precision (2 bytes per parameter), a 7B model will require roughly 14 GB of memory just for the weights.
Because of this, "quantization" is used to shrink models. Quantization involves storing weights in lower precision, such as 8-bit or 4-bit, to save space. An 8B model needs approximately 16 GB at FP16, about 8 GB at 8-bit, and roughly 4–5 GB at 4-bit.
When planning your hardware, you should also account for 1 to 2 GB of runtime overhead for the framework and activations. For a business owner in India setting up a local workstation, it is better to over-provision memory than to run into "out of memory" errors during a customer interaction.
Why Context Length Impacts Memory
When a model generates text, it doesn't re-read the entire conversation from scratch for every single word it produces. Instead, it uses something called the KV cache. This stores key and value vectors for past tokens so they don't need to be recomputed, allowing the model to "remember" the conversation history efficiently.
However, this cache comes at a cost. The KV cache grows linearly with the context length (the number of tokens in the conversation) and the number of users being served at once. For a single user, a long conversation can require a significant amount of extra memory beyond the model weights themselves.
For example, in a Llama 3.1 8B style model, a conversation with 131,072 tokens of context can require about 16 GiB of memory for the cache alone—which is actually more than the size of the 4-bit model weights.
Fortunately, there are ways to manage this:
- Grouped-query attention (GQA): This architecture allows several query heads to share one KV head, which can make the cache up to four times smaller than older designs.
- KV Cache Quantization: Many modern runtimes allow you to quantize the KV cache to 8-bit, which can halve its size with minimal loss in quality.
Choosing Your Hardware Tier
Hardware choice depends on whether you prioritize raw speed, massive model capacity, or power efficiency. Understanding the difference between hardware tiers is essential for budgeting.
| GPU Tier | Memory Type/Capacity | Best Use Case |
|---|---|---|
| Consumer (e.g., RTX 4090) | 24 GB GDDR6X | Best bandwidth per rupee; good for models up to 32B at 4-bit. |
| Workstation (e.g., RTX PRO 6000) | 96 GB GDDR7 (ECC) | High VRAM for large models, MoE, and professional serving. |
| Datacenter (e.g., H100/H200) | 80–141 GB HBM3/e | High-throughput serving and training for many users. |
Apple Silicon: Mac computers with M-series chips use unified memory. This allows a Mac to load very large models into system RAM that would otherwise require multiple expensive GPUs. While the bandwidth is lower than a high-end NVIDIA card, the capacity is exceptionally high and the system is very efficient.
CPUs: You can run quantized models on a standard CPU using tools like llama.cpp. However, CPUs are limited by much lower memory bandwidth. A 40 GB model might only reach 2 tokens per second on a typical desktop CPU, making it better for background tasks rather than interactive chat.
Why Bandwidth Decides Token Speed
A common misconception is that a faster "compute" score (TFLOPS) always means a faster AI. For single-user text generation (decoding), the chip spends most of its time waiting on memory, not performing calculations. This is often referred to as being "memory-bound."
During the generation of each new token, the chip must read every single active weight from the memory. Therefore, your maximum speed is roughly limited by: memory bandwidth ÷ bytes read per token.
This is why GPUs are so much faster than CPUs for AI. A GPU has thousands of simple cores designed to move and multiply data in parallel, whereas a CPU is designed for complex logic. If you want faster text generation, prioritize memory bandwidth over raw compute power.
Speeding Up Inference with MoE
Mixture-of-Experts (MoE) models offer a clever way to get high performance without the massive compute cost of a giant model. Instead of activating every single parameter for every word, an MoE model splits its layers into sub-networks (experts) and only routes the token to a few of them.
This creates a unique trade-off: MoE models trade memory capacity for speed.
Because all the "experts" must stay loaded in memory to be ready at any time, you need enough VRAM to hold the full model. However, because only a fraction of the parameters are "active" during any single calculation, the model decodes at a speed much closer to a much smaller model. For example, a 26B parameter MoE model might only activate about 3.8 billion parameters per token, allowing it to run very quickly on hardware that can fit the full 26B weights.
Should You Buy or Rent Hardware?
Deciding whether to invest in physical hardware or use cloud services is a key business decision. The right choice depends on your expected usage patterns.
| Situation | Better Choice | Reasoning |
|---|---|---|
| Exploring models or short bursts of work | Rent | No upfront cost; easy to try different hardware. |
| Bulk generation (many hours every day) | Buy or Lease | Fixed costs are cheaper than hourly cloud rates over time. |
| Sensitive or private data | Buy (On-premise) | You maintain total control over data privacy. |
| Spiky or unpredictable public traffic | Rent or API | Scale up only when demand arrives; avoid paying for idle hardware. |
A simple way to decide is to estimate your monthly GPU hours. If your hardware would be busy most of the day for a year or more, owning a dedicated machine usually wins. If the machine would sit idle most of the time, renting is much more cost-effective.
Frequently Asked Questions
- How much VRAM do I need for an 8B model?
- At 4-bit quantization, you need about 4 to 5 GB for the weights, plus additional memory for the KV cache and runtime overhead.
- What is the difference between a CPU and a GPU for AI?
- CPUs have a few powerful cores for complex logic, while GPUs have thousands of simpler cores designed for parallel matrix multiplication, which is the core of AI work.
- What are NPUs used for in mobile devices?
- NPUs are small accelerators designed for low-power tasks like summarizing notifications, transcription, and smart replies using 1B to 4B parameter models.
- Does context length affect memory usage?
- Yes, the KV cache grows linearly with context length. A long conversation can require significantly more memory than the model weights themselves.
- Why is memory bandwidth important for LLMs?
- During token generation, the chip must read every active weight from memory. High bandwidth allows the chip to read these weights faster, increasing tokens per second.

Questions people ask
How much VRAM do I need for an 8B model?
At 4-bit quantization, you need about 4 to 5 GB for the weights, plus additional memory for the KV cache and runtime overhead.
What is the difference between a CPU and a GPU for AI?
CPUs have a few powerful cores for complex logic, while GPUs have thousands of simpler cores designed for parallel matrix multiplication, which is the core of AI work.
What are NPUs used for in mobile devices?
NPUs are small accelerators designed for low-power tasks like summarizing notifications, transcription, and smart replies using 1B to 4B parameter models.
Does context length affect memory usage?
Yes, the KV cache grows linearly with context length. A long conversation can require significantly more memory than the model weights themselves.
Why is memory bandwidth important for LLMs?
During token generation, the chip must read every active weight from memory. High bandwidth allows the chip to read these weights faster, increasing tokens per second.
