What is Edge AI Hardware?
Edge AI involves running models on local hardware like GPUs, Apple Silicon, or NPUs rather than relying solely on the cloud. To run a large language model (LLM), you need enough fast memory to hold the model weights plus the KV cache for your context length. Once the model fits, memory bandwidth decides how many tokens per second you get.
Edge AI involves running models on local hardware like GPUs, Apple Silicon, or NPUs rather than relying solely on the cloud. The primary goal is to fit model weights and the KV cache into available memory. Performance is determined by two main factors: memory capacity (to hold the model) and memory bandwidth (to determine token speed).
A neural network is mostly matrix multiplication. Generating one token means multiplying the current input by billions of weights. Each multiplication is independent of the others, so the work can be split across thousands of small cores at once. A CPU has a few dozen powerful cores designed for branching and varied logic, while a GPU has thousands of simpler cores designed to do the same operation on large blocks of numbers in parallel. This is exactly the shape of AI work.
The Memory Rule of Thumb
When choosing hardware, you must calculate your total memory requirements accurately. A common mistake is forgetting that the model weights are only one part of the equation. You must also account for the context length and the overhead required to run the software.
The basic formula is: Total memory needed = (Parameters × Bytes per parameter) + KV cache + 1-2 GB runtime overhead.
To manage memory, builders use quantization—storing weights in lower precision to save space. Here is how common quantization levels affect your requirements:
- FP16/BF16: Uses 2 bytes per parameter.
- 8-bit: Uses 1 byte per parameter.
A model that "just fits" in VRAM will slow down or fail once long conversations fill the KV cache. Always leave 20-30% headroom for context length and system overhead.
| Model Size | FP16 / BF16 Weights | 8-bit Weights | 4-bit Weights | Typical Single GPU (4-bit) |
|---|---|---|---|---|
| 3B | ≈ 6 GB | ≈ 3 GB | ≈ 1.5–2 GB | Any 6–8 GB card, laptops, phones |
| 8B | ≈ 16 GB | ≈ 8 GB | ≈ 4–5 GB | 8–12 GB card |
| 14B | ≈ 28 GB | ≈ 14 GB | ≈ 7–9 GB | 12–16 GB card |
| 32B | ≈ 64 GB | ≈ 32 GB | ≈ 16–20 GB | 24–32 GB card |
| 70B | ≈ 140 GB | ≈ 70 GB | ≈ 35–42 GB | 48 GB or larger, or two GPUs |
Understanding the KV Cache
While the model weights are a fixed cost, the KV cache is a dynamic cost. While generating text, a transformer stores a key and a value vector for every past token in every layer so it does not recompute them. This allows the model to generate new text without recalculating everything from scratch.
The cache grows linearly with context length and the number of users served at once. For a very large context window, such as 131,072 tokens, the KV cache can require about 16 GiB of memory, which is more than the 4-bit weights of an 8B model itself.
To manage this, modern architectures use Grouped-query attention (GQA). In older designs, every attention head had its own keys and values. GQA lets several query heads share one KV head. This can make the cache up to four times smaller than it would otherwise be. Additionally, many runtimes can quantize the KV cache to 8-bit, halving the memory requirement again with a small quality cost.
Why Bandwidth Dictates Speed
During the decoding phase (generating one token at a time), the chip spends most of its time waiting on memory to read weights. Because the arithmetic per weight is tiny, the process is "memory-bound." A useful speed ceiling for your hardware is:
Max tokens per second ≈ memory bandwidth ÷ bytes read per token
GPUs beat CPUs because they have much higher memory bandwidth and thousands of parallel cores for matrix multiplication. For example, an 8B model at 4-bit on an RTX 4090 (1,008 GB/s) has a theoretical ceiling of about 220 tokens per second. On a typical desktop CPU with DDR5 memory, that same model might top out at only 2 tokens per second.
Mixture-of-Experts (MoE) models offer a unique way to trade memory for speed. An MoE model splits layers into many expert sub-networks and routes each token to only a few. All weights must sit in memory, but each token only reads the active ones. This allows a large model to decode at a speed closer to a much smaller model.
Comparing AI Hardware Tiers
Choosing the right hardware depends on whether you need capacity (to fit big models) or bandwidth (to get fast responses). Different tiers serve different business needs.
| GPU Tier | Example Hardware | Memory Bandwidth | Best For |
|---|---|---|---|
| Consumer | GeForce RTX 4090 | 1,008 GB/s | Best bandwidth per rupee; up to 32B models at 4-bit |
| Workstation | RTX PRO 6000 Blackwell | 1,792 GB/s | More VRAM (96 GB) and ECC; 70B models, small-team serving |
| Datacenter | H100 / H200 (SXM) | 3.35 – 4.8 TB/s | High-throughput serving, training, many concurrent users |
Apple Silicon uses a unified memory architecture. This allows a Mac to load massive models that would require several consumer GPUs by using system RAM. While the bandwidth is lower than high-end NVIDIA cards, the capacity is extremely high and the system is efficient for on-device AI tasks.
NPUs and Mobile AI
Neural Processing Units (NPUs) are small accelerators found in phones and laptops. They are designed for low-power, specific tasks rather than heavy lifting. You will typically see them used for features like transcription, smart replies, or summarizing notifications. Because phone memory is shared with the rest of the system, these models are often compact (1B to 4B parameters) and heavily quantized to fit within available memory.
Should You Rent or Buy Hardware?
For business owners, the decision to invest in hardware or use the cloud is a matter of economics and data privacy. If you are building private AI solutions, the choice becomes even more critical.
| Situation | Better Choice | Why |
|---|---|---|
| Exploring models / Short bursts | Rent (Cloud/API) | No upfront cost; easy to test different GPUs |
| Bulk generation / Daily use | Buy (On-premise) | Fixed cost beats per-token/hour bills at high use |
| Sensitive data requirements | Buy (On-premise) | You control exactly where your data goes |
| Spiky public traffic | Rent or API | Scale up only when demand arrives |
A simple way to decide: estimate your monthly GPU hours. If the hardware would be busy most of the day for a year or more, owning the hardware usually wins. If it would sit idle, renting is the smarter path.

Questions people ask
How much VRAM do I need for an 8B parameter model?
At 4-bit quantization, an 8B model needs about 4-5 GB. You should also account for the KV cache and 1-2 GB of runtime overhead.
Why is my AI generation slow even with a good GPU?
Speed is often limited by memory bandwidth rather than raw compute. If the model is too large and spills into system RAM, performance drops significantly.
What is the difference between a CPU and a GPU for AI?
A CPU has a few powerful cores for complex logic, while a GPU has thousands of simpler cores designed for the parallel matrix multiplications required by AI.
Does quantization affect model quality?
Yes, quantization (like 4-bit or 8-bit) reduces the memory required per weight, which can result in a small quality cost compared to FP16.
What is the role of the KV cache?
The KV cache stores mathematical vectors for previous tokens in a conversation, allowing the model to generate new text without re-calculating everything from scratch.
