Let's talk

On-device AI

Local LLM on Android

Running large language models directly on mobile devices requires understanding the balance between model size, quantization, and NPU capabilities.

By Divyaprakash JhaUpdated 4 min readTopic: AI hardware

Photo for Local LLM on Android

Mobile Hardware and NPUs

Running local LLMs on Android requires balancing model size with the device's specific hardware. For the best experience, focus on small models that fit within the phone's shared memory and leverage the NPU for efficiency.

Android phones are equipped with Neural Processing Units (NPUs), which are small accelerators designed specifically for low-power AI tasks. Unlike a general-purpose CPU, an NPU is optimized to handle the heavy mathematical workloads required by neural networks without draining the battery as quickly.

Because mobile hardware is constrained, NPUs are typically optimized for compact models, usually falling in the range of 1B to 4B parameters. These models are ideal for specific on-device features such as:

  • Summarizing long text notifications.
  • Transcribing audio recordings locally.
  • Generating smart replies in messaging apps.

A critical factor for builders to remember is that mobile memory is shared with the rest of the system. This means the RAM used by your LLM is the same RAM used by the operating system and other apps, which strictly limits the size of the models you can run effectively.

Why Quantization is Essential

If you attempt to run a full-precision model on a smartphone, you will likely run out of memory immediately. Mobile devices have significantly limited RAM compared to the high-end workstations or desktops used for AI development.

This is where quantization becomes a necessity for mobile deployment. Quantization is the process of storing model weights in lower precision, such as 8-bit or 4-bit, rather than the standard 16-bit format. By doing this, you drastically reduce the memory footprint of the model.

For example, a 4-bit model requires significantly less space than a full FP16 model, making it viable for mobile hardware. Beyond just saving space, lower precision allows for more mathematical operations per second on mobile chips, which can improve the speed of the model's responses.

Estimating Memory Needs

For builders, the most important calculation is determining if a model will fit on a target device. The fundamental rule of thumb for memory is: parameters × bytes per parameter.

When using quantization, the number of bytes per parameter drops significantly. Here is how the memory requirements scale for an 8B parameter model:

Precision | Format Bytes per Parameter Approx. Memory for 8B Model
FP16 (Full Precision) 2 bytes 16 GB
8-bit 1 byte 8 GB

However, you cannot simply match the model size to the available RAM. You must also account for the KV cache. The KV cache is the memory used to store the context of a conversation so the model doesn't have to re-read the entire history every time you send a new message. This cache grows linearly with the length of the conversation.

Always leave headroom for the KV cache. If a model "just fits" into your available memory, the application will likely slow down or fail entirely once the conversation reaches a certain length. For a smooth user experience, you should always aim for 20% to 30% more memory than the model weights alone require.

Bandwidth vs. Compute

When users ask why their mobile LLM feels slow, the answer is rarely about raw processing power (compute) and almost always about memory bandwidth. For LLM inference, memory bandwidth often determines how many tokens per second you get.

During the "decoding" phase—when the model is streaming text word by word—the chip spends much of its time waiting on memory to read the model weights. The arithmetic performed on each weight is tiny, meaning the processor is often sitting idle, waiting for the next chunk of data to arrive from the RAM.

While NPUs accelerate specific tasks, the actual speed of text generation is heavily tied to how fast the device can read model weights from memory. This is why high-end hardware like a GPU or a Mac with unified memory can feel so much faster than a standard CPU-based setup.

When to Use Local LLMs on Android

Not every AI feature needs to be local. For business owners, the decision to run a model on-device should be driven by the specific requirements of the task. Local LLMs are ideal for privacy-sensitive tasks where data must stay on the device to meet compliance or user trust standards.

They are also best suited for small, specific tasks. For instance, a retail app in India might use a local model to summarize customer feedback or provide smart replies to common queries without needing a constant internet connection. This reduces latency and saves on cloud API costs.

Conversely, if your application requires very large models (such as a 70B parameter model) for complex reasoning, mobile devices are not the right tool. Those use cases are better suited for dedicated hardware or cloud APIs. For more information on how different hardware handles these loads, see our guide on VRAM for LLMs.

Implementation Tips for Builders

If you are building an Android application with on-device AI, follow these practical guidelines to ensure a high-quality user experience:

  1. Focus on small models: Target models in the 1B to 4B parameter range. These are the "sweet spot" for mobile NPUs and provide a balance of intelligence and speed.
  2. Prioritize quantization: Use quantized formats to ensure the model fits into the shared system memory without crashing the app.
  3. Manage your context: Be mindful of the trade-off between model intelligence (parameter count) and device responsiveness. A larger model might be smarter, but its lower memory bandwidth will make it feel sluggish.
  4. Plan for headroom: Always leave extra RAM available for the system and the KV cache to prevent performance drops during long user interactions.

By understanding these hardware constraints, you can build AI features that feel fast, private, and reliable for your users.

Close-up of a computer processor chip mounted in a socket.
Photo: File:AMD Fiji GPU package with GPU, HBM memory and interpose by C. Spille/pcgameshardware.de, CC BY-SA 4.0

Questions people ask

Can I run large models like Llama 3 70B on an Android phone?

It is highly unlikely. A 70B model at 4-bit requires 35-42 GB of memory, which exceeds the capacity of most mobile devices. Android NPUs are designed for much smaller models (1B-4B parameters).

What is the benefit of using an NPU for local LLMs?

NPUs are small accelerators designed for low-power AI tasks. They allow the phone to perform tasks like transcription or summarization efficiently without draining the battery as much as a CPU would.

How does quantization affect my mobile LLM?

Quantization reduces the number of bits used per weight (e.g., from 16-bit to 4-bit). This makes the model small enough to fit into a phone's shared memory, though it may slightly impact quality.

Why does my LLM slow down during long conversations on mobile?

As the conversation grows, the KV cache (which stores past tokens) grows linearly. This consumes more memory and can lead to performance drops if you run out of headroom.

Do I need more RAM or more NPU speed for mobile AI?

For LLM inference, memory capacity is the first limit you hit (to fit the model), but memory bandwidth determines the actual speed of text generation.