Fast On-device LLM Inference with NPUs

On-device LLM tasks read long personalized prompts and emit a handful of tokens, so PREFILL — not decoding — eats 88–99% of latency, and the one mobile chip built for exactly that math, the NPU, sat unused because its static graphs, per-tensor-only quantization, and weak floating point clash with how LLMs work; llm.npu closes all three gaps with fixed-size chunk-sharing graphs, shadow execution of the 0.1–0.3% outlier channels on the CPU, and out-of-order subgraph scheduling — delivering 22.4× faster prefill, 30.7× energy savings, and the first 1,000+ tokens/sec prefill on a phone.

Why this problem is worth solving

The on-device LLM story is usually told as a privacy story — your emails, screenshots, and messages never leave the phone — but this paper’s contribution begins with a measurement that inverts the field’s assumptions. Datacenter LLM serving is dominated by the decode stage (autoregressive, memory-bound, one token per pass), and this site’s datacenter papers optimize accordingly. Mobile workloads flip the ratio: a UI-automation agent ingests an entire screen’s view hierarchy to emit a three-token action; an email auto-replier reads 1,800 tokens of history to write two sentences. Long input, tiny output — so

That reframing points at underused silicon. Every flagship SoC ships an

What people did before, and where it broke

• Decode-side acceleration (activation sparsity like PowerInfer-v2, speculative decoding): real gains, wrong stage — on DroidTask-style workloads, decode is 1–12% of latency.
• CPU/GPU engines (llama.cpp, MNN, TFLite, MLC-LLM): mature, but mobile CPUs/GPUs are parallelism-starved by design; the 8.1-second UI step is llama.cpp on a flagship CPU doing its best.
• The NPU’s three refusals. (1) Static shapes: NPU toolchains compile and heavily optimize a fixed-shape graph; per-prompt-length rebuilds cost ~11 s (Gemma-2B) — longer than the inference — and max-length padding wastes the TOPS. (2) Per-tensor quantization only: LLM activations carry
  rare activation channels with magnitudes far beyond the rest — a transformer quirk that wrecks uniform INT8 quantization, because one scale factor stretched to cover outliers crushes everything else’s precision. State-of-the-art quantizers answer with per-GROUP scales, which mobile NPUs cannot execute natively
  ; emulating per-group MatMul as sub-MatMuls plus float reductions costs up to 10.7×. (3) Floating point remains: LayerNorm, attention softmax, and dequantization stay FP even in aggressively quantized models, and NPUs handle FP poorly — naive CPU offload of those ops serializes the pipeline.

How llm.npu works

The idea

Static hardware, dynamic workload — so quantize the dynamism. The NPU’s compile-once model and the prompt’s variable length look irreconcilable until you notice that causal attention already processes text left-to-right: nothing about correctness requires the whole prompt in one graph. Chunking discretizes prompt length into a single reusable graph shape — the CNN-era constraint is satisfied by re-batching the language rather than re-engineering the hardware. The memory insight completes it: what varies per chunk is only attention’s KV geometry; the parameter-heavy operators (FFN, projections) are shape-identical across chunks and can be loaded once. The chunk size itself (256) is an empirical sweet spot — long enough to saturate the NPU’s parallelism, short enough to bound last-chunk padding.

Split the tensor where the hardware splits. Shadow outlier execution is an exact algebraic decomposition, not an approximation: clip-to-INT8 on the NPU plus the clipped-off remainder in float on the CPU, summed at the end. What makes it a systems idea is recognizing that the remainder is tiny (0.1–0.3% of channels), stable (hot channels persist across inputs), and mostly ignorable (importance profiling licenses pruning 85% of layers’ outliers outright) — so the accurate path costs almost nothing in time, memory, or synchronization. This is the same shape of insight as per-group quantization, but re-partitioned along the hardware boundary instead of the tensor boundary: NPUs get the uniform math they like, CPUs get the sparse exceptional math they’re fine at.

Chunking’s second dividend: an instruction window. Once execution is (chunk × subgraph) units with explicit dependencies, the classic ILP playbook applies — llm.npu’s out-of-order scheduler is a dataflow engine over transformer pieces, filling NPU stalls with whichever CPU-side subgraph unlocks the most downstream NPU work. The heuristic’s framing is the mature part: since the NPU is the critical path (its work is ~2× the CPU’s), the objective is not CPU utilization but NPU stall reduction — another instance of optimizing the end-to-end surface rather than a per-processor proxy.

A 1,024-token UI-automation prompt, through llm.npu

graph prep: offline, once — not per prompt
shared static ops: one copy of FFN/linear weights across all chunks

Why it works

• Mobile tasks read much and write little → prefill dominates (88–99%) and prefill is compute-bound → the NPU’s integer TOPS are the right resource, and the whole problem reduces to feeding them despite CNN-era interface constraints.
• Causal attention processes tokens left-to-right anyway → fixed-size chunks on pre-built graphs preserve semantics exactly → the static-shape constraint is satisfied without rebuild latency or padding waste, and sharing length-independent operators keeps memory at ~1× model size.
• The quantization-hostile part of activations is tiny, spatially concentrated, and mostly unimportant → exact algebraic splitting sends 99.8% of the work to the NPU as uniform INT8 and 0.2% to the CPU as sparse float → per-group accuracy at per-tensor speed, with sync eliminated for 85% of layers by importance pruning.
• Chunking multiplies schedulable units with clean dependencies → out-of-order execution fills the 37% bubbles that in-order CPU/NPU overlap leaves → the heterogeneous pipeline behaves like a dataflow machine whose objective is NPU-stall minimization, the true critical path.
• Everything is post-training and engine-level → no model retraining, no hardware changes, five model families on commodity phones → the gap closed is purely systems impedance-matching, which is why it generalizes.

What it costs

• Vendor-stack coupling. Built on Qualcomm QNN and Hexagon; the techniques are principled but the 10K lines of C/C++/ assembly are per-ecosystem work, and other NPUs (Edge TPU, Apple ANE) have their own toolchains and quirks.
• Prefill-only. Decoding — memory-bound, GEMV-shaped — is untouched and stays on CPU/GPU; workloads with long outputs (chat generation) see proportionally less benefit than UI-automation-style read-heavy tasks.
• Offline profiling per model. Outlier importance and hot channels are profiled on large corpora offline; chunk length is tuned per device. Distribution-shifted inputs could in principle surface unprofiled outliers.
• Approximation at the margins. Pruning 85% of layers’ outliers and clipping to INT8 is validated to under-1% accuracy loss on the evaluated benchmarks — empirical, per-model assurance rather than a guarantee.
• Disk-in-the-loop. Cold outlier weights stream from storage; the overlap hides it in measurements, but it couples inference latency to flash behavior under memory pressure.
• Chunk-size padding. The last chunk pads to 256; very short prompts lose proportionally more, which is why the wins are quoted at long-prompt operating points.

The evaluation

Five mobile-sized models (Qwen1.5-1.8B, Gemma-2B, phi-2 2.7B, LLaMA-2-7B, Mistral-7B), four benchmark suites spanning the motivating applications (DroidTask UI automation, LongBench context-aware generation, Persona-Chat summarization), two commodity phones, five baselines covering industrial engines and research prototypes. Accuracy is held within 1% of FP16 throughout — the quantization tricks are validated, not assumed.

Check yourself

Sources

• arXiv preprint 2407.05858 (v2) — full text used for this explainer
• ACM DL: Fast On-device LLM Inference with NPUs — ASPLOS 2025
• mllm engine on GitHub