MoE-APEX: An Efficient MoE Inference System with Adaptive Precision Expert Offloading

Peng Tang, Jiacheng Liu, Xiaofeng Hou, Yifei Pu, Jing Wang, Pheng-Ann Heng and 2 others

Mixture-of-Experts language models are efficient because each word only uses a few "expert" sub-networks, but the full set of experts is huge, so on a phone or other small device you must keep most experts in slow memory and load them on demand — and that loading stalls the model; this paper (Shanghai Jiao Tong University and the Chinese University of Hong Kong — Peng Tang, Chao Li, Minyi Guo, and colleagues) presents MoE-APEX, which loads a smaller low-precision version of the less-important experts so they arrive faster, with almost no accuracy loss, speeding decoding up to 9.75 times.

The argument in one glance

Problem

Mixture-of-Experts models are efficient to compute but too big to fit on a small device

Mixture-of-Experts (MoE) is a popular way to build large language models efficiently: instead of one big feed-forward network, each layer has many "expert" sub-networks and a router that sends each token to only a few of them. This means a model can have a huge total number of parameters — great for quality — while only a small fraction are actually used to process any given token, keeping the computation cheap. The catch is memory. Even though only a few experts run per token, the model still has to store all of them, and the full set of experts is enormous. On a server with lots of GPU memory that is fine, but on a memory-constrained edge device — a phone, a laptop, an embedded system — the complete set of experts simply does not fit. So deploying MoE models where they are most useful, close to the user, runs straight into a memory wall: the whole point of MoE is to get high quality without high compute, but the storage cost undoes that on small devices. The question is how to run an MoE model whose experts are far larger than the device's fast memory, without either giving up the model or grinding to a halt.
What people did

Offload experts to slow memory and load them on demand

The standard fix is expert offloading: keep the experts in a larger but slower memory — the CPU's RAM, or storage — and load only the experts a token actually needs into fast device memory, on demand, as the model runs. A small cache of recently-used experts stays resident in fast memory, and when a token routes to an expert that is already there (a cache hit) it runs immediately. This lets a model whose experts vastly exceed the fast memory run on a device that could never hold them all at once.
Why it fell short

Loading a missing expert stalls inference, and every miss pays full cost

Offloading solves the memory problem but creates a latency problem. When a token routes to an expert that is not currently resident — a cache miss — the system must fetch that expert's full weights from the slow memory into fast memory before the token can proceed, and that transfer is slow. On an edge device, where the link to the slow memory has limited bandwidth, these loads stall inference badly, so traditional offloading can dramatically increase latency. Worse, the standard approach pays the same full loading cost for every missing expert, treating them all as equally important. But experts are not equally important to a given token's output: some contribute much more than others, and loading a large, less-important expert at full precision, just to have it barely affect the result, is a poor trade. What was missing was a way to make the unavoidable expert loads cheaper — especially for the experts that matter least — rather than paying the full transfer cost every time, and to orchestrate loading, prefetching, and caching intelligently across the levels of MoE computation.
This paper's idea

Load a low-precision version of the less-critical missing experts

MoE-APEX is an inference system whose core innovation is adaptive precision: it dynamically manages the numerical precision of the experts it loads. The key idea is that when a cache-miss expert is less critical to the current token, MoE-APEX loads a low-precision (quantized) variant of it instead of the full-precision one. A low-precision expert is smaller, so it transfers from slow memory much faster, cutting the loading latency — and because the expert was less critical anyway, using a coarser version of it barely affects the model's accuracy. So the system spends full precision (and full loading cost) only where it matters, and uses cheap low-precision loads for the experts that do not, turning the fixed loading cost into an accuracy-aware, adjustable one. To exploit this mixed-precision offloading fully, MoE-APEX adds three techniques that map onto the natural hierarchy of MoE computation — token, layer, and sequence. A token-level dynamic expert loading mechanism decides, for each token, which experts to load and at what precision. A layer-level adaptive expert prefetching technique fetches the experts an upcoming layer will need ahead of time, so the transfer overlaps with computation instead of stalling it. And a sequence-level cost-aware expert caching policy decides which experts to keep resident across the whole sequence, weighing the cost of reloading a given expert against the memory it occupies. Implemented on top of Llama.cpp, MoE-APEX reports decoding speedups from 1.34 to 9.75 times over state-of-the-art MoE offloading systems across diverse edge devices, while maintaining accuracy.

Why this problem is worth solving

Running capable language models locally on personal devices is increasingly desirable, and MoE is a leading way to get quality cheaply — if its memory cost can be tamed.

There are strong reasons to run language models on the device rather than in the cloud: privacy, offline use, low latency, and cost. Mixture-of-Experts is one of the most promising architectures for this, because it delivers the quality of a very large model while only computing with a small active fraction per token, which suits a device’s limited compute budget. But the architecture’s memory footprint — it must store every expert even though it uses few — is exactly what edge devices cannot afford. So the very design that makes MoE attractive for the edge is also what blocks it there, and bridging that gap determines whether powerful MoE models can actually run on the hardware in people’s hands. Because on-device inference is a large and growing use case, and because MoE is central to modern high-quality models, making MoE deployable on constrained devices unlocks a broad class of capabilities where they are most valuable. The efficiency of the offloading system, not just the model, is what decides whether it is usable.

The insight is that not all experts deserve equal effort, so precision should follow importance rather than being uniform.

Traditional offloading treats every expert load identically: whatever expert a token needs, if it is missing, fetch its full weights at full precision. That uniformity is the waste. In an MoE layer, the router does not just pick experts, it weights them, and the experts differ greatly in how much they influence a token’s output — some are pivotal, others marginal. Loading a marginal expert at full precision, paying the full transfer cost, so that it makes a negligible difference to the result, is spending the scarcest resource (memory bandwidth on the edge) on the least valuable work. MoE-APEX’s core realization is that precision is a knob you can turn per expert per token: keep it high for the critical experts, whose accuracy you need, and drop it for the less critical ones, whose smaller low-precision form loads far faster while costing almost nothing in accuracy. This converts the loading bottleneck from a fixed tax into something the system can trade against accuracy, expert by expert. The reason the three hierarchical techniques matter is that this trade has to be made at the right granularity — which experts and precisions per token, what to prefetch per layer, what to keep cached per sequence — because MoE computation naturally has those three levels, and orchestrating the mixed-precision loads across all of them is what turns the idea into a full speedup.

What people did before, and where it broke

Keep the whole MoE model in fast memory. Store all experts on the device’s GPU or accelerator memory. Where it broke. The full set of experts is far too large for a memory-constrained edge device, so the model does not fit.
Offload experts and load them on demand. Keep experts in slow memory and fetch the needed ones into a small fast-memory cache. Where it broke. A cache miss must transfer an expert’s full weights over a slow, low-bandwidth link, stalling inference and greatly increasing latency.
Pay full precision for every loaded expert. Fetch each missing expert at full precision regardless of importance. Where it broke. It spends the same scarce loading bandwidth on marginal experts as on critical ones, wasting effort where accuracy barely benefits.
Load each expert only when the token reaches it. React to the router’s choice at the moment it happens. Where it broke. Loading synchronously at the point of need serializes the transfer with computation, so the stall is fully exposed.
The gap. No offloading system varied expert precision by importance to cut loading cost, and orchestrated loading, prefetching, and caching across the token, layer, and sequence levels of MoE computation.

How MoE-APEX works

Mixed-precision expert offloading across the MoE computation hierarchy

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Offload the model to slow memory and stream only what each token needs. MoE-APEX keeps the huge expert set in larger, slower memory and loads the active experts on demand, so a model far bigger than fast memory can run on a small device. Keeping a model too large for fast memory in a larger backing store and streaming in only the pieces the current computation needs is the same approach the work on streaming weights for modern DNN workloads on mobile GPUs (FlashMem) covered elsewhere on this site takes, treating capacity as something to stream rather than to hold. MoE-APEX applies it to MoE experts, where the natural unit to stream is the expert a token routes to.

Use lower precision where accuracy can spare it. MoE-APEX loads less-critical experts as smaller low-precision variants, spending full precision only on the experts that matter. Reducing numerical precision selectively to cut the cost of large-model inference, where a coarser representation is good enough, is the same principle the work on a metadata-augmented microscaling format for efficient low-bit LLM inference (M2XFP) covered elsewhere on this site is built on, trading bits for efficiency where the model tolerates it. MoE-APEX turns precision into a per-expert, per-token knob tied to importance and to loading cost.

Prefetch what is about to be needed to hide the transfer. MoE-APEX’s layer-level prefetching fetches an upcoming layer’s experts ahead of time so the slow load overlaps with computation rather than stalling it. Predicting what a computation will need and fetching it in advance to hide the access latency behind useful work is the same idea the work on critical-slice prefetching (CRISP) covered elsewhere on this site develops, running ahead of demand to mask a slow access. MoE-APEX prefetches experts a layer ahead, guided by the routing, so the expensive transfers do not sit on the critical path.

Following a token through MoE-APEX

Decoding a token with mixed-precision expert offloading

An illustrative walkthrough, based on the paper's abstract (this explainer is written without the full paper text; exact details are in the paper).

Step 1 of 5

The router selects experts for the token

As decoding reaches an MoE layer, the router picks the few experts this token should use and how much each should contribute, so some are more critical to the output than others.

Expert importance varies, which is what MoE-APEX exploits.

Why it works

Low-precision experts load faster. A cache miss must transfer an expert’s full weights over a slow link → but loading a smaller low-precision variant of a less-critical expert → so far less data moves and the loading latency drops.
Importance-matched precision protects accuracy. Dropping precision everywhere would hurt quality → but lowering precision only for experts that contribute little to the token → so accuracy is maintained while the cheap loads happen where they are safe.
Prefetching hides the transfer. Loading an expert at the moment it is needed stalls the pipeline → but fetching an upcoming layer’s experts ahead of time → so the slow transfer overlaps with computation instead of exposing the stall.
Cost-aware caching keeps the right experts resident. A naive cache might evict an expert that is expensive to reload → but weighing reload cost against occupied memory across the sequence → so fast memory holds the experts whose absence would hurt most.
Matching the three levels exploits MoE’s structure. A single global policy misses the structure → but making loading decisions per token, prefetching per layer, and caching per sequence → so the mixed-precision strategy is applied at every granularity MoE computation naturally has.

What it costs (trade-offs and scope)

Low-precision experts do lose some accuracy. Using coarser variants of less-critical experts is a real approximation; the design relies on those experts being marginal enough that the loss is negligible, so a workload where the router’s less-weighted experts still matter could see quality slip.
It needs low-precision variants of experts available. Adaptive precision assumes quantized versions of experts exist to load; producing and storing these variants is extra setup and storage the plain full-precision model does not require.
Judging expert criticality must be fast and accurate. The gains depend on correctly identifying which cache-miss experts are less critical for a token; a wrong judgment either wastes bandwidth on a full-precision load or degrades a critical expert.
Prefetching and caching add orchestration. The token, layer, and sequence policies introduce coordination and prediction that a simple load-on-demand system lacks, and mispredicted prefetches or evictions cost bandwidth and memory.
Reported speedups are device- and model-specific. The 1.34-to-9.75-times range is wide, reflecting that the benefit varies a great deal across edge devices, models, and workloads; the exact gain depends on the memory constraints, the bandwidth, and how much low precision the model tolerates.

The evaluation

Running Mixture-of-Experts inference on memory-constrained edge devices

Property	Full model in fast memory	Traditional expert offloading	MoE-APEX
Fits on the device
Cache-miss loading
Latency hiding
Caching policy
Decoding speed

A comparison of ways to run MoE inference on edge devices. By loading low-precision variants of less-critical cache-miss experts (adaptive precision), and orchestrating loading per token, prefetching per layer, and cost-aware caching per sequence, MoE-APEX (built on Llama.cpp) reports decoding speedups from 1.34 to 9.75 times over state-of-the-art MoE offloading systems across diverse edge devices while maintaining accuracy. This explainer is written from the paper's abstract; exact figures and mechanisms should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is written from the paper’s abstract (the full text was not accessible at the time of writing — the ASPLOS 2026 paper is open access under CC BY, but the copy was reachable only through a publisher page that could not be retrieved, and no author-hosted copy or preprint was found). The described system is the paper’s: MoE-APEX is an Adaptive Precision EXpert offloading system that optimizes Mixture-of-Experts inference for edge architectures by dynamically managing expert precision. Its core innovation is to replace less critical cache-miss experts with low-precision variants, reducing loading latency while maintaining accuracy. It introduces three techniques mapping the natural hierarchy of MoE computation: (1) a token-level dynamic expert loading mechanism, (2) a layer-level adaptive expert prefetching technique, and (3) a sequence-level cost-aware expert caching policy, which together let it exploit mixed-precision expert inference. Implemented atop Llama.cpp, the paper reports decoding speedups ranging from 1.34 to 9.75 times compared to state-of-the-art MoE offloading systems across diverse edge devices. The background above on MoE routing, expert offloading, cache hits and misses, quantization/precision, and prefetching is standard context provided to make the approach intuitive; the paper’s own account of how expert criticality is judged, how the low-precision variants are produced and selected, and the details of the three techniques is the authority. The step-by-step walkthrough is illustrative, not a claim about the paper’s exact implementation. Exact mechanisms, the accuracy analysis, and full quantitative results should be read from the paper.

Background

Why running big models on small devices keeps becoming a memory-movement problem

It helps to see MoE-APEX within a broad pattern: as models grow far larger than the fast memory of the devices meant to run them, the dominant cost of inference shifts from computing to moving the model’s parameters into place. This is especially acute at the edge, where memory is small and the links to any larger backing store are slow, so the parameters that do not fit must be streamed in, and the time to move them dwarfs the time to use them. The recurring toolkit is the same across settings: keep the bulk in a larger store, bring in only what the current step needs, prefetch it ahead so the transfer hides behind computation, cache what will be reused, and — increasingly — shrink what must move by using a cheaper representation where accuracy allows. MoE-APEX combines all of these, with the distinctive twist that Mixture-of-Experts gives it a natural place to apply cheaper representations: the experts a token barely uses can be loaded coarsely without hurting the result. This effort to make demanding language-model inference fit and run on memory-limited edge devices is the same terrain the work on turbocharging NLP inference at the edge via elastic pipelining (STI) covered elsewhere on this site occupies, wringing usable inference out of tight device budgets. MoE-APEX’s contribution is to show that, for Mixture-of-Experts, adapting each loaded expert’s precision to its importance — and orchestrating that across the token, layer, and sequence levels — turns expert offloading from a latency trap into a fast, accuracy-preserving way to run big MoE models on small hardware.

If you remember only three things

1. Mixture-of-Experts (MoE) language models route each token to only a few of many expert sub-networks, so they compute cheaply, but they must store every expert, and the full set is far too large for a memory-constrained edge device. The standard fix, expert offloading, keeps experts in slower memory and loads the needed ones into a small fast-memory cache on demand — but a cache miss must transfer an expert's full weights over a slow, low-bandwidth link, which stalls inference and greatly increases latency. Traditional offloading pays this full loading cost for every missing expert, treating them all as equally important, even though experts differ greatly in how much they contribute to a token's output.
2. MoE-APEX's core innovation is adaptive precision: when a cache-miss expert is less critical to the current token, it loads a smaller low-precision (quantized) variant instead of the full-precision one. The low-precision expert transfers much faster, cutting loading latency, and because the expert was marginal anyway, using a coarser version barely affects accuracy — so precision (and loading cost) is spent only where it matters. To exploit this fully, MoE-APEX adds three techniques matched to the token, layer, and sequence levels of MoE computation: a token-level dynamic expert loading mechanism (what to load and at what precision), a layer-level adaptive prefetching technique (fetch an upcoming layer's experts ahead so the transfer overlaps computation), and a sequence-level cost-aware caching policy (keep resident the experts most costly to reload).
3. Implemented on top of Llama.cpp, the paper reports MoE-APEX achieves decoding speedups from 1.34 to 9.75 times over state-of-the-art MoE offloading systems across diverse edge devices, while maintaining accuracy. The trade-offs: low-precision experts do lose some accuracy (the design bets they are marginal enough that it is negligible), quantized variants of experts must be available and stored, judging which experts are less critical must be fast and correct, and the prefetching and caching add orchestration and prediction that a simple load-on-demand system lacks. The wide speedup range reflects strong dependence on the device, model, and how much low precision the workload tolerates. This explainer is based on the abstract; full details are in the paper.

Check yourself

Check your understanding

1. Why is deploying a Mixture-of-Experts model on a memory-constrained edge device hard, even though MoE is computationally efficient?
2. What is the latency problem with traditional expert offloading, and what does it fail to exploit?
3. What is MoE-APEX's core innovation of "adaptive precision"?
4. What are MoE-APEX's three techniques, and what do they achieve?

Sources

The paper is “MoE-APEX: An Efficient MoE Inference System with Adaptive Precision Expert Offloading” by Peng Tang, Jiacheng Liu, Xiaofeng Hou, Yifei Pu, Jing Wang, Pheng-Ann Heng, Chao Li, and Minyi Guo (Shanghai Jiao Tong University and the Chinese University of Hong Kong), published at ASPLOS 2026, DOI 10.1145/3779212.3790187. The paper is open access under CC BY.

This explainer is written from the paper’s abstract; the full text was not accessible at the time of writing (the open-access copy was reachable only through a publisher page that could not be retrieved, and no author-hosted copy or preprint was found). MoE-APEX is an Adaptive Precision EXpert offloading system for Mixture-of-Experts inference on edge devices that dynamically manages expert precision. Its core innovation is to replace less critical cache-miss experts with low-precision variants, reducing loading latency while maintaining accuracy. It introduces three techniques mapping the natural hierarchy of MoE computation: a token-level dynamic expert loading mechanism, a layer-level adaptive expert prefetching technique, and a sequence-level cost-aware expert caching policy, which together let it exploit mixed-precision expert inference. Implemented atop Llama.cpp, the paper reports decoding speedups ranging from 1.34 to 9.75 times compared to state-of-the-art MoE offloading systems across diverse edge devices. The background on MoE routing, expert offloading, quantization, and prefetching is standard context; exact mechanisms and full results should be confirmed in the paper.