EARTH: An Efficient MoE Accelerator with Entropy-Aware Speculative Prefetch and Result Reuse

Fangxin Liu, Ning Yang, Jingkui Yang, Zongwu Wang, Chenyang Guan, Yu Feng and 2 others

A Mixture-of-Experts language model saves computation by routing each word to only a few of its many expert sub-networks, but all those experts still have to live in memory, and shuttling the right ones onto the chip for every token becomes the real bottleneck; this paper (Shanghai Jiao Tong) presents EARTH, which stores each expert as a small always-cached base plus a precision-restoring delta, then speculatively preloads the bases of the experts it predicts and fetches deltas only where accuracy demands — reusing delta patterns it has seen before — and pairs this with a matching accelerator, reaching up to 2.10 times speedup while keeping accuracy high.

The argument in one glance

Problem

MoE saves computation but the experts still have to fit in memory

A Mixture-of-Experts model is a large language model whose feed-forward layers are split into many parallel sub-networks called experts. For each input token a small router picks just a few experts to run, so the amount of arithmetic per token stays low even as the model grows. The catch is that all of the experts must still exist somewhere in memory, and the total size of their parameters is enormous. On a big server this is manageable; on resource-constrained hardware it is not, because the experts do not all fit in fast on-chip or device memory. So the system must keep most experts in slower memory and move the ones a token actually needs onto the chip on demand. Which experts a token needs is decided at run time by the router, token by token, so this movement is unpredictable and frequent, and it is the data movement, not the computation, that now dominates the cost.
What people did

Offload experts to slow memory and prefetch or compress them

The standard responses are to offload experts to host or slower memory and fetch them when selected, to prefetch experts predicted to be needed so the transfer overlaps with computation, and to compress or quantize experts so each transfer moves fewer bytes.
Why it fell short

Each existing strategy fails a different way

Compressing experts too aggressively, or simply dropping some, loses model accuracy. Fetching full-size experts on demand or prefetching the wrong ones moves prohibitively large amounts of data, so memory traffic becomes the bottleneck the MoE structure was supposed to avoid. And schemes that add heavy bookkeeping or prediction work at decode time inflate the per-token latency until the compute savings are eaten up. No existing offloading or prefetching approach is simultaneously accurate, low-traffic, and low-overhead, which is what keeps MoE models off resource-constrained hardware.
This paper's idea

Split each expert into a cheap base and a precision-restoring delta, then prefetch and reuse cleverly

EARTH encodes each expert as two parts: a compact, always-available base that captures its high-information content, and a delta that restores full precision when needed, with precision managed adaptively. It then speculatively preloads the bases of predicted experts and fetches deltas only selectively — running important experts at full precision by fetching their delta, while unimportant experts reuse only their cached base to save bandwidth — and it reuses previously computed delta patterns to cut redundant traffic and work. A co-designed hardware accelerator schedules and parallelizes this encoding-and-prefetch scheme, reaching up to 2.10 times speedup while keeping accuracy high.

Why this problem is worth solving

The whole appeal of the architecture is that it decouples model size from per-token computation, and that same decoupling is what creates the memory problem.

A Mixture-of-Experts model replaces a dense feed-forward layer with many parallel expert sub-networks plus a small gating network, or router, that for each token selects a few experts to actually run. Because only a handful of experts fire per token, the computation per token is far smaller than the model total parameter count would suggest, which lets the model grow to enormous capacity without a proportional growth in arithmetic. The price is memory: every expert, whether or not it fires for a given token, must be stored, so the parameter footprint is large and dominated by experts that are idle at any moment. On hardware that cannot hold them all in fast memory, the experts a token needs must be brought in from slower storage on demand, and because the router chooses them dynamically and differently for each token, this traffic is hard to predict and expensive. The architecture trades computation for memory pressure, which is exactly the pressure EARTH sets out to relieve.

The reason offloading is even viable is that not every expert, and not every bit of an expert, matters equally to the output.

Adaptive precision is the idea of spending numerical accuracy where it changes the result and saving it where it does not, rather than representing every value at the same fixed width. In a neural network, some computations are sensitive — a small error propagates into a wrong output — while others are robust to approximation. Adaptive-precision methods keep the sensitive parts at high precision and represent the rest coarsely, cutting storage and data movement in proportion to how much precision is actually discarded. EARTH applies this idea at the granularity of an expert and of an expert internal decomposition: a low-precision base is enough for experts or computations that tolerate approximation, while a delta adds back precision only where it is needed. Deciding where to spend precision, rather than uniformly compressing, is what lets the scheme shrink memory traffic without paying for it in accuracy.

What people did before, and where it broke

Offload experts and fetch them on demand. Keep the full expert set in host or slower memory and transfer each selected expert to the accelerator when the router picks it. Where it broke. The transfers are large, unpredictable, and on the critical path of every token, so memory traffic and latency dominate, which is the very bottleneck the MoE compute savings were meant to sidestep.
Prefetch predicted experts. Guess which experts a token will need and start moving them early so the transfer overlaps computation. Where it broke. Predictions are imperfect, and moving full-size experts speculatively multiplies traffic when the guess is wrong; the mechanism helps latency only when accurate and otherwise wastes bandwidth.
Quantize or compress experts. Shrink each expert so every transfer moves fewer bytes and more experts fit in fast memory. Where it broke. Compress too little and the traffic is still large; compress too much, or drop experts entirely, and model accuracy falls — a uniform compression cannot tell the parts that matter from the parts that do not.
Add decode-time machinery to manage all this. Layer prediction, caching, and scheduling logic on top at inference time to coordinate offloading. Where it broke. The extra bookkeeping and prediction work inflate per-token decode overhead, and past a point the coordination costs more than the compute it saves.
The gap. No approach delivered all three of accuracy, low memory traffic, and low overhead at once, and none exploited the fact that an expert can be split into a cheaply-cached approximate part and a small precision-restoring correction, nor that the corrections recur in reusable patterns — leaving the accurate-yet-lightweight offloading that resource-constrained MoE needs unbuilt.

How EARTH is put together

Base-plus-delta experts, speculatively prefetched and reused on co-designed hardware

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Do not store an expert at one precision — split it into a cheap base and a precision-restoring delta. The failure mode of compression is that it is uniform: shrink every expert the same way and you either barely reduce traffic or you lose accuracy. EARTH central encoding instead decomposes each expert into two components of different information content — a compact base that holds the high-information part and a delta that adds back the precision needed for full accuracy — and manages precision adaptively across them. The consequence is that the two parts can be treated completely differently by the memory system: the base is small enough to keep cached and to move freely, while the delta is fetched only when the accuracy of a particular expert justifies its cost. Splitting a value into an always-available coarse version and an on-demand refinement, so the system can choose per case how much precision to pay for, is the same accuracy-scaling idea the inference-serving work covered elsewhere on this site is built on, dialing precision up only where it changes the answer and running coarser everywhere else to save resources.

Speculatively prefetch the cheap part, and only pay full price where it matters. Ordinary expert prefetching is risky because it moves whole experts on a guess, so a wrong prediction wastes large bandwidth. EARTH defuses that risk by prefetching only bases speculatively: because a base is small, preloading the bases of predicted experts costs little even when some predictions miss, so the latency-hiding benefit of prefetching is kept without its bandwidth penalty. Then, at the point where accuracy is actually decided, it fetches deltas selectively — important experts get their delta and run at full precision, while unimportant experts run on their cached base alone — so the expensive, precise transfers happen only where they change the result. Hiding memory latency by speculatively moving data ahead of when it is needed, while keeping the speculation cheap enough that mispredictions do not hurt, is the same discipline the tree-based speculative LLM-serving work covered elsewhere on this site relies on, speculating aggressively precisely because the cost of a wrong guess has been made small.

Notice that the corrections repeat, and reuse them. Even selective delta fetching would be wasteful if every token recomputed and re-fetched its deltas from scratch, so EARTH exploits a structural observation: the delta patterns recur across tokens, and a delta computed once can serve again. By caching previously computed delta patterns and reusing them, it removes both redundant memory transfers and redundant computation, compounding the savings from the base-plus-delta split and the speculative prefetch. Prefetching and preparing data before it is needed, and reusing what has already been brought in rather than re-fetching it, is the same memory-management instinct the unified-memory tensor migration-and-prefetching work covered elsewhere on this site is built on, moving and caching data ahead of demand so the same bytes are not paid for twice.

Following a token through EARTH

Serving one MoE token under offloading

An illustrative trace, based on the paper's abstract, of how EARTH handles a token whose router selects several experts that do not all reside in fast memory. Internal specifics beyond the abstract were not available for this draft.

Step 1 of 5

The bases of predicted experts are already resident

Before the token arrives, EARTH has speculatively preloaded the bases of the experts it predicts upcoming tokens will need. Because bases are compact, this preloading is inexpensive, so when the router selects experts, their approximate versions are already available in fast memory.

This is what removes the large, on-critical-path transfer that plain on-demand offloading pays for every token.

Why it works

Base-plus-delta breaks the accuracy-versus-traffic tie. Uniform compression must choose between too little traffic reduction and too much accuracy loss → but splitting an expert into a compact base and a precision-restoring delta lets each part be handled separately → so the system moves little by default and restores full precision only where it is needed.
Speculating on the cheap part removes the prefetch penalty. Prefetching whole experts wastes large bandwidth whenever a prediction misses → but speculatively preloading only small bases makes a wrong guess nearly free → so prefetching hides latency without the traffic blow-up that made it risky.
Selective deltas spend precision where it changes the output. Running every selected expert at full precision moves maximal data → but fetching deltas only for important experts and running unimportant ones on their base alone → so bandwidth is spent in proportion to how much accuracy each expert actually contributes.
Reusing delta patterns removes repeated work. Recomputing and re-fetching deltas every token duplicates effort that recurs across tokens → but caching and reusing previously computed delta patterns eliminates the repeats → so both redundant memory traffic and redundant computation are cut.
Co-designing the hardware realizes the savings. A clever encoding and prefetch scheme can stall on a general accelerator that cannot overlap its irregular transfers → but a co-designed accelerator tuned for this encoding, execution order, and parallelism keeps memory and compute busy together → so the paper reports the strategy turning into up to 2.10 times end-to-end speedup at high accuracy.

What it costs (trade-offs and scope)

This is a draft written without the paper’s main text. At the time of writing, the full paper was behind a publisher access wall (the ACM Digital Library PDF was not retrievable, there is no arXiv version, and the authors’ listed paper link is a placeholder), so this account is built from the paper’s abstract and the publisher’s abstract page. The one reported number, up to 2.10 times speedup, is the abstract’s own; the encoding details, precision levels, prediction method, accelerator microarchitecture, benchmark suite, and accuracy figures are not reproduced and should be read from the paper.
Unimportant experts run approximate by design. The bandwidth savings come partly from running some experts on their base alone, at reduced precision. The abstract states accuracy is maintained, but the trade-off is real: how important an expert is must be judged correctly, and a mis-judgment means either wasted precision or a degraded output; the robustness of that judgment across models and inputs is a question for the full evaluation.
The gains depend on prediction and pattern recurrence. Speculative prefetching helps to the extent expert usage is predictable, and delta reuse helps to the extent delta patterns actually recur; workloads whose routing is highly irregular or whose deltas rarely repeat would see less benefit, and the abstract does not characterize where those limits lie.
The encoding adds its own machinery. Decomposing experts into base and delta with adaptive precision, tracking importance, and caching delta patterns all require storage and control; the net win assumes this overhead stays well below the memory traffic it removes, which the co-designed hardware is meant to ensure but which this draft cannot quantify.
It targets a hardware accelerator, not a drop-in software library. The full benefit is reported for a co-designed accelerator, so the approach is a hardware-software co-design rather than a change deployable on commodity devices unchanged; how much of the benefit survives on existing hardware is outside what the abstract states.

The evaluation

Approaches to running an offloaded MoE model

Property	On-demand offloading	Compress or full-expert prefetch
What moves per selected expert
Effect on accuracy
Prefetch risk
Redundant work across tokens		Repeated fetches or compute
Reported end-to-end speedup	Baseline

A qualitative comparison based on the paper's abstract; the paper's main body, including its encoding details, prediction and importance criteria, accelerator microarchitecture, benchmarks, and accuracy measurements, was not available for this draft. The up-to-2.10-times speedup is the abstract's reported figure.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This is an honest draft assembled from the paper’s abstract and the publisher’s abstract page; the full text was not accessible, so the internal design — the dual-entropy encoding math, the adaptive precision levels, how experts and deltas are predicted and judged important, the delta-reuse mechanism, and the co-designed accelerator’s microarchitecture — is not reproduced, and no numbers beyond the abstract’s own have been introduced. The single reported result is up to 2.10 times speedup over state-of-the-art baselines while maintaining high accuracy. What is solid is the shape of the idea: store each MoE expert as a compact, cacheable base plus a precision-restoring delta; speculatively preload the cheap bases of predicted experts; fetch deltas only for experts important enough to warrant full precision, and run the rest on their base alone; reuse recurring delta patterns to cut redundant traffic and compute; and co-design an accelerator to schedule it all. Read the mechanism, the precision-versus-accuracy trade-off, and the evaluation against the paper itself; treat this page as a faithful account of the approach at the level of its abstract, not of its implementation or measured results.

Background

Why MoE serving is a memory problem, not a compute problem

It helps to see why a paper about a Mixture-of-Experts model spends its effort on memory rather than math. Large language model inference in general is memory-bound: generating each token requires reading a great many parameters from memory, and the arithmetic per byte read is low, so the processor spends much of its time waiting on memory rather than computing. Mixture-of-Experts makes this sharper. It was invented precisely to cut computation — activate only a few experts per token — but in doing so it multiplies the parameters that must be stored while leaving most of them idle at any instant, so the model total footprint balloons even as its per-token compute shrinks. On a device that cannot hold every expert in fast memory, the question of the hour is no longer how fast can we multiply but which parameters do we move, and when. That reframing, where serving a model well is chiefly about feeding memory to the compute rather than the compute itself, is the same pressure the memory-centric LLM-inference acceleration work covered elsewhere on this site responds to, putting computation closer to memory so the parameter traffic stops being the wall. EARTH answer is to make the traffic itself smaller and smarter — move a cheap approximation always, and the expensive precision only where it counts.

If you remember only three things

1. Mixture-of-Experts models cut per-token computation by activating only a few experts, but every expert must still be stored, so on hardware that cannot hold them all the system must offload experts and fetch the router-selected ones on demand; existing offloading, prefetching, and compression strategies each fail differently — losing accuracy, moving too much data, or adding too much decode overhead — which keeps MoE models off resource-constrained hardware.
2. EARTH is a hardware-software co-design with three parts: a dual-entropy encoding that splits each expert into a compact high-information base and a precision-restoring delta with adaptive precision; a delta-aware speculative prefetch-and-reuse mechanism that preloads the cheap bases of predicted experts, fetches deltas only for important experts (running unimportant ones on the base alone), and reuses recurring delta patterns; and a co-designed accelerator that optimizes execution order, parallelism, and memory use.
3. The abstract reports up to 2.10 times speedup over state-of-the-art baselines while maintaining high accuracy, by making the per-expert data movement both smaller (base plus optional delta) and smarter (speculate cheaply, spend precision selectively, reuse repeats). This explainer is a draft from the abstract without the paper body, so the encoding details, importance and prediction criteria, accelerator microarchitecture, benchmarks, and accuracy numbers should be confirmed against the paper.

Check yourself

Check your understanding

1. Why do Mixture-of-Experts models create a memory bottleneck even though they reduce computation?
2. What is the dual-entropy (base-plus-delta) encoding, and why does it help?
3. How does EARTH make speculative prefetching worthwhile despite imperfect predictions?
4. What should a reader keep in mind about this particular explainer?

Sources

The paper is “EARTH: An Efficient MoE Accelerator with Entropy-Aware Speculative Prefetch and Result Reuse” by Fangxin Liu, Ning Yang, Jingkui Yang, Zongwu Wang, Chenyang Guan, Yu Feng, Li Jiang, and Haibing Guan (Shanghai Jiao Tong University), published at ASPLOS 2026 (Volume 2), DOI 10.1145/3779212.3790155.

This explainer is a draft. The paper’s main text was not accessible at the time of writing (the ACM Digital Library PDF returned a bot-verification challenge, there is no arXiv version, no author preprint was found, and the authors’ own publication listing links to the paper with a placeholder), so it is written from the paper’s abstract and the publisher’s abstract page. The abstract supplies the problem framing (MoE models reduce computation by activating a subset of experts per token but introduce severe memory bottlenecks from the large number of expert parameters, with existing offloading and prefetching strategies incurring accuracy loss, prohibitively high memory traffic, or high decoding overhead) and the three innovations: a dual-entropy encoding that decomposes each expert into a high-information base and a delta component for compact storage with accuracy preserved via adaptive precision management; a delta-aware speculative prefetching and reuse mechanism that preloads base components of predicted experts and selectively fetches deltas, reusing previously computed delta patterns to reduce memory traffic and redundant computation; and a co-designed hardware accelerator that optimizes execution order, parallelism, and memory utilization. The publisher’s abstract page adds that the prefetch mechanism loads delta parts to enable full-precision computation for important experts while reusing only cached base parts for unimportant experts to save bandwidth. The reported result — up to 2.10 times speedup over state-of-the-art baselines while maintaining high model accuracy — is the abstract’s own figure. The paper’s encoding details, precision levels, prediction and importance criteria, accelerator microarchitecture, benchmark workloads, and accuracy measurements are not reproduced here and should be read from the paper; no numbers beyond the abstract’s have been introduced.