LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training

Xinyi Liu, Yujie Wang, Fangcheng Fu, Xuefeng Xiao, Huixia Li, Jiashi Li and 1 other

Mixture-of-Experts models train fast because each token only uses a few "expert" sub-networks, but the router sends far more tokens to some experts than others, so the devices holding those hot experts become stragglers everyone waits for — and rebalancing by moving experts around is so expensive it is done rarely; this paper (Peking University and ByteDance — Xinyi Liu, Yujie Wang, Bin Cui, and colleagues) presents LAER-MoE, which shards every expert across all devices so an expert can be re-placed almost for free, letting the system rebalance the load every single iteration.

The argument in one glance

Problem

A few overloaded experts stall the whole Mixture-of-Experts training run

Mixture-of-Experts (MoE) models are how you train enormous language models affordably: instead of one big feed-forward layer, an MoE layer has many "expert" sub-networks plus a gating network that routes each token to only a few experts, so a model can hold trillions of parameters while each token activates only a small fraction of them. Because the experts together are far too big for one device, MoE is trained with expert parallelism — each device stores a subset of the experts, and in every layer tokens are shuffled across the network (an All-to-All communication) to the devices holding their chosen experts, computed there, and shuffled back. The problem is that the gating network does not spread tokens evenly. Dynamic routing sends far more tokens to a handful of "hot" experts than to the rest, and which experts are hot changes from iteration to iteration. The devices hosting those overloaded experts have much more work to do, so they take longer, and because every device must synchronize at the end of each iteration, everyone else sits idle waiting for the stragglers. This load imbalance shows up as large tail latency and communication delays, and it recurs at almost every iteration of training — the paper measures it while training Mixtral 8x7B. A small number of overloaded experts thus become the bottleneck for the entire, very expensive training run.
What people did

Rebalance by replicating hot experts or relocating experts across devices

The natural fixes rearrange where experts live. Expert replication copies a hot expert onto more devices so its tokens can be split among the copies, spreading the load. Expert relocation migrates experts between devices to even things out — moving an expert's parameters (and optimizer states) to a less busy device. Systems like SmartMoE, Prophet, and FlexMoE combine these: they dynamically adjust which experts are replicated and where experts are placed, trying to track the shifting load.
Why it fell short

Re-layout is treated as a costly separate phase, so rebalancing is done rarely

Both fixes are expensive, and prior systems treat the rearrangement — the re-layout — as its own separate phase, which forces a painful trade-off. Expert replication needs extra communication to synchronize the gradients of the replicated experts, and that communication is itself imbalanced because experts are not replicated on every device. Expert relocation is worse: migrating an expert's parameters and optimizer states moves a communication volume of roughly six times the expert's parameter size, and it inflates peak memory because the send and receive buffers must be allocated at once. Because re-layout is so costly, existing methods have to hold back: SmartMoE only relocates every few hundred iterations, while Prophet and FlexMoE penalize any adjustment that would incur high overhead. The result is that they cannot respond in time to a routing distribution that changes every iteration — they are always rebalancing for a load that has already moved on. The field was stuck in a trade-off between how well you balance and how much the balancing costs. What was missing was a way to make re-layout so cheap that you could do it every iteration, in step with the constantly shifting load.
This paper's idea

Shard every expert across all devices so re-layout becomes almost free

LAER-MoE breaks the trade-off with a new parallel paradigm called Fully Sharded Expert Parallel (FSEP). Instead of each expert living entirely on one device, FSEP fully partitions every expert's parameters across all devices — each device holds only a slice of every expert — and then restores the experts it needs, at expert granularity, on demand through All-to-All communication during training. The consequence is transformative: because an expert's parameters already live, in shards, on every device, deciding which device should compute an expert no longer requires migrating six-times-the-parameters of data. Re-layout becomes a matter of where you gather the shards, and that gather is folded into the All-to-All communication the MoE layer already performs. So re-layout is no longer a costly separate phase; it is essentially free and can be done every iteration. On top of FSEP, LAER-MoE adds two pieces. It performs fine-grained scheduling of the communication operations to minimize the overhead of all these gathers, overlapping and ordering them carefully. And it uses a load-balancing planner that, each iteration, formulates both the re-layout strategy for the experts (which device computes each expert) and the routing scheme for the tokens (which tokens go where), in response to the current, measured load. Because it can rebalance in lockstep with the changing routing distribution rather than lagging far behind it, LAER-MoE keeps devices evenly loaded. On an A100 cluster, it reports up to 1.69 times faster training than state-of-the-art MoE training systems.

Why this problem is worth solving

MoE is how the largest models are trained affordably, and load imbalance directly wastes a large share of that training’s very expensive compute.

Mixture-of-Experts is the dominant way to scale model parameters without scaling compute proportionally, which is why the biggest models use it — but the whole efficiency argument depends on the experts actually being kept busy. When a few overloaded experts stall their devices every iteration, the rest of a large, costly GPU cluster sits idle waiting, so a substantial fraction of the training run is simply wasted. Because these runs occupy thousands of expensive accelerators for weeks, even a moderate improvement in how evenly the load is spread translates into a large amount of reclaimed compute and time. And the imbalance is not an occasional hiccup; the paper shows overloaded experts appearing at nearly every iteration, with the hot set shifting constantly, so the waste is continuous. Fixing it is therefore not a corner case but a first-order lever on the cost of training the models that most depend on MoE. A system that keeps the experts evenly loaded turns the theoretical efficiency of MoE into realized efficiency on the hardware.

The crux is that the load changes faster than expensive rebalancing can keep up, so the rebalancing itself must become cheap.

The routing distribution in MoE training is genuinely dynamic — which experts are hot changes from one iteration to the next — so any effective balancing scheme has to adjust continuously. But every prior approach makes adjustment expensive: relocating an expert moves several times its parameter size across the network and spikes memory, and replicating one adds imbalanced gradient-synchronization traffic. When rebalancing is costly, you are forced to do it rarely, which means you are always correcting for a load that has already changed — you chase the distribution and never catch it. This is the trap the field was in, and it is why prior systems capped their relocation frequency or penalized high-overhead moves. The only way out is to change the cost of re-layout itself, not just schedule it more cleverly. LAER-MoE’s insight is that if every expert’s parameters are already sharded across all devices, then moving an expert’s computation somewhere else is nearly free, because nothing has to be migrated — the data is already there, and gathering it is the communication the layer was going to do anyway. Once re-layout is cheap, the trade-off between balancing quality and balancing cost dissolves, and you can finally rebalance every iteration, in time with the load. That is why the paradigm change, rather than a better scheduler on top of the old paradigm, is the essential move.

What people did before, and where it broke

Static expert parallelism, one expert per device. Assign each expert to a device and route tokens to it. Where it broke. Dynamic routing overloads a few experts, so their devices become stragglers and everyone waits at each iteration’s synchronization.
Replicate hot experts across more devices. Copy an overloaded expert so its tokens split among copies. Where it broke. It adds gradient-synchronization communication for the replicas, and that traffic is itself imbalanced because experts are not replicated on every device.
Relocate experts to less busy devices. Migrate expert parameters and optimizer states to rebalance. Where it broke. Moving an expert costs about six times its parameter size in communication and spikes peak memory from simultaneous send and receive buffers.
Treat re-layout as a separate phase and do it sparingly. Adjust every few hundred iterations, or penalize high-overhead moves (SmartMoE, Prophet, FlexMoE). Where it broke. The load changes every iteration, so infrequent rebalancing always lags the actual distribution.
The gap. No system made re-layout cheap enough to rebalance every iteration in step with the constantly shifting routing distribution.

How LAER-MoE works

Fully sharding experts so the load can be rebalanced every iteration

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Do not own the whole parameter; keep it sharded and gather what you need. FSEP’s foundation is that no device holds a whole expert; every expert is partitioned across all devices and reconstructed on demand, so an expert’s computation can move anywhere without migrating its parameters. Keeping parameters distributed and gathering only the piece needed for the current work, rather than statically owning the whole thing, is the same approach the work on tensor migration and prefetching in unified memory (DeepUM) covered elsewhere on this site takes, fetching the needed tensors on demand instead of holding everything in place. LAER-MoE applies it across devices to experts, which is exactly what makes re-placing an expert cheap.

Rebalance an uneven workload by shifting where the work runs, continuously. LAER-MoE’s planner reacts to the measured per-expert load each iteration, moving expert computation to even out the stragglers as the hot set changes. Continuously shifting load away from overloaded units toward idle ones, tracking a demand that keeps moving, is the same idea the work on dynamic load shifting among microservice containers (Embracing Imbalance) covered elsewhere on this site develops, redistributing work to match imbalance as it evolves. LAER-MoE does this for MoE experts, and its cheap re-layout is what lets the shifting happen every iteration rather than rarely.

Hide the cost of the extra communication behind careful scheduling. Reconstructing experts by All-to-All adds communication, so LAER-MoE finely schedules and overlaps those operations to keep the overhead small. Scheduling and overlapping communication so its cost is masked rather than paid serially is the same principle the work on overlapping communication with dependent computation covered elsewhere on this site is built on, ordering the transfers so they hide behind other work. LAER-MoE’s fine-grained communication scheduling is what keeps the gathers from eating the gains that flexible re-layout provides.

Following an iteration through LAER-MoE

Rebalancing overloaded experts within a single training iteration

A trace, drawn from the paper, of how LAER-MoE keeps devices evenly loaded each iteration.

Step 1 of 5

The router produces an uneven token distribution

In this iteration the gating network routes far more tokens to a few hot experts than to the rest, so if the experts were placed statically, the devices computing the hot experts would become stragglers.

The hot set differs from last iteration, as routing is dynamic.

Why it works

Full sharding removes the cost of moving an expert. Relocating a whole expert moves about six times its parameters → but with every expert already sharded on every device → so re-placing an expert is just a gather, not a migration.
Cheap re-layout dissolves the balancing trade-off. Prior systems balanced rarely because balancing was expensive → but folding re-layout into the layer’s existing All-to-All → so rebalancing can happen every iteration without a costly separate phase.
Per-iteration planning tracks the shifting load. The hot experts change each iteration, so infrequent rebalancing lags → but re-planning re-layout and routing every iteration from the measured load → so the balance stays matched to the current distribution.
Scheduling keeps the extra communication cheap. Reconstructing experts by All-to-All adds traffic that could erode the gains → but fine-grained scheduling and overlapping of those operations → so the gather overhead is masked and the net effect is faster training.
Spreading hot experts cuts the tail that stalls everyone. One overloaded device forces all others to wait at synchronization → but distributing the hot experts’ work across idle devices → so devices finish together and the tail latency that dominated iterations shrinks.

What it costs (trade-offs and scope)

Full sharding turns expert access into communication. Because no device owns a whole expert, every use of an expert requires gathering its shards over the network; LAER-MoE schedules this carefully, but the paradigm trades local parameter ownership for extra All-to-All traffic, so it depends on that communication being well hidden.
The gains depend on real, exploitable imbalance. LAER-MoE profits from spreading hot experts across idle devices; a workload whose routing happens to be already balanced offers little to rebalance, so the benefit scales with how uneven and dynamic the routing is.
The planner must be fast and good. Deciding re-layout and token routing every iteration adds planning work on the critical path, and a poor plan wastes the flexibility; the approach relies on the planner producing good strategies quickly from the measured load.
It targets expert-parallel MoE training specifically. FSEP is a paradigm for distributing MoE experts across devices; it is not a general technique for other model structures, and its benefits are tied to the MoE routing-and-All-to-All pattern.
Reported speedups are cluster- and model-specific. The up-to-1.69-times acceleration is measured on an A100 cluster against particular state-of-the-art systems and models; the exact gain depends on the model, the degree of imbalance, the cluster, and the interconnect.

The evaluation

Handling expert load imbalance in Mixture-of-Experts training

Property	Static expert parallelism	Replication or relocation (SmartMoE, FlexMoE)	LAER-MoE
Where an expert lives
Cost of re-layout
How often it rebalances
Tail latency from stragglers
Reported training speed

A comparison of ways to handle expert load imbalance in MoE training. By fully sharding every expert across all devices (FSEP) and restoring experts on demand through All-to-All, LAER-MoE makes expert re-layout nearly free — folded into the layer's existing communication — so it can rebalance every iteration in step with the dynamic routing distribution, with fine-grained communication scheduling and a per-iteration load-balancing planner. The paper reports up to 1.69 times faster training than state-of-the-art systems on an A100 cluster. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ arXiv copy, 2602.11686, of the ASPLOS 2026 paper). The described system is the paper’s: LAER-MoE is an efficient Mixture-of-Experts training framework built on a new parallel paradigm, Fully Sharded Expert Parallel (FSEP), which fully partitions each expert’s parameters across all devices and restores partial experts at expert granularity through All-to-All communication during training, enabling flexible re-layout of expert parameters to improve load balancing. It performs fine-grained scheduling of communication operations to minimize communication overhead, and uses a load-balancing planner to formulate re-layout strategies for experts and routing schemes for tokens during training. The paper motivates the work with the observation that dynamic routing causes significant, iteration-to-iteration load imbalance among experts (measured while training Mixtral 8x7B), that overloaded experts become stragglers causing tail latency, and that prior approaches (expert replication, expert relocation, and systems such as SmartMoE, Prophet, and FlexMoE) treat re-layout as a costly separate phase — relocation moving a communication volume of roughly six times the expert parameter size — forcing them to rebalance infrequently. The paper reports experiments on an A100 cluster showing up to 1.69 times acceleration over state-of-the-art training systems, and provides source code at github.com/PKU-DAIR/Hetu-Galvatron (laer-moe branch). The MoE and expert-parallelism background above is standard context provided to make the approach intuitive. The step-by-step walkthrough is an illustration of the described mechanism. Exact algorithms, the FSEP communication design, the planner, and the full evaluation should be read from the paper.

Background

Why scaling model training keeps turning into a communication and load-balancing problem

It helps to see LAER-MoE within a broad pattern: as models are spread across more and more devices, the hard part of training stops being raw computation and becomes keeping every device busy and every communication cheap. A distributed training run is only as fast as its slowest participant at each synchronization, so any imbalance — a straggler, a hot shard, an uneven partition — directly wastes the whole cluster, and any data that must move between devices competes with the useful work. Mixture-of-Experts sharpens both problems at once: its routing is deliberately sparse and therefore deliberately uneven, and its experts must be shuffled and gathered across the network constantly. The recurring lesson, visible across parallel-training research, is that the layout of parameters and the scheduling of communication are not implementation details but the central design problem, and that the best solutions often come from rethinking the parallel paradigm itself rather than patching an existing one. LAER-MoE is a clean instance: it does not add a smarter rebalancer on top of static expert parallelism, it changes how experts are laid out so that rebalancing becomes cheap in the first place. This effort to compose and rethink how a model is partitioned across devices to make distributed training efficient is the same terrain the work on composing SPMD partitioning strategies for machine learning (PartIR) covered elsewhere on this site occupies, treating the parallel layout as the thing to get right. LAER-MoE’s contribution is to show that fully sharding experts, and folding their re-layout into the communication the model already does, turns MoE’s notorious load imbalance from a fixed tax into something a system can correct every single iteration.

If you remember only three things

1. Mixture-of-Experts (MoE) models scale parameters cheaply by routing each token to only a few expert sub-networks, and they are trained with expert parallelism — each device holds some experts, and tokens are shuffled across the network (All-to-All) to the devices holding their experts. But the gating network's dynamic routing overloads a handful of experts, and which ones are hot changes every iteration, so the devices holding overloaded experts become stragglers that everyone waits for at each synchronization — large tail latency that recurs almost every iteration. Existing fixes (replicating hot experts, relocating experts) treat this rearrangement as a costly separate phase: relocating an expert moves about six times its parameter size and spikes memory, so systems rebalance only rarely and always lag the shifting load.
2. LAER-MoE introduces Fully Sharded Expert Parallel (FSEP): instead of whole experts living on single devices, every expert's parameters are partitioned across all devices, and the experts a device needs are restored on demand, at expert granularity, through All-to-All communication. Because an expert's parameters already live in shards everywhere, deciding which device computes an expert no longer requires migrating six-times-the-parameters of data — re-layout is just where you gather the shards, folded into the communication the MoE layer already does, so it is nearly free and can happen every iteration. On top of FSEP, LAER-MoE finely schedules the communication operations to minimize overhead, and uses a load-balancing planner that formulates the expert re-layout and token routing each iteration from the measured load.
3. On an A100 cluster, the paper reports LAER-MoE trains up to 1.69 times faster than state-of-the-art MoE training systems, by keeping devices evenly loaded in step with the constantly shifting routing rather than lagging behind it. The trade-offs: full sharding turns every expert access into communication that must be well hidden by scheduling, the gains depend on real and dynamic imbalance to exploit, the per-iteration planner adds work on the critical path and must produce good plans quickly, and the approach targets expert-parallel MoE training specifically. The reported speedup depends on the model, the degree of imbalance, and the cluster; the source code is public.

Check yourself

Check your understanding

1. Why does load imbalance arise in Mixture-of-Experts training, and why does it slow the whole run?
2. Why did prior approaches (expert replication and relocation) fail to keep up with the imbalance?
3. What is Fully Sharded Expert Parallel (FSEP), and why does it make re-layout cheap?
4. Beyond FSEP, what else does LAER-MoE add, and what does it achieve?

Sources

The paper is “LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training” by Xinyi Liu, Yujie Wang, Fangcheng Fu, Xuefeng Xiao, Huixia Li, Jiashi Li, and Bin Cui (Peking University, Shanghai Jiao Tong University, and ByteDance Seed), published at ASPLOS 2026, DOI 10.1145/3779212.3790180. It is available on arXiv as 2602.11686, with source code at github.com/PKU-DAIR/Hetu-Galvatron (laer-moe branch).

This explainer is written from the paper’s full text (the authors’ arXiv copy). LAER-MoE is an efficient MoE training framework built on Fully Sharded Expert Parallel (FSEP), which fully partitions each expert’s parameters across all devices and restores partial experts at expert granularity through All-to-All communication during training, enabling flexible, low-cost re-layout of experts to improve load balancing. It performs fine-grained scheduling of communication operations to minimize overhead, and uses a load-balancing planner to formulate expert re-layout strategies and token routing schemes each iteration. It addresses the load imbalance from dynamic MoE routing — where a few overloaded experts become stragglers causing tail latency — that prior methods (expert replication and relocation, in systems like SmartMoE, Prophet, and FlexMoE) handle only infrequently because they treat re-layout as a costly separate phase (relocation moving about six times the expert parameter size). Evaluated on an A100 cluster, the paper reports up to 1.69 times acceleration over state-of-the-art training systems. Numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.