MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training

Training a giant language model across tens of thousands of machines needs frequent checkpoints so a crash does not throw away days of work, but the popular Mixture-of-Experts design blows up the model's parameter count — and so its checkpoints — without adding compute, making the saves too big to hide behind training; this work from HKUST-Guangzhou builds MoC-System, which saves only a rotating subset of the experts each time (the sparse model barely notices the few missed updates), shrinking each checkpoint to the size of an ordinary model's and cutting the per-checkpoint overhead by up to 98.9 percent with no loss of accuracy.

The argument in one glance

Problem

Mixture-of-Experts models make checkpoints enormous

Training giant language models across tens of thousands of machines needs frequent checkpoints — periodic saves of the model state — so a crash does not throw away days of work. The popular Mixture-of-Experts design adds many expert sub-networks to raise quality without adding compute, but its parameter count, and so its checkpoint size, balloons, and the save can no longer hide behind training.
What people did

Shard and overlap the checkpoint save

Existing fault-tolerance work spreads each checkpoint across machines (sharding) and writes it in the background while training continues (asynchronous checkpointing), to hide the save cost.
Why it fell short

The MoE checkpoint is still too big to hide

Even sharded and asynchronous, the Mixture-of-Experts checkpoint is so large — because it must store all the many expert parameters — that its save cannot be fully overlapped with training, so it adds directly to the total training time and strains the storage system. Prior sharding also lacks tuning for the expert-parallel layouts MoE training uses.
This paper's idea

Save only a rotating subset of experts each time

MoC-System saves only a chosen subset of experts per layer each checkpoint (Partial Experts Checkpointing), fully saving the rest of the model, shrinking the checkpoint toward dense-model size. Because the sparse model barely notices a few missed expert updates, accuracy holds — bounded by a Lost-Tokens metric. Even sharding and a two-level memory-then-storage save then hide the rest, cutting per-checkpoint overhead by up to 98.9 percent.

Why this problem is worth solving

At the scale of modern training, saving state to survive crashes is unavoidable, and MoE makes it costly.

Training a large language model can run for weeks across thousands or tens of thousands of machines, and at that scale hardware faults happen regularly — a machine crashes, a network link drops. If training simply restarted from the beginning after each fault, it would never finish. So systems checkpoint: at regular intervals they save the model state — the parameters and the optimizer state, which together capture everything needed to resume — to durable storage. After a fault, training reloads the most recent checkpoint and continues from there, losing only the work since that save. The cost of checkpointing is the cost of writing that state out, and it must be paid repeatedly. For an ordinary dense model this is manageable, but the sparse Mixture-of-Experts design changes the picture. A Mixture-of-Experts model raises quality by adding a large number of expert sub-networks, and it stays cheap to run because only a few experts are used per input. The consequence is that its parameter count — and therefore its checkpoint — is vastly larger than a dense model with the same compute cost. Writing that huge checkpoint strains the distributed storage system, and even with the usual tricks of sharding and asynchronous saving, the save takes so long that it cannot be fully hidden behind ongoing training, so it lengthens the whole training run. Cutting that cost is what this work is about.

The key opening is that not all of those parameters need saving every time.

A Mixture-of-Experts layer replaces a single feed-forward network with many of them, called experts, plus a small gating network. For each input token, the gate picks only the top few experts to actually run, so the model has enormous capacity but activates only a sparse slice of it per token — hence sparse Mixture-of-Experts. This is why parameter count can grow far faster than compute. The insight this paper builds on comes from how MoE models behave under limited updates. Prior work on fine-tuning found that updating only the non-expert parameters can match the accuracy of updating everything, while updating only the experts hurts accuracy badly — meaning the experts individually are less sensitive to any particular batch of updates. This is attributed to the sparsity of MoE: because each expert is trained by only the tokens routed to it, and MoE models are trained on very large data, missing a limited number of an expert’s updates barely changes the model. That tolerance is the lever. If a checkpoint can skip saving some experts, the only thing lost on recovery is a limited number of those experts’ recent updates — and if that loss is kept small enough, the recovered model is just as accurate. The paper makes this precise with a metric, the Proportion of Lost Tokens, and finds accuracy stays at the fault-free level as long as that proportion stays at or below about 3.75 percent.

What people did before, and where it broke

Periodic full checkpointing (for dense models). Save the entire model state at intervals. Where it broke. For a Mixture-of-Experts model this checkpoint is enormous because of all the expert parameters, so it strains storage and cannot be hidden behind training.
Sharded checkpointing. Split the checkpoint across machines so each writes a piece. Where it broke. It helps, but the total MoE checkpoint is still huge, and existing sharding was not tuned for the expert-parallel layouts MoE training uses, so the workload is spread unevenly.
Asynchronous checkpointing. Snapshot the state and write it to storage in the background while training continues. Where it broke. The enlarged MoE save is too big to fully overlap with training, so part of it still stalls training and adds to the total time.
The gap. No approach reduced the fundamental size of the MoE checkpoint itself while keeping accuracy on recovery. Doing that — by saving only a subset of experts each time and bounding the accuracy impact, then sharding and tiering the smaller save — is what MoC-System provides.

How MoC-System works

From an unhideable full save to a small, tiered, bounded one

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Skip saving most experts, because the sparse model tolerates it. The heart of MoC-System is Partial Experts Checkpointing: rather than write out every expert every time, it saves the non-expert parameters fully but only a rotating subset of the experts, so the checkpoint shrinks toward the size of a dense model. This is safe precisely because a sparse Mixture-of-Experts model is insensitive to a limited number of missed expert updates — each expert sees only its share of tokens, and the model trains on so much data that a little staleness barely registers. Exploiting a model’s tolerance for reduced fidelity to make the common operation dramatically cheaper is the same bet the low-bit language-model serving work covered elsewhere on this site makes, running weights and activations at very low precision because the model absorbs the imprecision, and paying full cost only where it truly matters.

Bound the damage with a principled metric, not a guess. Skipping expert saves loses some updates, and that could hurt accuracy, so the method needs to know how far it can push. MoC-System defines the Proportion of Lost Tokens, which measures exactly how much of the experts’ training the checkpoint would forfeit on recovery, and it holds that quantity under a safe threshold — the paper finds accuracy stays at the fault-free level as long as it stays at or below about 3.75 percent. Governing an aggressive shortcut by a measured threshold that says exactly how far you can cut before quality suffers is the same discipline the early-termination inference work covered elsewhere on this site follows, using a statistical test to decide the precise moment it is safe to stop rather than stopping on a hunch.

Save through a fast tier and a reliable tier, asynchronously. Even a smaller checkpoint should not stall training, so MoC-System separates the save into two levels: a fast in-memory snapshot that captures the state quickly, and a slower persist that writes it to durable distributed storage in the background. The snapshot uses the high bandwidth of memory; the persist uses the reliability of storage; and running the persist asynchronously overlaps it with computation. Staging data through a fast tier in front of a slower, more durable backing store, and doing the slow part off the critical path, is the same layering the datacenter-tracing work covered elsewhere on this site relies on, capturing into a cheap fast tier and moving the bulk to the durable tier without blocking the work.

Following a checkpoint and a recovery through MoC

Partial checkpointing and recovery for an MoE model

Tracing one save and one crash shows why skipping experts is safe.

Step 1 of 5

A checkpoint step arrives

Training a Mixture-of-Experts model with 16 experts per layer reaches a checkpoint step. Saving all 16 experts per layer, plus optimizer state, every time would be enormous.

The experts dominate the parameter count, though only a few run per token.

Why it works

MoE bloats the checkpoint, not the compute. Because a sparse Mixture-of-Experts model has far more parameters than a dense model at similar compute → its checkpoint is far larger and cannot be hidden behind training → so checkpointing becomes the fault-tolerance bottleneck.
Experts tolerate missed saves. Because MoE routing is sparse and experts train on enormous data, so a limited number of missed updates barely changes the model → saving only a subset of experts each time loses little → so the checkpoint shrinks toward dense size at negligible accuracy cost.
A metric keeps the risk bounded. Because the Proportion of Lost Tokens measures exactly how much expert training is forfeited → keeping it under a safe threshold ensures recovery accuracy matches the fault-free case → so the aggressive saving is principled rather than a gamble.
Even sharding removes stragglers. Because the reduced checkpoint work is distributed evenly across all ranks, including the expert-parallel ones → no single machine is overloaded with the save → so the checkpoint completes faster.
Two tiers hide the remaining cost. Because a fast in-memory snapshot captures the state and a slow persist to reliable storage runs asynchronously → most of the checkpoint cost overlaps with training → so per-checkpoint overhead drops by up to 98.9 percent and iterations with checkpointing speed up by up to 5.12 times.

What it costs (trade-offs and scope)

Recovery reverts some experts to slightly older values. Skipping expert saves means a fault loses a bounded amount of the un-saved experts’ recent training. The paper shows this is harmless below the lost-token threshold, but it is a real, if small, loss compared with a full checkpoint.
It relies on the accuracy threshold holding. The safety of Partial Experts Checkpointing rests on the finding that accuracy is preserved while the Proportion of Lost Tokens stays at or below about 3.75 percent. Pushing past that, or a model that is more sensitive, would erode accuracy.
It is specific to Mixture-of-Experts. The core saving comes from the experts being numerous, sparse, and tolerant of missed updates. Dense models, whose every parameter matters on every token, do not offer the same opportunity, though the sharding and tiering parts are more general.
It adds scheduling machinery. Choosing which experts to save, tracking the lost-token proportion, sharding evenly, and coordinating snapshot and persist across ranks all add complexity and some overhead to the training system.
The gains depend on the configuration. The reported reductions come from particular models, parallel strategies (data parallelism with optimizer sharding plus expert parallelism), and the Megatron-DeepSpeed framework; different setups would shift the numbers.
A limited update loss helping accuracy is a bonus, not a guarantee. The observed small downstream-accuracy gain is plausibly a mild regularization effect, but it should be read as evidence that the loss is harmless, not as a reliable way to improve models.

The evaluation

Three ways to checkpoint a Mixture-of-Experts model

Property	Full checkpoint	Sharded and async
Checkpoint size for MoE		Huge, all experts
Overlaps with training
Accuracy after recovery	Full	Full
Overhead per checkpoint	Baseline	Reduced
Iteration speedup with checkpointing	Baseline	Some

Three ways to checkpoint a sparse Mixture-of-Experts model during training, evaluated on the Megatron-DeepSpeed framework with data parallelism (with optimizer-state sharding) plus expert parallelism. A full checkpoint must save all the many expert parameters, making it enormous and unable to hide behind training. Sharding and asynchronous writing help but the checkpoint is still too big to fully overlap. MoC-System saves only a rotating subset of experts per layer (Partial Experts Checkpointing), shrinking one model's total checkpoint by 57.7 percent, and bounds the accuracy impact with the Proportion of Lost Tokens metric — accuracy stays at the fault-free level while that proportion stays at or below about 3.75 percent, with downstream accuracy even 1.08 percent higher on average. It then shards the smaller save evenly and splits it into a fast in-memory snapshot and an asynchronous persist. Overall it cuts per-checkpoint overhead by up to 98.9 percent and speeds up each training iteration that includes checkpointing by up to 5.12 times.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

MoC-System is a neat algorithm-system co-design: it takes a systems problem — Mixture-of-Experts checkpoints are too big to save cheaply — and solves it by exploiting a property of the model, that its many sparse experts tolerate a limited number of missed updates. Partial Experts Checkpointing saves only a rotating subset of experts each time, shrinking the checkpoint toward dense-model size, and the Proportion of Lost Tokens metric turns what could have been a risky shortcut into a principled one by bounding the accuracy impact — accuracy holds at the fault-free level while that proportion stays at or below about 3.75 percent, and downstream accuracy was even slightly higher. The remaining machinery, fully sharded checkpointing and a two-level snapshot-then-persist save, distributes and hides the smaller checkpoint. The results are strong: up to a 98.9 percent cut in per-checkpoint overhead and up to a 5.12 times speedup of iterations that include checkpointing, on the Megatron-DeepSpeed framework. Read it with its scope. The approach is specific to Mixture-of-Experts, where numerous sparse experts create the opportunity; dense models do not benefit the same way. Its safety rests on the lost-token threshold holding for the models and settings tested, recovery does forfeit a small, bounded amount of expert training, and the reported numbers depend on the particular models, parallel strategies, and framework. The observed accuracy bump is best read as confirmation the loss is harmless, not as a training trick. The durable lesson is that co-designing the systems layer with the model’s tolerances can slash a cost that looked fixed.

Background

Why co-designing systems with model tolerances keeps winning

It helps to see MoC-System as one more case of a pattern that is reshaping machine-learning infrastructure: the biggest systems savings often come from exploiting what the model can tolerate, rather than treating the model as a black box that must be served exactly. A checkpoint looks like data that must be saved in full, but once you know the experts tolerate missed updates, most of it need not be saved every time. The wave-scheduled training work covered elsewhere on this site makes a kindred move, shaping the training schedule around the model’s own learning behavior rather than treating every stage as equal, to use the cluster better. In both, the leverage comes from opening up the boundary between the learning algorithm and the system that runs it, and letting each inform the other. The general point is that large-model training and serving are full of costs that seem fixed — the size of a checkpoint, the precision of a weight, the length of a computation — but are actually elastic once you account for how much fidelity the model truly needs. As models keep growing, the teams that co-design the algorithm and the system together, and measure exactly how much they can cut, will keep finding large savings that a system blind to the model cannot.

If you remember only three things

1. Training giant language models across tens of thousands of machines requires frequent checkpoints — saving the model parameters and optimizer state so a fault does not lose days of work. The sparse Mixture-of-Experts design raises quality by adding many expert sub-networks, of which only a few run per token, so parameter count and checkpoint size explode while compute stays comparable to a dense model. The result is that MoE checkpoints are too large to hide behind training even with sharding and asynchronous writing, so they lengthen the whole run.
2. MoC-System reduces the checkpoint itself. Its Partial Experts Checkpointing saves only a rotating subset of experts per layer each time, fully saving the non-expert parameters, shrinking the checkpoint toward dense-model size. This is safe because a sparse MoE model tolerates a limited number of missed expert updates, and the Proportion of Lost Tokens metric bounds the risk — accuracy stays at the fault-free level while that proportion stays at or below about 3.75 percent. On top, fully sharded checkpointing spreads the smaller save evenly across ranks including expert-parallel ones, and a two-level manager splits it into a fast in-memory snapshot and an asynchronous persist to storage.
3. On the Megatron-DeepSpeed framework, MoC-System cuts per-checkpoint overhead by up to 98.9 percent and speeds up each training iteration that includes checkpointing by up to 5.12 times, with Partial Experts Checkpointing reducing one model's total checkpoint size by 57.7 percent while preserving accuracy (downstream accuracy was even 1.08 percent higher on average). Its scope: the saving is specific to Mixture-of-Experts, its safety depends on the lost-token threshold holding, recovery forfeits a small bounded amount of expert training, and the numbers depend on the models, parallel strategies, and framework used.

Check yourself

Check your understanding

1. Why does the Mixture-of-Experts design make checkpointing especially expensive?
2. What is Partial Experts Checkpointing, and why is it safe?
3. What role does the "Proportion of Lost Tokens" (PLT) metric play?
4. A team wants to apply Partial Experts Checkpointing to a standard dense model (no experts). What does the paper's reasoning imply?

Sources

The paper is “MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training” by Weilin Cai, Le Qin, and Jiayi Huang, published at ASPLOS 2025, DOI 10.1145/3676641.3716006. The authors are at the Hong Kong University of Science and Technology (Guangzhou); Jiayi Huang is the corresponding author. MoC-System is built on the Megatron-DeepSpeed framework.

This explainer is written from the full text of the paper, obtained from the authors’ arXiv preprint (2408.04307), which matches the ASPLOS 2025 proceedings. Everything above comes from the paper’s own sections: the introduction and background on distributed large-language-model training and checkpoint-based fault tolerance, and on sparse Mixture-of-Experts models (experts as feed-forward networks with a top-k gating network, and the resulting large parameter count at comparable compute) and why their checkpoints strain storage and cannot be fully overlapped with training; the Mixture-of-Checkpoint System design with its Partial Experts Checkpointing mechanism (saving a selected subset of experts per layer while fully saving non-expert parameters, inspired by the fine-tuning observation that non-expert parameters carry the accuracy while experts tolerate limited missed updates), the Proportion of Lost Tokens metric and its finding that accuracy is maintained while the proportion stays at or below about 3.75 percent, the fully sharded checkpointing strategies for expert-parallel layouts, and the two-level snapshot-and-persist management with snapshot-Partial-Experts-Checkpointing and persist-Partial-Experts-Checkpointing; and the evaluation on Megatron-DeepSpeed with data parallelism (with optimizer-state sharding) plus expert parallelism, reporting a 57.7 percent reduction in total checkpoint size on the GPT-350M-16E model, comparable validation loss and a 1.08 percent average downstream-accuracy increase on recovery, up to a 98.9 percent reduction in per-checkpoint overhead, and up to a 5.12 times speedup of training iterations that include checkpointing.