Training a large model on cheap preemptible cloud machines only works if you save your progress often, but every existing checkpointing system lets exactly one save be in flight at a time, so asking for more frequent saves means the next one waits on the last one and the accelerators sit idle; PCcheck (ETH Zurich) allows several saves to overlap, guarded by a lock-free algorithm that guarantees one complete checkpoint always survives a crash — saving every ten iterations for a 3 percent cost, though the honest peak-to-peak comparison is 1.25 times rather than the headline 2.86.
The argument in one glance
-
Problem
Long training runs on unreliable hardware lose work constantly
A large model trains for days or weeks across many accelerators, and at that scale something always breaks — hardware faults, network problems, out-of-memory crashes, software bugs. One study of a production cluster finds half of all training jobs hit a failure within sixteen minutes. It gets much worse on purpose: preemptible cloud machines cost 60 to 90 percent less, and one measured cluster of 64 of them saw 127 separate preemptions in a single day.
-
What people did
Save the model state periodically, and overlap the saving with training
The standard answer is to write the weights and optimiser state to durable storage every so often, and resume from the last one after a crash. The naive version stops training while it saves. The state of the art overlaps the work — copy the state off the accelerator while training continues, then flush it to disk in the background — and a competing approach sends it over the network to the memory of another machine instead of to disk.
-
Why it fell short
Ask for saves more often and throughput collapses
Both of those systems allow exactly one checkpoint to be in flight. The next save cannot begin copying until the previous one has finished being written, so at high frequency the accelerator simply waits. Checkpointing one model every iteration made training 57 times slower; even every 50 iterations cost more than 10 percent. And the deadline is real — a preemption warning gives 30 seconds, while a 16 gigabyte checkpoint takes 37 seconds to write.
-
This paper's idea
Let several checkpoints be in flight at once
The bottleneck was never the storage bandwidth; it was the rule that saves must not overlap. PCcheck allows a small number of concurrent checkpoints, hands out free memory regions from a lock-free queue, persists each one with several threads, and publishes the newest completed checkpoint with a single atomic pointer swap — maintaining the invariant that a complete, valid checkpoint always survives a crash. Saving every ten iterations then costs about 3 percent.
Why this problem is worth solving
The economics of training push jobs onto exactly the hardware most likely to disappear. Periodically writing a training job’s model weights and optimiser state to durable storage, so that after a crash the job resumes from the last saved point rather than from the beginning. The frequency is a direct trade-off — saving often costs throughput but loses little work to a failure, while saving rarely costs almost nothing until something breaks and hours of computation have to be repeated. Checkpoints also serve a second purpose entirely, letting practitioners inspect how training is behaving when accuracy starts diverging.
Which means the metric that matters is not the one usually reported. Useful throughput — batches processed per second, discounting any batch that is a recomputation of work already done before a failure. It is the honest measure for a job on unreliable hardware, because raw throughput can be made to look excellent by never checkpointing, right up to the moment a crash throws away an hour. Plotted against checkpoint frequency it forms a curve with a peak in the middle, since saving too often wastes time saving and saving too rarely wastes time redoing.
What people did before, and where it broke
- Stop training, copy the weights, write them out, resume. What the standard frameworks do by default. Where it broke. Every phase is sequential, so training stalls for the entire duration of the save. Fine at low frequency, ruinous at high.
- Overlap the copy and the flush with training. The established improvement — training continues while the state is copied off the accelerator and while that copy is written to storage. Where it broke. Only one checkpoint may exist at a time, so the next save’s copy waits for the previous save’s write to complete. On one vision model, checkpointing every iteration made training 57 times slower, and every 10 to 100 iterations still cost between 5.7 times and 1.2 times.
- Send the state to another machine’s memory instead of to disk. A distributed alternative that avoids slow storage entirely by using the network and remote memory. Where it broke. It assumes a fast network. The measured bandwidth on the cloud machines used here is about 15 gigabits per second, which is ordinary for public cloud, and at that speed the transfer is not hidden. It also allows only one checkpoint at a time, so it inherits the same stall.
- Copy directly from accelerator to persistent memory using its own kernels. A general persistent-memory system, not built for training. Where it broke. It stops the accelerator entirely for each save, so its overhead grows worse than the overlapping approach as checkpoints get larger.
- Only checkpoint when a failure is actually detected. An elegant idea — healthy workers notice a peer has died and save their own state at that moment, relying on replication across parallel workers to reconstruct what was lost. Where it broke. It assumes simultaneous multi-node failures are extremely rare. On preemptible machines, bulk preemptions of many machines at once are exactly what happens.
- The gap. Everyone optimised the cost of one checkpoint. Nobody questioned the constraint that only one may be outstanding, which is the thing that actually breaks at high frequency, and removing it requires an algorithm that keeps several partially-written checkpoints straight while guaranteeing a crash at any moment still finds a complete one.
How PCcheck saves state without stalling
From a training iteration to a durable checkpoint, several at a time
-
-
new
-
new
-
new
-
new
-
new
The idea
The stall was structural, not physical, so remove the rule rather than optimise around it. Everyone before had assumed the cost of frequent checkpointing was a bandwidth problem and attacked it by making a single checkpoint cheaper — overlapping phases with training, or routing around slow disks to remote memory. The measurement that reframes it is that the overhead does not track the device speed; it tracks the moment when a new save arrives before the previous one has finished, which is a queueing property of the design rather than a property of the hardware. Allowing several saves in flight is conceptually trivial and operationally delicate, because now several partially-written checkpoints coexist on the device and a crash can land anywhere among them. The algorithm holds one invariant — the newest complete checkpoint among the last few is always intact and can never be overwritten — enforced by persisting the state first, then a small record pointing at it, and only then swapping in that record with a single atomic operation. Getting that ordering backwards would publish a pointer to data that has not reached durable storage, which is precisely the class of bug that the persistency race detection work covered elsewhere on this site exists to find.
The shortest path from accelerator to disk is not the fastest one. There is a tempting simplification available here — hardware exists to move data straight from accelerator memory to storage without passing through host memory at all, and one of the baselines does something like it. The authors tried it and went the other way, because the destination is the slow part. Staging through host memory lets a fast copy off the accelerator overlap with a slow write to storage, whereas a direct path runs at the speed of the slowest link end to end. The rest of the path is tuned with similar care rather than assumed — copy engines rather than compute kernels, since they use no accelerator compute and their setup cost vanishes against gigabyte transfers; pinned memory so transfers go by direct memory access; cache-bypassing stores followed by a fence for persistent memory, measured at 4.01 against 2.46 gigabytes per second for the write-back alternative, because the data is written once and never read; and a single system call to flush for disks, where one thread can cover writes made by all of them, unlike persistent-memory fences which are internal to each processor. Measuring which path actually goes fastest rather than which one looks most direct is the same discipline as the accelerator-side persistent memory work covered elsewhere on this site, one of the systems compared against here.
Optimise for useful work, and accept that concurrency loses more of it per failure. The paper’s honesty is in the accounting. Allowing several saves in flight raises throughput but also means a crash can roll training back further, because the newest complete checkpoint may be several saves old. Their own worked example makes this explicit — on one model at the highest frequency, PCcheck redoes 550 iterations after failures across the trace where the single-checkpoint baseline redoes 257 — and the argument is simply that the extra 293 iterations of lost work are bought with 2733 extra iterations of progress. That is the right way to argue it, and it is why the evaluation replays a real preemption trace from a spot instance cluster rather than reporting throughput. It also produces the paper’s most useful caveat, since comparing peak against peak across all checkpoint frequencies rather than at a fixed one shrinks the headline 2.86 times down to 1.25. Treating unreliable cheap hardware as the design target rather than a degraded case is the same framing as the preemptible-instance serving work covered elsewhere on this site.
Walking one checkpoint through the algorithm
Saving state while two earlier saves are still being written
A training job configured for at most four concurrent checkpoints, saving every ten iterations.
Step 1 of 5
Take a ticket and a slot
The coordinator reads which checkpoint is currently the official latest one, then atomically increments a global counter to get a number for this new save. It then dequeues a free host-memory region from a lock-free queue, retrying until one is available.
The order of those two reads matters for correctness. Sampling the current latest checkpoint before taking the counter guarantees this save number is larger than the one it will later try to replace, which is what makes the eventual pointer swap legal rather than a race. The queue never contains the region holding the latest valid checkpoint, so that one cannot be handed out and overwritten.
Why it works
- The bottleneck was a rule, and rules are free to change. Overhead at high frequency came from a new save waiting on the previous one rather than from the device being busy → so permitting several in flight removes the wait without needing any faster hardware → and the remaining cost is only the contention those saves create with each other.
- A small amount of concurrency is all the device can absorb. Each additional outstanding checkpoint competes for the same write bandwidth → so past about four the device is saturated and more concurrency stops helping → which conveniently bounds the memory and storage the design has to reserve.
- Persist ordering turns many partial writes into one safe answer. Data is made durable, then a record pointing at it, then one atomic swap publishes it → so a crash between any two of those steps leaves the previous complete checkpoint untouched and still official → and no partially-written state is ever reachable.
- Staging through host memory beats going direct. The write to storage is much slower than the copy off the accelerator → so passing through host memory lets the fast half overlap the slow half instead of running at the slow half’s speed end to end → which is why the more direct hardware path was measured and rejected.
- Frequency buys recovery time at a price you can now afford. Recovery cost is roughly proportional to how many iterations happened since the last complete save → so cutting the interval from 100 iterations to 10 cuts recovery roughly tenfold → and this only became possible once frequent saving stopped costing throughput.
What it costs (trade-offs and scope)
- The headline number and the fair number are different. At a fixed checkpoint frequency PCcheck reaches up to 2.86 times the goodput of the best-known baseline. Comparing each system at its own best frequency — which is what a deployer would actually choose — the margins are 1.25, 1.27 and 1.44 times against the three baselines. Both numbers are in the paper.
- Concurrency loses more work per failure. Because the newest complete checkpoint may be several saves behind, a crash rolls back further. The paper’s own example has PCcheck redoing 550 iterations against a baseline’s 257, offset by far more progress made in between.
- It uses more storage. Supporting several outstanding checkpoints requires room for one more than that number, against two checkpoints’ worth for the single-checkpoint systems. Host memory is more forgiving — halving the buffer to one checkpoint’s size costs at most 7 percent.
- The distributed coordination is preliminary and labelled as such. Peers report their save numbers to a designated leader and wait. The paper notes that all peers happened to report in the same order in their experiments, and that robustness and efficiency at larger scale is still being worked on.
- One of the named techniques barely matters. Splitting checkpoints into chunks to pipeline copying against writing gives only slightly higher throughput, with differences the paper describes as quite small.
- A stall was left in deliberately. The weight update still blocks the next iteration, and the paper explains it could be removed by keeping a second copy of the weights in accelerator memory, but judges that memory too expensive to spend. Reasonable, and it means the reported overhead is not the floor.
- Configuration is static. The profiling tool picks a concurrency level and interval once. The paper acknowledges the best frequency may drift during training, since vision workloads are input-bound and language workloads offload activations, and lists adaptive adjustment as future work.
- One baseline was reimplemented rather than run. The remote-memory system has no public implementation, so the authors built it from the paper description, and evaluated it on a 15 gigabit per second cloud network where its central assumption of a fast interconnect does not hold.
- Scale and hardware are modest relative to the motivation. The largest configuration is a 7 billion parameter model across six cloud machines, not the thousand-accelerator deployments the introduction cites. The persistent memory results use hardware whose manufacturer has discontinued it, which the paper notes, arguing the mechanism carries over to newer coherent interconnects.
The evaluation
Three ways to keep a training job recoverable
| Property | Overlapped checkpointing to storage | Snapshotting to a remote machine memory | PCcheck (several checkpoints in flight) |
|---|---|---|---|
| How many saves may be outstanding | |||
| Cost at a high save rate | |||
| Recovery time after a failure | 250 seconds on the largest model at comparable overhead | ||
| Extra storage required | Two checkpoints worth | ||
| Goodput on a replayed preemption trace |
Check yourself
Check your understanding
1. Why did prior checkpointing systems slow training down so much at high frequency?
2. What guarantees that a crash never leaves the job without a usable checkpoint?
3. Why does the data go through host memory instead of straight from the accelerator to storage?
4. How should the reported speedups be read?
Sources
The paper is “PCcheck: Persistent Concurrent Checkpointing for ML” by Foteini Strati, Michal Friedman and Ana Klimovic at ETH Zurich, published at ASPLOS 2025 Volume 1, pages 811-827, DOI 10.1145/3669940.3707255. The first two authors contributed equally. It is open access under a Creative Commons Attribution licence, and the implementation is public.
This explainer is written from the full text of the published paper. Everything above comes from its own sections 2 through 5: the failure statistics motivating frequent checkpointing, including the production-cluster study finding half of jobs failing within sixteen minutes, the 45 minute mean time between failures reported for a multi-tenant cluster, the 127 preemptions in a day observed on 64 spot machines, and the observation that a 30 second preemption warning does not cover a 37 second write; the survey of prior mechanisms and the identification in section 2.2 that their common overhead is a checkpoint stalling on the previous one, with the specific rejection of failure-triggered checkpointing on the grounds that bulk preemptions break its rarity assumption; the section 3 architecture with its coordinator and persistent manager, the free-slot queue, the chunked copy and multi-threaded write, the memory and storage footprint comparison, and the explicit decision not to spend accelerator memory on a second weight copy; the section 3.3 measurements behind the data path choices, including copy engines with pinned memory and cache-aware I/O, the rejection of direct accelerator-to-storage transfer, and the 4.01 against 2.46 gigabytes per second comparison for persistent memory store instructions; the section 3.4 configuration model relating checkpoint interval, concurrency and acceptable slowdown, and its acknowledgement that the optimum may drift during training; the section 4 algorithm with its global counter, descriptor records, lock-free queue, ordered persistence and compare-and-swap publication with all three outcomes handled, the recovery bounds for each system, and the distributed coordination step the authors describe as still being improved; and the section 5 evaluation across six models on cloud and on-premise hardware, including the throughput and recovery results, the goodput replay of a recorded preemption trace with both the fixed-frequency and peak-to-peak comparisons, the worked iteration accounting showing more work lost per failure, and the sensitivity studies on concurrency, writer threads, chunking and memory budget.