SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips

Training a large language model that will not fit in GPU memory often means offloading part of the work to the CPU, and every existing offloading system is built around the fact that the GPU-to-CPU link (PCIe) is painfully slow, so they obsess over moving as little data as possible; new Superchips like NVIDIA's Grace Hopper glue a GPU and CPU onto one package with a link thirty times faster, which quietly invalidates that whole design philosophy — this work from the DeepSpeed team and UIUC rebuilds offloading around the fast link, keeping the GPU, CPU, and interconnect all busy at once through five coordinated techniques, and trains models up to 2.5 times faster.

The argument in one glance

Problem

Offloading LLM training assumes a slow GPU-CPU link

When a model does not fit in GPU memory, offloading moves part of its state and computation to the CPU. Every mature offloading system is built around PCIe being slow, so its whole design goal is to minimize data crossing the GPU-CPU link, even at the cost of extra work.
What people did

Run those systems on a fast Superchip

Take an existing offloading system, built for slow PCIe, and run it unchanged on a Superchip like Grace Hopper, where a GPU and CPU share a package joined by a link about thirty times faster.
Why it fell short

The old assumptions waste the new hardware

On a Superchip the link is no longer the bottleneck, but the systems still minimize data movement as if it were, so they badly underuse the fast GPU, the capable CPU, and the huge interconnect bandwidth — and a much larger GPU-to-CPU compute gap exposes the CPU's optimizer work on the critical path.
This paper's idea

Rebuild offloading around the fast link (SuperOffload)

SuperOffload is a Superchip-centric offloading system that keeps the GPU, CPU, and interconnect busy at once. Five coordinated techniques — adaptive weight offloading, bucket repartitioning, speculation-then-validation, Superchip-aware casting, and a tuned Grace-CPU Adam — deliver up to 2.5 times higher training throughput and train a 25-billion-parameter model on one Superchip.

Why this problem is worth solving

Training large language models is fundamentally limited by memory, and offloading is the main way smaller teams get around that limit.

The defining constraint in training large language models is the memory wall: model sizes have grown exponentially while GPU memory capacity and bandwidth have grown far more slowly, so training is very often bottlenecked by how much fits in GPU memory. When a model exceeds a single GPU’s memory, one option is to shard it across many GPUs with distributed-training techniques, but the large number of GPUs required is a barrier for researchers and institutions on modest budgets — and this matters well beyond the original pre-training, because models are refined through post-training phases like long-context extension, continued training, supervised fine-tuning, instruction tuning, and human-preference alignment, all of which require training with far fewer GPUs than pre-training used. Offloading is the leading way to make that feasible: it moves a portion of the model’s state (its parameters, gradients, and especially the optimizer states) and some of the computation off the GPU and onto the CPU, which has abundant memory, so both processors are used together to train a large model without needing a wall of GPUs. It is a mainstream capability, supported in PyTorch’s fully-sharded data parallel and in DeepSpeed.

Every one of those offloading systems, however, was designed for a world where the GPU and CPU are far apart, and new hardware has changed that world.

Existing offloading rests on one key assumption: the GPU is an accelerator attached to the CPU over PCIe, a link with limited bandwidth — on the order of tens of gigabytes per second. Because that link is so slow, prior systems make minimizing the data crossing it their central objective, and will happily do extra computation or use clever tricks to avoid moving bytes over PCIe, since the link is the bottleneck that determines performance. Superchips overturn this. NVIDIA’s Grace Hopper, for instance, integrates a Hopper GPU and an ARM-based Grace CPU on the same package, joined by an NVLink chip-to-chip interconnect delivering around 900 gigabytes per second between them — roughly thirty times the bandwidth of a traditional PCIe connection — and multiple Superchips can be linked into a large tightly-coupled system. On this hardware the premise that the link is the bottleneck is simply false. Yet the offloading systems, still minimizing data movement as if PCIe were in the way, leave the Hopper GPU, the Grace CPU, and the vast interconnect bandwidth all underutilized. Worse, the GPU-to-CPU compute gap is now far larger than before, which introduces new imbalance problems the old designs never had to face.

What people did before, and where it broke

Minimize GPU-CPU data movement above all. Design offloading to cross the PCIe link as little as possible, trading extra work to save bandwidth. Where it broke. On a Superchip the link is roughly thirty times faster and no longer the bottleneck, so this trade now wastes compute to save bandwidth that is abundant.
Transfer only low-precision data, then cast on the CPU. In mixed-precision training, greedily move low-precision tensors to cut PCIe volume and convert precisions on the CPU. Where it broke. With fast interconnect this greedy volume-minimizing choice is suboptimal, and the casting operations it forces have real cost and interact badly with pinned memory.
Offload the whole optimizer step to the CPU and overlap with buckets. Split state into buckets so CPU optimizer work overlaps GPU compute. Where it broke. The Grace-to-Hopper compute gap is about 330 to 1, far larger than on prior servers, so the last buckets’ optimizer work cannot overlap and sits exposed on the critical path, stalling the GPU.
Synchronize before every optimizer step. Wait for all gradients so global operations like gradient-norm clipping and NaN/Inf checks are correct. Where it broke. This synchronization exposes the whole optimizer step on the critical path, preventing it from overlapping with the backward pass, even though those global checks almost never change anything.
The gap. No system used the Superchip’s fast link and both processors together, or handled its larger compute gap and mixed-precision costs. Providing that is what SuperOffload does.

How SuperOffload works

From PCIe-era offloading to a Superchip-centric design

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

When the link stops being the bottleneck, redesign around moving data freely. SuperOffload’s foundation is recognizing that the Superchip’s fast interconnect, roughly thirty times a PCIe link, dissolves the constraint every prior offloading system was built to respect, so instead of hoarding bandwidth it can move parameters and gradients between the Hopper GPU and Grace CPU liberally to keep both processors busy. It represents the whole training step as a Superchip-aware data-flow graph — each operator annotated with its cost on the GPU versus the CPU, each dependency with its communication cost — and treats an offloading plan as a partition of that graph, so placement decisions are made with the real, fast link in mind. Exploiting a tight, high-bandwidth coupling between the GPU and CPU to migrate tensors freely across the boundary, rather than treating that boundary as an expensive wall, is the same move the unified-memory tensor-migration work covered elsewhere on this site makes, using a closer GPU-CPU coupling to shuttle data back and forth on demand instead of laboring to avoid ever crossing between them.

Balance work across a lopsided pair of processors. The Grace CPU and Hopper GPU differ enormously in compute — the GPU is about 330 times faster — a far wider gap than on earlier servers, and it wrecks the naive plan of overlapping the CPU’s optimizer step with GPU compute: the last buckets of work cannot overlap because the next iteration’s first forward step depends on the last backward gradient, leaving the CPU’s slow optimizer update exposed on the critical path and stalling the GPU. SuperOffload rebalances by placing those last buckets’ gradients and optimizer states on the GPU when they fit, and by adaptively deciding whether to keep weights on the GPU or flow them from the CPU based on which is more efficient for the model size, batch, and sequence length at hand. Carefully partitioning a heterogeneous training workload across a fast accelerator and a slower general processor to keep both busy despite their imbalance is the same discipline the offloading-based large-model fine-tuning work covered elsewhere on this site is built on, dividing computation and state between GPU and CPU so the combination trains models neither could hold alone without either processor idling.

Assume the common case and fix the rare one afterward. The optimizer step normally must wait for all gradients so it can do global operations — clipping the gradient norm, checking for NaN or Inf values in mixed precision — and that synchronization pins the whole optimizer step to the critical path. But SuperOffload observes that these global checks almost never change anything: clipping rarely triggers after warm-up, and a healthy run rarely produces NaN or Inf. So it runs the optimizer step speculatively with the gradients already available, overlapping it with the next forward pass, while a background process validates in parallel and, on the rare violation, rolls the step back in place and re-executes — an exact optimization that preserves correctness while hiding the synchronization cost. Speculatively executing the likely-correct path and validating it concurrently, rolling back only in the uncommon case, is the same design the speculative LLM-serving work covered elsewhere on this site is built on, betting on the common outcome to overlap work that a strict dependency would otherwise serialize, and paying a correction cost only when the bet is wrong.

Following a training step through SuperOffload

Training an LLM step on a Grace Hopper Superchip

A trace of SuperOffload executing one training iteration, following the paper's Superchip-centric design.

Step 1 of 5

Plan placement on the cost graph

SuperOffload views the step as a Superchip-aware data-flow graph and decides, from a cost model over the fast interconnect, whether to keep weights on the Hopper GPU or flow them from the Grace CPU, and how to bucket the state — choosing the plan that keeps both processors busy.

The fast link makes moving data a viable option, not a last resort.

Why it works

The fast link changes the optimization target. Because the Superchip interconnect is about thirty times a PCIe link → minimizing data movement is no longer the goal and both processors can be fed → so offloading should maximize utilization, not conserve a bottleneck that no longer exists.
Graph-based planning matches work to the right processor. Because each operator has a different cost on the GPU versus the CPU → modeling training as a cost-annotated data-flow graph lets placement be chosen deliberately → so the adaptive weight and bucket decisions fit each model, batch, and sequence length.
Moving the last buckets onto the GPU removes the stall. Because the next forward pass depends on the last backward gradient and the CPU is about 330 times slower → doing those final updates on the GPU avoids a round-trip on the critical path → so the GPU no longer idles waiting for the CPU optimizer to finish.
Speculation hides synchronization that rarely matters. Because gradient clipping and NaN/Inf checks almost never fire → speculatively running the optimizer step and validating in the background overlaps it with the forward pass → so the synchronization cost is paid only in the rare case that needs a rollback.
Tuning for the actual CPU and casts recovers wasted time. Because the Grace ARM CPU lacked a fast Adam and mixed-precision casts have real cost → a tuned GraceAdam and Superchip-aware casting use the CPU and link efficiently → so effort the old assumptions ignored is reclaimed.

What it costs (trade-offs and scope)

It is specific to tightly-coupled Superchips. SuperOffload’s whole premise is the fast on-package GPU-CPU link. On traditional PCIe-connected servers the old bandwidth-minimizing designs remain appropriate, so this is not a drop-in improvement for that hardware.
Speculation adds machinery and occasional rework. The speculation-then-validation scheme needs a background validation process, in-place rollback support, and the ability to re-execute a step; it preserves exact correctness, but the rare NaN/Inf or clipping violation costs a skipped iteration or a redo, and the mechanism is more complex than a plain synchronize-then-step.
The adaptive choices depend on the scenario. Whether weight-stationary or weight-flow wins, and how to bucket, depends on model size, batch size, and sequence length; the cost model guides this, but the best configuration is workload-dependent rather than one fixed setting.
Placing last buckets on the GPU needs spare GPU memory. The repartitioning that removes the critical-path stall keeps the final buckets’ gradients and optimizer states on the GPU only when they fit, so its benefit is bounded by available GPU memory, which is itself the scarce resource offloading exists to stretch.
Results are on NVIDIA GH200 and its software stack. The up-to 2.5 times throughput, the 25-billion-parameter single-Superchip training, and the million-token 8-Superchip result are measured on Grace Hopper with DeepSpeed; other Superchips (for example AMD’s MI300A) share the architecture but would need their own tuning, and numbers would shift.
It targets training throughput, within offloading. SuperOffload optimizes offloading-based training; it is complementary to, not a replacement for, pure multi-GPU sharding when many GPUs are available, and its aim is efficient training on limited hardware rather than the absolute fastest training at any scale.

The evaluation

Offloading LLM training: PCIe-era design versus Superchip-centric

Property	PCIe-era offloading	Naive port to a Superchip	SuperOffload
Design objective
Uses the fast interconnect	No such link assumed
Handles the large compute gap	Gap was smaller
Optimizer-step overlap	Exposed by synchronization
Training throughput	Baseline

Offloading-based LLM training on a Superchip, as compared in the paper. PCIe-era offloading systems assume the GPU-CPU link is slow, so their entire design minimizes data crossing it, trading extra work to save bandwidth. Run unchanged on a Grace Hopper Superchip — where the on-package NVLink chip-to-chip interconnect reaches about 900 gigabytes per second, roughly thirty times PCIe — those systems keep minimizing movement and so underuse the Hopper GPU, the Grace CPU, and the interconnect, while the much larger GPU-to-CPU compute gap (about 330 to 1) leaves the final buckets' CPU optimizer work stalling the GPU on the critical path. SuperOffload is a Superchip-centric redesign that represents training as a Superchip-aware data-flow graph and applies five techniques: adaptive weight-stationary versus weight-flow offloading chosen by a compute-versus-communication cost model; fine-grained bucketization repartitioning that keeps the last buckets' gradients and optimizer states on the GPU to remove the critical-path stall; a speculation-then-validation scheme that runs the optimizer step speculatively and overlaps it with the next forward pass, validating NaN/Inf and gradient clipping in the background and rolling back the rare time it must (an exact optimization); Superchip-aware casting that rethinks where mixed-precision conversions happen for the fast link; and GraceAdam, an Adam optimizer tuned for the Grace ARM CPU. On NVIDIA GH200 this delivers up to 2.5 times higher training throughput than state-of-the-art offloading systems and trains a 25-billion-parameter model on a single Superchip; extended with ZeRO-style data parallelism and DeepSpeed-Ulysses sequence parallelism, it trains a 13-billion-parameter model with sequences up to a million tokens on eight GH200 at 55 percent model-FLOPs utilization.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

SuperOffload is a timely, well-engineered systems paper that catches a hardware transition at exactly the moment the software assumptions behind it go stale. The problem is real: training large language models is memory-bound, offloading to the CPU is how teams without a wall of GPUs cope, and every mature offloading system was built for a world where the GPU and CPU communicate over slow PCIe, so its central design goal is to move as little data across that link as possible. Superchips like NVIDIA’s Grace Hopper break that premise by fusing a GPU and CPU onto one package with an interconnect roughly thirty times faster, and the paper’s core insight is that this does not just make offloading faster — it invalidates the design philosophy, because minimizing data movement is now optimizing for a bottleneck that no longer exists, while the fast GPU, capable CPU, and huge interconnect sit underused and a much larger compute gap creates new stalls. SuperOffload’s response is a coherent set of five techniques: model training as a cost-annotated data-flow graph and adaptively choose what to offload; repartition buckets so the last, un-overlappable optimizer work happens on the GPU; speculatively run the optimizer step and validate the rare failure cases in the background; rethink mixed-precision casting for the fast link; and provide a properly tuned Adam for the Grace ARM CPU. The results are strong — up to 2.5 times higher throughput, a 25-billion-parameter model on a single Superchip, and million-token sequences across eight — and it is integrated into DeepSpeed behind a few lines of code. Read the scope precisely. The whole approach is predicated on tightly-coupled Superchip hardware and does not apply to conventional PCIe servers, where the older designs remain right; speculation adds complexity and occasional rollback; the adaptive choices and the last-bucket-on-GPU trick depend on the workload and on spare GPU memory; and the numbers are specific to GH200 with DeepSpeed, with other Superchips needing their own tuning. The durable idea is that hardware assumptions are baked deep into systems software, and when a tightly-coupled architecture replaces a loosely-coupled one, the right response is not to port the old system but to revisit the assumptions it was built on — here, that the link is precious — and redesign around what the new hardware actually makes cheap.

Background

Why tighter GPU-CPU coupling keeps rewriting the training playbook

It helps to see SuperOffload within a broader hardware trend: the boundary between the GPU and the CPU is steadily dissolving, and each step of that convergence forces the software that spans the boundary to be rethought. For years the GPU was a distant accelerator reached over a slow bus, and an entire generation of training systems — offloading, memory management, parallelism strategies — encoded that distance as a fundamental cost to be minimized. The large-scale model-training work covered elsewhere on this site reflects the same era’s response from the parallelism side, combining data, tensor, and pipeline parallelism to spread a model across many separate devices whose interconnects were the limiting resource. Superchips move in the opposite direction, pulling the GPU and CPU onto one package with a link fast enough that they behave almost like one tightly-coupled machine, which reopens design decisions that were considered settled. The common thread is that distributed and heterogeneous training systems are shaped, often invisibly, by the relative costs of compute, memory, and communication in the hardware of their moment, and when those costs shift sharply — as a thirty-fold jump in GPU-CPU bandwidth does — the optimal software can change qualitatively, not just quantitatively. As tightly-coupled architectures like Grace Hopper, Grace Blackwell, and MI300A proliferate, the pattern SuperOffload exemplifies, of revisiting bedrock assumptions when the hardware coupling tightens, is likely to recur across the training stack.

If you remember only three things

1. Training large language models is bounded by the memory wall — models outgrow GPU memory faster than that memory grows — so training a model that does not fit means either many GPUs (costly, and often unnecessary for fine-tuning and post-training) or offloading, which moves part of the model's state (parameters, gradients, and especially optimizer states) and computation from the GPU to the CPU's abundant memory. Every mature offloading system, in PyTorch and DeepSpeed, was designed around one assumption: the GPU connects to the CPU over PCIe, a link so slow (tens of gigabytes per second) that the design goal is to minimize data crossing it, even at the cost of extra work. Superchips like NVIDIA's Grace Hopper break this: they fuse a Hopper GPU and an ARM-based Grace CPU on one package with an NVLink chip-to-chip interconnect around 900 gigabytes per second — roughly thirty times PCIe — so the link is no longer the bottleneck. Run unchanged, existing systems keep minimizing movement and badly underuse the GPU, CPU, and interconnect, while a much larger GPU-to-CPU compute gap (about 330 to 1) creates new critical-path stalls.
2. SuperOffload is a Superchip-centric offloading system that models a training step as a Superchip-aware data-flow graph (each operator carrying its GPU-versus-CPU cost, each dependency its communication cost) and keeps the GPU, CPU, and fast link all busy through five coordinated techniques. Adaptive weight offloading chooses, via a compute-versus-communication cost model, whether to keep weights on the GPU (weight-stationary) or flow them part-by-part from the CPU (weight-flow) for the given model, batch, and sequence length. Fine-grained bucketization repartitioning places the last buckets' gradients and optimizer states on the GPU (when they fit) so their update needs no CPU round-trip, removing the stall the huge compute gap would otherwise expose. Speculation-then-validation runs the optimizer step speculatively with the gradients on hand — overlapping the next forward pass — and validates gradient clipping and NaN/Inf in the background, rolling back the rare time it must, an exact optimization that preserves correctness. Superchip-aware casting rethinks where mixed-precision conversions happen for the fast link, and GraceAdam is an Adam optimizer tuned for the Grace ARM CPU.
3. On NVIDIA GH200, SuperOffload delivers up to 2.5 times higher training throughput than state-of-the-art offloading systems and trains a 25-billion-parameter model on a single Superchip; extended with ZeRO-style data parallelism and DeepSpeed-Ulysses sequence parallelism, it trains a 13-billion-parameter model with sequences up to a million tokens on eight GH200 at 55 percent model-FLOPs utilization (eight times longer sequences than Ulysses alone), and it is integrated into DeepSpeed behind a few lines of code. Scope: the approach is predicated on tightly-coupled Superchip hardware and does not apply to conventional PCIe servers where the old bandwidth-minimizing designs remain right; speculation adds complexity and occasional rollback; the adaptive choices and the last-bucket-on-GPU trick depend on workload and spare GPU memory; and the numbers are specific to GH200 with DeepSpeed. The durable lesson is that hardware assumptions are baked deep into systems software, so when a tightly-coupled architecture replaces a loosely-coupled one, the right move is to revisit the assumptions — here, that the GPU-CPU link is precious — and redesign around what the new hardware makes cheap.

Check yourself

Check your understanding

1. What assumption underlies existing GPU-CPU offloading systems, and why do Superchips break it?
2. Why does the large GPU-to-CPU compute gap on a Superchip cause a problem, and how does SuperOffload address it?
3. What is the speculation-then-validation technique, and why is it correct?
4. What did SuperOffload achieve, and what are its limits?

Sources

The paper is “SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips” by Xinyu Lian, Masahiro Tanaka, Olatunji Ruwase, and Minjia Zhang, published at ASPLOS 2026, DOI 10.1145/3760250.3762217. The authors are affiliated with the University of Illinois Urbana-Champaign, Anyscale, and Snowflake.

This explainer is based on the full text of the paper, obtained from the authors’ arXiv version (arXiv:2509.21271, a Creative Commons Attribution / CC-BY version). The problem framing (the memory wall in LLM training; offloading model state and computation from GPU to CPU as an alternative to many-GPU sharding, especially for post-training; the assumption in existing systems that the GPU-CPU link is slow PCIe and the resulting objective of minimizing data movement; and how Superchips such as NVIDIA Grace Hopper GH200, integrating a Hopper GPU and an ARM Grace CPU on one package via an NVLink chip-to-chip interconnect at around 900 gigabytes per second, roughly thirty times PCIe, break that assumption and leave the GPU, CPU, and interconnect underutilized), the SuperOffload design (the Superchip-aware data-flow graph; adaptive weight-stationary versus weight-flow offloading chosen by a compute-versus-communication cost model; fine-grained bucketization repartitioning that places the last buckets’ gradients and optimizer states on the GPU to remove the critical-path stall caused by the roughly 330-to-1 Hopper-to-Grace compute gap; the speculation-then-validation algorithm that runs the optimizer step speculatively and validates gradient clipping and NaN/Inf in the background with in-place rollback, preserving exact correctness; Superchip-aware casting for mixed-precision training; and GraceAdam, an Adam optimizer tuned for the Grace ARM CPU), and the results (up to 2.5 times higher training throughput than state-of-the-art offloading systems on NVIDIA GH200, training a 25-billion-parameter model on a single Superchip, and, extended with ZeRO-style data parallelism and DeepSpeed-Ulysses sequence parallelism, training a 13-billion-parameter model with sequence lengths up to a million tokens on eight GH200 at 55 percent model-FLOPs utilization, integrated into DeepSpeed) are all drawn directly from the paper. Background explanations of LLM training, optimizer states, mixed-precision training, gradient clipping, and GPU-CPU offloading are standard and provided to make the paper’s contribution understandable.