DIP: Efficient Large Multimodal Model Training with Dynamic Interleaved Pipeline

Zhenliang Xue, Hanpeng Hu, Xing Chen, Yimin Jiang, Yixin Song, Zeyu Mi and 4 others

Large multimodal models are trained by splitting the model across GPUs as a pipeline, but they are stitched together from very different pieces (image encoders, video encoders, a language backbone) and fed batches whose mix of text, images, and video changes constantly — so the pipeline stages are chronically unbalanced and GPUs sit idle waiting on the slowest one; this paper (Shanghai Jiao Tong University and StepFun — Zhenliang Xue, Zeyu Mi, Haibo Chen, and colleagues) presents DIP, which gives each modality its own balanced pipeline segment, splits data to even out the stages, and generates a fresh schedule for every batch on idle CPUs, raising training throughput by up to 97.3 percent.

The argument in one glance

Problem

Multimodal models are patchworks, and their training pipeline never stays balanced

A large multimodal model (LMM) is not one uniform network; it is assembled from several very different pieces — an encoder for text, another for images, another for video, one or more decoders, and a large language-model backbone in the middle, all connected by adapters. This lets it accept flexible mixes of input (a prompt with an image and a video clip, say) and produce text or audio. To train such a huge model, systems use pipeline parallelism: they cut the model into consecutive stages, place each stage on a different machine, and stream batches of data through the stages like an assembly line, so many machines work at once. Pipelines only run efficiently when the stages take roughly equal time; otherwise the fast stages finish early and sit idle waiting for the slow one, and that idle time is called a pipeline bubble. Multimodal models make balancing the stages very hard, for two compounding reasons. First, the modality modules have wildly different computational costs and operator types (an image encoder does convolutions, the backbone does attention and matrix multiplies), so no clean split of layers into equal stages exists — even the best partitioning of a 37-billion-parameter LMM still leaves about 22.8 percent of the time as bubbles. Second, the data itself keeps changing: one batch is mostly text, the next is heavy with images or video, and the biggest sample can demand about 4.15 times the compute of the smallest. So the balance that was right for one batch is wrong for the next. Together these produce dynamic imbalance, which can inflate training time by up to 40.3 percent.
What people did

Use one static pipeline schedule, or balance the data batches

The standard practice is to compute a single, static pipeline schedule once — for example Megatron-LM's 1F1B (one-forward-one-backward) schedule — and reuse it for every iteration. To reduce the imbalance from variable data, systems also apply data packing, bundling samples together to make batches more uniform, and some training systems for dynamic text lengths adaptively reorder microbatches to shave bubbles.
Why it fell short

A fixed schedule cannot follow a moving target, and text tricks miss modality imbalance

These fall short because the imbalance is dynamic and modality-specific. A single static schedule is computed for one assumed workload, but the actual per-batch mix of modalities changes constantly, so the schedule is mismatched for most batches and the bubbles persist. Data packing makes batches somewhat more uniform overall but does not remove the underlying computational imbalance, because packing cannot equalize how much work an image encoder does versus the backbone. And the techniques built for unimodal language models — which handle variable text sequence lengths by reordering or repacking microbatches — assume the workload variation affects all layers uniformly; they cannot address inter-modality imbalance, where adding more images overloads the image encoder specifically while leaving other modules underused. What was missing was a scheduler that attacks the imbalance at its source (the fact that different modalities have different costs) and adapts to each batch's actual modality composition, without the adaptation itself slowing training down.
This paper's idea

Give each modality its own segment, split data to balance them, and re-plan every batch

DIP is a dynamic, modality-aware pipeline scheduling framework for LMM training, built on three principles. First, modality-aware partitioning tackles the imbalance at its source. Rather than co-locating different modalities' computations in one pipeline pass — where the slowest modality bottlenecks the rest — DIP uses separated partitioning, dedicating a distinct pipeline segment (a complete forward or backward pass across all stages) to each modality, so a slow image encoder no longer stalls the language stages. Even after separation the segments differ in cost, so DIP adds modality-aware batching: it splits a batch into smaller, modality-specific sub-microbatches, so that, for example, a slow vision segment runs several short stages whose latency matches a single language stage, aligning the whole pipeline. Second, because the right schedule depends on each batch's modality mix, DIP performs asynchronous schedule generation: for every upcoming batch it generates a tailored schedule on the fly, running the planning on idle CPU cores in parallel with the GPUs so the search stays off the critical path and never stalls training. Third, since searching the schedule space exactly (as an integer linear program) would take minutes to hours — far more than a 10-to-60-second training iteration allows — DIP uses a decomposed search: it breaks the problem into pipeline-segment reordering (explored with Monte Carlo Tree Search) and then stage interleaving, guided by a fast training simulator that estimates stage latencies and memory. Across five LMMs from 12 to 94 billion parameters, DIP achieves up to 97.3 percent higher training throughput than state-of-the-art systems.

Why this problem is worth solving

Multimodal models are where large-model capability is heading, and training them is enormously expensive, so wasted pipeline time is wasted money at scale.

Large multimodal models — able to understand and generate across text, images, video, and audio — are becoming the foundational architecture for the next generation of AI systems, powering tasks from document understanding to multi-turn dialog. But training them consumes vast fleets of GPUs for long stretches, and at that scale even a modest inefficiency translates into large amounts of hardware time and energy. Pipeline parallelism is the standard way to fit and train a model too big for one machine, but its Achilles’ heel is the bubble: every unit of time a GPU spends idle waiting for a slower stage is compute paid for and thrown away. When bubbles can consume tens of percent of training time, as they do for these models, closing that gap directly cuts the cost and time to train a model. So making LMM pipeline training efficient is not a marginal tuning problem; it is about reclaiming a large fraction of a very expensive resource.

What makes it genuinely hard is that the imbalance is both structural and constantly moving, which defeats any fixed plan.

Two problems combine into one. The structural problem is that an LMM is heterogeneous by construction: its modality modules do fundamentally different kinds of computation, with different operator types, shapes, and memory patterns, so there is no way to cut the model into equal-cost pipeline stages — the pieces simply are not equal, and even an exhaustive search over layer splits leaves a big residual imbalance. The moving problem is that the training data is diverse: the proportion of text, images, and video varies dramatically from batch to batch, so the amount of work each modality module must do swings up and down over the course of training. Crucially, these interact: a batch with many images does not just add uniform work, it overloads the image encoder specifically while other modules idle, so the location of the bottleneck shifts with the data. A single static schedule, computed once, is therefore right for at most one workload and wrong for the rest, and generic fixes for text-length variation cannot help because they assume the variation is uniform across the model. The only way to keep the pipeline balanced is to change the plan as the data changes — but re-planning constantly risks costing more than it saves, so it must be done without stalling the training. Solving that is the crux.

What people did before, and where it broke

Reuse one static pipeline schedule (Megatron-LM 1F1B). Compute a schedule once and apply it every iteration. Where it broke. The per-batch modality mix changes, so a fixed schedule is mismatched for most batches and leaves large bubbles.
Pack data into more uniform batches. Bundle samples together to smooth out batch-to-batch variation. Where it broke. Packing cannot equalize the intrinsic cost difference between modalities, so computational imbalance persists even in a well-packed batch.
Reorder or repack microbatches as in unimodal LLM training. Apply techniques built for variable text sequence lengths. Where it broke. Those assume the workload variation hits all layers uniformly; they cannot resolve inter-modality imbalance, where more images overload one specific module.
Search for the optimal schedule with an exact solver. Formulate scheduling as an integer linear program and solve it. Where it broke. Exact solving takes minutes to hours, far beyond the 10-to-60-second budget of a single training iteration, so it cannot adapt in real time.
The gap. No scheduler attacked the imbalance at its modality-specific source and adapted to each batch’s composition without the adaptation itself stalling training.

How DIP works

Keeping a multimodal training pipeline balanced, batch by batch

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Partition a heterogeneous model into balanced pipeline stages. DIP’s foundation is pipeline parallelism done carefully: it decides how to cut a large, uneven model into stages and segments so the assembly line stays as balanced as possible, because balance is what determines how much of the pipeline is wasted on bubbles. Splitting a big model across devices as a pipeline and shaping the partition to minimize the idle time between stages is the same undertaking the work on optimizing pipeline parallelism with adaptive recomputation (AdaPipe) covered elsewhere on this site pursues, treating the partition itself as the lever for efficiency. DIP does this for multimodal models, where the pieces are heterogeneous, by giving each modality its own segment and sizing sub-microbatches to match latencies.

Build the system to expect and absorb dynamism. The reason DIP re-plans every batch is that the workload is not fixed: the modality composition of the data is a moving target, so the system is designed around change rather than a single assumed shape. Treating dynamism — inputs whose shape and cost vary at runtime — as a first-class thing to handle, instead of assuming a static workload, is the same principle the work on composable abstractions for end-to-end dynamic machine learning (Relax) covered elsewhere on this site is built on, making the system adapt to what actually arrives. DIP tailors a schedule to each batch’s real modality mix, so the plan tracks the data instead of lagging behind it.

Search a vast configuration space fast with a cost model instead of solving it exactly. DIP cannot afford an exact optimizer, so it explores the space of schedules heuristically, using a fast simulator to estimate the quality of candidates and steer the search toward good ones within a tight time budget. Replacing an intractable exact search with a guided exploration that leans on a quick performance estimate to find near-optimal configurations is the same idea the work on a fast Bayesian compiler optimization framework (BaCO) covered elsewhere on this site develops, trading exhaustive optimality for a model-guided search that finishes in time. DIP applies this with Monte Carlo Tree Search over segment orderings, guided by its training simulator.

Following a training iteration through DIP

Scheduling one batch without stalling the GPUs

A trace, drawn from the paper, of how DIP produces a tailored schedule for a training batch.

Step 1 of 5

The planner prefetches the next batch's metadata

On idle CPU cores, DIP's online planner fetches metadata for the upcoming batch — such as token counts and the number of images — so it knows the batch's modality composition before the GPUs need the schedule.

This runs in parallel with GPU training, off the critical path.

Why it works

Separating modalities removes the worst source of bubbles. Co-locating different modalities in one pipeline pass lets the slowest one stall the rest → but dedicating a segment to each modality isolates them → so a slow encoder no longer bottlenecks the language stages.
Sub-microbatching aligns the segments that remain unequal. Even separated segments differ in cost → but splitting data into modality-specific sub-microbatches sizes each segment’s stages to match → so latencies across the pipeline line up and bubbles shrink.
Per-batch scheduling tracks the moving workload. A static schedule is right for only one modality mix → but generating a fresh schedule per batch adapts to the actual composition → so every batch, not just the average, runs with a fitted schedule.
Asynchronous planning hides the search cost. Re-planning every batch could easily cost more than it saves → but running the search on idle CPUs in parallel with the GPUs keeps it off the critical path → so adaptation adds no stall to training.
Decomposition plus a simulator makes the search fit the budget. An exact schedule solver takes minutes to hours → but decomposing into MCTS reordering and interleaving, scored by a fast simulator → finds good schedules within a 10-to-60-second iteration.

What it costs (trade-offs and scope)

It depends on having idle CPU capacity. DIP hides the scheduling search by running it asynchronously on spare CPU cores; if those CPUs are saturated by other work, the planning can no longer fully overlap with GPU training, weakening the benefit.
It relies on the simulator’s accuracy. The search is guided by a training simulator’s estimates of stage latency and memory, so a mismatch between the simulator and the real hardware yields worse schedules; the system allows custom estimation rules to override the cost model, which is itself extra tuning.
It adds substantial system machinery. DIP introduces an offline model-chunk partitioner, an online sub-microbatch partitioner and schedule searcher, a training simulator, and a reconfigurable pipeline runtime built on Megatron-LM — considerably more complex than reusing a single static 1F1B schedule.
The search is heuristic, not optimal. To meet the per-iteration time budget, DIP decomposes the problem and uses Monte Carlo Tree Search rather than solving the exact integer program, so the schedules it finds are high-quality but not guaranteed optimal.
It targets multimodal pipeline parallelism specifically. The gains come from modality heterogeneity and data dynamicity under pipeline parallelism; a unimodal model, a workload with a stable data mix, or a different parallelization strategy would see smaller benefit, and sub-microbatch splitting has practical granularity limits.

The evaluation

Training a large multimodal model with pipeline parallelism

Property	Static 1F1B + data packing	Unimodal reorder/repack tricks	DIP
Handles modality cost differences
Adapts to each batch's mix
Planning cost on the training loop
Search method
Training throughput

A comparison of ways to schedule multimodal model training under pipeline parallelism. By giving each modality its own balanced pipeline segment, splitting data into modality-specific sub-microbatches, and generating a tailored schedule per batch asynchronously on idle CPUs (searched with MCTS and a training simulator), DIP eliminates the dynamic imbalance a static schedule cannot, reaching up to 97.3 percent higher throughput across five LMMs from 12 to 94 billion parameters.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ arXiv copy of the ASPLOS 2026 paper). The system described is the paper’s: DIP is a dynamic, modality-aware pipeline scheduling framework for large multimodal model training that addresses dynamic imbalance from two sources — pipeline stage imbalance due to heterogeneous modality modules, and training data dynamicity from diverse multimodal data. It uses modality-aware partitioning: separated partitioning dedicates a distinct pipeline segment (a complete forward or backward pass across all stages) to each modality to remove intra-segment imbalance, and modality-aware batching splits data into smaller modality-specific sub-microbatches to balance the segments. It generates a tailored schedule for each batch via asynchronous schedule generation on idle CPU cores, off the critical path, and searches efficiently with a decomposed algorithm (Monte Carlo Tree Search for pipeline-segment reordering, then stage interleaving) guided by a training simulator, since an exact integer-linear-program search is too slow for a 10-to-60-second iteration. The paper reports that even an optimal static partitioning of a 37-billion-parameter LMM leaves about 22.8 percent bubbles, that dynamic imbalance can inflate training time by up to 40.3 percent, that the largest data sample can require about 4.15 times the compute of the smallest, and that DIP achieves up to 97.3 percent higher training throughput than state-of-the-art systems across five LMMs from 12 to 94 billion parameters (vision-language and diffusion models). DIP is implemented on Megatron-LM. Exact algorithms, the simulator, the partitioning and search details, and full results should be read from the paper.

Background

Why scaling model training keeps turning into a scheduling problem

It helps to see DIP within a broad trend in large-scale machine learning: as models outgrow any single device, the central difficulty shifts from the math of the model to the logistics of running it across a fleet, and that logistics is fundamentally a scheduling and partitioning problem. How you cut a model into pieces, how you place those pieces on machines, how you stream data through them, and how you overlap computation with communication together determine how much of the expensive hardware actually does useful work. Every advance in this space is a different answer to the same question — how to keep thousands of accelerators busy rather than waiting on each other — and the answers keep having to grow more dynamic, because modern workloads are less uniform than the dense, regular training of a few years ago. Multimodal models push this to an extreme: they are heterogeneous inside and fed heterogeneous data, so the old assumption that one carefully chosen static plan suffices simply breaks. The broad, long-running effort to design and optimize the full stack for training ever-larger models efficiently is the same terrain the work on the journey of designing and optimizing a full-stack machine-learning platform (AWS Trainium) covered elsewhere on this site occupies, wrestling with the same goal of maximizing useful work across a large training system. DIP’s contribution is to show that for multimodal training, the plan itself must become dynamic — recomputed for every batch, off the critical path — so that the pipeline stays balanced even as the data mix refuses to hold still.

If you remember only three things

1. Large multimodal models are assembled from heterogeneous modality modules (text, image, and video encoders, decoders, and a language backbone), and trained with pipeline parallelism by cutting the model into stages across GPUs. But the modules have very different computational costs, so the stages cannot be balanced (even an optimal static partition of a 37-billion-parameter model leaves about 22.8 percent bubbles), and the modality mix of each data batch fluctuates, so a fixed schedule is wrong for most batches. This dynamic imbalance can inflate training time by up to 40.3 percent, and a single static schedule cannot fix it.
2. DIP is a dynamic, modality-aware pipeline scheduler with three ideas. Separated partitioning dedicates a distinct pipeline segment to each modality so a slow encoder no longer bottlenecks the language stages, and modality-aware sub-microbatching splits data so each segment's stage latency matches the others. For each batch it generates a fresh schedule tailored to that batch's modality mix, running the search asynchronously on idle CPU cores so it stays off the critical path. Because an exact schedule solver is far too slow for a 10-to-60-second iteration, it decomposes the search into Monte Carlo Tree Search segment reordering plus stage interleaving, guided by a fast training simulator.
3. Across five LMMs from 12 to 94 billion parameters (vision-language and diffusion models), DIP achieves up to 97.3 percent higher training throughput than state-of-the-art systems. The trade-offs: it depends on having idle CPU capacity to hide the planning, relies on the simulator's accuracy (with custom cost rules as an override), adds substantial system machinery on top of Megatron-LM, uses a heuristic rather than optimal search to meet the time budget, and targets multimodal pipeline parallelism specifically, so unimodal or stable-workload training benefits less.

Check yourself

Check your understanding

1. What is "dynamic imbalance" in large multimodal model training, and why does it cause pipeline bubbles?
2. How does DIP's modality-aware partitioning reduce pipeline bubbles at their source?
3. Why does DIP generate a new schedule for every batch, and how does it avoid this slowing training down?
4. Predict: DIP needs a schedule within a 10-to-60-second training iteration, but the exact scheduling problem is an integer linear program that can take minutes to hours to solve. How does DIP get a good schedule in time?

Sources

The paper is “DIP: Efficient Large Multimodal Model Training with Dynamic Interleaved Pipeline” by Zhenliang Xue, Hanpeng Hu, Xing Chen, Yimin Jiang, Yixin Song, Zeyu Mi, Yibo Zhu, Daxin Jiang, Yubin Xia, and Haibo Chen (Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University, and StepFun), published at ASPLOS 2026, DOI 10.1145/3779212.3790154. It is licensed CC BY, with an author copy available at arXiv:2504.14145.

This explainer is written from the paper’s full text (the authors’ arXiv copy). DIP is a dynamic, modality-aware pipeline scheduling framework for large multimodal model training that addresses dynamic imbalance, which arises from pipeline stage imbalance (heterogeneous modality modules with different computational costs) and training data dynamicity (diverse, fluctuating multimodal data). DIP separates the computations of different modalities into dedicated pipeline segments to remove intra-segment imbalance, and dynamically splits input data into finer-grained, modality-specific sub-microbatches to balance workloads across segments. It asynchronously generates a tailored pipeline schedule for each batch on idle CPU resources in parallel with GPU training, keeping the search off the critical path, and uses a decomposed search (Monte Carlo Tree Search for pipeline-segment reordering, then stage interleaving) guided by a training simulator to meet the tight per-iteration time budget. The paper reports that DIP achieves up to 97.3 percent higher training throughput than state-of-the-art systems across five LMMs ranging from 12 to 94 billion parameters (including vision-language and diffusion models), and that dynamic imbalance can otherwise inflate training time by up to 40.3 percent. Numbers are the paper’s own; exact algorithms, the simulator, and full results should be confirmed in the paper.