TetriServe: Efficiently Serving Mixed DiT Workloads

Runyu Lu, Shiqi He, Wenxuan Tan, Shenggui Li, Ruofan Wu, Jeff J. Ma and 2 others

Diffusion Transformer image models denoise step by step and get very expensive at high resolution, so serving them under per-request deadlines on a shared GPU pool means choosing how many GPUs each request gets — but existing systems fix that number, which is wasteful for small images and too slow for large ones; this paper (University of Michigan — Runyu Lu, Ang Chen, Mosharaf Chowdhury, and colleagues) presents TetriServe, which adjusts the parallelism of each request step by step and packs requests into fixed time rounds like Tetris pieces to hit the most deadlines — raising deadline attainment by up to 32 percent without hurting image quality.

The argument in one glance

Problem

Diffusion image models are expensive, and one fixed parallelism suits no request

Diffusion Transformer (DiT) models generate images by starting from noise and refining it over many denoising steps, each of which is a full transformer pass over all the image tokens. This is compute-bound and expensive, and the cost explodes with resolution: the paper notes that a 2048 by 2048 image on a single H100 GPU can take up to a minute, and a 4096 by 4096 image can exceed ten minutes. In production, these models are served online under strict service-level objectives — a deadline for each request — while sharing a fixed pool of GPUs across many users to keep costs down. To hit tight deadlines on large images, you must split a request across multiple GPUs, and the standard way is sequence parallelism: partition the image tokens across GPUs so they work on the denoising together. The trouble is that requests arrive with mixed resolutions and deadlines, and existing systems assign every request the same fixed degree of parallelism. No single degree works: a low degree (one or two GPUs) is efficient for small images but leaves GPUs idle and runs too slowly on large ones, while a high degree (four or eight GPUs) speeds up large images but wastes huge communication overhead on small ones, clogging the queue. Worse, these systems are non-preemptive — once a request starts on its GPUs, it holds them until it finishes — so a badly-sized request blocks better scheduling of everything behind it.
What people did

Pick a fixed degree of sequence parallelism and run each request to completion

The established approach, used by DiT serving engines like xDiT, is fixed-degree sequence parallelism: choose a number of GPUs to split each request across, and apply it uniformly. A deployment picks a degree — say two or four GPUs per request — and every request runs at that degree from its first denoising step to its last, holding those GPUs for the whole run. The engine may offer a few fixed configurations, but within a request the parallelism never changes, and a running request cannot be interrupted to make room for a more urgent one.
Why it fell short

A fixed degree is wasteful for small images and too slow for large ones, and blocks the queue

A single, static parallelism degree cannot fit a workload of mixed resolutions and deadlines, because the ideal degree depends heavily on the image size. Set it low, and small requests run efficiently but large ones crawl and leave most of the cluster idle, missing their deadlines. Set it high, and large requests are fast but small ones pay so much inter-GPU communication overhead — splitting a tiny amount of work across many GPUs — that they slow down and block the head of the queue. Either way, a large fraction of requests miss their SLOs and GPU utilization is poor. The non-preemptive nature compounds this: once a request begins at its chosen degree, it occupies its GPUs until done, so even when a later, more urgent request arrives, or when finishing one request early would let two others make their deadlines, the scheduler is stuck. The result is a system that cannot adapt the resource given to a request to what that request actually needs, nor rearrange work as deadlines demand. What was missing was the ability to change a request's parallelism as it runs and to schedule the shared GPUs with deadlines explicitly in view.
This paper's idea

Adjust parallelism step by step, and pack requests into time rounds to hit deadlines

TetriServe reframes DiT serving as a fine-grained GPU scheduling problem and exploits a property of DiT inference that LLM serving lacks: because each request is a sequence of denoising steps and DiT is stateless (no KV cache to carry between steps), the number of GPUs a request uses can be changed between steps at essentially no migration cost. This is step-level sequence parallelism: give a request more GPUs when it is large or its deadline is near, fewer when it is small or has slack, adjusting across steps. The catch is that finding the globally optimal step-level schedule under a fixed GPU budget is NP-hard, and requests arrive online needing millisecond scheduling decisions, so exhaustive optimization is impossible. TetriServe makes it tractable with a deadline-aware round-based scheduler: it chops continuous time into fixed-duration rounds, and at each round decides which requests to run and at what parallelism degree. A cost model profiles per-step latency as a function of GPU count and finds, for each request, the minimal GPU allocation that can still meet its deadline; the scheduler then packs these allocations into the round — like fitting Tetris pieces of different widths and heights onto the GPU grid — using dynamic programming to minimize the number of requests that would become late. It adds selective continuous batching to merge small requests' steps and reduce overhead, and keeps idle GPUs busy without disrupting running jobs. On FLUX.1-dev and SD3 across H100 and A40 nodes, TetriServe achieves up to 32 percent higher SLO attainment than the fixed-degree engine xDiT, without degrading image quality.

Why this problem is worth solving

Image generation is becoming a mainstream online service, and unlike text, its cost per request varies by orders of magnitude with resolution, which makes serving it efficiently a distinct and pressing problem.

A great deal of systems work has gone into serving large language models, but DiT image serving is not the same problem, and the paper is explicit about why. First, DiT inference is stateless: there is no KV cache accumulating across the generation, so nothing has to be carried forward, which is what later makes changing a request’s parallelism mid-flight cheap. Second, it is compute-bound rather than memory-bound: every denoising step does a full transformer pass over all the image tokens, so the work is heavy and grows with resolution, and the models are small enough to fit on a single GPU, so the pressure is on compute, not memory capacity. Third, the cost per request spans an enormous range — a small image is quick, a large one can take minutes — so a fleet serving mixed requests faces wildly heterogeneous jobs sharing the same GPUs under individual deadlines. These differences mean the tricks that make LLM serving efficient do not transfer, and that the central lever is how many GPUs to spend on each request and when. Getting that right determines whether a shared GPU pool can meet its customers’ deadlines at acceptable cost, which is exactly the economic question a provider of image generation faces. As these services scale, the efficiency of DiT serving translates directly into how many requests a fixed, expensive GPU fleet can satisfy on time.

The core realization is that the resource a request needs is not a constant to be fixed once, but a knob to be turned continuously as the request runs and as deadlines shift.

Fixed-degree parallelism fails because it answers a per-request, per-moment question with a single global constant. The right amount of parallelism for a request depends on its resolution and how much time is left before its deadline, and both the mix of requests and the urgency of each change constantly as new requests arrive and others complete. TetriServe’s insight is that DiT’s structure — a sequence of independent, stateless steps — makes the parallelism a genuinely adjustable knob: you can give a request eight GPUs for a few steps when its deadline looms and then drop it to one when there is slack, without paying a migration penalty, because there is no state to move. That turns serving into a scheduling problem where the scheduler can, round by round, allocate exactly the parallelism each request needs to just meet its deadline and no more, freeing the rest for others. The reason this is hard, and worth a paper, is that doing it optimally is NP-hard and must be decided in milliseconds online. The contribution is not just the observation that parallelism should be dynamic, but a way to make dynamic, deadline-aware allocation practical: discretize time into rounds so the problem becomes a tractable packing, use a cost model to find each request’s minimal deadline-meeting allocation, and pack with dynamic programming to minimize misses. Recognizing that the elasticity is available and building the machinery to exploit it under real-time constraints is what converts a theoretical win into 32 percent more met deadlines.

What people did before, and where it broke

Fixed-degree sequence parallelism (xDiT). Split every request across a fixed number of GPUs, chosen for the deployment. Where it broke. The ideal degree depends on resolution, so one fixed degree is wasteful for small images (idle GPUs) or too slow and communication-heavy for large ones — poor utilization and missed deadlines either way.
Non-preemptive execution. Once a request starts at its degree, let it hold its GPUs until it finishes. Where it broke. A badly-sized or long request blocks better scheduling of the queue, so more urgent requests behind it miss their deadlines.
Borrowing LLM serving techniques. Apply LLM-serving optimizations to DiT. Where it broke. DiT is stateless, compute-bound, and single-GPU-sized — fundamentally different from LLMs — so KV-cache-centric LLM techniques do not address DiT’s scheduling problem.
Static, deadline-agnostic scheduling. Schedule requests without explicitly reasoning about their deadlines and GPU cost. Where it broke. Without a cost model and deadline awareness, the system cannot allocate just enough parallelism to meet each deadline, so it either overspends GPUs or misses SLOs.
The gap. No system adjusted a request’s parallelism as it ran and scheduled the shared GPUs with deadlines explicitly in view, so mixed DiT workloads suffered low SLO attainment on expensive GPU pools.

How TetriServe works

From fixed-degree, non-preemptive serving to step-level, round-packed scheduling

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Serve generative requests on shared GPUs with deadlines as the goal. TetriServe is fundamentally a QoS-aware inference serving system: it shares a fixed GPU pool across many requests and schedules them to meet per-request service-level objectives. Scheduling inference requests on shared GPUs with quality-of-service targets as the explicit objective, rather than raw throughput alone, is the same QoS-aware serving goal the work on QoS-aware and resource-efficient multi-stage GPU serving (Astraea) covered elsewhere on this site is built on, treating deadlines as first-class. TetriServe brings that discipline to DiT image generation, where the per-request cost varies enormously with resolution.

Give each request elastically just enough resource to meet its deadline. Rather than a fixed allocation, TetriServe sizes each request’s GPU count to the minimum that meets its deadline and adjusts it over time, freeing resources for others. Elastically allocating exactly the resources a job needs to hit its deadline, and reshaping allocations as demands and deadlines change, is the same deadline-driven elasticity the work on an elastic platform for distributed deep learning under deadlines (ElasticFlow) covered elsewhere on this site is built on, scaling resources to satisfy time constraints. TetriServe applies this per denoising step, exploiting DiT’s statelessness to make the elasticity cheap.

Pack variable-sized work into limited capacity as an optimization. TetriServe’s round scheduler packs requests of different GPU-widths and durations onto the GPU grid to minimize misses, solving a Tetris-like packing with dynamic programming. Framing the placement of variable-sized demands into a bounded resource as a packing problem solved by optimization, rather than by a greedy rule, is the same allocation-as-packing approach the work on lightweight memory allocation for hardware-accelerated machine learning (MinimAlloc) covered elsewhere on this site takes, fitting rectangles into limited space without overrunning it. TetriServe packs in the dimensions of GPUs and time, minimizing late completions.

Following a request through TetriServe

From an arriving image request to a met deadline

A trace, drawn from the paper, of how TetriServe schedules one Diffusion Transformer request across denoising steps and rounds.

Step 1 of 5

A request arrives with a resolution and deadline

A user asks for an image at some resolution, with a deadline by which it must be ready. TetriServe must decide how many of the shared GPUs to devote to it, and when, so it finishes in time without starving other requests.

Larger resolutions and tighter deadlines demand more GPUs.

Why it works

Statelessness makes re-parallelizing cheap. Changing a request’s GPU count normally means migrating state → but DiT carries no KV cache between steps → so parallelism can shift step to step at essentially no cost.
Minimal deadline-meeting allocation frees resources. Fixed degrees over- or under-serve every request → but giving each only the GPUs it needs to just meet its deadline → so the rest stay free for other requests to also succeed.
Rounds make an NP-hard problem tractable. Optimal continuous step-level scheduling is NP-hard and online → but discretizing time into fixed rounds → so each round becomes a bounded packing solvable in milliseconds.
DP packing directly minimizes misses. Greedy allocation cannot see the whole round’s tradeoffs → but packing allocations with dynamic programming → so the scheduler chooses the combination that leaves the fewest requests late.
Batching and scale-up recover wasted capacity. Small requests and idle GPUs waste the cluster → but merging small requests’ steps and filling idle GPUs without remapping → so utilization stays high alongside deadline satisfaction.

What it costs (trade-offs and scope)

Rounds trade optimality for tractability. Discretizing time into fixed rounds makes scheduling solvable in milliseconds but is an approximation of the true continuous, NP-hard optimum, so the schedule is good rather than provably best.
It leans on an accurate cost model. Finding each request’s minimal deadline-meeting allocation depends on profiled per-step latencies as a function of GPU count; mispredicted costs would lead to under- or over-allocation.
The statelessness assumption is specific to DiT. The cheap step-level re-parallelization works because DiT has no KV cache; the technique does not transfer to stateful models like LLMs where changing parallelism would require moving state.
Sequence parallelism has communication limits. Splitting a request across many GPUs adds inter-GPU communication, so even optimal packing cannot make a tiny request benefit from a large degree — the cost model must respect these diminishing returns.
Results are on specific models and hardware. The up-to-32-percent SLO-attainment gain and the quality preservation are measured on FLUX.1-dev and SD3 on H100 and A40 nodes against xDiT; other models, hardware, or workload mixes may differ.

The comparison

Serving mixed-resolution DiT requests under deadlines

Property	Fixed-degree parallelism (xDiT)	Deadline-agnostic scheduling	TetriServe
Parallelism per request
Can change mid-request
Deadline awareness
Scheduling method
Reported SLO attainment

A comparison of ways to serve mixed-resolution DiT requests under deadlines. Fixed-degree sequence parallelism (xDiT) gives every request the same number of GPUs, wasting them on small images and slowing large ones, and holds GPUs non-preemptively; deadline-agnostic scheduling lacks a cost model to size allocations. TetriServe adjusts parallelism step by step (cheap because DiT is stateless), uses a cost model to find each request's minimal deadline-meeting allocation, and packs requests into fixed rounds with dynamic programming to minimize late completions — achieving up to 32 percent higher SLO attainment than xDiT on FLUX.1-dev and SD3 without degrading image quality. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ arXiv copy, 2510.01565, of the ASPLOS 2026 paper; code at github.com/DiT-Serving/TetriServe). The described system is the paper’s: TetriServe serves mixed Diffusion Transformer (DiT) image-generation workloads under per-request deadlines on a shared GPU pool, using step-level sequence parallelism — dynamically adjusting each request’s degree of parallelism across denoising steps — which is cheap because DiT inference is stateless (no KV cache), compute-bound, and single-GPU-sized. It proves that optimal step-level scheduling under a fixed GPU budget is NP-hard, and makes it tractable with a deadline-aware round-based scheduler that discretizes time into fixed-duration rounds, uses a cost model of per-step latency versus GPU count to find each request’s minimal deadline-meeting allocation, and packs requests with dynamic programming to minimize late completions; it adds selective continuous batching for small requests and work-conserving elastic scale-up. The headline result — up to 32 percent higher SLO attainment than the fixed-degree engine xDiT without degrading image quality — is from the paper’s evaluation on FLUX.1-dev and SD3 across 8-GPU H100 and 4-GPU A40 nodes, including robustness to bursty arrivals and diverse mixes. Background on diffusion models, sequence parallelism, and SLO-based serving is standard context. The step-by-step walkthrough illustrates the described mechanism. Exact cost-model and scheduling algorithm details and full results should be read from the paper.

Background

Why serving generative models keeps becoming a scheduling problem

It helps to see TetriServe within the broader effort to serve machine-learning models efficiently on shared, expensive GPUs. As inference has moved into production, the core challenge has shifted from making a single model fast to sharing a fixed fleet across many requests that each carry their own latency target, at the lowest cost. This is fundamentally a scheduling and resource-allocation problem: which request runs where, with how much hardware, and when, so that as many as possible meet their service-level objectives while the fleet stays busy. The recurring themes are treating quality-of-service targets as first-class objectives rather than afterthoughts, sizing each job’s resources to what it actually needs instead of a one-size-fits-all allocation, and packing heterogeneous jobs onto shared hardware without waste. Generative models sharpen the challenge because their per-request cost varies so much — a short prompt versus a long one, a small image versus a huge one — so heterogeneity is the norm, not the exception. This drive to serve many demanding, heterogeneous model requests on shared GPUs while honoring each one’s performance target is the same terrain the work on high-performance multi-tenant deep learning serving (Veltair) covered elsewhere on this site occupies, packing diverse inference onto shared accelerators. TetriServe’s particular contribution is to notice that Diffusion Transformers, being stateless and step-structured, let the scheduler change each request’s parallelism on the fly, and to build the round-based, deadline-aware machinery that turns that flexibility into substantially more met deadlines.

If you remember only three things

1. Diffusion Transformer (DiT) models generate images through many denoising steps and are compute-bound, with cost exploding at high resolution (a 4096 by 4096 image can take over ten minutes on one GPU). Served online under per-request deadlines on a shared GPU pool, they demand parallelism — usually sequence parallelism, splitting image tokens across GPUs. But existing engines like xDiT assign every request the same fixed degree of parallelism, and no single degree fits a workload of mixed resolutions: a low degree wastes GPUs and is too slow on large images, a high degree drowns small images in communication overhead and blocks the queue. Because these engines are non-preemptive, a badly-sized request holds its GPUs to completion and stalls better scheduling of everything behind it, so many requests miss their deadlines.
2. TetriServe reframes DiT serving as a fine-grained GPU scheduling problem and exploits that DiT inference is stateless — with no KV cache to carry between steps, a request's GPU count can be changed between denoising steps at essentially no migration cost. This step-level sequence parallelism gives a request more GPUs when it is large or its deadline is near and fewer when it has slack. Since optimal step-level scheduling under a fixed GPU budget is NP-hard and requests arrive online, TetriServe makes it tractable with a deadline-aware round-based scheduler: it discretizes time into fixed-duration rounds, uses a cost model of per-step latency versus GPU count to find each request's minimal deadline-meeting allocation, and packs these allocations onto the GPU grid with dynamic programming — like fitting Tetris pieces — to minimize how many requests become late. Selective continuous batching and work-conserving scale-up keep the cluster busy.
3. The key insight is that the right amount of parallelism is a per-request, per-moment knob, not a global constant, and DiT's stateless step structure makes that knob cheap to turn — so serving becomes a deadline-aware packing problem the round scheduler can solve in milliseconds. On FLUX.1-dev and SD3 across H100 and A40 nodes, TetriServe achieves up to 32 percent higher SLO attainment than the fixed-degree engine xDiT without degrading image quality, and is robust to bursty arrivals and diverse mixes. The trade-offs: rounds approximate the true continuous optimum, the scheme leans on an accurate per-step cost model, the cheap re-parallelization is specific to stateless DiT (not stateful LLMs), and sequence parallelism's communication limits bound how much a small request can gain from more GPUs.

Check yourself

Check your understanding

1. Why does fixed-degree sequence parallelism serve mixed DiT workloads poorly?
2. What property of DiT inference does TetriServe exploit, and what does it enable?
3. How does TetriServe make the scheduling tractable, given it is NP-hard?
4. What did TetriServe achieve, and what is a key limitation of its core trick?

Sources

The paper is “TetriServe: Efficiently Serving Mixed DiT Workloads” by Runyu Lu, Shiqi He, Wenxuan Tan, Shenggui Li, Ruofan Wu, Jeff J. Ma, Ang Chen, and Mosharaf Chowdhury (University of Michigan), published at ASPLOS 2026, DOI 10.1145/3779212.3790233. It is available on arXiv as 2510.01565, with code at github.com/DiT-Serving/TetriServe.

This explainer is written from the paper’s full text (the authors’ arXiv copy). TetriServe serves mixed Diffusion Transformer image-generation workloads under per-request deadlines on a shared GPU pool via step-level sequence parallelism — dynamically adjusting each request’s degree of parallelism across denoising steps, which is cheap because DiT inference is stateless (no KV cache), compute-bound, and single-GPU-sized. The paper proves optimal step-level scheduling is NP-hard and makes it tractable with a deadline-aware round-based scheduler that discretizes time into fixed rounds, uses a per-step cost model to find each request’s minimal deadline-meeting GPU allocation, and packs requests with dynamic programming to minimize late completions, adding selective continuous batching and work-conserving scale-up. Evaluated on FLUX.1-dev and SD3 across H100 and A40 nodes, the paper reports up to 32 percent higher SLO attainment than the fixed-degree engine xDiT without degrading image quality. Background on diffusion models, sequence parallelism, and SLO-based serving is standard context. The numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.