Towards High-Goodput LLM Serving with Prefill-decode Multiplexing

Yukang Chen, Weihao Cui, Han Zhao, Ziyi Xu, Xiaoze Fan, Xusheng Chen and 4 others

An LLM answers in two very different phases — a compute-heavy prefill that reads your whole prompt, and a memory-bound decode that emits tokens one by one — and each has its own deadline, so running them on the same GPU makes a long prefill stall decode and blow its latency target; this paper (SJTU, NUS, Huawei — Yukang Chen, Quan Chen, Bingsheng He, and colleagues) presents MuxWise, which runs prefill and decode on separate parts of one GPU at the same time, sharing memory so the KV cache stays large, and carefully schedules them to avoid idle gaps and interference — for 2.2 times higher throughput under latency guarantees on average.

The argument in one glance

Problem

An LLM's two phases have different deadlines and fight over the same GPU

An LLM answers a request in two phases with very different characters. The prefill phase reads the whole input prompt at once and produces the first token; it is compute-heavy and parallel. The decode phase then generates the remaining tokens one at a time, each step light on computation but bottlenecked on memory bandwidth. Serving has strict deadlines for both: a chatbot might require the first token within 500 milliseconds (a Time-To-First-Token target) and each subsequent token within 100 milliseconds (a Time-Between-Tokens target). The metric that matters is goodput — the peak throughput a system can sustain while still meeting these deadlines. The conflict is that on a shared GPU the two phases interleave and interfere. To keep the memory-bound decode efficient, servers batch many decodes together, but to add a new request they must stall the ongoing decode to run its prefill first. A long prefill therefore delays the waiting decodes, potentially pushing them past their per-token deadline. Multi-turn workloads make this worse and more valuable to get right, because consecutive turns share context, so systems reuse the stored key-value (KV) cache across requests to avoid recomputing it. Meeting both phases' deadlines while keeping the GPU busy and the KV cache large is the crux of high-goodput serving.
What people did

Separate the phases onto different GPUs, or fuse prefill chunks into decode

Two families of solution exist. Disaggregated serving (as in Splitwise) puts prefill and decode on distinct GPU instances, so a long prefill on one set of GPUs cannot stall decode on another, protecting each phase's deadline. Chunked prefill instead keeps both on the same GPU but splits the prefill into small chunks and fuses each chunk with a decode iteration, capping a combined token budget so the fused step stays short enough to meet the decode deadline. Both are widely used ways to keep the phases from trampling each other's latency.
Why it fell short

Disaggregation wastes resources and shrinks the cache; chunking couples latency to utilization

Each approach breaks in its own way. Disaggregation is rigid and wasteful: GPUs are statically split between prefill and decode at startup, so under fluctuating load one pool sits idle while the other is overwhelmed. Worse, it fragments memory — each instance gets its own KV cache pool, so the effective cache is smaller, the cross-request cache hit rate collapses (the paper cites a fall from 36.6 percent to 4.2 percent), and the misses force expensive recomputation that erodes goodput. Chunked prefill keeps the full shared cache but creates a dilemma. Because a prefill chunk and a decode iteration run together as one step, a single token budget governs both the decode deadline and how fully the GPU is used, and no value satisfies both: the paper notes that a 70-billion-parameter model on eight A100 GPUs needs a token budget of about 4,000 to saturate the GPU, which is eight times larger than the budget of 256 that a 100-millisecond decode deadline allows. And each prefill chunk must re-read the KV cache of all earlier chunks, which inflates the per-token time; with the very long reused contexts common in multi-turn workloads, chunked prefill can fail the deadline outright. So one approach cannot adapt and shrinks the cache, and the other cannot separate the deadline from utilization — neither sustains high goodput.
This paper's idea

Run prefill and decode side by side on separate cores of one GPU

MuxWise introduces intra-GPU prefill-decode multiplexing: rather than making prefill and decode take turns or fuse into one step, it runs them at the same time on different parts of the same GPU — different streaming multiprocessors (SMs), the GPU's compute cores — via spatial sharing. This gets the best of both prior approaches. Because the two phases run on separate compute partitions, they execute independently, so a long prefill no longer stalls decode and the deadline is decoupled from GPU utilization, escaping chunked prefill's dilemma. Because they run on the same GPU, they share its memory, so the KV cache pool stays full-sized and cross-request reuse stays effective, escaping disaggregation's cache fragmentation. And the compute split between prefill and decode can be reconfigured cheaply as load shifts, escaping disaggregation's rigidity. Making this work is non-trivial, so MuxWise adds three parts. A bubble-less multiplex engine partitions prefill into per-layer pieces so its execution latency aligns with decode's and neither leaves the other's cores idle. A contention-tolerant estimator predicts worst-case latency under the interference that spatial sharing causes — chiefly for memory bandwidth, which SM partitioning does not control — by combining a solo-run predictor with a contention guard learned from one-time offline profiling. And an SLO-aware dispatcher uses the engine and estimator to pick, for the current mix of requests, the multiplexing plan that maximizes goodput while respecting the deadlines. Built on the SGLang serving system, MuxWise improves peak throughput under deadline guarantees by an average of 2.20 times, and up to 3.06 times, over state-of-the-art baselines.

Why this problem is worth solving

Serving is where the cost and quality of every LLM deployment are decided, and the prefill-decode tension sits at the center of it.

It is tempting to measure a serving system by raw tokens per second, but that number is misleading if the tokens arrive too late to be useful. Interactive LLM services live and die by latency: a user waiting for the first word of a response, or watching it stutter out token by token, is judging the system against deadlines, and a request that misses its deadline is, for practical purposes, a failure even if the GPU was busy producing it. Goodput captures this correctly — it is the throughput the system can sustain while still meeting the deadlines — and it is the metric a provider actually cares about, because it determines how many paying, satisfied users a fixed, expensive GPU fleet can serve. The reason the prefill-decode structure makes this hard is that the two phases have opposite resource profiles and opposite deadline pressures, yet must share the same hardware. Prefill is a big compute burst that, if run carelessly, blocks the steady trickle of decode that users are watching; decode is cheap per step but must never stall. Balancing them well is the difference between a system that saturates its GPUs at low latency and one that either wastes hardware or misses deadlines. Because LLM serving is now an enormous and growing cost, and because latency is what users feel, getting this balance right is directly worth money and user experience at scale.

The deeper insight is that the field had been forced to choose between independence and sharing, and that spatial multiplexing on one GPU dissolves that false choice.

Look closely and the two prior approaches are trading the same two goods against each other. Disaggregation buys independence — the phases cannot interfere because they are on different machines — but pays for it by splitting memory, which fragments the KV cache that multi-turn workloads depend on, and by fixing the compute split, which cannot follow changing load. Chunked prefill buys shared memory and a full cache by keeping everything on one GPU, but pays by fusing the phases into a single step whose one knob, the token budget, must simultaneously be small enough for the decode deadline and large enough to saturate the GPU — an impossible demand. MuxWise’s reframing is that these goods are not actually in conflict; they only appeared to be because the phases were forced to either live on separate hardware or execute as one operation. A modern GPU has many independent compute cores, so the phases can run at the same time on different cores of the same chip: independent in execution, like disaggregation, yet sharing memory, like chunking, and with a compute split that can be adjusted on the fly. That is the whole idea, and it is powerful because it removes the coupling that made the old dilemma unavoidable. The remaining difficulty is engineering — keeping the concurrently-running phases from leaving cores idle, and taming the memory-bandwidth contention that spatial sharing introduces — which is precisely what MuxWise’s three components address. Recognizing that concurrency on one GPU is a third option, not a compromise between the other two, is the conceptual unlock.

What people did before, and where it broke

Disaggregated serving (Splitwise). Put prefill and decode on separate GPU instances so neither stalls the other. Where it broke. The static split cannot adapt to fluctuating load (idle resources), and separate KV cache pools shrink the effective cache, collapsing cross-request hit rate and forcing recomputation that lowers goodput.
Dynamic disaggregation without cross-request reuse (LoongServe). Reallocate GPUs by request length and phase. Where it broke. It cannot reuse the KV cache across requests, so multi-turn workloads pay heavy recomputation overhead.
Chunked prefill. Split prefill into chunks and fuse each with a decode iteration under one token budget. Where it broke. The single budget must be small for the decode deadline yet large to saturate the GPU (256 versus about 4,000 in the paper’s example), and re-reading prior chunks’ KV inflates per-token time, so long reused contexts can miss the deadline outright.
Naive GPU spatial sharing. Partition the GPU’s compute cores between tasks. Where it broke. Differing phase latencies leave cores idle (bubbles), and partitioning compute gives no control over shared memory bandwidth, so contention makes latency unpredictable and deadlines hard to guarantee.
The gap. No system ran prefill and decode independently yet on shared memory with adaptive compute, while controlling the bubbles and contention that concurrency introduces — which is what high goodput requires.

How MuxWise works

From taking turns or fusing to concurrent prefill and decode on one GPU

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Maximize throughput while honoring latency deadlines. MuxWise is a serving system whose goal is goodput: pushing as many requests through the GPUs as possible without missing their deadlines. Driving up how many requests a serving system handles while respecting each request’s service-level target is the same throughput-under-guarantees goal the work on a high-throughput inference serving system (Proteus) covered elsewhere on this site is built on, treating deadlines as the constraint on throughput. MuxWise pursues it for the specific structure of LLM inference, where the prefill and decode phases must both be kept within their targets.

Share one GPU between concurrent work without letting them collide. MuxWise’s mechanism is to run two different computations on one GPU at once, on separate cores, while controlling their mutual interference. Co-locating multiple deep-learning computations on a single GPU and isolating their performance so they do not degrade each other is the same concurrent-GPU-sharing challenge the work on non-intrusive performance isolation for concurrent deep learning (Tally) covered elsewhere on this site addresses, keeping co-running jobs from interfering. MuxWise applies this to the two phases of one LLM request, adding a contention model precisely because naive spatial sharing does not isolate memory bandwidth.

Predict behavior under interference to keep guarantees. MuxWise’s dispatcher relies on estimating worst-case latency under contention, so it can schedule aggressively yet still meet deadlines. Managing shared resources by predicting performance under uncertainty and interference, so that quality targets are met despite co-location, is the same uncertainty-aware resource-management approach the work on quality-of-service and uncertainty-aware resource management (Aquatope) covered elsewhere on this site takes, planning against the worst case rather than the average. MuxWise’s contention-tolerant estimator plays that role, turning unpredictable spatial-sharing interference into a bound the dispatcher can plan around.

Following a request mix through MuxWise

From an arriving mix of prefills and decodes to a deadline-meeting schedule

A trace, drawn from the paper, of how MuxWise multiplexes prefill and decode on one GPU for a batch of requests.

Step 1 of 5

A mix of new and ongoing requests arrives

Some requests need their prompt processed (prefill) while others are mid-generation (decode), and multi-turn requests carry reusable KV cache from earlier turns. All must meet their first-token and per-token deadlines on the shared GPU.

The shared full-size KV cache lets multi-turn reuse stay effective.

Why it works

Concurrency decouples the deadline from utilization. Fusing prefill and decode ties one budget to both → but running them on separate cores at once → so decode meets its deadline while prefill still saturates the GPU.
Shared memory keeps the KV cache large. Disaggregation splits memory and collapses cache reuse → but multiplexing on one GPU shares its memory → so the full cache pool and cross-request reuse are preserved.
A reconfigurable core split adapts to load. A static prefill-decode allocation idles resources under shifting demand → but cheaply adjusting the compute partition → so the GPU follows the changing prefill-to-decode ratio.
Per-layer prefill erases the bubbles. Concurrent phases of different lengths leave cores idle → but partitioning prefill into layers to align latencies → so neither phase wastes the other’s cores.
A contention bound makes deadlines guaranteeable. Spatial sharing causes unpredictable memory-bandwidth interference → but predicting worst-case latency with a profiled contention guard → so the dispatcher can schedule aggressively and still keep the deadlines.

What it costs (trade-offs and scope)

Spatial sharing still causes real interference. Running two phases on one GPU contends for memory bandwidth that core partitioning does not isolate; MuxWise plans around it with a worst-case estimator rather than eliminating it, so it accepts some efficiency loss to keep guarantees.
The estimator needs offline profiling. The contention guard comes from one-time offline profiling; a workload or hardware far from what was profiled could make the worst-case bound loose or inaccurate, affecting how aggressively the dispatcher can pack work.
It adds scheduling machinery to the serving path. The multiplex engine, estimator, and dispatcher are extra components and decisions on the critical path compared with a simpler fused or disaggregated server, a complexity cost the goodput gain must justify.
It targets GPUs with controllable core partitioning. Intra-GPU multiplexing assumes the ability to split streaming multiprocessors between phases; hardware or software stacks without that capability would need adaptation.
Results are on specific models, hardware, and workloads. The 2.20-times average and 3.06-times peak goodput gains are over particular baselines on particular LLMs and real-world traces via SGLang; other setups may differ.

The comparison

Managing the prefill and decode phases in LLM serving

Property	Disaggregation	Chunked prefill	MuxWise
Where the phases run
KV cache pool
Deadline versus utilization
Adapts to load
Reported goodput

A comparison of ways to manage LLM prefill and decode. Disaggregation runs the phases on separate GPUs (independent but with fragmented, smaller KV caches and a rigid split); chunked prefill fuses prefill chunks into decode on one GPU (full cache but a single token budget that cannot both meet the decode deadline and saturate the GPU). MuxWise runs the phases concurrently on separate cores of one GPU — independent execution with a shared full-size cache and a reconfigurable core split — using a bubble-less engine, a contention-tolerant estimator, and an SLO-aware dispatcher to raise goodput by 2.20 times on average (up to 3.06 times). Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ arXiv copy, 2504.14489, of the ASPLOS 2026 paper). The described system is the paper’s: MuxWise is an LLM serving framework that introduces intra-GPU prefill-decode multiplexing, running the prefill and decode phases concurrently on different streaming multiprocessors of one GPU, so they execute independently (avoiding the deadline-versus-utilization tradeoff of chunked prefill) while sharing GPU memory (avoiding the KV-cache fragmentation of disaggregation) with a compute split that can be reconfigured with low overhead. It comprises a bubble-less multiplex engine (partitioning prefill into layers to align latencies and avoid idle cores), a contention-tolerant estimator (worst-case latency predictions combining a solo-run predictor with a contention guard from one-time offline profiling, addressing memory-bandwidth interference that core partitioning does not control), and an SLO-aware dispatcher (selecting multiplexing plans to maximize goodput under deadlines). The paper motivates this with concrete figures — a cross-request cache hit rate falling from 36.6 percent to 4.2 percent under disaggregation, and a 70-billion-parameter model on eight A100 GPUs needing a token budget of about 4,000 to saturate the GPU versus 256 for a 100-millisecond decode deadline. Implemented on SGLang, MuxWise reports an average 2.20 times (up to 3.06 times) improvement in peak throughput under SLO guarantees over state-of-the-art baselines, on small and large LLMs with real-world workloads. Background on LLM prefill and decode, TTFT and TBT deadlines, KV caching, GPU streaming multiprocessors, and continuous batching is standard context. The step-by-step walkthrough illustrates the described mechanism. Exact algorithms and full results should be read from the paper.

Background

Why LLM serving keeps reorganizing around the two phases

It helps to see MuxWise within the fast-moving effort to serve LLMs efficiently, much of which is a sustained argument about how to handle the prefill and decode phases. The two phases are so different — one a compute-heavy burst, the other a memory-bound trickle, each with its own latency target — that nearly every serving-system advance is, in some form, a new answer to how they should coexist on the hardware. Continuous batching learned to pack decodes together; chunked prefill learned to interleave prefill with them; disaggregation learned to pull them apart onto different machines; and now spatial multiplexing learns to run them side by side on one chip. Underneath is a recurring systems theme: two workloads with complementary resource profiles can, in principle, share hardware efficiently, but only if you can keep them from interfering and schedule them to meet their guarantees. That is a general problem in serving co-located work, sharpened here by the tight latency budgets and the value of the shared KV cache. This drive to serve interactive AI workloads at high throughput without violating their latency guarantees is the same terrain the work on accelerating LLM serving for multi-turn dialogues (multi-turn dialogue serving) covered elsewhere on this site occupies, squeezing more served requests out of the same GPUs. MuxWise’s particular contribution is to show that running the two phases concurrently on one GPU, with careful bubble and contention control, is a distinct and stronger point in this design space than either separating or fusing them.

If you remember only three things

1. An LLM serves each request in two phases with opposite profiles: a compute-heavy prefill that reads the whole prompt and produces the first token, and a memory-bound decode that emits the rest one at a time. Both carry latency deadlines (for instance, first token within 500 milliseconds, each later token within 100 milliseconds), and the metric that matters is goodput — throughput sustained while meeting those deadlines. On a shared GPU the phases interfere: to add a request, a server stalls ongoing decodes to run the new prefill, and a long prefill can push waiting decodes past their per-token deadline. Prior fixes each fail: disaggregation puts the phases on separate GPUs but is statically rigid and fragments the KV cache (collapsing cross-request reuse, e.g. a hit rate falling from 36.6 to 4.2 percent), while chunked prefill fuses prefill chunks into decode under one token budget that cannot be both small enough for the deadline and large enough to saturate the GPU (256 versus about 4,000 for a 70B model on eight A100s).
2. MuxWise introduces intra-GPU prefill-decode multiplexing: it runs the two phases at the same time on different streaming multiprocessors (compute cores) of one GPU. This is independent execution, like disaggregation, so a long prefill no longer stalls decode and the deadline is decoupled from utilization; but it shares the GPU's memory, like keeping everything on one GPU, so the KV cache stays full-sized and cross-request reuse remains effective; and the compute split between the phases can be reconfigured cheaply as load shifts. Realizing it takes three components: a bubble-less multiplex engine that partitions prefill into per-layer pieces so its latency aligns with decode and neither phase idles the other's cores; a contention-tolerant estimator that predicts worst-case latency under the memory-bandwidth interference spatial sharing causes (a solo-run predictor plus a profiled contention guard); and an SLO-aware dispatcher that selects the multiplexing plan maximizing goodput under the deadlines.
3. The key reframing is that independence and shared memory were only in conflict because the phases were forced to live on separate hardware or execute as one fused step; a GPU's many independent cores make concurrency a third option that is strictly better than either. Built on the SGLang serving system, MuxWise improves peak throughput under deadline guarantees by an average of 2.20 times, and up to 3.06 times, over state-of-the-art baselines across small and large LLMs on real workloads. The trade-offs: spatial sharing still causes memory-bandwidth interference that MuxWise plans around rather than eliminates, the estimator depends on one-time offline profiling, the approach adds scheduling machinery and assumes controllable GPU core partitioning, and the reported gains are on specific models, hardware, and workloads.

Check yourself

Check your understanding

1. Why do an LLM's prefill and decode phases conflict when served on a shared GPU?
2. What are the drawbacks of disaggregation and chunked prefill?
3. What is intra-GPU prefill-decode multiplexing, and why does it beat the alternatives?
4. What three components does MuxWise use, and what did it achieve?

Sources

The paper is “Towards High-Goodput LLM Serving with Prefill-decode Multiplexing” by Yukang Chen, Weihao Cui, Han Zhao, Ziyi Xu, Xiaoze Fan, Xusheng Chen, Yangjie Zhou, Shixuan Sun, Bingsheng He, and Quan Chen (Shanghai Jiao Tong University, the National University of Singapore, and Huawei, among the affiliations), published at ASPLOS 2026, DOI 10.1145/3779212.3790236. It is available on arXiv as 2504.14489.

This explainer is written from the paper’s full text (the authors’ arXiv copy). The system, MuxWise, introduces intra-GPU prefill-decode multiplexing: it runs the prefill and decode phases of LLM inference concurrently on different streaming multiprocessors of one GPU, so they execute independently (avoiding chunked prefill’s deadline-versus-utilization coupling) while sharing GPU memory (avoiding disaggregation’s KV-cache fragmentation) with a reconfigurable compute split. It comprises a bubble-less multiplex engine (per-layer prefill partitioning to align latencies), a contention-tolerant estimator (worst-case latency under memory-bandwidth interference, via a solo-run predictor and an offline-profiled contention guard), and an SLO-aware dispatcher (selecting goodput-maximizing multiplexing plans). Built on SGLang, the paper reports an average 2.20 times (up to 3.06 times) improvement in peak throughput under SLO guarantees over state-of-the-art baselines. Background on LLM prefill and decode, TTFT and TBT deadlines, KV caching, and GPU streaming multiprocessors is standard context. The numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.