QoServe: Breaking the Silos of LLM Inference Serving

Kanishk Goel, Jayashree Mohan, Nipun Kwatra, Ravi Shreyas Anupindi and Ramachandran Ramjee

Different LLM applications need wildly different response speeds — an autocomplete must reply instantly, a batch summary can wait — but today's serving systems split the hardware into just two rigid silos, interactive and batch, which wastes GPUs and cannot honor finer latency promises; this paper (Microsoft Research — Kanishk Goel, Jayashree Mohan, Nipun Kwatra, Ravi Shreyas Anupindi, Ramachandran Ramjee) presents QoServe, which co-schedules all these workloads on shared GPUs with fine-grained Quality-of-Service classes, dynamically resizing the prefill chunks it processes to spend the deadline slack of relaxed requests on throughput while still meeting the strict ones, and gracefully shedding a little service under overload — raising serving capacity by 23 percent, lifting per-replica goodput by up to 2.4 times over Sarathi, and cutting deadline violations by an order of magnitude under extreme load.

The argument in one glance

Problem

One LLM cluster must serve apps with wildly different speed needs

A single large language model backend serves many kinds of applications, and their latency needs are all over the map. A coding autocomplete must reply almost instantly or it is useless; a chat assistant needs a fast first token and a smooth stream after; a nightly batch summarization job can take as long as it likes. Serving all of these well on the same expensive GPU fleet is the goal, because the alternative — dedicating separate hardware to each need — is wasteful. The catch is that the knobs that make serving fast for one class make it slow or costly for another, so an operator must somehow honor each application's specific latency promise while keeping the GPUs busy and the cost down. How you schedule the work, and how you carve the fleet, decides whether you can do both at once.
What people did

Split the fleet into interactive and batch silos

The common practice is to segregate workloads into two silos on separate infrastructure: an interactive pool and a batch pool. The interactive pool uses small prefill chunks — splitting each prompt into little pieces so decodes are not stalled — to keep latency low, and the batch pool uses large chunks to maximize throughput because latency does not matter there. Each pool is tuned for its class, and requests are routed to the appropriate silo.
Why it fell short

Two rigid silos waste GPUs and cannot express finer QoS

This coarse, two-bucket segregation is inefficient on both axes. On cost: the interactive silo's small chunks buy low latency at the price of throughput (around 28 percent lower), and because each silo is provisioned separately it cannot lend its idle capacity to the other, so GPUs sit underused while the neighboring pool is overloaded. In one real setting, meeting three distinct latency tiers with silos took thirteen GPUs, seven for the strict tier and three each for the looser two. On expressiveness: only two classes exist, so an application whose latency need falls between interactive and batch cannot be served precisely, and neither silo adapts its behavior to the moment-to-moment load. The result is that the fleet is both overprovisioned and under-differentiated — the opposite of what serving diverse LLM traffic efficiently requires.
This paper's idea

Co-schedule everything on shared GPUs with QoS-aware dynamic chunking

QoServe breaks the silos by co-scheduling all workloads on shared infrastructure under fine-grained Quality-of-Service classes, letting each application state a precise latency requirement and adapting scheduling to the real-time system state. Its engine is dynamic chunking: because LLM inference has predictable execution (distinct prefill and decode phases, and a very predictable prefill), QoServe adjusts the prefill chunk size on the fly per the current load and each request's deadline, spending the deadline slack of relaxed requests to run bursts of larger chunks that lift throughput, while still delivering the strict requests on time. To stay stable under pressure, it adds hybrid prioritization — a policy that smoothly interpolates between earliest-deadline-first and shortest-job-first so it minimizes deadline violations at both low and high load — and selective request relegation, eagerly and selectively degrading service for a small subset of requests so the rest keep their guarantees during overload. On an A100 cluster QoServe raises serving capacity by 23 percent over siloed deployments, improves per-replica goodput by up to 2.4 times over Sarathi on a shared cluster, and under extreme load cuts deadline violations by an order of magnitude.

Why this problem is worth solving

LLM serving is where most of the cost and user experience of deployed models is decided, and latency requirements are genuinely diverse.

The same model backend powers applications with completely different notions of fast enough. What matters is captured by a few latency metrics. Time to first token measures how long from submitting a request until the first output word appears; for a chatbot or coding assistant this dominates the feeling of responsiveness, so it must be small. Time between tokens measures the gap between successive generated tokens; it governs how smoothly the response streams, again crucial for interactive use. Time to last token measures how long the whole response takes; for a batch job — summarizing a corpus overnight, generating reports — this is all that matters, and it can be large. Because applications weight these differently, a coding autocomplete might demand a very tight time to first token while a batch summarizer is happy with a loose time to last token. A serving system that treats all requests the same either overspends to give everyone the strictest latency or underserves the ones that need it. Meeting each application’s specific latency target, at the lowest hardware cost, is therefore the core objective of LLM serving, and it is exactly what a rigid, one-size split cannot deliver.

The reason the standard fix falls short is that segregating workloads throws away the flexibility that shared hardware could provide.

LLM inference proceeds in two phases: a prefill that processes the input prompt, and a decode that generates output tokens one at a time. A key scheduling technique, chunked prefill, splits a prompt’s prefill into equal-sized chunks and interleaves them with ongoing decodes, so long prompts do not stall other requests. The chunk size is a knob: small chunks keep latency low because no single step is long, but they reduce throughput; large chunks pack more work per step for high throughput but raise latency. Siloed serving hard-codes this knob per pool — small in the interactive silo, large in the batch silo — which is why each pool is efficient only for its own class and wasteful for anything else. Worse, because the pools are physically separate, the slack in one cannot help the other: when the interactive pool is quiet and the batch pool is slammed, the idle interactive GPUs cannot pitch in. The opportunity QoServe sees is that if you co-serve mixed classes on shared hardware, the chunk size no longer has to be fixed — you can raise it when strict requests have deadline slack and lower it when they are tight — turning a static compromise into a dynamic optimization. Capturing that requires putting the diverse workloads together rather than apart, which is the shift the paper makes.

What people did before, and where it broke

Interactive and batch silos. Run two separate pools, small chunks for low-latency interactive traffic and large chunks for high-throughput batch traffic. Where it broke. Two coarse classes cannot express finer latency needs, and physically separate pools cannot share idle capacity, so GPUs are wasted and mid-latency apps are served poorly.
A single global chunk size for the strictest class. Serve everything on shared hardware but fix the chunk size small enough for the tightest latency requirement. Where it broke. It meets every deadline but at low throughput and high cost for all classes, because the relaxed requests are forced to run at the strict class’s expensive settings.
Shortest-job-first under overload. When overloaded, prioritize short requests, exploiting that system load grows quadratically with request length. Where it broke. It reduces load but ignores the QoS of longer jobs, so long requests systematically miss their deadlines.
Earliest-deadline-first scheduling. Always run whichever request has the nearest deadline. Where it broke. It is optimal at low load but collapses when load slightly exceeds capacity, producing a cascade of deadline violations rather than degrading gracefully.
The gap. No system co-scheduled fine-grained QoS classes on shared GPUs while dynamically resizing chunks to trade relaxed requests’ slack for throughput, and handled overload with a policy that stays stable across both low and high load — which is what serving diverse LLM traffic efficiently actually needs.

How QoServe works

Co-scheduling mixed QoS classes on shared GPUs with dynamic chunking

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Put the workloads together so their slack can be shared. The foundational move is to stop segregating traffic into interactive and batch silos and instead co-schedule every QoS class on one shared fleet. Silos waste hardware because each is provisioned for its own peak and cannot lend idle capacity to a busy neighbor, and they cannot express latency needs between the two extremes. By mixing classes on shared GPUs, QoServe turns the deadline slack of relaxed requests into a resource the strict requests’ throughput can draw on, which is impossible when they live on separate machines. Serving heterogeneous inference requests efficiently on shared infrastructure, rather than dedicating pools per workload, is the same consolidation instinct the work on a native serverless system for low-latency, high-throughput inference covered elsewhere on this site is built on, pooling capacity so it flexes with demand. QoServe carries that instinct into LLM serving and makes the QoS classes fine-grained rather than binary.

Turn the chunk size from a fixed compromise into a live control knob. The engine that makes shared serving pay off is dynamic chunking. Chunked prefill exposes a throughput-latency knob — small chunks are gentle on latency but waste throughput, large chunks do the reverse — and siloed systems freeze this knob per pool. QoServe instead exploits the predictability of LLM prefill to set the chunk size moment to moment, using the deadline slack of relaxed requests to run larger, more efficient chunks when it can afford to, and shrinking them when strict deadlines loom. Reorganizing how LLM inference executes in response to the live shape of the workload, rather than running one static configuration, is the same adaptivity the work on shift parallelism for low-latency, high-throughput LLM inference under dynamic workloads covered elsewhere on this site pursues, reshaping execution to fit demand. QoServe’s knob is the chunk size, and its lever is the slack that co-serving mixed deadlines creates.

Fail gracefully instead of cascading. A serving system is judged as much by its behavior under overload as at steady state, and QoServe treats overload as a first-class case. It replaces brittle single policies — earliest-deadline-first, which collapses just past capacity, and shortest-job-first, which starves long jobs — with a hybrid that interpolates between them to stay stable across load levels, and it adds eager, selective relegation that sacrifices a small, chosen set of requests so the majority keep their guarantees. Shedding or degrading a little service deliberately so the system as a whole stays within its guarantees under stress, rather than letting failures cascade, is the same principle the work on cooperative graceful degradation in containerized clouds covered elsewhere on this site is built on, choosing controlled partial loss over total collapse. QoServe applies it to LLM request deadlines, which is why its violations rise gently rather than exploding when load spikes.

Following a mixed batch through QoServe

Serving strict and relaxed requests together on one shared cluster

A trace, drawn from the paper, of how QoServe schedules a mix of QoS classes on shared GPUs.

Step 1 of 5

Requests arrive tagged with precise latency targets

A coding-autocomplete request with a tight time-to-first-token target, a chat request with a moderate one, and a batch summarization request with a loose time-to-last-token target all enter the same shared QoServe cluster, each carrying its QoS class.

Fine-grained classes replace the coarse interactive-versus-batch split.

Why it works

Sharing beats siloing because slack becomes usable. Separate pools strand idle capacity while a neighbor is overloaded → but co-scheduling all classes on shared GPUs lets one class’s slack serve another → so the same hardware meets more deadlines, cutting the fleet by 23 percent.
Dynamic chunking exploits real, predictable slack. A fixed chunk size is a permanent compromise between latency and throughput → but LLM prefill is predictable enough to resize chunks per deadline and load → so throughput rises whenever strict requests have room to spare.
The hybrid policy is stable where pure policies break. Earliest-deadline-first collapses just past capacity and shortest-job-first starves long jobs → but interpolating between them tracks the best of each as load changes → so deadline violations stay low across both low and high load.
Selective relegation contains overload. Letting every request slip when load spikes cascades into mass violations → but deliberately degrading a small chosen subset protects the rest → so the system degrades gracefully instead of collapsing.
Fine-grained QoS matches cost to need. Two buckets force mid-latency apps into an ill-fitting silo → but precise per-request latency targets let the scheduler spend effort exactly where deadlines demand → so no request is over- or under-served relative to its requirement.

What it costs (trade-offs and scope)

Relegation means some requests are deliberately degraded. Graceful degradation under overload works by sacrificing service for a selected minority of requests; those requests get worse latency or are shed, so the guarantee is that the system stays stable and most requests are met, not that every request always is.
The gains depend on a mix of QoS classes with slack. Dynamic chunking’s throughput boost comes from spending relaxed requests’ deadline slack; a workload that is uniformly strict, or has no relaxed traffic to exploit, offers less slack to convert into throughput, so the benefit over a well-tuned interactive silo shrinks.
It relies on LLM execution being predictable. The chunk-size decisions lean on the predictability of prefill timing; models, hardware, or features that make execution time harder to predict would weaken the scheduler’s ability to size chunks confidently against deadlines.
The reported numbers are for specific baselines and hardware. The 23 percent capacity gain, up to 2.4 times per-replica goodput over Sarathi, and order-of-magnitude reduction in violations are measured on an A100 cluster against particular siloed and shared baselines; other models, traffic mixes, QoS-tier distributions, and hardware would shift them.
It is a scheduling-and-serving layer, not a model or kernel change. QoServe improves how requests are classified, chunked, prioritized, and shed; it does not change the model or the low-level attention kernels, so it composes with, rather than replaces, per-request execution optimizations.

The evaluation

Serving diverse-latency LLM workloads

Property	Interactive/batch silos	Shared, fixed chunk size	QoServe
QoS granularity
Shares capacity across classes
Chunk size
Overload behavior
Reported outcome

A qualitative comparison of LLM serving strategies for diverse latency needs. On an A100 cluster QoServe raises serving capacity by 23 percent over siloed deployments, improves per-replica goodput by up to 2.4 times over Sarathi on a shared cluster, and reduces deadline violations by an order of magnitude under extreme load.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the Microsoft Research camera-ready PDF of the ASPLOS 2026 paper; the work was previously circulated under the name Niyama, arXiv:2503.22562, so some sources use that title). The mechanism described is the paper’s: QoServe co-schedules diverse workloads on shared infrastructure with fine-grained QoS classification (applications specify precise latency requirements), dynamically adapting to real-time system state; it uses dynamic chunking, adjusting prefill chunk sizes based on the predictable execution of LLM inference and the deadline slack of relaxed requests to raise throughput while meeting strict deadlines; it uses hybrid prioritization that interpolates between earliest-deadline-first and shortest-job-first to stay stable across load levels; and it uses selective (eager) request relegation to degrade a small subset of requests for graceful degradation under overload. The headline figures are the paper’s own: on an A100 cluster, a 23 percent increase in serving capacity over siloed deployments, up to 2.4 times higher per-replica goodput than Sarathi on a shared cluster, and an order-of-magnitude reduction in SLO violations under extreme load. Exact algorithms, the QoS model, and the full evaluation should be read from the paper.

Background

Why LLM serving is becoming a scheduling problem

It helps to see QoServe as part of a shift in what makes LLM deployment hard: increasingly, the challenge is not running a single model fast but orchestrating a shared, expensive GPU fleet across a flood of heterogeneous requests. Inference is costly and bursty, user expectations for latency are high and varied, and the hardware is scarce, so the systems questions — how to batch, chunk, prioritize, place, and shed requests — have become as important as the model kernels themselves. A recurring theme is that treating all requests uniformly, or carving the fleet into rigid pools, leaves large amounts of efficiency and quality of service on the table, and that adapting to the live workload recovers it. This is why LLM serving has drawn so much systems research into request scheduling, admission control, and resource management specific to the peculiar two-phase, autoregressive structure of inference. The broader effort to schedule LLM inference across a cloud fleet under real operational constraints is the same problem space the work on thermal- and power-aware scheduling for LLM inference in cloud platforms covered elsewhere on this site operates in, managing the fleet as a whole rather than one request at a time. QoServe’s contribution is to make quality of service the organizing principle, and to show that co-serving mixed classes with dynamic chunking beats siloing.

If you remember only three things

1. LLM applications have very different latency needs — instant autocomplete, smooth chat, or leisurely batch summaries — but existing serving systems segregate traffic into just two silos, interactive and batch, on separate hardware. This wastes GPUs (each silo is provisioned alone and cannot lend idle capacity to a busy neighbor) and cannot express finer QoS: interactive silos use small prefill chunks for low latency at about 28 percent lower throughput, batch silos use large chunks for throughput, and neither adapts to real-time load.
2. QoServe co-schedules all QoS classes on shared GPUs. It lets applications specify precise latency requirements and adapts to real-time state; its dynamic chunking exploits the predictability of LLM prefill to resize chunks on the fly, spending the deadline slack of relaxed requests on bursts of larger, higher-throughput chunks while still meeting strict deadlines. For overload it uses hybrid prioritization (smoothly interpolating between earliest-deadline-first and shortest-job-first to stay stable at both low and high load) and selective, eager request relegation (degrading a small subset so the rest keep their guarantees).
3. On an A100 cluster QoServe raises serving capacity by 23 percent over siloed deployments, improves per-replica goodput by up to 2.4 times over Sarathi on a shared cluster, and cuts SLO violations by an order of magnitude under extreme load. Its gains depend on a mix of QoS classes with exploitable slack and on predictable execution, relegation deliberately degrades a minority of requests, and it is a scheduling-and-serving layer that composes with model- and kernel-level optimizations.

Check yourself

Check your understanding

1. Why is the standard interactive-versus-batch siloing of LLM serving inefficient?
2. What is QoServe's core mechanism for raising throughput without missing deadlines?
3. How does QoServe handle overload gracefully?
4. What did QoServe achieve, and what are its main trade-offs?

Sources

The paper is “QoServe: Breaking the Silos of LLM Inference Serving” by Kanishk Goel, Jayashree Mohan, Nipun Kwatra, Ravi Shreyas Anupindi, and Ramachandran Ramjee (Microsoft Research), published at ASPLOS 2026 (Volume 2), DOI 10.1145/3779212.3790206. It is licensed CC BY. The work was previously circulated under the name Niyama (arXiv:2503.22562), so some listings use that title.

This explainer is written from the paper’s full text (the Microsoft Research camera-ready PDF). The paper addresses LLM inference serving for applications with very different latency requirements, which existing frameworks handle with siloed infrastructure and coarse interactive-versus-batch segregation, leading to inefficient resource utilization and limited QoS differentiation. QoServe is a QoS-driven serving system that co-schedules diverse workloads on shared infrastructure: it introduces fine-grained QoS classification allowing applications to specify precise latency requirements and dynamically adapts scheduling to real-time system state; it uses dynamic chunking, leveraging the predictable execution of LLM inference (distinct prefill and decode phases, predictable prefill) to adjust prefill chunk sizes and exploit the deadline slack of relaxed requests for higher throughput while meeting strict deadlines; it uses hybrid prioritization that interpolates between earliest-deadline-first and shortest-job-first to minimize SLO violations across low and high load; and it employs selective, eager request relegation for graceful service degradation during overload. The reported results are the paper’s own: on an A100 cluster, a 23 percent increase in serving capacity over siloed deployments while maintaining QoS guarantees, up to 2.4 times higher per-replica goodput than Sarathi on a shared cluster, and an order-of-magnitude reduction in SLO violations under extreme load. No numbers beyond the paper’s own have been introduced; exact algorithms, the QoS model, and the full evaluation should be confirmed in the paper.