Bullet: Boosting GPU Utilization for LLM Serving via Dynamic Spatial-Temporal Orchestration

Serving a large language model has two phases that stress opposite parts of a GPU — the prefill that reads your whole prompt is compute-heavy, and the token-by-token decode is memory-heavy — so running one at a time leaves half the chip idle, and the usual fix of blending them into one batch forces a bad choice between speed and latency; this work from Sun Yat-sen University runs prefill and decode on the same GPU at the same time, slicing the chip's cores between them and re-slicing on the fly with a performance model so each fills the other's idle capacity, delivering about 1.26 times more throughput while still meeting latency targets.

The argument in one glance

Problem

LLM serving's two phases stress opposite parts of the GPU

Prefill (reading the whole prompt for the first token) is compute-heavy but light on memory bandwidth; decode (generating tokens one at a time) is memory-heavy but leaves compute cores idle. A GPU running one phase at a time wastes the resource that phase does not need, so utilization is poor.
What people did

Blend prefill and decode into a hybrid batch

Serving systems put prefill and decode into one hybrid batch, often chunking a long prefill into pieces interleaved with decodes, to keep the GPU busy across both phases.
Why it fell short

Hybrid batching forces a bad speed-versus-latency tradeoff

Prioritizing prefill lifts throughput but hurts per-token latency; chunking to protect latency sacrifices throughput, so resources stay idle either way. And even prefill under-uses compute due to wave quantization and attention bottlenecks, while hybrid batching leans too far toward latency, wasting both compute and memory bandwidth.
This paper's idea

Run prefill and decode concurrently, re-slicing the GPU live (Bullet)

Bullet runs prefill and decode on the same GPU at the same time, partitioning the cores between them so the compute-heavy and memory-heavy work fill each other's idle capacity. A performance model decides the split, an SLO-aware scheduler protects latency, and microsecond-fast core masking re-slices the GPU as the workload shifts, for about 1.26 times more throughput while meeting latency targets.

Why this problem is worth solving

Serving large language models is now a dominant, expensive GPU workload, and the way its two phases use the hardware makes efficiency stubbornly hard.

Answering a request with a large language model happens in two phases. First, the prefill phase reads the entire input prompt at once and processes it in parallel to produce the first output token; this is dominated by large matrix multiplications, so it is compute-intensive — it can saturate the GPU’s arithmetic units (its streaming multiprocessors), and the latency users feel here is the time to first token. Second, the decode phase generates the rest of the output one token at a time, autoregressively, each new token attending to all the previous ones; each step is a small matrix-vector operation dominated by reading the model weights and the cached context from memory, so it is memory-bandwidth-bound — it saturates memory bandwidth but leaves most of the compute cores idle, and the latency here is the time between output tokens. These are opposite resource profiles on the same chip: prefill wants compute and wastes memory bandwidth, decode wants memory bandwidth and wastes compute. A GPU that runs one phase at a time therefore always leaves one of its two major resources idle, which is why LLM serving struggles to keep the expensive hardware well utilized.

The standard fix trades one inefficiency for another, and the underlying waste runs deeper than it first appears.

To avoid running the phases separately, serving systems combine prefill and decode requests into a single hybrid batch, commonly splitting a long prefill into smaller chunks that are interleaved with ongoing decodes so the GPU always has some of each. But this only relocates the problem into a throughput-versus-latency tradeoff. If the batch prioritizes prefill, throughput rises but the interleaved decodes are delayed, hurting per-token latency; if prefill is chunked finely to protect latency, throughput falls, and the tradeoff scales poorly. Either way, substantial GPU resources sit idle. The paper digs into why and finds two deeper root causes. First, even the compute-heavy prefill does not fully use the compute, because of wave quantization — GPU kernels launch thread blocks in waves across the streaming multiprocessors, but the number of blocks rarely divides evenly by the GPU’s core count (which is not a power of two, for example 108 or 132), so the final wave leaves many cores idle, an effect especially bad for attention and small matrix multiplies. Second, hybrid batching disproportionately prioritizes latency over throughput, wasting both compute and memory bandwidth. The conclusion is that keeping one phase’s leftover resources idle, rather than filling them with the other phase’s complementary needs, is the real inefficiency.

What people did before, and where it broke

Run prefill and decode in separate time slices. Process one phase, then the other, on the GPU. Where it broke. Each phase leaves the resource it does not need idle — prefill wastes memory bandwidth, decode wastes compute — so half the chip is underused at any moment.
Blend them into a hybrid batch with chunked prefill. Interleave chunks of prefill with decodes in one batch to keep both busy. Where it broke. It creates a throughput-versus-latency tradeoff that scales sub-linearly: favoring prefill hurts decode latency, chunking for latency cuts throughput, and resources stay idle either way.
Disaggregate prefill and decode onto separate GPUs. Give each phase its own dedicated pool of GPUs. Where it broke. It needs careful per-workload tuning of how many GPUs each phase gets, cannot reconfigure quickly when the request load fluctuates, and requires heavy high-bandwidth interconnect to migrate the cached state between the pools.
Optimize the kernels within chunked prefill. Speed up the individual operators. Where it broke. Operator-level optimization does not remove the fundamental latency penalty that chunking imposes, so the intrinsic tradeoff remains.
The gap. No system ran the two complementary phases concurrently on one GPU with fine-grained, latency-aware control of how the chip is split between them. Providing that is what Bullet does.

How Bullet works

From time-sliced or hybrid batching to concurrent, re-sliced execution

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Run the two complementary phases at once so each fills the other’s idle resource. Bullet’s foundation is to stop treating prefill and decode as things that must take turns, and instead execute them concurrently on the same GPU, because their resource profiles are opposites: the compute-heavy prefill leaves memory bandwidth idle, the memory-heavy decode leaves compute cores idle, and running them together lets each consume what the other wastes. This turns the phase mismatch from a liability into an opportunity, filling the chip rather than leaving half of it dark. Co-locating workloads with complementary resource demands on the same accelerator so their combined execution uses hardware that either alone would leave idle is the same move the multi-tenant deep-learning-services work covered elsewhere on this site makes, packing jobs together on shared hardware so that one job’s slack is taken up by another’s demand and overall utilization climbs.

Model the contention precisely to decide how to split the chip. Concurrency alone is not enough — if prefill and decode contend for the cores uncontrolled, neither runs well — so Bullet spatially partitions the streaming multiprocessors between them and chooses the partition with an SM-scaling roofline model that captures how the two kernels’ performance changes as each gets more or fewer cores, calibrated from just a few samples. This lets it pick the split that maximizes combined utilization rather than guessing. Building a performance model of how a workload behaves as its share of the hardware’s compute resources varies, and using that model to make provisioning decisions, is the same discipline the roofline-based operator-analysis work covered elsewhere on this site follows, modeling how computation and resource allocation interact so that where to spend the hardware is decided from an accurate account rather than trial and error.

Steer toward throughput while guaranteeing latency, and reconfigure instantly. Because a serving system must respect each request’s latency target, Bullet couples the split decision to an SLO-aware scheduler that maximizes throughput without breaching the time-to-first-token and per-token latency constraints, and it reconfigures the core partition in microseconds using lightweight compute-unit masking as the request mix changes, so the allocation continuously tracks the live workload. Navigating the throughput-versus-latency tradeoff dynamically to serve each request within its constraints, rather than fixing one operating point, is the same goal the constraint-aware LLM scheduling work covered elsewhere on this site pursues, treating latency targets as constraints to satisfy while pushing throughput as high as the hardware and those constraints allow, moment by moment.

Following a request mix through Bullet

Serving concurrent prefill and decode on one GPU

A trace of Bullet orchestrating LLM requests, following the paper's spatial-temporal sharing design.

Step 1 of 5

Prefill and decode requests arrive together

At any moment the server has some requests needing prefill (reading new prompts) and many in decode (generating tokens). Their resource needs are opposite: prefill wants compute, decode wants memory bandwidth.

Running either alone leaves the other resource idle.

Why it works

Complementary phases fill each other’s idle resource. Because prefill wastes memory bandwidth and decode wastes compute → running them concurrently lets each consume what the other leaves idle → so the GPU’s two major resources are used at once instead of one at a time.
A partition prevents uncontrolled contention. Because two concurrent kernels would otherwise fight for the cores → spatially dividing the streaming multiprocessors gives each a sized slice → so both phases run predictably rather than degrading each other.
The roofline model picks the right split. Because the best core partition depends on how each kernel scales with its share → an SM-scaling model calibrated from few samples predicts the contention → so Bullet provisions from an accurate account rather than guesswork.
SLO-awareness keeps latency safe while chasing throughput. Because serving must meet latency targets → tying the split to a scheduler that respects time-to-first-token and per-token limits → so throughput is maximized without breaching the constraints users feel.
Microsecond reconfiguration tracks the live workload. Because the mix of prefill and decode shifts continuously → re-partitioning the cores in microseconds via lightweight masking → so the allocation stays optimal as load fluctuates instead of lagging behind.

What it costs (trade-offs and scope)

It needs fine-grained control over GPU cores. Bullet relies on compute-unit (streaming-multiprocessor) masking to partition and reconfigure the GPU, using a modified low-level library. This depends on the hardware and software exposing that control, which current GPUs support only through such specialized mechanisms rather than as a first-class feature.
Concurrent execution has contention that must be modeled well. Running two kernels at once on partitioned cores still shares memory bandwidth, caches, and other resources, so the benefit depends on the roofline model accurately capturing the contention; a poor model would misprovision and lose the gains.
The performance model needs calibration. The SM-scaling roofline is calibrated from samples; while the paper reports minimal sampling and low overhead, the model must be fit for the models and hardware in use, and unusual kernels or configurations could need recalibration.
The gains are moderate and workload-dependent. The reported 1.26 times average (up to 1.55 times) throughput improvement is meaningful but not dramatic, and depends on the request mix having enough concurrent prefill and decode with complementary idle resources to exploit; workloads dominated by one phase offer less headroom.
It targets single-device sharing. Bullet’s spatial-temporal sharing is within a GPU; it is complementary to, not a replacement for, multi-GPU parallelism and disaggregation, and combining it with those at scale is beyond the core mechanism.
Results are against specific baselines and settings. The improvements are over particular state-of-the-art chunked-prefill systems (such as vLLM and SGLang at given chunk sizes) on particular models and workloads; other baselines, models, or GPUs would shift the exact numbers.

The evaluation

Ways to serve LLM prefill and decode on a GPU

Property	Hybrid / chunked-prefill batch	Prefill-decode disaggregation	Bullet (spatial-temporal sharing)
Where phases run
Fills complementary idle resources
Throughput-latency tradeoff
Reconfigures under load shifts	Rebatches
Throughput vs state-of-the-art	Baseline

Ways to serve large language model prefill and decode on GPUs, as compared in the paper. Blending them into a hybrid, chunked-prefill batch keeps the GPU busy but forces a throughput-versus-latency tradeoff that scales sub-linearly, so resources stay idle. Disaggregating the phases onto separate GPU pools avoids the tradeoff within each pool but needs careful per-workload GPU tuning, reconfigures slowly under load changes, and demands heavy interconnect to migrate cached state. Bullet runs prefill and decode concurrently on the same GPU via spatial-temporal sharing: it partitions the streaming multiprocessors between the two kernels so the compute-heavy prefill and memory-heavy decode fill each other's idle resource; an SM-scaling roofline model calibrated from few samples predicts how they contend at each partition; an SLO-aware scheduler picks the split that maximizes throughput within the time-to-first-token and per-token latency targets; and a resource manager reconfigures the partition in microseconds via lightweight compute-unit (SM) masking as the request mix shifts. It also addresses prefill's own under-utilization from wave quantization (thread blocks not dividing evenly across the non-power-of-two core count) and attention bottlenecks. On real-world workloads Bullet delivers 1.26 times average throughput gains (up to 1.55 times) over state-of-the-art chunked-prefill systems while consistently meeting latency constraints, with markedly lower time-to-first-token at modest per-token cost; the implementation is open-source.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

Bullet is a well-targeted systems paper that turns a structural inefficiency of LLM serving into a source of speedup. The setup is by now familiar but genuinely hard: an LLM request has a compute-heavy prefill phase and a memory-bound decode phase, so a GPU running one at a time always leaves half its capability idle, and the common fix of blending them into a chunked hybrid batch just relocates the problem into a throughput-versus-latency tradeoff that wastes resources either way. The paper’s contribution is twofold. Diagnostically, it identifies that even prefill under-uses the compute because of wave quantization — thread blocks not dividing evenly across a GPU’s non-power-of-two core count, leaving idle cores on the tail wave — and attention bottlenecks, and that hybrid batching leans too far toward latency. Constructively, it runs prefill and decode concurrently on one GPU and controls the sharing precisely: it partitions the streaming multiprocessors between them, uses an SM-scaling roofline model calibrated from a few samples to predict their contention and pick the split, ties that to an SLO-aware scheduler so latency targets hold, and reconfigures the partition in microseconds via lightweight compute-unit masking as the workload shifts. The payoff is a 1.26 times average (up to 1.55 times) throughput gain over strong chunked-prefill baselines while meeting latency, notably by slashing time-to-first-token at modest per-token cost, and the code is open. Read the scope with care. The mechanism depends on fine-grained SM-masking control that today’s GPUs expose only through specialized libraries; concurrent execution still shares memory bandwidth and caches, so the benefit hinges on the contention model being accurate; the gains are moderate and depend on the request mix offering complementary prefill and decode to overlap; and it optimizes sharing within a single device, complementary to multi-GPU disaggregation rather than replacing it. The durable idea is that when a workload has phases with opposite resource appetites, the highest-leverage move is not to schedule them apart but to overlap them so each consumes the other’s slack — provided you can model the contention and re-slice the hardware fast enough to keep the balance right as the load changes.

Background

Why LLM serving keeps chasing GPU utilization

It helps to see Bullet within the intense, ongoing effort to make large language model serving efficient, which has become one of the defining systems problems of the era because inference now dominates AI cost and GPUs are scarce and expensive. The recurring obstacle is that LLM inference’s two phases pull the hardware in opposite directions — prefill toward compute, decode toward memory bandwidth — under unpredictable, bursty traffic, and almost every serving advance is a new way to keep the GPU fully fed despite that. The dynamic-parallelism LLM serving work covered elsewhere on this site attacks the same underlying tension from the parallelism angle, switching how the model is spread across the hardware to match whether traffic is latency-sensitive or throughput-sensitive, where Bullet instead shares a single GPU between the phases in space and time. The common thread is that serving efficiency comes from refusing a single fixed operating point: the phases, the traffic, and the latency targets all vary, so the wins go to systems that can continuously rebalance how the hardware is used. As models grow and serving traffic diversifies, techniques like Bullet that extract idle capacity by overlapping complementary work, guided by performance models and latency constraints, are likely to remain central to getting the most out of every GPU.

If you remember only three things

1. A large language model serves each request in two phases with opposite resource appetites. Prefill reads the whole prompt at once to produce the first token and is compute-intensive — it saturates the GPU's arithmetic cores (streaming multiprocessors) but barely uses memory bandwidth; its latency is the time to first token. Decode then generates output tokens one at a time and is memory-bandwidth-bound — it saturates bandwidth streaming weights and cached context but leaves most compute cores idle; its latency is the time between tokens. A GPU running one phase at a time always leaves one major resource idle, so utilization is poor. The common fix blends prefill and decode into a hybrid, chunked-prefill batch, but that just relocates the problem into a throughput-versus-latency tradeoff that scales sub-linearly, leaving resources idle either way. The paper also finds deeper waste: even prefill under-uses compute due to wave quantization (thread blocks not dividing evenly across the non-power-of-two core count, for example 108 or 132, so the last wave idles cores) and attention bottlenecks, while hybrid batching over-prioritizes latency, wasting both compute and bandwidth.
2. Bullet runs prefill and decode concurrently on the same GPU via fine-grained spatial-temporal sharing, so the compute-heavy and memory-heavy work fill each other's idle resource. It controls the sharing precisely with four components: a performance estimator using an SM-scaling roofline model, calibrated from a few samples, that predicts how the concurrent prefill and decode kernels contend at different partitions of the streaming multiprocessors; an SLO-aware scheduler that picks the core split maximizing throughput while keeping each request within its time-to-first-token and per-token latency targets; a resource manager that reconfigures the partition in microseconds using lightweight compute-unit (SM) masking (via a modified libsmctrl library) as the request mix shifts; and a concurrent execution engine that launches the two kernels on their assigned core slices. This spatially partitions the GPU between the phases and temporally re-slices it live, rather than time-slicing the phases or disaggregating them across separate GPU pools.
3. On real-world workloads, Bullet delivers 1.26 times average throughput gains (up to 1.55 times) over state-of-the-art chunked-prefill serving systems while consistently meeting latency constraints, notably by markedly reducing time-to-first-token at modest per-token cost, and the implementation is open-source. Scope: it depends on fine-grained SM-masking control that current GPUs expose only through specialized libraries; concurrent execution still shares memory bandwidth and caches, so gains hinge on the contention model being accurate; the improvements are moderate and depend on the request mix offering complementary prefill and decode to overlap; and it optimizes sharing within one device, complementary to multi-GPU disaggregation. The durable lesson is that when a workload has phases with opposite resource appetites, the biggest win is to overlap them so each consumes the other's slack — provided you can model the contention and re-slice the hardware fast enough to keep the balance right as the load changes.

Check yourself

Check your understanding

1. Why does LLM serving underutilize the GPU, and why does the common hybrid-batching fix not fully solve it?
2. What is wave quantization, and why does it mean even the compute-heavy prefill under-uses the GPU?
3. How does Bullet run prefill and decode together, and how does it decide and adjust the GPU split?
4. What did Bullet achieve, and what are its main limitations?

Sources

The paper is “Bullet: Boosting GPU Utilization for LLM Serving via Dynamic Spatial-Temporal Orchestration” by Zejia Lin, Hongxin Xu, Guanyi Chen, Zhiguang Chen, Yutong Lu, and Xianwei Zhang, published at ASPLOS 2026, DOI 10.1145/3779212.3790135. The authors are at Sun Yat-sen University, and the system is open-source (github.com/zejia-lin/Bullet).

This explainer is based on the full text of the paper, obtained from the authors’ arXiv version (arXiv:2504.19516, which also appears under the title “Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing”). The problem framing (LLM inference’s compute-intensive prefill phase, measured by time to first token, and memory-bound decode phase, measured by time between tokens, and the resulting structural under-utilization when a GPU runs one at a time; the hybrid-batching and chunked-prefill practice and its throughput-versus-latency tradeoff; and the two root causes of prefill’s wave quantization and attention bottlenecks and hybrid batching’s over-prioritization of latency), the analysis of prior approaches (prefill-decode disaggregation requiring per-workload GPU tuning, slow reconfiguration, and heavy interconnect for state migration; and chunked prefill’s intrinsic latency penalty), the Bullet design (concurrent execution of prefill and decode via spatial-temporal sharing; the four components of a performance estimator with an SM-scaling roofline model calibrated from minimal samples, an SLO-aware task scheduler, a resource manager offering fast reconfiguration through lightweight compute-unit masking with microsecond overhead via a modified libsmctrl library, and a concurrent execution engine; and the fine-grained partitioning of streaming multiprocessors between phases), and the results (1.26 times average throughput gains, up to 1.55 times, over state-of-the-art chunked-prefill systems including vLLM and SGLang while meeting latency constraints, with reduced time to first token at modest per-token increments) are all drawn directly from the paper. Background explanations of GPUs, streaming multiprocessors, thread blocks and waves, LLM prefill and decode, the KV cache, and roofline modeling are standard and provided to make the paper’s contribution understandable.