BlendServe: Optimizing Offline Inference with Resource-Aware Batching

Yilong Zhao, Shuo Yang, Kan Zhu, Lianmin Zheng, Baris Kasikci, Yifan Qiao and 3 others

Running a giant batch of LLM requests overnight (offline inference) is cheap because latency does not matter, so the only goal is throughput — and the trick is that a GPU has two separate resources, compute and memory bandwidth, that can run at once, so a batch mixing compute-heavy requests (long prompts) with memory-heavy ones (long outputs) keeps both busy; but reordering requests to balance those resources fights with prefix sharing, which wants requests that share a prompt prefix kept together to reuse work; this paper (UC Berkeley — Yilong Zhao, Ion Stoica, and colleagues) presents BlendServe, which reconciles the two by organizing requests in a resource-aware prefix tree and using a dual-scanning algorithm to build balanced, prefix-sharing batches — up to 1.44 times the throughput of vLLM and SGLang and 20.84 percent over the state-of-the-art NanoFlow.

The argument in one glance

Problem

To maximize throughput you must balance two GPU resources — but that fights another optimization

Offline batch inference runs a huge pile of large-language-model (LLM) requests within a relaxed time window (for example, a 24-hour turnaround), which is far cheaper than instant online serving — used for model evaluation, data curation, and benchmarking. Because latency does not matter, the only goal is maximum throughput. Here is the key fact: an LLM request has two phases, a compute-intensive prefill (reading the input prompt) and a memory-bandwidth-intensive decode (generating output tokens one at a time), and a GPU can do compute and memory work at the same time. So a batch that mixes compute-heavy requests (long prompts, short outputs) with memory-heavy ones (short prompts, long outputs) keeps both the compute units and the memory bandwidth busy at once. The trouble is twofold. Applications have wildly different resource profiles, so a batch dominated by one type wastes the other resource. And the natural fix — reorder requests to blend the two types — collides with prefix sharing, the standard optimization where requests that share a common prompt prefix reuse its cached computation and must therefore be grouped together. Balancing resources and sharing prefixes want different orderings, and pursuing one hurts the other.
What people did

Optimize throughput within each batch, and share prefixes where they line up

Prior throughput work optimizes within a batch — overlapping prefill and decode, or multiplexing operators on the GPU, to use compute and memory together. Separately, systems apply prefix sharing: group requests with a common prompt prefix so the prefix is processed once and its key-value cache is reused. Each technique is applied on its own, on whatever batches happen to form.
Why it fell short

Within-batch tricks fail on imbalanced batches, and the two goals conflict

Optimizing within a fixed batch is not enough, because it cannot fix the batch's composition. If the requests in a batch are mostly memory-intensive, the system quickly runs out of compute-heavy prefill work and leaves the GPU's compute units idle during the long decode phase; if mostly compute-intensive, the memory bandwidth sits underused. Overlapping only helps when a batch already contains a good mix, and nothing was arranging that mix across batches. The obvious remedy is to reorder requests so every batch has balanced compute and memory demand — but that reordering conflicts with prefix sharing, which wants requests sharing a prefix kept adjacent so the prefix is computed once. A schedule that maximizes resource overlap can scatter prefix-sharing groups, and a schedule that maximizes prefix sharing can leave batches resource-imbalanced. Prior systems picked one and paid for the other, so neither reached the throughput that offline inference could achieve. What was missing was a way to satisfy both at once.
This paper's idea

Blend requests with a resource-aware prefix tree and a dual scan

BlendServe harmonizes resource overlapping and prefix sharing to maximize offline throughput. First, it builds a performance model that estimates each request's resource demand as a compute density — how compute-heavy versus memory-heavy it is — from the model's dimensions and the request's input and (estimated) output lengths. Then it organizes all requests in a resource-aware prefix tree: a prefix tree captures which requests share prompt prefixes (preserving the prefix-sharing structure), and each node is annotated with the aggregate compute density of the requests in its subtree, so the tree encodes resource demand and sharing together. Finally, a dual scanning algorithm produces the schedule: it sorts the tree's nodes by compute density (compute-intensive on one side, memory-intensive on the other) and scans inward from both ends at once, pairing compute-heavy with memory-heavy requests to form batches whose compute and memory demands are balanced — while keeping prefix-sharing groups intact. The result is a request order that both blends resources for full overlap and preserves prefix reuse, resolving the conflict instead of trading one for the other. Prototyped on the NanoFlow inference engine, BlendServe delivers up to 1.44 times the throughput of the industry-standard vLLM and SGLang, and 20.84 percent higher throughput on average than NanoFlow, the prior state-of-the-art.

Why this problem is worth solving

Offline batch inference is becoming a major, cost-sensitive way to run LLMs, and its economics are decided almost entirely by throughput.

Not every LLM request needs an instant answer. A great deal of LLM work is offline and latency-insensitive: evaluating a model on a benchmark, labeling or curating a huge dataset, generating synthetic data. For these, you submit an enormous batch and accept results within a long window — OpenAI’s Batch API, for instance, promises a 24-hour turnaround at roughly half the price of online serving. Because the deadline is so loose, the only thing that matters is how many requests you can push through per unit of hardware time, so throughput is everything, and throughput is set by how fully you use the GPU. Crucially, an LLM inference request stresses two different GPU resources at different times. The prefill phase, which ingests the input prompt, is compute-intensive: it does a lot of arithmetic. The decode phase, which emits output tokens one at a time, is memory-bandwidth-intensive: each step mostly reads the model weights and the key-value cache. These two resources — compute throughput and memory bandwidth — are physically separate on the GPU and can be exercised simultaneously. So if a batch contains a healthy mix of compute-heavy and memory-heavy work, the GPU can overlap them and stay fully utilized. If not, one resource idles while the other is the bottleneck. That is why controlling the resource mix of the work is the lever for offline throughput, and why leaving it to chance leaves large amounts of performance, and money, on the table.

The reason this is hard is that the obvious fix collides head-on with an optimization everyone already relies on.

Two things pull the request ordering in opposite directions. To balance resources, you want to reorder requests so each batch blends compute-intensive and memory-intensive work, letting the GPU overlap compute and memory bandwidth; a batch that is all one type wastes the other, and prior within-batch tricks cannot help because they operate on whatever batch already formed — if it is all memory-intensive, the system runs out of prefill work and the compute units sit idle through the long decode. To reuse computation, you want prefix sharing: many requests share a common prompt prefix — a shared system prompt, a set of few-shot examples, a common document — and if you group those requests together, the prefix is processed once and its key-value cache is reused by all of them, saving substantial compute. But prefix sharing wants requests with the same prefix adjacent, while resource balancing wants requests reordered by their compute density; these are generally different orderings. Maximize overlap and you scatter the prefix-sharing groups, losing reuse; maximize prefix sharing and you get resource-imbalanced batches, losing overlap. Because both are large effects, giving up either one costs throughput, and prior systems, optimizing one at a time, could not have both. Reconciling them in a single schedule is the real problem BlendServe has to solve.

What people did before, and where it broke

Optimize throughput within a batch (overlap prefill and decode). Overlap the compute-heavy and memory-heavy phases, or multiplex operators, inside whatever batch is given. Where it broke. It cannot fix an imbalanced batch: if the batch is mostly memory-intensive, prefill work runs out and the GPU’s compute units idle through the decode; overlap only helps when the mix is already good.
Apply prefix sharing on the given order. Group requests that share a prompt prefix so the prefix is computed once and its key-value cache is reused. Where it broke. It saves compute only when sharing requests happen to be adjacent, and it takes no account of whether the resulting batches are resource-balanced.
Reorder requests to balance resources. Sort requests by compute density to blend compute-heavy and memory-heavy work in each batch. Where it broke. This ordering scatters prefix-sharing groups, so it wins resource overlap but loses prefix reuse — trading one large optimization for another.
Pick one goal and accept the loss on the other. Optimize either resource overlap or prefix sharing, whichever seems dominant. Where it broke. Both effects are big, so sacrificing either leaves substantial throughput unrealized in offline inference.
The gap. No system produced a single request schedule that both balanced compute and memory demand across batches (for overlap) and preserved prefix-sharing groups (for reuse), so offline throughput fell short of what the hardware allows.

How BlendServe works

Turning a pile of requests into balanced, prefix-sharing batches

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Schedule the requests, not just the batch, to balance the GPU’s two resources. BlendServe’s foundational move is to treat the whole offline workload as something to reorder, choosing which requests share a batch so that compute-heavy and memory-heavy work always coexist and the GPU’s separate compute and memory-bandwidth resources are both saturated. Deciding, across many requests with different resource profiles, how to group them so a shared resource is used efficiently rather than left imbalanced is the same problem the work on constraint-aware resource scheduling for large-language-model inference covered elsewhere on this site tackles, scheduling requests against resource limits to raise utilization. BlendServe specializes this to offline throughput, where the freedom to reorder the entire batch makes deliberate resource blending possible.

Fill one resource’s idle time with another’s demand by overlapping them. The reason blending pays off is that compute and memory bandwidth are independent and can run concurrently, so pairing a request that needs one with a request that needs the other keeps both busy instead of one waiting. Overlapping two distinct kinds of work so the idle time of one hardware resource is covered by the demand of another is the same principle the work on efficient scheduling for communication-computation overlap covered elsewhere on this site is built on, hiding one resource’s stalls behind another’s activity. BlendServe applies it to compute versus memory bandwidth within LLM inference, and its whole batching design exists to create the balanced mixes that make the overlap possible.

Reuse shared prompt prefixes instead of recomputing them. The other half of BlendServe’s throughput comes from prefix sharing: requests that begin with the same prompt reuse that prefix’s computed key-value cache rather than each processing it from scratch. Exploiting the overlap between related requests to reuse cached computation instead of repeating it is the same idea the work on accelerating large-language-model serving for multi-turn dialogues with efficient resource reuse covered elsewhere on this site is built on, keeping and reusing the shared context across requests. BlendServe’s contribution is to preserve this reuse while simultaneously balancing resources, by scheduling over a prefix tree rather than forcing a choice between the two.

Following a mixed workload through BlendServe

Scheduling summarization and generation requests that share a system prompt

A trace, drawn from the paper, of how BlendServe blends resources while preserving prefix sharing.

Step 1 of 5

A workload mixes compute-heavy and memory-heavy requests

The batch contains document-summarization requests (long inputs, short outputs — compute-heavy) and text-generation requests (short inputs, long outputs — memory-heavy), many of which begin with the same shared system prompt.

The workload has both a resource imbalance and prefix-sharing opportunities.

Why it works

Blended batches keep both GPU resources busy. Compute and memory bandwidth are separate and can run at once → but only if a batch contains both compute-heavy and memory-heavy work → so pairing the two types via scheduling saturates both instead of leaving one idle.
A compute-density model tells the scheduler what to pair. Requests vary from compute-heavy to memory-heavy → but a performance model quantifies each as a density from lengths and model size → so the scheduler knows which requests to blend for balance.
The prefix tree carries sharing and resources at once. Balancing and prefix sharing normally want different orders → but annotating a prefix tree with subtree density holds both kinds of information together → so a single structure can be scheduled for both goals.
Dual scanning resolves the conflict in one pass. Sorting by density and scanning from both ends pairs opposite resource types → while operating over the tree keeps prefix groups intact → so the schedule achieves overlap and reuse simultaneously rather than trading them.
Both wins compound into higher throughput. Full resource overlap and preserved prefix reuse each raise throughput → and BlendServe secures both at the same time → so it beats systems that captured only one, by up to 1.44 times over vLLM and SGLang.

What it costs (trade-offs and scope)

It needs to estimate output length, which is not always known. Compute density depends on how many tokens a request will generate, and output length is dynamic and predicted (except for image or video generation, where it is fixed); the paper builds in robustness to inaccurate estimates, but the scheduling quality still depends on those predictions being reasonable.
The benefit depends on workload diversity. BlendServe’s gains come from blending compute-heavy and memory-heavy requests and from shared prefixes; a workload that is uniformly one resource type, or has little prefix sharing, offers less to blend or reuse and will benefit less.
It targets offline, latency-relaxed inference. The whole approach assumes the freedom to reorder the entire batch within a long deadline; it does not apply to online serving with tight per-request latency targets, where such global reordering is not permitted.
It is a scheduling layer on top of an engine. BlendServe is prototyped on NanoFlow (with chunked prefill) and relies on the underlying engine’s execution; its improvements are in how requests are ordered and batched, so realizing them requires integration with a capable inference engine.
The numbers are on specific models, datasets, and baselines. The up-to-1.44-times and 20.84-percent figures are against vLLM, SGLang, and NanoFlow on models like Llama-3 and Qwen-2.5 and datasets such as OpenVid, MMLU, and LIMO; other models, workloads, or hardware could shift the results.

The evaluation

Maximizing throughput in offline LLM batch inference

Property	Within-batch overlap only	Prefix sharing only	BlendServe
Balances compute and memory across batches
Preserves prefix sharing
Handles imbalanced workloads
Resolves overlap-vs-sharing conflict
Throughput

A comparison of offline LLM inference strategies. BlendServe uses a resource-aware prefix tree and a dual scanning algorithm to build batches that both balance compute and memory demand and preserve prefix sharing, reaching up to 1.44 times the throughput of vLLM and SGLang and 20.84 percent over the state-of-the-art NanoFlow.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (arXiv:2411.16102). The mechanism described is the paper’s: BlendServe optimizes offline LLM batch inference by harmonizing resource overlapping and prefix sharing to maximize throughput. It builds a performance model to estimate each request’s resource demand (a compute density reflecting compute-intensive prefill versus memory-bandwidth-intensive decode), organizes all requests in a resource-aware prefix tree whose nodes are annotated with the aggregate compute density of their subtree, and uses a dual scanning algorithm (sorting nodes by density and scanning inward from both ends) to produce a request schedule that forms resource-balanced batches while preserving prefix-sharing groups. It addresses the conflict between the schedule that maximizes resource overlap and the schedule that maximizes prefix sharing. It is prototyped on NanoFlow (with chunked prefill), supports data and tensor parallelism, and includes robustness to inaccurate output-length estimation. Evaluated on models including Llama-3-8B, Llama-3-70B, and Qwen-2.5-7B and datasets such as OpenVid, MMLU, and LIMO, it reports up to 1.44 times the throughput of vLLM and SGLang and an average 20.84 percent higher throughput than NanoFlow. Exact performance-model details, the scheduling algorithm, and full results should be read from the paper.

Background

Why LLM serving keeps turning into a scheduling problem

It helps to see BlendServe as part of a broad realization that serving large language models efficiently is, more than anything, a scheduling and resource-management problem. The models themselves are fixed and enormously expensive to run, so the way to cut cost is to squeeze more useful work out of each GPU-second, and that comes down to deciding what to run together and when. Inference has an unusual structure that makes this rich: it has two phases with opposite resource appetites (compute-heavy prefill and memory-heavy decode), requests differ enormously in shape, and requests often share context that can be reused. Every one of these becomes a lever — overlap the phases, batch complementary requests, reuse shared prefixes, disaggregate prefill from decode — and the levers interact, sometimes in tension, as resource balancing and prefix sharing do here. Offline inference sharpens the opportunity because the relaxed deadline grants total freedom to reorder the workload, so scheduling can be global and aggressive. The broader push to build high-throughput inference-serving systems by rethinking how requests are scheduled and resources are shared is the same undertaking the work on a high-throughput inference serving system with accuracy scaling covered elsewhere on this site pursues, treating serving as a systems problem rather than a model one. BlendServe contributes a scheduler that finally reconciles two of the biggest levers at once, arguing that the path to cheap offline inference runs through smarter batching.

If you remember only three things

1. Offline LLM batch inference is a cheap, latency-insensitive service (for evaluation, data curation, benchmarking) where the only goal is throughput. A GPU has two separable resources — compute and memory bandwidth — and LLM requests stress them differently (compute-heavy prefill, memory-heavy decode), so a batch mixing compute-heavy and memory-heavy requests keeps both busy. But applications have imbalanced resource demands, so a batch dominated by one type wastes the other, and reordering requests to balance them conflicts with prefix sharing, which groups requests that share a prompt prefix to reuse its computation.
2. BlendServe harmonizes both. It models each request's compute density (compute-versus-memory demand) from model dimensions and input/output lengths, organizes all requests in a resource-aware prefix tree (capturing prefix sharing, with each node tagged by the aggregate density of its subtree), and runs a dual scanning algorithm that sorts nodes by density and scans inward from both ends to pair compute-heavy with memory-heavy requests into balanced batches — without breaking the prefix-sharing groups. This produces one schedule that achieves resource overlap and prefix reuse simultaneously, resolving the conflict prior systems could not.
3. Prototyped on the NanoFlow engine, BlendServe delivers up to 1.44 times the throughput of the industry-standard vLLM and SGLang and 20.84 percent more than the prior state-of-the-art NanoFlow, on models like Llama-3 and Qwen-2.5. It depends on estimating output length (with built-in robustness to errors), benefits most on diverse workloads with prefix sharing, and targets offline inference specifically, where the relaxed deadline permits global reordering of the whole batch.

Check yourself

Check your understanding

1. Why does maximizing throughput in offline LLM inference come down to the resource mix of each batch?
2. What is the core conflict BlendServe has to resolve?
3. How does BlendServe satisfy both resource balancing and prefix sharing at once?
4. Predict: you run BlendServe on a workload that is almost entirely memory-intensive requests with no shared prompt prefixes. How much benefit should you expect, and why?

Sources

The paper is “BlendServe: Optimizing Offline Inference with Resource-Aware Batching” by Yilong Zhao, Shuo Yang, Kan Zhu, Lianmin Zheng, Baris Kasikci, Yifan Qiao, Yang Zhou, Jiarong Xing, and Ion Stoica (University of California, Berkeley and collaborators), published at ASPLOS 2026, DOI 10.1145/3779212.3790133. It is licensed CC BY.

This explainer is written from the paper’s full text (arXiv:2411.16102). Offline batch inference is a cost-effective, latency-insensitive way to run large language models, where maximizing throughput is the primary goal; previous studies optimized throughput only within a batch, but the diverse resource demands (compute-intensive versus memory-intensive) across applications make those approaches less effective when batches are resource-imbalanced. BlendServe first builds a performance model to analyze request resource demands, then designs a scheduler that harmonizes both resource overlapping and prefix sharing to maximize throughput. Resource overlapping (reordering requests to balance compute and memory demand) conflicts with prefix sharing (grouping requests that share a prompt prefix to reuse its computation), so BlendServe organizes all requests in a resource-aware prefix tree — annotating each node with the aggregate compute density of its subtree — and proposes a dual scanning algorithm (sorting nodes by density and scanning inward from both ends) to obtain a request schedule that forms resource-balanced batches while preserving prefix-sharing groups. Prototyped on NanoFlow (with chunked prefill) and supporting data and tensor parallelism with robustness to inaccurate output-length estimation, BlendServe achieves up to 1.44 times the throughput of the industry-standard vLLM and SGLang and an average 20.84 percent higher throughput than NanoFlow, the prior state-of-the-art, on models including Llama-3-8B, Llama-3-70B, and Qwen-2.5-7B and datasets such as OpenVid, MMLU, and LIMO. No numbers beyond the paper’s own have been introduced; exact performance-model and algorithm details and full results should be confirmed in the paper.