SwiftSpec: Disaggregated Speculative Decoding and Fused Kernels for Low-Latency LLM Inference

Ziyi Zhang, Ziheng Jiang, Chengquan Jiang, Menghan Yu, Size Zheng, Haibin Lin and 2 others

When one person waits on a chatbot or code assistant, the model generates tokens one at a time on a single request, and speculative decoding — a small draft model proposing tokens for a big target model to check — speeds that up, but the draft still sits on the critical path and spreading the work across GPUs stalls on communication; this paper (ByteDance Seed, University of Chicago — Ziyi Zhang, Henry Hoffmann, and colleagues) presents SwiftSpec, which puts the draft and target models on separate GPUs so they run asynchronously, keeps their KV caches consistent with an evolving token tree, and fuses custom CUDA kernels that hide communication — serving Llama-3-70B at 347 tokens per second on an eight-GPU node, 1.75 times faster on average than prior speculative decoding.

The argument in one glance

Problem

One person waiting on an LLM needs speed that throughput-tuned systems do not give

Most LLM-serving systems are tuned to maximize throughput: they batch many requests together so the expensive model weights are reused across users, which is efficient but not fast for any single request. Interactive uses — a chatbot, a code assistant, a robot — are the opposite case: one user, one request, and a hard limit on how long they will wait. Here latency is everything, and it is stubbornly high because decoding is sequential: the model produces one token at a time, each requiring a full pass over the model, and modern outputs are long. The paper notes that a 4-bit Llama-70B on eight NVIDIA H800 GPUs can take around 30 seconds to generate a 3000-token response with popular frameworks, and chain-of-thought reasoning can decode tens of thousands of tokens. Two known techniques each help. Speculative decoding uses a small, fast draft model to guess several tokens ahead, which the big target model then checks in one pass, so multiple tokens can be accepted per expensive step. Tensor parallelism splits the model across GPUs so each does part of every layer, cutting the time of a single forward pass. The trouble is combining them.
What people did

Run speculative decoding and split the models across GPUs with tensor parallelism

The obvious plan is to do both at once: use speculative decoding to get multiple tokens per target step, and use tensor parallelism to make each draft and each target step faster by spreading it over all the GPUs in the node. In the usual arrangement, the draft and target models are co-located on the same GPUs, and each takes its turn: the draft runs, generating candidate tokens (sometimes a tree of them), then the target verifies, then the draft runs again, all sharing the same tensor-parallel hardware.
Why it fell short

Draft and target do not share hardware gracefully, and small batches choke on communication

Co-locating the two models and giving them the same parallelism does not work well, for several reasons the paper identifies. First, the draft and target have very different compute needs: the target is huge and benefits from being split across many GPUs, but the draft is tiny, and splitting a tiny model across more GPUs makes it slower, not faster, because the communication between GPUs outweighs the shrinking compute. The paper measures this — going from two to four GPUs, a small Llama-3B draft model's time per step actually rises, while the Llama-70B target's falls — so no single tensor-parallel degree suits both. Second, because they take turns on the same hardware, the draft sits on the critical path: every millisecond the draft spends is a millisecond the target waits, added directly to latency. Third, at the batch size of a single request, tensor parallelism hits diminishing returns fast, because inter-GPU communication — not computation — comes to dominate each step, so adding GPUs stops helping. And keeping the draft and target KV caches consistent as speculation is accepted or rejected adds its own complexity. The result is that the two accelerations interfere instead of compounding.
This paper's idea

Pull draft and target onto separate GPUs, run them asynchronously, and fuse away the seams

SwiftSpec redesigns speculative decoding to be asynchronous and disaggregated. Instead of the draft and target taking turns on shared hardware, it partitions the node's GPUs into two groups — a draft group and a verification group — each running its model with its own parallelism, sized to its needs. Because they are on separate GPUs, the draft group can run continuously and asynchronously, generating candidate token trees for future steps while the target group verifies the current one, so the draft is taken off the critical path entirely. Three techniques make this work. Parallel tree generation lets the draft group keep producing speculative token trees ahead of time rather than on demand. A tree-aware, evolving KV cache keeps the draft and target caches consistent as speculation is accepted or rejected, and maximizes reuse between them, so the disaggregated models stay in sync without redundant recomputation. And a set of fused, latency-optimized CUDA kernels — about 3000 lines, using low-latency GPU communication primitives directly inside the core GEMM (matrix multiply) and attention kernels — hides the communication that otherwise dominates at single-request batch sizes. On one eight-GPU H800 node, SwiftSpec serves Llama-3-70B at 347 tokens per second, which the paper reports is 1.3 times faster than NVIDIA's own benchmarks on the higher-performance H200, and averages 1.75 times faster decoding than state-of-the-art speculative decoding across five model families and six datasets.

Why this problem is worth solving

The single-request, latency-first regime is a genuinely different problem from the throughput serving most systems are built for, and it is exactly the regime interactive AI lives in.

Throughput-oriented serving wins by batching: with many requests in flight, the cost of loading the model’s weights and running each layer is amortized across all of them, so the machine stays busy and efficient. But an interactive user is a batch of one. There is nothing to amortize against, so the same hardware that is efficient at scale is underused for a lone request, and the user still waits for every token to be produced in sequence. As outputs get longer — long chatbot answers, whole files of generated code, extended chain-of-thought reasoning that can run to tens of thousands of tokens — this sequential per-token latency compounds into seconds or minutes of waiting. Systems tuned to maximize tokens-per-second across many users do not automatically minimize time-to-finish for one, and the paper shows popular frameworks leaving large latency on the table in this setting. That gap matters because the most demanding interactive applications — coding assistants, conversational agents, robots acting in real time — are precisely single-request and precisely latency-bound. Building a system specifically for this case, rather than borrowing a throughput system, is what makes ultra-low-latency decoding achievable.

The deeper insight is that the two obvious accelerations pull in different directions when forced onto the same hardware, and separating them is what lets both work.

Speculative decoding and tensor parallelism each attack latency, but they make incompatible demands. Speculative decoding pairs a tiny draft model with a huge target model, and those two have opposite parallelism profiles: the giant target wants to be spread across as many GPUs as possible to shrink each pass, while the tiny draft is fastest on few GPUs, because once it is small enough the communication cost of splitting it further outweighs any compute saved. Forcing both onto the same tensor-parallel configuration means one of them is always mis-provisioned. Worse, when they share hardware and alternate, the draft’s time is added straight onto the target’s — it is on the critical path — and the whole thing runs at a single-request batch size where inter-GPU communication, not math, is the bottleneck. The reframing SwiftSpec makes is that draft and target are different workloads that should live on different hardware, each scaled to its own needs, and run at the same time rather than in turn. Once they are disaggregated, the draft can work ahead asynchronously and vanish from the critical path, and each model can use the parallelism that actually helps it. The remaining challenge becomes keeping the two in sync and hiding the communication — which is what the rest of the system addresses — but the conceptual unlock is refusing to make one piece of hardware serve two mismatched jobs.

What people did before, and where it broke

Throughput-oriented batched serving. Batch many requests so model weights are reused across users. Where it broke. A single interactive request has nothing to batch against, so these systems leave large latency on the table for the one user who is waiting.
Speculative decoding on shared hardware. Alternate a small draft and a large target model on the same GPUs, accepting multiple tokens per target pass. Where it broke. The draft sits on the critical path — its time adds directly to latency — and it must share a parallelism setting that cannot suit both a tiny and a huge model.
Tensor parallelism for a single request. Split each model across all GPUs to speed up every forward pass. Where it broke. At a batch size of one, inter-GPU communication dominates and adding GPUs hits diminishing returns fast; splitting a small draft model can even make it slower.
Naively combining speculation and tensor parallelism. Apply both at once on co-located models. Where it broke. Imbalanced compute between draft and target, KV-cache inconsistency, and small-batch communication overhead make the two techniques interfere rather than compound.
The gap. No system let the draft and target each use the parallelism it needs, run at the same time off each other’s critical path, stay KV-consistent, and avoid drowning in communication at single-request batch sizes.

How SwiftSpec works

From alternating co-located models to asynchronous disaggregated decoding

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Take the draft model off the critical path with speculation done right. SwiftSpec keeps the core bet of speculative decoding — a cheap draft model proposing tokens a costly target checks in bulk — but restructures it so the draft never makes the target wait. Pairing a small, fast draft model with a large, accurate target model so the expensive model verifies many cheap guesses at once is the same speculative-decoding bargain the work on a heterogeneous architecture for speculative decoding with a draft model (DFVG) covered elsewhere on this site is built on, spending cheap draft compute to save expensive target compute. SwiftSpec’s twist is to run the draft asynchronously on its own hardware so the bargain no longer costs critical-path time.

Disaggregate mismatched components onto hardware each can use well. Rather than force a tiny draft and a huge target to share one GPU configuration, SwiftSpec separates them into independently-scaled groups. Splitting resources that were bundled together into separate pools, each provisioned and scaled for its own demand instead of compromising on a shared allocation, is the same disaggregation principle the work on a hardware-software co-designed disaggregated memory system (Clio) covered elsewhere on this site is built on, decoupling components so each scales on its own terms. SwiftSpec disaggregates the draft and target models the way disaggregated systems separate compute from memory, so each model runs on the parallelism that actually helps it.

Fuse communication into the compute so the small-batch bottleneck disappears. Because single-request decoding is dominated by inter-GPU communication rather than math, SwiftSpec builds kernels that carry the communication inside the GEMM and attention computation instead of alongside it. Weaving communication and computation together so the two overlap and the communication is hidden behind useful work, rather than stalling for it, is the same fine-grained overlap idea the work on transparent tracking and triggering for computation-communication overlap (T3) covered elsewhere on this site pursues, folding data movement into the compute it feeds. SwiftSpec’s fused CUDA kernels apply this to the GEMM-with-all-reduce and attention operations that dominate low-latency decoding.

Following a decoding round through SwiftSpec

From a partial output to several more accepted tokens, off the critical path

A trace, drawn from the paper, of one round of SwiftSpec's asynchronous disaggregated speculative decoding across a draft GPU group and a target GPU group.

Step 1 of 5

The draft group builds a token tree ahead of time

On its own few GPUs, the draft model runs continuously, generating a tree of candidate continuations for upcoming steps — parallel tree generation — so it is already working on what might come next rather than waiting to be asked.

The draft runs asynchronously, so its time is not added to the target's.

Why it works

Asynchronous drafting removes the draft from the critical path. On shared hardware the target waits for the draft each round → but giving the draft its own GPUs to run ahead → so drafting overlaps verification instead of adding to latency.
Right-sized parallelism suits each model. One tensor-parallel degree cannot fit both a tiny and a huge model → but disaggregating them into separately-scaled groups → so the target spreads wide while the draft stays compact and fast.
The tree cache keeps disaggregated models consistent. Splitting draft and target risks divergent, redundant KV state → but a tree-aware evolving cache tracking accept and reject → so the two stay in sync and reuse shared computation.
Fused kernels hide the small-batch communication cost. At batch size one, inter-GPU communication dominates each step → but folding low-latency communication into the GEMM and attention kernels → so the exchange overlaps compute rather than stalling it.
Parallel tree generation keeps the target fed. An on-demand draft would leave the target waiting for candidates → but generating token trees ahead of time → so the target almost always has speculations ready to verify.

What it costs (trade-offs and scope)

Disaggregation spends GPUs on the draft. Dedicating part of the node to a draft group means those GPUs are not verifying; the design bets that removing the draft from the critical path is worth partitioning the hardware, which suits a latency-first, single-request goal more than a throughput one.
It is engineered for single-request, low-latency decoding. SwiftSpec targets the batch-of-one interactive case; its wins come from that regime’s specific bottlenecks (small-batch communication, sequential latency) and do not directly transfer to high-throughput batched serving.
The custom kernels are substantial, specialized work. About 3000 lines of hand-written CUDA using low-latency communication primitives is significant engineering tied to particular GPU hardware and collective libraries, not a portable drop-in.
Speculative decoding still depends on draft quality. The speedup rests on the draft’s guesses being accepted often; a poorly-matched draft model would lower the acceptance rate and erode the gains, as in any speculative-decoding system.
Results are on specific hardware and models. The 347 tokens per second, the 1.3-times-over-H200 claim, and the 1.75-times average are measured on particular Hopper GPU nodes across the paper’s model families and datasets; other setups may differ.

The comparison

Accelerating single-request LLM decoding

Property	Speculative decoding alone	Naive speculation plus tensor parallelism	SwiftSpec
Where the draft model runs
Parallelism per model
Small-batch communication
KV-cache consistency
Reported speed

A comparison of ways to speed up single-request LLM decoding. Speculative decoding alone leaves the draft on the critical path; naively adding tensor parallelism forces a shared parallelism setting that suits neither model and drowns in small-batch communication. SwiftSpec disaggregates the draft and target onto separate GPU groups running asynchronously, keeps their KV caches consistent with an evolving tree cache, and fuses low-latency communication into the GEMM and attention kernels — serving Llama-3-70B at 347 tokens per second on an eight-GPU node, reportedly 1.3 times faster than NVIDIA's H200 benchmarks and 1.75 times faster on average than state-of-the-art speculative decoding. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ arXiv copy, 2506.11309, titled “SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding,” the same system as the ASPLOS 2026 paper). The described system is the paper’s: SwiftSpec redesigns speculative decoding in an asynchronous, disaggregated manner, partitioning a node’s GPUs into a draft group and a verification group so each model uses its own tensor-parallel degree and the draft runs off the critical path. It contributes parallel tree generation (the draft group produces tree-structured candidate tokens ahead of time), tree-aware KV-cache management (an evolving tree cache that keeps the draft and target caches consistent and maximizes reuse as speculation is accepted or rejected), and fused, latency-optimized CUDA kernels (about 3000 lines, using low-latency GPU communication primitives inside the core GEMM and attention kernels to hide communication at single-request batch sizes). The headline results — 347 tokens per second for Llama-3-70B on an eight-GPU H800 node, reported as 1.3 times faster than NVIDIA’s own benchmarks on an eight-GPU H200 setup, and an average 1.75 times speedup over state-of-the-art speculative decoding across five model families and six datasets, with a 1.7 times speedup at the 95th percentile over the best open-source baseline across 480 Llama-3-70B queries — are from the paper’s evaluation. The arXiv version reports 348 tokens per second; the ASPLOS abstract states 347. Background on speculative decoding, tensor parallelism, KV cache, and GPU collective communication is standard context. The step-by-step walkthrough illustrates the described mechanism. Exact kernel designs, the tree-cache algorithm, and full results should be read from the paper.

Background

Why fast LLM serving keeps splitting the work and fusing the seams

It helps to see SwiftSpec within the broader effort to serve large models fast. Two recurring moves show up across this work. The first is disaggregation: rather than run every part of an inference on one uniform pool of hardware, systems increasingly separate stages or components — prefill from decode, compute from memory, here draft from target — so each can be scaled and scheduled for its own very different demands. The second is fighting communication: once a model is split across GPUs, the exchanges between them can dominate, especially at the small batch sizes of latency-sensitive serving, so a great deal of systems work goes into overlapping, fusing, or otherwise hiding that communication behind computation. SwiftSpec combines both, disaggregating the draft and target and then fusing communication into its kernels so the disaggregation does not cost latency. This drive to hit tight latency targets for model serving by carefully structuring the system around the workload’s bottlenecks is the same terrain the work on a native serverless system for low-latency, high-throughput inference (INFless) covered elsewhere on this site occupies, engineering the serving stack to meet strict latency demands. SwiftSpec’s particular contribution is to show that the two accelerations of speculative decoding and tensor parallelism, which fight when co-located, compound when the models are pulled apart and the seams between them are fused shut.

If you remember only three things

1. Interactive LLM applications — chatbots, code assistants, robots — are single-request and latency-bound, a regime throughput-oriented batched serving does not serve well, because a batch of one has nothing to amortize against and decoding is sequential (a 4-bit Llama-70B can take about 30 seconds for a 3000-token answer). Two accelerations each help but resist combining: speculative decoding (a small draft model proposes tokens a large target verifies in bulk) leaves the draft on the critical path, and tensor parallelism (splitting models across GPUs) stalls on inter-GPU communication at single-request batch sizes. Forcing both onto co-located models means one tensor-parallel setting cannot suit both a tiny draft and a huge target, KV caches drift out of sync, and communication overhead cancels the gains.
2. SwiftSpec redesigns speculative decoding to be asynchronous and disaggregated: it partitions a node's GPUs into a draft group and a verification group, each with its own parallelism sized to its model, so the draft runs continuously and ahead — off the target's critical path — instead of taking turns. Three techniques make the split work: parallel tree generation, where the draft group produces tree-structured candidate tokens in advance so the target rarely waits; a tree-aware evolving KV cache that keeps the draft and target caches consistent and reused as speculations are accepted or rejected; and about 3000 lines of fused, latency-optimized CUDA kernels that fold low-latency GPU communication directly into the GEMM and attention computation, hiding the communication that dominates at batch size one.
3. The payoff is that speculative decoding and tensor parallelism, which interfere when co-located, compound when the models are disaggregated and the communication seams are fused shut. On one eight-GPU H800 node, SwiftSpec serves Llama-3-70B at 347 tokens per second — reported as 1.3 times faster than NVIDIA's own benchmarks on a higher-performance H200 node — and averages 1.75 times faster decoding than state-of-the-art speculative decoding across five model families and six datasets, with a 1.7 times speedup at the 95th percentile over the best open-source baseline. The trade-offs: it dedicates GPUs to the draft, targets the single-request low-latency case specifically rather than high-throughput serving, relies on substantial hardware-specific custom CUDA, and still depends on the draft model's acceptance rate.

Check yourself

Check your understanding

1. Why do throughput-oriented serving systems fall short for interactive LLM applications?
2. Why is it hard to combine speculative decoding with tensor parallelism on shared hardware?
3. What is SwiftSpec's central design change?
4. What three techniques let SwiftSpec make disaggregation work, and what did it achieve?

Sources

The paper is “SwiftSpec: Disaggregated Speculative Decoding and Fused Kernels for Low-Latency LLM Inference” by Ziyi Zhang, Ziheng Jiang, Chengquan Jiang, Menghan Yu, Size Zheng, Haibin Lin, Xin Liu, and Henry Hoffmann (ByteDance Seed and University of Chicago), published at ASPLOS 2026, DOI 10.1145/3779212.3790246. It is available on arXiv as 2506.11309, titled “SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding.”

This explainer is written from the paper’s full text (the authors’ arXiv copy). SwiftSpec redesigns speculative decoding in an asynchronous, disaggregated manner: it partitions a node’s GPUs into a draft group and a verification group so each model uses its own tensor-parallel degree and the draft runs off the critical path. Its techniques are parallel tree generation, tree-aware KV-cache management (an evolving tree cache for consistency and reuse), and fused, latency-optimized CUDA kernels (about 3000 lines, using low-latency GPU communication primitives inside the GEMM and attention kernels). Evaluated on Hopper GPU nodes, the paper reports 347 tokens per second for Llama-3-70B on eight H800 GPUs (the arXiv version states 348), reported as 1.3 times faster than NVIDIA’s benchmarks on an eight-GPU H200 node, and an average 1.75 times speedup over state-of-the-art speculative decoding across five model families and six datasets, with a 1.7 times speedup at the 95th percentile over the best open-source baseline across 480 Llama-3-70B queries. Background on speculative decoding, tensor parallelism, KV cache, and GPU communication is standard context. The numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.