Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads

Mert Hidayetoglu, Aurick Qiao, Michael Wyatt, Jeff Rasley, Yuxiong He and Samyam Rajbhandari

Serving a large language model across many GPUs forces a painful choice — tensor parallelism splits each layer to answer a single request fast but wastes throughput on constant cross-GPU chatter, while data parallelism runs independent replicas for high throughput but cannot speed up one request; you cannot mix them because their key-value caches are laid out incompatibly, so real traffic that swings between quiet interactive moments and heavy batch bursts is always served sub-optimally; this work from Snowflake notices that a third scheme, sequence parallelism, is fast like data parallelism yet stores its cache exactly like tensor parallelism, so a server can flip between the two on the fly with no cache reshuffling, getting low latency when traffic is light and full throughput when it is heavy.

The argument in one glance

Problem

LLM serving forces a latency-versus-throughput choice

Tensor parallelism (TP) splits each layer to answer one request fast, but its per-layer cross-GPU communication caps throughput. Data parallelism (DP) runs replicas for high throughput but cannot speed up a single request. Real traffic swings between light interactive and heavy batch, so one fixed choice is always wrong for half the time.
What people did

Switch between TP and DP as traffic changes

Use TP when traffic is light for low latency, and DP when traffic is heavy for high throughput, flipping between them to match the moment.
Why it fell short

TP and DP store the KV cache incompatibly

TP and DP lay out the attention key-value cache in memory in incompatible ways, so you cannot combine them, and converting between the two layouts on the fly is complex and costly — making dynamic switching impractical.
This paper's idea

Use sequence parallelism as the throughput mode (Shift Parallelism)

Sequence parallelism (SP) behaves like DP for throughput, avoiding TP's costly all-reduce, but crucially stores the KV cache in the same layout as TP. So a server can switch between TP and SP with no cache reshuffling. Shift Parallelism flips between them by traffic — low latency when light, full throughput when heavy — up to 1.51 times faster response and 50 percent more throughput than TP alone.

Why this problem is worth solving

Serving large language models is now the dominant AI workload, and how you spread a model across GPUs determines both its speed and its cost.

A large language model is too big and too compute-heavy to serve from a single GPU at scale, so it is partitioned across many GPUs working together on each request. Inference has two distinct quality-of-service concerns. For interactive use — a chatbot, an agent, a coding assistant — what matters is latency: the time to the first token of the response (TTFT), and the time between each subsequent token (TPOT), because a person or a calling program is waiting on each piece. For batch use — summarizing or translating thousands of documents at once — individual latency barely matters, and what matters is combined throughput, the total tokens per second the system processes, because that sets the cost per token. Crucially, the parallelism techniques used for inference were largely inherited from training, where the workload is stable, homogeneous, and cares only about throughput. Inference is the opposite: bursty, dynamic, with unpredictable traffic that swings between quiet interactive periods and heavy batch bursts, and different requests have different requirements. A serving system must somehow do well on both axes at once, under traffic it cannot predict.

The two standard ways to parallelize each sit at opposite, unsatisfying ends of that tradeoff, and the obvious fix is blocked by a low-level detail.

Tensor parallelism (TP) partitions the weight matrices and computation inside each layer across the GPUs, so all of them collaborate on a single request and answer it quickly. But splitting a layer means the partial results must be recombined every layer with an all-reduce, a collective communication whose cost grows with the number of GPUs; the more you parallelize, the larger the fraction of time spent communicating rather than computing, which drags down combined throughput. Data parallelism (DP) takes the opposite approach: each GPU or group holds a full replica of the model and serves different requests independently, an embarrassingly parallel arrangement with no per-layer communication, so throughput is high — but because a single request runs on a single replica, DP does nothing to lower that request’s latency. The natural idea is to switch: run TP when traffic is light to minimize latency, and DP when traffic is heavy to maximize throughput. This is blocked by memory layout. TP and DP arrange the attention key-value cache — the stored keys and values from previous tokens that every new token attends to — in fundamentally incompatible ways, so you cannot combine them, and reshuffling the entire KV cache to convert from one layout to the other mid-serving is complex and prohibitively costly.

What people did before, and where it broke

Use tensor parallelism for everything. Partition every layer across the GPUs to keep single-request latency low. Where it broke. The per-layer all-reduce communication grows with the parallelism degree, so combined throughput suffers, wasting capacity during heavy batch bursts.
Use data parallelism for everything. Run independent replicas for maximum throughput. Where it broke. A single request runs on one replica and gets no speedup, so latency is poor — bad for interactive traffic where a user waits on every token.
Statically pick one parallelism per deployment. Choose TP or DP up front based on the expected workload. Where it broke. Real enterprise traffic is dynamic and mixed, swinging between interactive and batch, so any fixed choice is wrong half the time, paying latency or throughput penalties it did not need to.
Try to switch between TP and DP on the fly. Flip parallelism as traffic changes to match the moment. Where it broke. TP and DP store the attention KV cache in incompatible memory layouts, so switching requires reshuffling the whole cache, which is complex and too costly to do dynamically.
The gap. No approach gave both low latency in light traffic and high throughput in heavy traffic while being able to switch cheaply. Providing that, by finding a throughput-friendly parallelism that shares TP’s cache layout, is what Shift Parallelism does.

How Shift Parallelism works

From a fixed parallelism to switching by traffic without cache reshuffling

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Match the parallelism to the moment, not to a guess. Shift Parallelism starts from the observation that enterprise inference traffic is dynamic and mixed — quiet interactive stretches punctuated by heavy batch bursts — so committing to a single parallelism means being wrong for whichever regime you did not optimize for. Instead, the server should run a latency-optimized mode when traffic is light and a throughput-optimized mode when it is heavy, switching as the traffic shifts. Adapting the serving strategy in response to conditions that change unpredictably at runtime, rather than fixing it at deployment for an average case, is the same move the preemptible-instance LLM serving work covered elsewhere on this site makes, reconfiguring how the model is served on the fly as the environment changes so the system stays efficient under conditions no static choice could anticipate.

Find the shared representation that lets two strategies interoperate. The reason TP and DP cannot be switched cheaply is a mismatch in one specific structure — the attention key-value cache is laid out differently, so converting between them means moving all of it. The paper’s key discovery is that sequence parallelism, though a different scheme, keeps the KV cache in exactly the same layout as tensor parallelism, so the expensive conversion simply disappears and switching becomes seamless. Identifying the common underlying representation that two otherwise-different strategies can share, so that combining or alternating them costs almost nothing, is the same insight the multi-dimensional parallelism work covered elsewhere on this site is built on, making distinct forms of parallelism cooperate by aligning how the model’s state is partitioned rather than forcing an expensive translation at every boundary.

Escape the tradeoff instead of picking a point on it. Prior systems chose a spot on the latency-versus-throughput curve: TP near the low-latency end, DP near the high-throughput end, each paying dearly on the other axis. By making both a latency mode and a throughput mode available at once and switching between them for free, Shift Parallelism moves the whole tradeoff curve outward — getting low latency in light traffic without giving up throughput in heavy traffic, rather than settling for one compromise. Refusing a fixed operating point and instead navigating the latency-throughput tradeoff dynamically to serve each request well is the same goal the constraint-aware LLM scheduling work covered elsewhere on this site pursues, treating the tradeoff as something to be steered request by request and moment by moment rather than a single dial set once for the whole deployment.

Following a request through Shift Parallelism

Serving light and heavy traffic on the same GPUs

A trace of Shift Parallelism handling changing traffic, following the paper's SP and TP switching design.

Step 1 of 5

The model spans a node of GPUs

A large model is served across the GPUs of a node. Shift Parallelism keeps two configurations ready: a base configuration using sequence parallelism (or mixed SP and TP), and a shift configuration using full tensor parallelism across the node.

Weights are replicated so both modes are available without reloading.

Why it works

Dynamic traffic makes any fixed parallelism wrong half the time. Because enterprise inference swings between light interactive and heavy batch traffic → a single static choice pays a latency or throughput penalty in the regime it did not optimize → so switching modes to match the traffic beats committing to one.
A shared cache layout makes switching free. Because SP and TP store the attention KV cache identically → flipping between them needs no costly cache reshuffling → so the switch that was impossible between TP and DP becomes seamless.
Sequence parallelism avoids the all-reduce tax. Because SP uses all-to-all communication whose cost does not grow with the parallelism degree, unlike TP’s per-layer all-reduce → it sustains high combined throughput like DP → so it is a genuine throughput mode, not just a relabeled TP.
Full TP minimizes single-request latency. Because in the shift configuration all GPUs collaborate on each request → the per-token and first-token latencies drop → so light interactive traffic is served fast when it matters.
Adapting to inference, not training, unlocks it. Because SP came from training and inference has its own nuances → extending it for grouped-query attention, small-batch load balancing, and cache replication beyond the head count → so SP works as a practical inference mode across real models.

What it costs (trade-offs and scope)

Sequence parallelism has the worst per-token latency of the three. SP does not parallelize the decoding step the way TP does, so in isolation its time-per-output-token is worse than both TP and DP. Shift Parallelism only uses it as the throughput mode and switches to TP when latency matters, but SP alone is not a latency solution.
Weights are replicated to keep both modes ready. To transition between the base and shift configurations without reloading, the paper replicates weights rather than slicing them on the fly, chosen after a memory-cost analysis. This costs extra GPU memory that could otherwise hold more KV cache or a larger batch.
Cache invariance needs careful engineering. The KV cache is not automatically invariant for arbitrary SP and TP combinations, because head ordering in head parallelism breaks it; the authors resolve this with a general process-to-data mapping, so the seamless switch depends on that mechanism being correct.
The gains are relative to a TP-only baseline on specific setups. The up-to 1.51 times faster response and 50 percent higher throughput are versus TP-only, measured on particular models and hardware (for example Llama-70B on eight H200 GPUs in FP8). Against DP it trades differently — faster generation in light traffic while giving up some throughput in heavy traffic.
It targets within-node parallelism switching. The design switches between configurations spanning a node of GPUs; scaling the idea across many nodes, or to very high parallelism degrees beyond the number of attention heads, brings the cache-replication cases the paper has to handle specially.
It is a serving-time mechanism, not a model change. Shift Parallelism changes how an existing model is parallelized and served, integrated into vLLM as a plug-in; it does not change the model’s quality, and its benefit is realized only when traffic actually varies enough to make switching worthwhile.

The evaluation

Three ways to parallelize LLM inference across GPUs

Property	Tensor parallelism	Data parallelism	Shift Parallelism
Single-request latency
Combined throughput
KV cache layout	TP layout
Handles dynamic mixed traffic
Versus TP-only	Baseline

Three ways to parallelize large language model inference across GPUs, as compared in the paper. Tensor parallelism (TP) splits each layer so all GPUs collaborate on one request, giving low latency, but its per-layer all-reduce communication grows with the GPU count and reduces combined throughput. Data parallelism (DP) runs independent replicas for high throughput but cannot speed up a single request, so latency is poor; and it cannot be cheaply switched with TP because their attention key-value cache layouts are incompatible. Shift Parallelism uses Ulysses sequence parallelism (SP) as its throughput mode: SP partitions the token sequence and uses all-to-all communication whose cost does not grow with the parallelism degree (throughput-friendly like DP), yet it stores the KV cache in the same layout as TP. Because of that shared layout, Shift Parallelism switches between a full-TP configuration for low latency in light traffic and an SP configuration for high throughput in heavy traffic with no cache reshuffling, driven by batch size and traffic. The authors adapt SP from training to inference (handling grouped-query attention, small-batch load balancing, mixing with TP, and cache replication when parallelism exceeds the head count), replicate weights so both modes are ready, and integrate it into vLLM. On production traces and synthetic benchmarks across models, context sizes, and arrival rates (for example Llama-70B on eight H200 GPUs in FP8), Shift Parallelism delivers up to 1.51 times faster response in interactive workloads and 50 percent higher throughput in batch workloads than TP-only, and versus DP it generates about 2 times faster in light traffic while losing only about 17 percent throughput in heavy traffic — a strictly better latency-throughput tradeoff.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

Shift Parallelism is a neat systems result whose appeal is that the key move is an observation, not a new mechanism. The setup is a real and expensive problem: serving large language models is now the dominant AI workload, and the parallelism techniques for it were inherited from training, which is stable and throughput-only, whereas inference is bursty, mixed, and cares about both latency and throughput. The two standard schemes sit at opposite ends — tensor parallelism gives low latency but its per-layer all-reduce communication caps throughput, data parallelism gives high throughput but cannot speed up a single request — and the obvious fix of switching between them as traffic changes is blocked because their attention key-value caches are laid out incompatibly, so converting mid-flight is too costly. The paper’s insight is that a third scheme, Ulysses sequence parallelism borrowed from training, behaves like data parallelism for throughput (cheap all-to-all communication that does not grow with the parallelism degree) but stores the KV cache in the same layout as tensor parallelism. That shared layout is the whole trick: it lets a server flip between a throughput mode and a latency mode with no cache reshuffling, matching the parallelism to the traffic moment by moment. The reported gains are strong — up to 1.51 times faster response and 50 percent more throughput than tensor parallelism alone, and a strictly better tradeoff than either baseline — and it is integrated into vLLM and deployed. Read the scope with care. Sequence parallelism on its own has the worst per-token latency of the three, so it is only ever the throughput mode; the seamless switch requires replicating weights, which costs GPU memory, and a careful process-to-data mapping to keep the cache invariant when head ordering would otherwise break it; the headline numbers are against a tensor-parallel baseline on specific models and hardware; and the benefit only materializes when traffic actually varies enough to reward switching. The durable idea is that when two strategies each win on one axis, the path to getting both is often not a cleverer algorithm but finding a representation they can share, so that moving between them becomes free.

Background

Why LLM serving keeps hunting for the latency-throughput sweet spot

It helps to see Shift Parallelism within the fast-moving effort to make large language model serving efficient, which has become one of the central systems problems in computing because inference now dominates AI cost. The recurring tension is between latency, what an interactive user or agent waits on, and throughput, what sets the cost per token for batch work, and almost every serving advance is a new way to navigate that tension under real, unpredictable traffic. The tree-based speculative LLM serving work covered elsewhere on this site attacks the same tension from the decoding side, restructuring how tokens are generated to raise throughput without hurting the latency an interactive user feels. The common thread is that serving efficiency is fundamentally about not being forced into a single operating point: interactive and batch traffic want opposite things, real traffic mixes them unpredictably, and the wins come from systems that can serve each regime well without paying the other’s cost. Shift Parallelism contributes a particularly clean instance of this — exploiting a shared cache layout so the system can be latency-optimized and throughput-optimized at different moments on the same hardware — and as LLM traffic keeps growing and diversifying, techniques that let one deployment gracefully span the whole latency-throughput space, rather than being provisioned for a single workload shape, are likely to remain central.

If you remember only three things

1. Serving a large language model means splitting it across many GPUs, and the two standard ways trade off badly. Tensor parallelism (TP) partitions each layer's weights and computation so all GPUs collaborate on one request, giving low latency, but every layer needs an all-reduce whose cost grows with the GPU count, which reduces combined throughput. Data parallelism (DP) runs independent model replicas for high throughput but cannot speed up any single request, so latency is poor. You would want to switch — TP when traffic is light for latency, DP when heavy for throughput — but TP and DP lay out the attention key-value (KV) cache in memory incompatibly, so switching requires an expensive full-cache conversion. Real enterprise traffic is bursty and mixed (quiet interactive periods and heavy batch bursts), so a system locked into one parallelism is always wrong for the regime it did not optimize.
2. Shift Parallelism's key observation is that Ulysses sequence parallelism (SP), a scheme from training, behaves like DP for throughput — it partitions the token sequence and uses all-to-all communication whose cost does not grow with the parallelism degree, avoiding TP's per-layer all-reduce — yet it stores the KV cache in exactly the same layout as TP. Because SP and TP share that layout, a server can switch between them seamlessly with no cache reshuffling. Shift Parallelism keeps two configurations ready (a full-TP shift configuration for low latency, and an SP or mixed base configuration for high throughput) and flips between them based on batch size and traffic. The authors adapt SP from training to inference (handling grouped-query attention, load-balancing at small batch sizes, combining with TP, and replicating the cache when the parallelism degree exceeds the number of attention heads), resolve head-ordering with a general process-to-data mapping, replicate weights so both modes are instantly available, and integrate it into vLLM as a plug-in already deployed in production.
3. On real production traces and synthetic benchmarks across models, context sizes, and arrival rates (for example Llama-70B on eight H200 GPUs in FP8), Shift Parallelism delivers up to 1.51 times faster response in interactive workloads and 50 percent higher throughput in batch workloads compared with TP-only, and versus DP it generates about 2 times faster in light traffic while losing only about 17 percent throughput in heavy traffic — a strictly better latency-versus-throughput tradeoff. Scope: SP alone has the worst per-token latency and is only used as the throughput mode, weight replication costs GPU memory, cache invariance needs careful engineering, and the headline gains are against a TP baseline on specific setups. The durable lesson is that when two strategies each win on one axis, the way to get both is often not a cleverer algorithm but finding a shared representation (here, the KV cache layout) that makes switching between them free.

Check yourself

Check your understanding

1. Why do tensor parallelism and data parallelism sit at opposite ends of the latency-throughput tradeoff?
2. What is the key observation that makes Shift Parallelism possible?
3. How does Shift Parallelism actually switch modes, and when does it use each?
4. What did Shift Parallelism achieve, and what are its main limits?

Sources

The paper is “Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads” by Mert Hidayetoglu, Aurick Qiao, Michael Wyatt, Jeff Rasley, Yuxiong He, and Samyam Rajbhandari, published at ASPLOS 2026, DOI 10.1145/3779212.3790219. The authors are at Snowflake AI Research.

This explainer is based on the full text of the paper, obtained from the authors’ arXiv version (arXiv:2509.16495, a Creative Commons Attribution / CC-BY version). The problem framing (LLM inference as the dominant AI workload; interactive versus batch workloads and the metrics of time-to-first-token, time-per-output-token, and combined throughput; parallelism techniques inherited from training being ill-suited to dynamic inference traffic; tensor parallelism giving low latency but reduced throughput due to per-layer all-reduce communication that grows with the GPU count; data parallelism giving high throughput but poor single-request latency; and the impossibility of cheaply switching TP and DP because of incompatible KV cache layouts), the Shift Parallelism design (the observation that Ulysses sequence parallelism has DP-like throughput via all-to-all communication that does not grow with the parallelism degree yet shares TP’s KV cache layout; adapting SP from training to inference including grouped-query attention, load-balancing at small batch sizes, combining with TP, and KV cache replication when the parallelism degree exceeds the number of attention heads; the base and shift configurations; the process-to-data mapping for cache invariance and weight replication for mode transitions; and integration into vLLM as a plug-in), and the results (up to 1.51 times faster response in interactive workloads and 50 percent higher throughput in batch workloads versus TP-only, and about 2 times faster generation than DP in light traffic while losing about 17 percent throughput in heavy traffic, evaluated on production traces and synthetic benchmarks with an example of Llama-70B on eight H200 GPUs in FP8) are all drawn directly from the paper. Background explanations of transformer inference, attention, the KV cache, tensor and data parallelism, and collective communications are standard and provided to make the paper’s contribution understandable.