Serving a giant language model usually assumes a rack of identical GPUs, but real clusters are a mix — a few A100s here, a pile of old T4s there, fast links inside a datacenter and slow ones between regions — and the two decisions you must make (which model layers live on which GPU, and how to route each request through them) are tangled together, so the balanced-FLOPs heuristics of prior systems create network bottlenecks that waste the fast GPUs; Helix (CMU) reframes the whole thing as a MAX-FLOW problem on a graph whose edges encode both GPU compute and link bandwidth, solves a mixed-integer program to place layers and route requests jointly for maximum throughput, and at runtime gives every request its own pipeline chosen by weighted round-robin — lifting serving throughput up to 3.3x and cutting prompt latency up to 66% versus the best heterogeneous-serving baselines.
The argument in one glance
-
Problem
A huge model, and a junk-drawer of mismatched GPUs to run it on
A modern LLM is too big for one GPU — LLaMA-2 70B needs about 7 A100s, or 30 L4s, or 12 H100s, depending on the card. But the GPUs you actually have are rarely a clean set of identical A100s. Cloud regions run short of the good cards, so you end up with a MIX: a few A100s, a pile of L4s, a heap of older T4s, sometimes spread across regions connected by slow links while GPUs inside a region talk fast. Serving the model well over that heterogeneous pool — using the fast GPUs fully without choking on the slow links — is the problem, and the systems everyone uses were not built for it.
-
What people did
Assume the cluster is homogeneous, or patch heterogeneity with heuristics
Mainstream serving systems (vLLM, Orca) assume every GPU is identical and split the model uniformly across them. The heterogeneity-aware systems that exist lean on simple rules: Petals runs decentralized volunteer GPUs with no global coordinator, and Swarm (from heterogeneous training) balances the FLOPs — give each GPU a share of layers proportional to its compute. Split the work in proportion to how fast each GPU is, the thinking goes, and everything balances out.
-
Why it fell short
Balancing compute ignores the network, and placement and routing are entangled
Balancing FLOPs optimizes the wrong quantity. If you hand each GPU a share of layers proportional to its speed but ignore the links between them, consecutive pipeline stages can land on either side of a slow inter-region link, and that link — not any GPU — becomes the bottleneck, starving the fast A100s waiting on data. Uniform partitioning is worse: it underuses the powerful GPUs entirely. And the deeper trouble is that the two decisions are ENTANGLED — where you place the model layers determines which GPU-to-GPU routes even exist and what they cost, and how you route requests determines whether a placement is actually good. Optimizing them one at a time, by heuristic, leaves throughput on the table, and neither Petals nor Swarm optimizes the real end-to-end objective at all.
-
This paper's idea
See serving as max-flow, and solve placement and routing together
Helix (Carnegie Mellon) reframes the entire problem as a classic one: MAX-FLOW on a directed weighted graph. Each GPU becomes a node, and every edge carries a capacity that captures BOTH the GPU's compute throughput AND the network bandwidth of the link — so the maximum flow from a source to a sink through this graph is exactly the cluster's maximum serving throughput (one unit of flow = one token per second). Because the model PLACEMENT decides which edges exist, Helix solves a mixed-integer linear program that jointly picks the layer placement AND the request routing to maximize that flow. At runtime it gives every request its OWN pipeline through the cluster, chosen by an interleaved weighted round-robin that follows the max-flow solution without bursting. The result: up to 3.3x higher throughput and up to 66% lower prompt latency than the best heterogeneous-serving baselines.
Why this problem is worth solving
Nobody serving large models at scale gets to choose a pristine
cluster of identical GPUs. The good cards are scarce and
unevenly distributed across cloud regions, so the realistic
situation is a
a serving pool made of
different GPU types at once — say a handful of powerful A100s,
a larger set of mid-tier L4s, and a pile of older, slower T4s —
often physically spread across regions. Two kinds of
heterogeneity matter. COMPUTE heterogeneity: the GPUs differ in
memory capacity and in how many tokens per second they can push
through a model layer, by large factors. NETWORK heterogeneity:
the links between them differ enormously, from fast interconnect
within a single datacenter to slow, high-latency connections
(as low as 100 Mbps) between regions. A model too big for any
one GPU must be split across several, so its layers form a
pipeline that spans both fast and slow GPUs and both fast and
slow links — and every request must traverse that pipeline. The
challenge is to use this ragtag pool as if it were a coherent
machine
The reason naive approaches fail is that they optimize a proxy
instead of the real target. Splitting the model to balance
compute feels right, but throughput on a heterogeneous cluster
is not set by compute alone; it is set by whichever resource
saturates first along the path a request takes, and that is
often a slow network link between two pipeline stages, not a
GPU. So the two decisions a serving system makes are not
separable. The
the fact that
model PLACEMENT (which contiguous block of layers each GPU
holds) and request SCHEDULING (which sequence of GPUs a given
request flows through) determine each other. Placement fixes the
graph of legal routes: a request can only go from GPU A to GPU B
if B holds the layers that come right after A’s, and the cost of
that hop is the bandwidth of the A-to-B link divided by the size
of the data that must cross it. Scheduling then decides how much
traffic actually flows along each legal route. A placement is
only as good as the routing it enables, and a routing is only as
good as the placement it runs on — so choosing one without the
other, or choosing both by a local heuristic, systematically
leaves throughput unclaimed
What people did before, and where it broke
- Homogeneous-cluster serving (vLLM, Orca). Assume every GPU is identical and partition the model uniformly. On a real mixed pool this underuses the fast GPUs (they wait on the slow ones) and there is no notion of network cost between stages at all. Excellent within their assumption, inapplicable outside it.
- Decentralized volunteer serving (Petals). Run inference across volunteer GPUs over the open internet. But it lacks the GLOBAL coordination needed to use GPU and network resources efficiently — each node acts locally, so the cluster never reaches a coherent optimum.
- FLOP-balancing heterogeneous systems (Swarm). Borrowed from heterogeneous training, give each GPU layers proportional to its compute. Optimizes a proxy (compute balance) that ignores the network, so consecutive stages land across slow links and those links become the bottleneck — Helix beats it by up to around 2.4x on throughput, and Swarm’s prompt latency blows up to 71 seconds (7.5x worse) under geo-distributed congestion.
- Separate fixed pipelines. Carve the cluster into a few fixed homogeneous-ish pipelines and route requests to whole pipelines. Simpler, but rigid — it cannot mix a fast and slow GPU within one request’s path, so it wastes the flexibility heterogeneity demands; Helix beats it by up to 2.24x on the most heterogeneous cluster.
- The gap. No prior system modeled BOTH compute and network heterogeneity in one objective, JOINTLY optimized model placement and request routing against the true end-to-end throughput, and then executed that plan smoothly at runtime. Helix does all three by casting the problem as max-flow.
How Helix works
Model the cluster as a flow graph, solve for placement and routing, then execute
-
Turn the cluster into a directed graph new
Every GPU instance becomes a node (split into an in-vertex and an out-vertex so the GPU's own throughput can be a capacity on the edge between them), plus a global SOURCE and SINK. A directed edge from GPU A to GPU B exists when B can hold the model layers that immediately follow A's. The graph encodes the whole heterogeneous cluster as one object.
-
Set capacities to capture BOTH bottlenecks new
Each GPU's internal edge gets capacity = min(its compute throughput, its network throughput) — whichever limits it. Each GPU-to-GPU edge gets capacity = link bandwidth divided by the size of the data crossing it (about 16 KB of activations per token between stages, about 4 bytes for a token id from the source). Now one number, capacity, speaks for compute and network heterogeneity at once.
-
Read serving throughput as max-flow new
The maximum flow from source to sink through this graph IS the maximum serving throughput of the cluster under a given placement — one unit of flow equals one token per second. This is the key reframing: a messy systems question ("how fast can this mixed cluster serve?") becomes a classic, solvable graph problem whose answer is exact.
-
Jointly optimize placement and routing (MILP) new
Because the model placement decides which edges exist, Helix does not take the graph as given — it solves a mixed-integer linear program whose variables are the layer assignment (which contiguous stage each GPU holds) AND the flow on every edge, maximizing total flow out of the source subject to flow-conservation, per-node compute limits, and link capacities. It must be integer, not a relaxed LP, because rounding a fractional placement would invalidate network connections and drastically change the max-flow.
-
Keep the MILP tractable new
The program has only a linear number of variables and constraints in cluster size (a 42-node cluster is 2,144 variables, 2,772 constraints), and three tricks keep it fast: prune slow network links so average node degree stays at or below 12 (shrinks the problem 36-46%), seed the solver with heuristic solutions (saves 8-43% of solve time), and early-stop when the solution nears the provable upper bound. Small clusters solve to optimality in under 5 minutes.
-
Execute with per-request pipelines and IWRR new
At runtime each request gets its OWN pipeline — any source-to-sink path — rather than a fixed one. An Interleaved Weighted Round-Robin scheduler at each node, weighted by that node's outgoing flow in the max-flow solution, picks the next hop; chaining these choices routes requests to match the optimal flow WITHOUT creating bursts. A KV-cache estimator masks out nodes near their memory high-water mark so no GPU runs out of memory.
The idea
Find the known problem hiding inside the messy one. The whole contribution turns on a reframing: the question “how fast can this heterogeneous cluster serve a model, and how should I place and route to make it fastest?” is, when you build the right graph, exactly the MAX-FLOW problem — a textbook optimization with exact solutions. Nodes are GPUs, edge capacities fold compute and network limits into one number, and the max source-to-sink flow is the serving throughput in tokens per second. Once the problem wears that shape, decades of optimization machinery apply and you can compute the true optimum instead of hoping a heuristic lands near it. This is the same map-your-problem-onto-a-solved-one discipline the site’s scheduling work keeps rewarding — the GPU-microservice page places pipeline stages with a graph MIN-CUT; Helix serves them with a graph MAX-FLOW — and the leverage comes from recognizing the classic problem underneath the systems noise.
Optimize the true objective, not a convenient proxy — and optimize the coupled decisions together. Swarm balances FLOPs because compute is the obvious thing to balance, but end-to-end throughput depends on the network too, so a compute-balanced plan bottlenecks on a slow link and the fast GPUs idle. Helix’s objective IS end-to-end throughput (the max-flow value), which by construction accounts for both compute and network, so it never trades an invisible network cost for a visible compute balance. And because placement and routing are entangled — placement defines the legal routes, routing decides their use — Helix refuses to separate them, folding both into one MILP. It is the recurring measure-what-actually-costs lesson applied to a two-variable problem: pick the objective that reflects reality, and optimize all the variables that reality couples.
Solve the hard problem once, offline; execute cheaply, online. A mixed-integer program is expensive, so Helix runs it ONCE to decide placement and the target flow, then makes runtime routing trivial: each request gets its own path, chosen hop by hop by an interleaved weighted round-robin whose weights are the max-flow edge values. IWRR realizes the optimal flow as a smooth stream of individual routing decisions, without the bursts a naive “send everything down the best path” would cause. The expensive global reasoning is amortized across all the requests that follow, and the per-request cost is a few weighted coin flips. It is the plan-expensively-then-execute-faithfully split the site sees whenever a system moves heavy optimization off the hot path and leaves a light mechanism to carry the plan out.
One heterogeneous cluster, placed and routed
Why balancing compute fails, and what max-flow does instead
A cluster of 4 A100s, 8 L4s, and 12 T4s serving LLaMA-2 70B, some nodes across a slow link. Numbers from Helix's evaluation.
Step 1 of 5
The model does not fit on one GPU
LLaMA-2 70B must be split into a pipeline of layer-stages spread across many GPUs. The cluster has a few fast A100s, more mid-tier L4s, and a lot of slow T4s, and some of them sit across a slow inter-region link. Every request has to flow through a full stack of layers, hopping GPU to GPU, so both the GPUs it lands on and the links it crosses shape its speed.
LLaMA-2 70B needs ~7 A100s, or ~30 L4s, or ~12 H100s — a mixed pool must combine types intra-region links are fast; the inter-region link can be as slow as 100 Mbps
Why it works
- Serving throughput on a heterogeneous cluster is set by the first resource to saturate along a request’s path → modeling the cluster as a graph whose edge capacities fold in BOTH compute and network → the max-flow value is the exact throughput, so the optimizer sees network bottlenecks a FLOP-balancer cannot.
- Model placement determines which routes exist and routing determines their use → folding both into one mixed-integer program that maximizes flow → placement and routing are chosen consistent with each other, not by two separate heuristics.
- Rounding a fractional layer placement would invalidate the network connections it implies → keeping the program integer (MILP, not a relaxed LP) → the solution’s placements are always legal and its predicted max-flow is real.
- The optimization is expensive but one-time → solve it offline for placement and target flow, then route online with cheap weighted round-robin → the heavy reasoning is amortized over every request and the per-request cost stays tiny.
- Sending all traffic down the single best path would burst and congest → interleaved weighted round-robin spreads each request along paths in proportion to the max-flow weights → the optimal flow is realized as a smooth stream, beating Swarm-style and random routing by 22% and 15%.
What it costs
- The MILP does not scale forever. A linear variable count keeps small and mid clusters fast (under 5 minutes to optimality), but beyond about 42 nodes the solver can take hours without the pruning and seeding tricks, and scaling to thousands of nodes needs heuristically partitioning the cluster into smaller sub-clusters — giving up global optimality.
- Placement is restricted to contiguous stages. Each node holds one continuous block of layers; allowing partial or interleaved inference (a node serving only some layers for some requests) would add linearization complexity the current formulation avoids.
- It assumes a stable, profiled network. Capacities come from profiling bandwidths ahead of time; Helix does not model dynamic congestion, link failures, or bandwidth that shifts during serving, so a volatile network could invalidate its plan.
- KV-cache is estimated, not known. Masking uses the AVERAGE output length to guess memory use, but the true length is unknown until a request finishes, so a node can be over- or under-provisioned, and decode throughput is ultimately capped by KV-cache VRAM with best-effort batching that may miss the optimal batch size.
- Narrow evaluation surface. Tested on LLaMA 30B and 70B with single-GPU nodes; multi-GPU nodes (needing tensor parallelism) and non-LLaMA architectures are future work, and the simulator, though within 5% of measured, is not the real system.
The evaluation
Clusters of 24 to 42 nodes spanning up to 7 GPU types, in single and geo-distributed (3-region, 100 Mbps inter-region) settings, serving LLaMA-1 30B and LLaMA-2 70B, against Swarm (FLOP- balancing), Separate Pipelines, and decentralized baselines, measuring offline and online throughput and prompt/decode latency.
Three ways to serve an LLM on a mixed GPU cluster
| Property | Homogeneous-assuming (vLLM / Orca) | FLOP-balancing heuristic (Swarm) | Helix (max-flow + MILP) |
|---|---|---|---|
| Handles mixed GPU types? | |||
| Accounts for network heterogeneity? | No notion of inter-stage link cost | ||
| Placement and scheduling | |||
| What it optimizes | Nothing global (assumption-bound) | ||
| Throughput / latency (heterogeneous) | Not applicable to mixed pools |
Check yourself
Check your understanding
1. Helix models the whole heterogeneous cluster as a graph and computes its MAX-FLOW. Predict why this reframing is the crux of the paper, rather than just a tidy way to describe the system.
2. Swarm balances the FLOPs — giving each GPU a share of layers proportional to its compute — and Helix beats it by up to ~2.4x, with Swarm's prompt latency sometimes blowing up to 71 seconds. Predict what Swarm is getting wrong.
3. Helix insists on solving a mixed-integer program (MILP) rather than relaxing it to a continuous linear program and rounding the result. Predict why integrality is essential here.
4. Helix solves an expensive MILP once, then routes each request at runtime with a cheap interleaved weighted round-robin (IWRR) weighted to the max-flow solution. Predict why this split — and IWRR specifically rather than "always send to the best path" — is the right runtime design.
Sources
- arXiv:2406.01566 — Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow — ASPLOS 2025 (full text read via ar5iv).
- ACM DL: Helix (ASPLOS 2025) — published version.
- Helix source code (Thesys-lab) — open-source implementation.