Modern AI services (image captioning, text-to-speech, cloud gaming) are pipelines of GPU stages wired together as microservices, but the microservice playbook built for CPUs breaks on GPUs — where passing data between two stages, even on the SAME chip, means copying it out to host memory and back (up to 45% of a request's time versus ~5% on CPUs), GPU memory is a hard capacity wall, and there is no hardware isolation so co-located stages silently fight over memory bandwidth; Astraea (Shanghai Jiao Tong) treats communication as a first-class scheduling factor, predicts each stage's contention, packs stages to cut cross-chip transfers, and lets same-GPU stages share data by passing a pointer instead of copying — raising the peak load a service can sustain within its latency target by up to 82.3%.
The argument in one glance
-
Problem
AI services are GPU pipelines, and pipelines have strict deadlines
A modern user-facing AI service is rarely one model — it is a chain of them. Image captioning is a convolutional encoder feeding an LSTM decoder; text-to-speech is a text model feeding a waveform synthesizer; cloud gaming chains game logic, rendering, and video encoding. Each stage runs on a GPU, and the stages are wired together as loosely-coupled MICROSERVICES. Because one stage often under-uses a whole GPU, several are packed onto shared GPUs at low load. But the whole service owes users a tail-latency target (99th-percentile within hundreds of milliseconds), and in a pipeline the stages are coupled by back-pressure: one slow or contended stage blows the deadline for the entire request.
-
What people did
Reuse the microservice playbook — and the single-GPU QoS work
Two bodies of prior work seem to apply. CPU microservice managers (FIRM, Sinan) predict each service's resource needs and reallocate cores, cache, and memory bandwidth to hold QoS, deploying services onto machines fairly freely. And single-application GPU QoS systems (Laius, Baymax) carefully allocate GPU compute units (SMs) so a latency-critical job survives co-location. Combine them, the thinking goes, and you can manage GPU microservices.
-
Why it fell short
Three GPU realities the CPU playbook and single-GPU tuning ignore
GPU microservices break both approaches on three counts. COMMUNICATION: passing data between two GPU stages costs up to 45% of end-to-end latency (versus ~5% for CPU microservices) — and even two stages on the SAME GPU must copy data out to host memory and back, because separate processes cannot share GPU memory directly. MEMORY: GPU global memory is a hard capacity wall (a stage with batch size over 256 can exhaust a whole GPU), where CPU scheduling treats RAM as effectively unlimited. ISOLATION: GPUs offer no cache or bandwidth partitioning, so even after you carefully allocate SMs, co-located stages silently contend for memory bandwidth and PCIe — contention offline profiling cannot see, causing QoS violations in 14 of 21 test cases. Random placement (CPU-style) and per-stage tuning (single-GPU-style) both fail.
-
This paper's idea
Make communication and contention first-class, and pass pointers not copies
Astraea (Shanghai Jiao Tong University) is a runtime that maximizes the load a multi-stage GPU service sustains within its tail-latency target. Three pieces. An online PERFORMANCE PREDICTOR (gradient-boosted trees, <1ms) forecasts each stage's duration, throughput, AND its global-memory-bandwidth usage under any SM allocation — the bandwidth prediction is what lets it reason about contention no hardware isolates. A DEPLOYMENT POLICY treats communication as a first-class factor: it solves a constrained optimization for SM quotas and instance counts to balance pipeline-stage throughputs, packs stages to minimize cross-node then cross-GPU transfers (a graph min-cut), and respects the memory-capacity wall. An AUTO-SCALING COMMUNICATION FRAMEWORK re-binds the fastest channel (global-memory pointer, NVLink, or RPC) whenever deployment changes, with no code change — and for same-GPU stages it passes an 8-byte memory HANDLE instead of copying data through the host, eliminating the transfer entirely. Result: up to 82.3% more sustainable peak load at the required QoS.
Why this problem is worth solving
The AI services people use every second are pipelines, and
pipelines are only as fast as their tightest deadline. When
a service is decomposed into
a multi-stage GPU application
built as loosely-coupled services — each stage (an encoder,
a decoder, a renderer) is an independent microservice that
communicates with the next by passing data. The appeal is the
usual microservice appeal: independent scaling, independent
deployment, packing several stages onto a shared GPU when
each alone under-uses it. The cost is that stages are coupled
by back-pressure — a downstream stage that stalls forces
upstream stages to wait — so the service’s tail latency is
governed by whichever stage is currently the bottleneck,
which shifts with load and contention
The trap is that this looks like a solved problem. CPU
microservice management is mature, and single-application GPU
QoS is well studied. But GPUs violate three assumptions those
solutions rest on, and the deepest is
the surprising fact that two
microservice stages on the SAME physical GPU cannot pass data
to each other directly. They run in separate OS processes, and
a GPU memory region owned by one process is not accessible to
another — so the data must be copied out of the first stage’s
GPU memory to host (CPU) main memory, then copied back into
the second stage’s GPU memory, a round trip over the PCIe bus.
For CPU microservices, communication is ~5% of latency; for
GPU microservices this copying is up to 45%, and it also burns
PCIe bandwidth that other co-located stages need
unlike CPUs — which
offer cache partitioning and memory-bandwidth controls (Intel
CAT, MBA) that let a scheduler carve out guaranteed slices —
commodity GPUs let you explicitly allocate only compute units
(streaming multiprocessors, SMs). Everything else — global
memory bandwidth, PCIe bandwidth — is shared with no
partitioning. So even after a scheduler assigns each stage its
SMs, co-located stages silently fight over bandwidth, and that
contention is invisible to the offline profiling that assumed
each stage runs alone
What people did before, and where it broke
- CPU microservice managers (FIRM, Sinan). Predict resource needs and reallocate cores/cache/bandwidth, then deploy services fairly freely (Kubernetes-style). Two failures on GPUs: they treat communication as negligible (~5% on CPUs) so they place stages randomly, incurring the 45% GPU transfer tax; and they lean on CPU isolation hardware (cache allocation) that GPUs simply do not have.
- Single-application GPU QoS (Laius, Baymax). Carefully allocate SMs so one latency-critical job survives co-location with best-effort work. Three failures: no way to split a whole-pipeline QoS target across stages; protecting one stage can starve co-running stages; and no notion of deploying a pipeline across multiple GPUs or nodes.
- Queuing/kernel-reordering GPU schedulers (GrandSLAm, Salus). Reorder or time-share GPU kernels for QoS — but built for time-sharing GPUs, not the modern spatial-sharing GPUs where kernels run concurrently on partitioned SMs and contend for bandwidth.
- Single-model batching systems (BatchMaker, Nexus). Optimize one RNN’s batching, or split an SLO across a single DNN’s stages — but ignore the general dependency graph and communication structure of a multi-microservice pipeline.
- The gap. No system treated inter-stage communication as a first-class scheduling factor, reasoned about the bandwidth contention GPUs do not isolate, respected the global-memory capacity wall, AND deployed a pipeline across GPUs and nodes to hold a whole-service tail-latency target. Astraea does all four.
How Astraea works
Predict, allocate, place, and re-bind communication
-
Online performance predictor new
For each microservice, a gradient-boosted decision tree predicts — from input size, batch size, and SM percentage — its processing duration, throughput, and (crucially) its global-memory-BANDWIDTH usage. The bandwidth prediction is the key novelty: since GPUs cannot isolate bandwidth, Astraea predicts each stage's bandwidth appetite so it can reason about contention. Trained online in under an hour, <1ms per prediction (GBDT chosen over random forest for the same accuracy at 5× lower latency).
-
Contention-aware SM allocation new
A constrained optimization sets each stage's SM quota and instance count to MAXIMIZE the minimum stage throughput (a pipeline is only as fast as its slowest stage), subject to five constraints: summed bandwidth demand ≤ GPU bandwidth (the anti-contention constraint), SM quotas ≤ available SMs, ≤48 MPS clients, memory footprint ≤ capacity, and critical-path latency ≤ QoS. Solved by a fast randomized-greedy (epsilon-greedy) search in ~10ms — the space is astronomically large (10⁴⁰ combinations), so exhaustive search is impossible.
-
Balance stages: scale-out plus SM tuning new
Some stages do not speed up with more SMs; some GPUs have few free SMs. So Astraea balances stage throughputs with two levers together — allocating SM percentage AND launching multiple instances of a slow stage — rather than SM tuning alone. The critical path is found ONLINE (weighted-longest-path) because contention shifts which stage is the bottleneck.
-
Topology-aware placement (cross-node then cross-GPU) new
Minimize the most expensive communication first. Cross-node placement is a graph MIN-CUT (a Balanced-Kmeans++ variant): stages are nodes, inter-stage data volume is edge weight, and the graph is partitioned across nodes to minimize cut weight while respecting per-node compute. Within a node, adaptive pruning co-locates high-traffic adjacent stages on the same or topologically-close GPUs, and packs onto GPUs with fewer free resources to avoid fragmentation.
-
Global-memory pointer-passing (the copy killer) new
For two stages on the same GPU, instead of copying data out to host memory and back, the producer stores its result in global memory and passes only an 8-byte memory HANDLE (via CUDA IPC) to the consumer, which reads the data directly. The whole PCIe round trip vanishes. A producer-consumer protocol with semaphores handles multiple instances; it also extends to cross-GPU transfers over NVLink (GPUDirect).
-
Auto-scaling communication framework new
A unified API (set stage id, append successors, publish/receive) lets stages communicate without knowing where their neighbors are. A central controller, told each instance's location after deployment, binds the fastest channel — global-memory handle (same GPU), NVLink (same node), or RPC (cross-node) — and RE-BINDS automatically when deployment changes at runtime, no source-code change. Like gRPC for CPUs, but topology-adaptive for GPUs.
The idea
On GPUs, communication is not overhead to amortize — it is the axis to schedule around. The CPU microservice mindset treats data movement as a small tax (5%) and so places services by resource fit, letting a generic RPC framework handle the wire. Astraea’s first move is to reject that: at 45% of latency, and with a same-GPU copy tax that most schedulers do not even know exists, communication cost DOMINATES the deployment decision. So placement is driven by the communication graph — minimize cross-node transfers (min-cut), then co-locate chatty adjacent stages — before resource fit is considered. This is the same “measure what actually costs, then optimize THAT” discipline the site keeps finding: the bottleneck migrated from compute to communication when the workload moved to GPUs, and the scheduler has to migrate with it.
Predict the resource you cannot isolate, so you can plan around it. The subtle novelty is the bandwidth predictor. Because GPUs let you allocate SMs but nothing else, a scheduler that allocates only SMs is blind to the memory- bandwidth contention that actually causes the QoS violations (14 of 21 cases). Astraea’s answer is not to isolate bandwidth (impossible on commodity GPUs) but to PREDICT each stage’s bandwidth appetite and add a constraint — summed demand must not exceed the GPU’s bandwidth — so the schedule never packs stages whose combined thirst would starve the pipeline. It converts an uncontrollable shared resource into a planned-for one by modeling it, the same move as any system that cannot partition a resource but can forecast its use and provision against the forecast.
Kill the copy by passing a name instead of the thing. The same-GPU copy tax exists only because two processes cannot share GPU memory — but they CAN share a pointer to it. Astraea stores the producer’s result in global memory and hands the consumer an 8-byte handle, so the data never moves and the PCIe round trip disappears. This is the address-indirection trick the site has now seen at several layers — pass a reference, let the consumer dereference, and the payload stays put — here dissolving a communication cost the way the on-NIC-memory and virtual-memory-stitching pages dissolve data movement elsewhere. It is a pure win above a tiny fixed IPC setup cost, which is why Astraea uses it for any transfer over ~0.02 MB (the batch-serving regime).
An image-captioning pipeline under rising load
Encoder feeds decoder, and the load climbs
Image-to-text: a VGG feature-extractor stage feeds an LSTM caption stage. Numbers from Astraea's evaluation across the seven real services.
Step 1 of 5
Low load: both stages on one GPU
At light load, both stages under-use a GPU, so Astraea packs them together and scales out whole copies of the two-stage pipeline across GPUs to serve more independent requests. Its predictor has forecast each stage's duration, throughput, and bandwidth appetite; the optimizer set SM quotas so the two stages have matched throughput (no bottleneck), and confirmed their summed bandwidth demand fits the GPU.
a pipeline runs fastest when stages have equal throughput — Astraea maximizes the MINIMUM stage throughput global-memory capacity is the top-priority resource: it packs onto GPUs with fewer free resources to avoid fragmentation
Why it works
- Inter-stage communication is up to 45% of GPU-pipeline latency → making it the first-class placement factor (min-cut across nodes, co-locate chatty stages) → the dominant cost is minimized before resource fit is even considered.
- Two same-GPU processes cannot share memory but can share a pointer → passing an 8-byte handle instead of copying through host RAM → the PCIe round trip and its bandwidth contention both vanish for any non-tiny tensor.
- GPUs isolate SMs but not memory bandwidth → predicting each stage’s bandwidth appetite and constraining summed demand ≤ GPU bandwidth → the schedule plans around the contention it cannot partition, killing the majority of QoS violations.
- A pipeline’s throughput is its slowest stage’s, and stages scale differently → maximizing the minimum stage throughput via SM quotas AND instance counts, with the critical path found online → stages stay balanced as load and contention shift.
- Optimal deployment changes with load, topology, and contention → an auto-scaling framework re-binds the fastest channel (handle / NVLink / RPC) per deployment with no code change → the communication method always matches the current placement.
What it costs
- Per-microservice online profiling. The predictor needs training data collected per stage (under an hour total, but not zero); a genuinely new microservice must be profiled before it can be scheduled well.
- Prediction-quality dependence. SM allocation and the bandwidth constraint rest on the GBDT’s accuracy (~95%); the paper reports no violations from misprediction, but a stage whose behavior the model captures poorly could break the plan (a compensation mechanism is noted as fallback).
- Pointer-passing has a floor and preconditions. CUDA IPC has a small fixed setup cost, so for tiny tensors (bytes) host-copy is faster; and cross-GPU handle-passing needs NVLink/GPUDirect, absent on commodity multi-GPU boxes that route through PCIe.
- Heuristic, not optimal. The 10⁴⁰-combination allocation space is searched by randomized greedy and the placement by Balanced-Kmeans++ — fast and good, but no optimality guarantee; adversarial pipelines could expose the gap.
- Co-location trade-offs remain. Weight-sharing cuts memory when instances of a stage share a GPU, but same-stage instances also contend for the same resources — Astraea navigates this trade-off but does not eliminate it, and the best answer is workload-dependent.
The evaluation
Seven real multi-stage GPU services plus 45 synthetic pipelines (three-stage and complex-graph, built from compute-/memory-/PCIe-intensive Rodinia kernels), on a 2-GPU 2080Ti server and three DGX-2 (V100) machines, against FIRM and Laius, measuring peak sustainable load within the 99%-ile latency target.
Three ways to manage a multi-stage GPU service
| Property | CPU-microservice manager (FIRM) | Single-GPU QoS (Laius) | Astraea |
|---|---|---|---|
| Treatment of communication | Not addressed (single-application focus) | ||
| Same-GPU inter-stage transfer | Host-memory round trip (PCIe) | ||
| Unisolated bandwidth contention | Assumes CPU cache/bandwidth isolation GPUs lack | ||
| Pipeline & multi-GPU awareness | Considers multiple instances but not the pipeline critical path | No pipeline QoS split; single GPU only | |
| Peak sustainable load at QoS |
Check yourself
Check your understanding
1. CPU microservice managers like FIRM place services fairly freely and let a generic RPC framework handle communication. Predict why this approach fails badly on GPU microservices, even after adapting resource allocation.
2. Astraea's performance predictor forecasts each stage's global-memory-BANDWIDTH usage, not just its duration and throughput. Predict why this specific prediction is the load-bearing novelty.
3. Astraea's copy-elimination trick: for two stages on the same GPU, pass an 8-byte memory handle instead of copying the data. Predict why the copy existed in the first place, and why passing a handle is safe and cheap.
4. A pipeline's throughput equals its slowest stage's throughput, so Astraea maximizes the MINIMUM stage throughput — but it uses two levers (SM percentage AND number of instances) rather than just allocating more SMs to slow stages. Predict why both levers are needed.