Aqua: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains

When a wave of requests hits a cloud-hosted language model, the GPU runs out of memory and newcomers get stuck in a queue, so the service stops responding — and the obvious fix, giving each request a quick turn, is ruined by the slow cost of shuffling each request's working data off the GPU and back; this work from Cornell builds Aqua, which parks that data on a neighboring GPU's spare memory over the fast NVLink cable instead of the slow link to main memory, making turn-taking cheap enough to keep the model 20 times more responsive under bursts while preserving throughput.

The argument in one glance

Problem

Bursts of requests exhaust GPU memory and make the model unresponsive

Serving a large language model is limited by GPU memory: each in-flight prompt needs memory for its working state, and a surge of requests exhausts it. Serving engines batch some prompts and queue the rest, but queueing causes head-of-line blocking, so during a burst the model stops responding — and even a few seconds of unresponsiveness drives users away.
What people did

Give every prompt a quick turn (preemptive scheduling)

Preemptively schedule prompts in time slices so each gets a share of GPU compute and the model stays responsive, pausing one prompt to run another and paging out the paused prompt's state to make room.
Why it fell short

Paging the state in and out is too slow

Preemption requires paging a prompt's working state out of GPU memory at each turn boundary and back later. Paging it to the host's main memory and back is bottlenecked by the slow link between CPU and GPU, so the overhead is high — improving responsiveness but cutting throughput by more than half. There is no way to get both.
This paper's idea

Page to a neighbor GPU over the fast interconnect (Aqua)

Aqua observes that at any moment some GPUs have spare memory while others are starved. It profiles models as memory producers or consumers, places producers alongside consumers in the same fast-interconnect domain, and pages a consumer's state to a producer GPU's spare memory over high-bandwidth NVLink instead of host memory over the slow link. Cheap paging makes preemptive fair scheduling practical, improving responsiveness by 20 times and single-long-prompt throughput by 4 times.

Why this problem is worth solving

Language-model serving lives and dies by GPU memory, and demand for it is spiky.

When a large language model answers a prompt, it does not just need the model’s weights in GPU memory — it needs additional memory for the prompt’s working state, chiefly the key-value cache: the attention information the model has computed for every token generated so far, which grows as the response gets longer. GPU memory is limited and expensive, so the number of prompts a server can handle at once is capped by how much of this working state fits. That is fine under steady load, but inference demand is bursty: a sudden increase in requests to a cloud-hosted model can exhaust GPU memory, and the prompts then contend for the little that is left. Modern serving engines handle this with admission control — they schedule a batch of prompts and queue the rest until memory frees up. The trouble is that batching with a queue causes head-of-line blocking: prompts waiting in the queue are stuck behind the batch being processed, so during a surge the model becomes unresponsive, and the time-to-first-token — how quickly the model starts replying — spikes. Since even a few seconds of unresponsiveness causes users to abandon a service, and autoscaling to more GPUs takes minutes to spin up, this unresponsiveness is a serious problem.

The obvious fix, letting prompts take turns, runs headlong into the cost of moving their state.

A natural way to stay responsive is preemptive scheduling in time slices: give each prompt a slice of GPU compute, then pause it and run the next, so every prompt makes steady progress and the model responds to all of them rather than making some wait in a queue. This also enables fine-grained fair scheduling, which today’s engines only approximate. But preemption has a hidden cost. When the scheduler pauses a prompt at a time-slice boundary to run another, it must page the paused prompt’s working state out of GPU memory to free space, and page it back in when the prompt resumes. The speed of that paging is bottlenecked by the link the data travels over. If the state is paged to the host’s main memory, it crosses the PCIe bus — the comparatively slow connection between CPU and GPU — in both directions. This overhead is large: when the authors implemented a preemptive fair scheduler in a popular serving engine, responsiveness improved, but the time to finish a prompt more than doubled, because paging over PCIe dominated. So naive preemption trades throughput for responsiveness, and you cannot have both.

What people did before, and where it broke

Batch prompts with admission control. Process a batch and queue the remaining prompts until GPU memory frees. Where it broke. Queueing causes head-of-line blocking, so during request bursts the model becomes unresponsive and time-to-first-token spikes.
Autoscale to more GPUs during a burst. Spin up additional GPU instances of the model to absorb the load. Where it broke. Scaling out takes minutes due to virtual-machine startup, during which the model stays unresponsive — long enough to lose users.
Preempt in time slices, paging state to host memory. Let prompts take turns, paging paused state to the host’s main memory. Where it broke. Paging over the slow CPU-to-GPU link is so costly that throughput more than halves, so responsiveness is bought at the price of throughput.
The gap. No approach made preemption cheap enough to deliver both responsiveness and throughput under memory pressure. Doing so by paging to spare GPU memory over a fast interconnect is what Aqua contributes.

How Aqua works

From queueing under memory pressure to fast paging over the GPU interconnect

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Harvest the spare memory that some GPUs have while others starve. Aqua’s founding observation is that GPU memory pressure is uneven: because different models and workloads are bottlenecked on different resources — some on compute, some on memory bandwidth, some on capacity — at any moment some inference jobs underutilize their GPU memory while others are starved for it. Aqua decouples memory allocation from compute allocation, profiling models as memory producers (with spare capacity to offer) or consumers (needing more), so one job’s idle memory can serve another’s shortage. Turning the capacity that one tenant is not using into a resource for another that needs it, instead of letting it sit idle, is the same harvesting the idle-flash-bandwidth storage work covered elsewhere on this site performs, reclaiming a resource going unused in one place to relieve a bottleneck in another.

Page over the fast interconnect, not the slow one. The reason preemption was too expensive was the path the paged state traveled — the slow CPU-to-GPU link to host memory. Aqua instead pages a consumer’s state to a producer GPU’s memory over the high-bandwidth NVLink interconnect that joins GPUs in a scale-up domain (a multi-GPU server, or now whole racks), which is far faster than host memory over that slow link, so preemption overhead collapses. Treating a peer’s memory reached over a fast interconnect as fast, near memory — rather than routing through the slow, distant path — is the same shift the disaggregated-memory work covered elsewhere on this site is built on, decoupling memory from compute and reaching remote memory over a fast coherent interconnect so it is cheap enough to use as if it were local.

Make preemption cheap so fair scheduling becomes practical. With fast paging in hand, Aqua builds a preemptive completely-fair scheduler for prompts, with a new algorithm that partitions each batch among prompts in different inference phases to keep the throughput benefit of batching while sharing compute fairly — something that was hopeless when every preemption paid a huge paging bill. Making an operation that was previously too costly to do often cheap enough to do routinely, so a scheduling policy that needs frequent switching finally becomes affordable, is the same enabling move the low-overhead-preemption work covered elsewhere on this site makes, driving down the cost of pausing and resuming work so that fine-grained, responsive scheduling stops being prohibitively expensive.

Following a request burst through Aqua

Keeping a model responsive when a wave of prompts arrives

A trace of Aqua handling a request burst, following the paper's preemptive scheduling over NVLink.

Step 1 of 5

A burst threatens to exhaust GPU memory

A cloud-hosted model is serving steadily when a burst of new prompts arrives, threatening to exhaust the GPU's memory and stall responses to everyone.

Under admission control, the newcomers would be queued and left waiting.

Why it works

Responsiveness and throughput conflict under memory pressure. Because batching maximizes throughput but queues prompts, while fair time-slicing keeps prompts responsive but must page state → the two goals pull against each other when GPU memory is scarce → so an engine is forced to sacrifice one until paging is made cheap.
Paging cost is set by the interconnect. Because moving working state out of GPU memory is bottlenecked by the link it travels over → paging to host memory over the slow CPU-to-GPU bus dominates the preemption overhead → so where the state is paged decides whether preemption is affordable.
The cluster’s memory is imbalanced. Because different models and loads leave some GPUs with spare memory and others starved → one job’s spare memory can serve another job that needs it → so decoupling memory from compute turns idle capacity into a usable resource.
A peer GPU over NVLink is a fast paging target. Because GPUs in a scale-up domain are joined by high-bandwidth NVLink, far faster than host memory over the slow bus → paging a consumer’s state to a producer’s spare memory is quick → so the preemption overhead shrinks dramatically.
Cheap preemption unlocks fair scheduling. Because the paging that preemption requires is now cheap → a preemptive completely-fair scheduler becomes practical without wrecking throughput → so the model stays responsive under bursts and absorbs load until autoscaling can catch up.

What it costs (trade-offs and scope)

It needs spare memory to exist and be nearby. Aqua works by borrowing a producer GPU’s spare memory; if every GPU in the domain is memory-saturated at once, there is no spare to page to, and the benefit shrinks. Its gains depend on the workload mix presenting an imbalance.
It is confined to a scale-up domain. The fast paging relies on high-bandwidth NVLink connecting GPUs in a scale-up domain — a multi-GPU server, or a rack in the newest architectures. Offloading beyond that domain would fall back to slower links, so the reach of the technique is bounded by how far the fast interconnect extends.
Borrowed memory is not free for the lender. Memory offloaded onto a producer GPU consumes capacity there, so the pairing must ensure the producer genuinely has room to spare; a producer whose own load rises could become a consumer, requiring the profiling and placement to adapt.
Placement adds a scheduling problem. Aqua must profile models and solve a placement that pairs producers with consumers in the same domain. This is an added optimization layer whose quality affects results, and mispredicted producer or consumer labels reduce its effectiveness.
The gains are workload- and metric-dependent. The headline 20 times responsiveness and 4 times single-long-prompt throughput come from specific models and workloads on 8 H100 GPUs; other model mixes, request patterns, and hardware would shift the numbers.
It complements, not replaces, autoscaling. Fair scheduling absorbs temporary bursts or holds sustained bursts until autoscaling adds instances; it does not remove the need to eventually add capacity when demand is sustained.

The evaluation

Three ways to serve LLM prompts under memory pressure

Property	Batch and queue	Preempt, page to host memory	Aqua
Responsive during bursts
High throughput
Where paused state is paged	Not paged, queued
Uses spare cluster memory	No
Result	Unresponsive bursts	Throughput drops over 2 times	20 times responsiveness, 4 times long-prompt throughput

Three ways to serve LLM prompts when GPU memory is under pressure, as evaluated in the paper. Batching with a queue (admission control) gives high throughput but becomes unresponsive during bursts because queued prompts are blocked behind the batch. Preemptive time-slicing keeps prompts responsive, but paging paused state to host memory over the slow CPU-to-GPU link more than doubles the time to finish a prompt. Aqua keeps preemption but pages a memory consumer's state to a memory producer GPU's spare capacity over high-bandwidth NVLink, chosen by profiling models as producers or consumers and placing them in the same interconnect domain. Because NVLink paging is fast, a preemptive completely-fair scheduler achieves both responsiveness and throughput. Evaluated on a server with 8 Nvidia H100 80G GPUs across text, audio, and vision models, Aqua improves responsiveness (time-to-first-token) by 20 times over the state of the art and improves throughput on a single long prompt by 4 times. The gains depend on there being spare memory to borrow within the fast-interconnect domain.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

Aqua is a well-targeted systems contribution to a real and growing pain in LLM serving. Its diagnosis is precise: GPU memory limits how many prompts a server can hold, bursts exhaust it, and the standard batch-and-queue response makes the model unresponsive exactly when users are most impatient. The natural cure — letting prompts take turns via preemptive scheduling — was undermined by the cost of paging each paused prompt’s working state over the slow link to host memory, which more than halved throughput. Aqua’s insight resolves the tension elegantly: because memory pressure across a GPU cluster is uneven, spare memory usually exists somewhere nearby, and a paused prompt’s state can be parked in a neighboring GPU’s memory over the high-bandwidth NVLink interconnect far faster than in host memory. It profiles models as memory producers or consumers, places them together in the same interconnect domain, manages the offloaded tensors with a library, and builds a preemptive fair scheduler on top that finally delivers both responsiveness and throughput. The results are strong — a 20 times improvement in responsiveness and a 4 times improvement in single-long-prompt throughput on 8 H100 GPUs across text, audio, and vision models. Read the scope carefully. The approach depends on spare memory existing within the fast-interconnect domain, so a uniformly memory-saturated cluster offers little to borrow; it is confined to the reach of NVLink (a server, or a rack in the newest designs); borrowed memory costs the lender capacity, so profiling and placement must adapt as loads shift; and the headline numbers are specific to the tested models and workloads. It complements autoscaling rather than replacing it, buying responsiveness while new capacity spins up. The durable idea is that the cost of moving data is set by the path it takes, and that a fast interconnect can turn a neighbor’s spare memory into near memory — cheap enough to make fine-grained, fair, responsive scheduling practical where it was not before.

Background

Why pooling nearby resources over a fast interconnect keeps paying off

It helps to see Aqua within a broad theme in machine-learning systems: efficiency comes not from adding hardware but from treating the resources already present in a cluster as a shared pool and moving work or data to wherever there is slack, over the fastest path available. Here the resource is GPU memory, the slack is the imbalance between memory-hungry and memory-light jobs, and the fast path is the NVLink interconnect that makes a neighbor’s memory usable as if it were local. The edge-training work covered elsewhere on this site pursues the same instinct from a different angle, harvesting the tidal idle capacity of servers built from many small chips and adapting the workload to the machine so that otherwise-wasted resources do useful work. The common thread is that clusters are almost never uniformly loaded — at any moment something is idle while something else is starved — and a system that can sense that imbalance and shuttle work or state across it, cheaply, extracts far more from the same hardware than static provisioning ever could. Aqua adds a sharpening lesson that the enabling factor is the interconnect: as fast links like NVLink expand from within a server to across an entire rack, the pool of resources that can be shared as if local grows with them, and designs that decouple memory from compute and page over these links will scale to ever-larger domains. As models grow and demand stays bursty, pooling nearby capacity over fast interconnects is likely to be a central tool for serving them affordably.

If you remember only three things

1. Serving a large language model is limited by GPU memory: each in-flight prompt needs memory for its working state (chiefly the key-value cache, which grows with response length), and a burst of requests exhausts it. Serving engines batch some prompts and queue the rest, but queueing causes head-of-line blocking, so during bursts the model becomes unresponsive and time-to-first-token spikes — and even a few seconds of unresponsiveness loses users, while autoscaling takes minutes. Preemptive time-slice scheduling would keep prompts responsive, but it requires paging a paused prompt's state out of GPU memory and back, and paging to host memory over the slow CPU-to-GPU link is so costly that throughput more than halves.
2. Aqua observes that GPU memory pressure across a cluster is uneven — at any moment some jobs underutilize their memory (producers) while others are starved (consumers) — so it decouples memory from compute and lets producers lend spare memory to consumers. Crucially, it pages a consumer's state to a producer GPU's memory over the high-bandwidth NVLink interconnect within a scale-up domain (a multi-GPU server, or a whole rack in the newest architectures), which is far faster than host memory over the slow bus, so preemption overhead collapses. It has three parts: a profiler that labels models as producers or consumers, a placer that pairs them in the same NVLink domain, and a library that manages the offloaded tensors. On top, it builds a preemptive completely-fair scheduler whose new batching algorithm keeps throughput while sharing compute fairly.
3. Because cheap paging makes preemption affordable, Aqua delivers both responsiveness and throughput under bursts. Evaluated on a server with 8 Nvidia H100 80G GPUs across text, audio, and vision models, it improves responsiveness (time-to-first-token) by 20 times over the state of the art and improves throughput on a single long prompt by 4 times. Scope: the gains depend on spare memory existing within the fast-interconnect domain, the technique is confined to the reach of NVLink, borrowed memory costs the lending GPU capacity so profiling and placement must adapt, and the numbers are specific to the tested models and workloads; it complements autoscaling rather than replacing it. The durable lesson is that the cost of moving data is set by the path it takes, and a fast interconnect can turn a neighbor's spare memory into near memory — cheap enough to make fine-grained, responsive, fair scheduling practical.

Check yourself

Check your understanding

1. Why does a burst of requests make a cloud-hosted LLM unresponsive?
2. Preemptive scheduling would keep prompts responsive. Why is it too costly in the naive form?
3. What is Aqua's key insight and mechanism?
4. Aqua reports improving both responsiveness and throughput. Why is that notable, and what does it depend on?

Sources

The paper is “Aqua: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains” by Abhishek Vijaya Kumar, Gianni Antichi, and Rachee Singh, published at ASPLOS 2025, DOI 10.1145/3676641.3715983. The authors are at Cornell University and Politecnico di Milano / Queen Mary University of London.

This explainer is based on the full text of the paper, obtained from the authors’ open-access arXiv version (arXiv:2407.21255), which corresponds to the CC-BY version of record. The problem framing (LLM inference constrained by GPU memory, the key-value cache as the growing working state, admission control and head-of-line blocking, the spike in time-to-first-token during bursts, the minutes-long delay of autoscaling, and the demonstration that preemptive scheduling improves responsiveness but paging state to host memory over PCIe more than doubles request completion time), the key insight (uneven GPU memory pressure across a cluster, and the decoupling of memory allocation from compute allocation by sharing spare memory over NVLink within a scale-up domain), the three components (Aqua-profiler labeling models as producers or consumers, Aqua-placer pairing them within the same NVLink domain, and Aqua-lib managing offloaded tensors), the preemptive completely-fair scheduler and its batch-partitioning algorithm, and the evaluation results (a server with 8 Nvidia H100 80G GPUs across text, audio, and vision models, a 20 times improvement in responsiveness measured by time-to-first-token, and a 4 times improvement in throughput on a single long prompt) are all drawn directly from the paper. The tool is open-sourced by the authors. Background explanations of the key-value cache, GPU memory, PCIe and NVLink interconnects, and preemptive scheduling are standard and provided to make the paper’s contribution understandable.