Asynchrony and GPUs: Bridging this Dichotomy for I/O with AGIO

Jihoon Han, Anand Sivasubramaniam, Chia-Hao Chang, Vikram Sharma Mailthody, Zaid Qureshi and Wen-Mei Hwu

GPUs run a mostly synchronous model built to hide latency by switching between thousands of threads, but when a GPU thread reads data from an SSD it stalls for microseconds — an eternity for a GPU — and there was almost no support for GPU threads to issue non-blocking I/O themselves; this paper (Penn State and NVIDIA — Jihoon Han, Anand Sivasubramaniam, and colleagues) presents AGIO, an API and runtime that lets GPU threads perform asynchronous, fully GPU-orchestrated I/O by decoupling when an I/O is started from when it completes, so a thread can do useful work or launch more I/O in between — running 65 percent faster than synchronous I/O when access patterns are known and 32 percent faster on data-dependent workloads by saturating SSD bandwidth, and matching synchronous performance with fewer than half the GPU's compute engines.

The argument in one glance

Problem

A GPU thread that reads from an SSD just freezes

A graphics processing unit (GPU) runs thousands of threads and is built around a largely synchronous model: a thread issues an operation and waits for it, and the GPU hides waiting by rapidly switching among its many threads so that some are always making progress. This works beautifully for memory latencies of a few hundred nanoseconds. But GPU applications increasingly need data that is too big for GPU or host memory and lives on solid-state drives (SSDs) — graph analytics, recommendation, and machine-learning workloads over huge datasets. When a GPU thread reads from an SSD using the usual blocking (synchronous) mechanism, it stalls for the SSD's latency, which is on the order of microseconds — thousands of times longer than a memory access, and far too long to hide by thread-switching alone. Worse, there is little hardware or systems support today for a GPU thread to issue a non-blocking (asynchronous) I/O directly and keep working. So the GPU sits idle waiting on storage, and the SSD's bandwidth goes underused, exactly when the data no longer fits in memory and storage access matters most.
What people did

Do blocking I/O, or route storage access through the CPU

The common approaches keep the synchronous model. A GPU thread performs a blocking read and simply waits for the SSD to return the data. Or the GPU hands storage requests off to the CPU, which manages the I/O and moves data in and out of GPU memory, so the GPU is not directly issuing storage operations at all.
Why it fell short

Blocking wastes GPU cycles; CPU-mediated I/O adds a middleman

Blocking I/O leaves the GPU's enormous parallelism stalled on microsecond-scale storage latency that thread-switching cannot cover, so compute engines idle and the SSD's bandwidth is not filled. Routing I/O through the CPU inserts a middleman on every access, adding coordination overhead and latency and preventing the GPU from directly and efficiently driving the storage the way it drives its own memory. Neither approach lets a GPU thread start a storage access and then get on with other work, or start many storage accesses at once to keep the SSD busy. The fundamental mismatch is that the GPU's synchronous execution model has no clean way to tolerate the long, variable latency of SSDs, and there was no direct, GPU-orchestrated asynchronous I/O mechanism to bridge that gap. What is missing is a way for GPU threads themselves to issue non-blocking I/O, overlap it with computation or with more I/O, and only collect the result once it is actually needed.
This paper's idea

Let GPU threads fire off I/O and keep going, collecting results later

AGIO provides APIs and a runtime that let GPU threads directly perform asynchronous I/O to SSDs, fully GPU-orchestrated with no CPU involvement. Its core idea is to decouple the initiation of an I/O from its completion, both in time and in space: a thread starts a storage request and immediately continues, doing useful work in between, and later checks for or collects the result when it is needed. This overlap hides much of the I/O latency. AGIO turns out to help in two distinct ways. First, when an application's access pattern is known at compile time, I/Os can be issued ahead of need, much like prefetching, so the data arrives by the time the thread wants it — yielding 65 percent better performance than the synchronous version. Second, and less obviously, for data-dependent accesses (where you cannot know in advance what to fetch), a thread that has issued one I/O can proceed to initiate more I/O even if there is little computation to overlap; this pro-active I/O issuance raises the number of storage requests in flight, more fully using the SSD's bandwidth, and yields 32 percent better performance than the synchronous alternative. Because initiation is decoupled from completion, AGIO adapts to the dataset on its own, so the programmer does not need advance knowledge of the inputs. AGIO can even match or beat synchronous performance using a GPU with fewer than half the compute engines, because it stops wasting compute on stalls.

Why this problem is worth solving

GPUs are increasingly the engine for data that no longer fits in memory, and the way they touch storage decides whether that engine runs or idles.

A GPU gets its speed from massive parallelism and a synchronous execution style: threads issue operations and wait, and the hardware keeps utilization high by switching among a huge number of threads so that whenever some are stalled on a memory access, others run. This latency-hiding trick works because memory latencies are short — a few hundred nanoseconds — so a modest amount of other work covers them. The trouble is scale of data. Modern GPU workloads in graph processing, recommendation, and machine learning operate on datasets far larger than GPU or even host memory, so the data must live on SSDs and be pulled in as needed. But SSD access latency is measured in microseconds, thousands of times longer than a memory access. No realistic amount of thread-switching can hide a stall that long, so when a GPU thread blocks on an SSD read, its compute engine sits idle. And because there is little support for GPU threads to issue non-blocking I/O directly, the natural fix — start the access, do other work, come back for the result — is not available. The consequence is stark: exactly when data outgrows memory and storage becomes central, the GPU’s expensive compute stalls and the SSD’s bandwidth goes underutilized. Fixing this is what lets GPUs remain efficient on the large, storage-resident datasets that increasingly define their workloads.

The deeper issue is a mismatch between how GPUs execute and how storage behaves, with no bridge between them.

Two things collide. The GPU’s model is synchronous and its latency-hiding assumes short waits; SSDs impose long, variable, microsecond-scale waits. The straightforward responses each fail. Blocking (synchronous) I/O from a GPU thread means the thread simply waits out the SSD latency, and since that latency dwarfs what thread-switching can cover, compute engines idle and only a few storage requests are ever in flight, so the SSD’s bandwidth is left largely unused. Alternatively, handing storage requests to the CPU — letting it perform the I/O and stage data into GPU memory — inserts a middleman into every access. That adds coordination and latency, keeps the GPU from directly and efficiently orchestrating storage, and does not give individual GPU threads a way to overlap their own I/O with their own work. What is genuinely needed is asynchrony brought into the GPU itself: a way for a GPU thread to initiate a storage access, continue executing, and reclaim the result later, and to issue many such accesses to keep the SSD saturated. Providing that — non-blocking, GPU-orchestrated I/O that decouples starting an access from finishing it — is the bridge that both blocking I/O and CPU mediation leave unbuilt, and it is the gap AGIO fills.

What people did before, and where it broke

Blocking (synchronous) I/O from GPU threads. A GPU thread issues a storage read and waits for the data to return. Where it broke. SSD latency is microseconds, thousands of times longer than memory latency, so thread-switching cannot hide it; compute engines idle and only a few requests are in flight, wasting SSD bandwidth.
Route storage access through the CPU. Have the CPU perform the I/O and stage data into and out of GPU memory. Where it broke. It inserts a middleman on every access, adding coordination overhead and latency and preventing the GPU from directly, efficiently orchestrating storage; individual GPU threads still cannot overlap their own I/O with their own work.
Rely on the GPU’s thread-level parallelism alone. Trust that having enough threads will keep the GPU busy despite some stalling on I/O. Where it broke. The stalls are too long to cover this way, so many threads block simultaneously and the machine still idles on storage latency.
The gap. There was little support for GPU threads to directly issue non-blocking, asynchronous I/O that decouples starting an access from completing it, so the GPU could neither overlap storage latency with useful work nor keep many storage requests in flight to fill the SSD’s bandwidth.

How AGIO works

Decoupling starting an I/O from finishing it, on the GPU

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Bring asynchrony into the GPU so a thread can start an access and keep working. AGIO’s foundational move is to give GPU threads non-blocking I/O that decouples initiation from completion, so the long SSD latency is overlapped with useful work instead of stalling a compute engine. Letting a GPU thread issue a storage request itself and drive the device directly, rather than blocking or deferring to the CPU, is the same capability the work on GPU-initiated on-demand high-throughput storage access in the BaM system covered elsewhere on this site established, putting the GPU in charge of its own storage accesses. AGIO builds on that direct-access foundation by making the accesses asynchronous, so the thread need not wait for each one to finish.

Hide long latency by overlapping it with other work. The reason decoupling helps is the classic latency-hiding principle: if you can do something useful during a long wait, the wait costs you little. AGIO issues I/O ahead of need (for known patterns) so data arrives just in time, exactly like prefetching. Overlapping a long-latency operation with independent computation so its cost is largely hidden is the same principle the work on transparent tracking and triggering for fine-grained overlap of computation and communication covered elsewhere on this site is built on, hiding one slow thing behind another useful thing. AGIO applies it to GPU storage I/O, and finds it pays off dramatically when the access pattern is predictable.

Keep the storage device saturated by issuing many requests at once. AGIO’s less obvious benefit is that decoupling lets a thread launch more I/O even without compute to overlap, raising the number of requests in flight and filling the SSD’s bandwidth. Driving a storage device with enough concurrent requests to use its full bandwidth, rather than trickling requests one at a time, is the same lever the work on maximizing SSD bandwidth utilization for huge embedding models covered elsewhere on this site pulls, turning many outstanding requests into throughput. AGIO does this from GPU threads for data-dependent workloads, where pro-active issuance is what turns idle bandwidth into a 32 percent speedup.

Following a data-dependent workload through AGIO

Traversing a graph too big for memory, without stalling

An illustrative walk-through, based on the paper's abstract, of how AGIO hides SSD latency for a data-dependent workload. Exact API and runtime details should be read from the full paper.

Step 1 of 5

A GPU thread needs a graph node stored on an SSD

The thread is traversing a graph whose data lives on SSDs, and it must read a node to know which neighbours to visit next — a data-dependent access, since the next read depends on this one's result. Blocking here would stall the thread for microseconds.

Data-dependent accesses cannot be perfectly prefetched, since each depends on the last.

Why it works

Decoupling lets latency be overlapped instead of stalling. SSD latency is too long for thread-switching to hide → but starting an I/O and continuing lets useful work fill the wait → so the compute engine keeps running instead of idling on storage.
Issuing ahead of need makes data arrive just in time. When the access pattern is known, the fetch can start early → so the data is present by the time the thread needs it → so, like prefetching, the latency is almost entirely hidden, giving 65 percent better performance.
Pro-active issuance saturates the SSD even without compute. Data-dependent workloads have little to overlap → but a thread can still launch more I/O instead of waiting → so many requests are in flight and the SSD’s bandwidth is filled, giving 32 percent better performance.
Decoupling makes it adaptive to the data. The programmer cannot always predict the inputs → but separating initiation from completion lets AGIO issue and reclaim I/O as the data dictates → so performance holds without advance knowledge of the dataset.
Not stalling means less compute is needed. Blocking wastes compute engines on waiting → but overlapping keeps them productive → so AGIO matches or beats synchronous performance with fewer than half the compute engines.

What it costs (trade-offs and scope)

This explainer is from the abstract only. The full paper was not available when this page was written, so the descriptions of AGIO’s APIs and runtime are at the level the abstract states; the internal design (how requests and completions are tracked GPU-side, queue structures, and the exact interface) is not reproduced here and should be read from the paper.
Asynchronous I/O is harder to program than blocking I/O. Decoupling initiation from completion means the programmer (or a compiler) must manage in-flight requests and collect results at the right time, which is more complex than a simple blocking read, even though AGIO’s adaptivity reduces the need for advance input knowledge.
The gains depend on the access pattern. The largest benefit (65 percent) comes when access patterns are known at compile time and I/O can be prefetched; data-dependent workloads gain less (32 percent) because pro-active issuance, not overlap with compute, is doing the work, and a workload with neither predictable patterns nor room to issue ahead would benefit less.
It relies on GPU-orchestrated direct storage access. AGIO assumes GPU threads can directly drive SSD I/O without the CPU; this builds on that class of capability and hardware, so it presumes a system where such direct GPU-storage access is available.
The numbers are on specific workloads and baselines. The 65 percent and 32 percent improvements and the fewer-than-half-compute-engines result are measured against synchronous counterparts on the paper’s workloads; other applications, SSDs, or GPUs could behave differently.

The evaluation

Accessing SSD data from a GPU

Property	Blocking GPU I/O	CPU-mediated I/O	AGIO (async, GPU-orchestrated)
GPU thread keeps working during I/O
Keeps many requests in flight
Uses SSD bandwidth fully
Performance vs synchronous
Compute engines needed

A comparison, drawn from the paper's abstract, of ways for a GPU to access SSD data. AGIO gives GPU threads asynchronous, fully GPU-orchestrated I/O that decouples initiation from completion, yielding 65 percent better performance for known access patterns and 32 percent for data-dependent ones, while matching synchronous performance with fewer than half the compute engines.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based only on the paper’s abstract; the full text was not available when it was written, so this page is marked as a draft. The claims stated here are the abstract’s: GPUs rely on a largely synchronous programming and execution model, and GPU threads incur significant latency when accessing SSD data with blocking/synchronous I/O, with little existing support for non-blocking/asynchronous operations directly from GPU threads. AGIO provides APIs and a runtime for GPU threads to directly perform asynchronous I/O, fully GPU-orchestrated without CPU involvement, decoupling I/O initiation from completion in time and space to allow useful work in between and hide latency. For applications with access patterns known at compile time, AGIO can issue I/Os ahead of need (like prefetching) for 65 percent better performance than synchronous; for data-dependent accesses, threads can proceed past initiation to issue more I/O, increasing I/O parallelism to more fully use bandwidth, for 32 percent better performance; decoupling makes AGIO adaptive to dataset characteristics without a priori input knowledge; and AGIO can match or better synchronous performance using a GPU with fewer than half the compute engines. The internal design of the APIs and runtime, the exact workloads and hardware, and the full results are not described in the abstract and have not been invented here; they should be confirmed in the full paper. The illustrative walk-through above is a plausible reconstruction, not a trace from the paper.

Background

Why GPUs are learning to talk to storage on their own

It helps to see AGIO within a broader shift in how GPUs handle data. For years, the GPU was a compute accelerator fed by the CPU: the CPU staged data into GPU memory and the GPU crunched it. That model breaks when datasets grow far beyond GPU or host memory, as they have in graph analytics, recommendation, and large-scale machine learning, because constantly routing data through the CPU becomes a bottleneck. The response has been to let GPUs reach out to storage directly and on demand, driving SSDs themselves at high throughput, so data can be pulled in exactly when needed without a CPU middleman. But direct access alone is not enough if each access still blocks a thread, because storage latency is far longer than the GPU’s synchronous model can hide. The natural next step, which AGIO takes, is to make that direct access asynchronous, so GPUs gain the same overlap-and-pipeline tricks that CPUs have long used for I/O. The broader move to feed GPUs from storage efficiently, prefetching and migrating data so computation is not starved when it outgrows memory, is the same terrain the work on tensor migration and prefetching in unified memory covered elsewhere on this site occupies, keeping the GPU busy over a larger-than-memory dataset. AGIO contributes the asynchrony that lets GPU threads themselves overlap and parallelize storage access, arguing that bridging the GPU-storage gap is as much about how you wait as about how you fetch.

If you remember only three things

1. GPUs use a largely synchronous model and hide short memory latencies by switching among thousands of threads, but SSD access latency is microseconds — thousands of times longer — so a GPU thread doing a blocking read stalls in a way thread-switching cannot cover, and there was little support for GPU threads to issue non-blocking I/O directly. The result is idle compute engines and underused SSD bandwidth exactly when datasets outgrow memory and storage access matters most.
2. AGIO gives GPU threads APIs and a runtime to perform asynchronous, fully GPU-orchestrated I/O (no CPU involvement), decoupling when an I/O is started from when it completes, in time and space, so the thread does useful work in between and hides much of the latency. It helps two ways: for access patterns known at compile time it issues I/O ahead of need like prefetching (65 percent faster than synchronous), and for data-dependent accesses a thread proactively issues more I/O even without compute to overlap, raising I/O parallelism to fill the SSD bandwidth (32 percent faster).
3. Because initiation is decoupled from completion, AGIO adapts to the dataset without the programmer knowing inputs in advance, and by not wasting cycles on stalls it can match or beat synchronous performance with fewer than half the GPU's compute engines. The trade-offs: asynchronous I/O is harder to program than blocking reads, the gains depend on the access pattern, and it assumes GPU-orchestrated direct storage access. (This page is written from the abstract only; the API and runtime internals and full results should be confirmed in the paper.)

Check yourself

Check your understanding

1. Why does a GPU stall badly when its threads read data from SSDs with blocking I/O?
2. What is AGIO's core mechanism?
3. How does AGIO help data-dependent workloads, where you cannot know in advance what to fetch?
4. Predict: two teams run the same storage-heavy GPU workload, one with blocking I/O and one with AGIO, on identical hardware. Beyond raw speed, what difference in resource efficiency should you expect, and why?

Sources

The paper is “Asynchrony and GPUs: Bridging this Dichotomy for I/O with AGIO” by Jihoon Han, Anand Sivasubramaniam, Chia-Hao Chang, Vikram Sharma Mailthody, Zaid Qureshi, and Wen-Mei Hwu (Pennsylvania State University and NVIDIA), published at ASPLOS 2026, DOI 10.1145/3779212.3790130. It is licensed CC BY.

This explainer is written from the paper’s abstract only; the full text was not available at the time of writing, so this page is marked as a draft. According to the abstract, GPUs rely on a largely synchronous programming and execution model, and with the increasing need to access data on SSDs, GPU threads incur significant latencies using blocking/synchronous I/O, with little hardware or systems support for non-blocking/asynchronous operations directly from GPU threads to tolerate microsecond-level SSD latencies. AGIO provides APIs and a runtime environment for GPU threads to directly perform asynchronous I/O operations, fully GPU-orchestrated without CPU involvement, decoupling I/O initiation from completion in time and space to allow useful computation in between and hide much of the I/O latency. For applications with access patterns known at compile time, AGIO I/Os can be introduced ahead of need (like prefetching), yielding 65 percent better performance than the synchronous counterpart; for applications with data-dependent accesses, threads can proceed beyond I/O initiation toward initiating more I/O, and this pro-active issuance increases I/O parallelism to more fully utilize bandwidth, yielding 32 percent better performance than the synchronous alternative. Decoupling initiation from completion makes AGIO adaptive to dataset characteristics without a priori input knowledge, and AGIO can meet or better the performance of synchronous counterparts using a GPU with fewer than half the compute engines. No numbers or mechanisms beyond the abstract have been introduced here; the internal design of the APIs and runtime, the workloads and hardware, and the full results should be confirmed in the full paper.