GMT: GPU Orchestrated Memory Tiering for the Big Data Era

A GPU working on data larger than its own memory has two known options — let host software shuttle pages for it, which cannot keep up when thousands of threads fault at once, or have the GPU issue storage commands directly and skip the host entirely, which is fast but throws away a tier of memory that sits between them; GMT (Penn State with NVIDIA and Illinois) puts host memory back as a middle tier while keeping the GPU in charge of moving data, and adds a predictor that decides for each evicted page whether it deserves that middle tier at all, since blindly demoting everything just evicts something more useful.

Why this problem is worth solving

The interesting constraint is not capacity but concurrency, and it is what makes GPU tiering a different problem from CPU tiering.

The second half of the problem is a question CPU memory managers rarely face.

What people did before, and where it broke

• Preload data onto the GPU before launching the kernel. The original model: host software works out what the computation needs, copies it over, then starts the kernel. Where it broke. High overhead, and it limits how much computation can be offloaded at once, forcing many more kernel launches.
• Extend the GPU address space into host memory with faulting. Unified virtual memory lets a GPU thread touch host-resident data, take a fault, and have driver code move the page. Where it broke. It only reaches host memory, not storage, and the transfers are initiated by host software.
• Extend that same mechanism down to storage. Several systems, including a mechanism now integrated into the Linux kernel, connect the GPU address space to the operating system page cache so the reach extends to SSDs. Where it broke. The host is still the intermediary. When thousands of GPU threads fault at once, a few host cores running page cache code cannot keep up, and the paper measures this design as roughly four and a half times slower than a GPU-orchestrated two-tier alternative.
• Give the GPU direct access to the SSD and skip the host. Placing storage command queues in GPU memory and mapping them across the interconnect lets GPU threads issue storage commands themselves, which removes the host bottleneck entirely and holds the performance record. Where it broke. It skips host memory in both directions. Fetching a page from host memory takes about 50 microseconds against about 130 for an SSD, so a tier that could serve a meaningful part of the working set is left unused.
• Cascade pages down the tiers, as a conventional hierarchy does. The obvious way to add the middle tier: whatever leaves GPU memory goes to host memory, whatever leaves host memory goes to SSD. Where it broke. The paper implements exactly this and finds it worth only 7% on average, because inserting a page with a distant reuse evicts one that would have been used sooner — and it generates the most wasted lookups and the most interconnect traffic of any policy tested.
• The gap. Nobody had built a three-tier hierarchy in which the GPU orchestrates the transfers, and nobody had asked whether an evicted page deserves the middle tier at all rather than assuming it does.

How GMT decides where an evicted page goes

The idea

A tier you bypass is a tier you cannot use, and the fix is not to reintroduce the host software with it. The design that set the record did two things at once — it put the GPU in charge of transfers, and it dropped host memory from the path — and it is easy to read those as a package. GMT separates them. Host memory sits at roughly 50 microseconds against 130 for an SSD, so a page found there is a real saving, and there is no reason the GPU cannot manage that tier itself using the same mechanism it uses for storage. The measurement that makes the case is a simple ablation: even placing evicted pages into host memory at random beats the two-tier design by 24%, which says most of the value is simply having the tier at all. Recovering a resource that an earlier design discarded as part of a bundle, by asking which half of the bundle actually mattered, is a direct continuation of the GPU-initiated storage access work covered elsewhere on this site.

With three tiers, eviction acquires a second question that two-tier systems never had to answer. On a conventional hierarchy the only decision is which page to evict; where it goes is obvious. Add a middle tier and there is now a placement decision, and the paper’s central point is that the obvious answer is wrong. Cascading everything downward means a page with a very distant reuse displaces one that would have been touched sooner, which is precisely the situation Belady’s optimal policy exists to avoid. So GMT compares a page’s estimated reuse distance against the tier sizes: shorter than GPU memory and it stays, shorter than host memory and it goes there, longer and it goes to storage or is simply dropped if unmodified. Deciding placement across tiers by predicting future access rather than by cascading on recency is the same reframing as the transparent page placement work for tiered memory covered elsewhere on this site, arrived at from the accelerator side.

Predicting reuse without instrumenting every access is where the engineering is. True reuse distance requires counting distinct pages between accesses, which needs state updated on every access to every page — hopeless at this scale and parallelism. GMT chains three cheap approximations instead. First, count all accesses rather than distinct ones, which needs only a counter and a per-page timestamp updated when that page is touched. Second, discover that this cheap proxy relates linearly to the real quantity, and fit the line from a small sample, offloading the regression to a host thread so GPU threads are not stalled. Third, predict the future value from history, using a three-state chain over which tier the page should have gone to on its last two evictions — which is checkable after the fact, since when a page comes back the system knows exactly how long it was away. Predicting what a GPU will need and when, rather than reacting to faults as they land, is the same instinct as the tensor migration and prefetching work for unified memory covered elsewhere on this site.

Walking one eviction through GMT

The saving is that a page timestamp is only written when that page is accessed. Real reuse distance requires updating state for every resident page on every access, because a distinct-page count changes for everyone. This proxy needs one counter and one write per access, and can lean on hardware counters where they exist.

Why it works

• Host memory is genuinely faster than storage, and the tier was simply unused. A page fetched from host memory takes about 50 microseconds against about 130 from the SSD → every page found in the middle tier is a saved storage access → even random placement into that tier beats the two-tier design by 24%, which shows most of the gain is having the tier at all.
• Keeping the GPU in charge is worth more than the extra tier. The host-orchestrated three-tier design has the same tiers available and is still slower than a GPU-orchestrated two-tier one → the paper tests an optimistic version with hit rates raised to match GMT and finds GMT still 90.3% faster → so throughput of transfer, not hit rate, is the dominant term.
• A third tier turns eviction into a placement problem worth solving. Cascading everything downward evicts middle-tier pages that would be used sooner → comparing predicted reuse against tier capacities routes each page to where it belongs → reuse-aware placement is worth 1.5 times against 1.07 for cascading.
• Counting all accesses is a usable stand-in for counting distinct ones. True reuse distance needs per-page state updated on every access → total access count needs one counter and one write to the page being touched → and the two are linearly related, so a fitted line recovers the quantity that matters.
• Whether a placement was right is knowable afterwards. When a page returns, the system knows exactly how long it was absent and therefore which tier it should have been in → that turns placement into a supervised prediction problem with free labels → two levels of history suffice, capturing both the constant and the alternating patterns real applications show.

What it costs (trade-offs and scope)

• One workload gets slower, and a random policy beats the clever one there. On the particle simulation with only 1.17% page reuse, GMT-Reuse suffers a 12% slowdown, because there is not enough history to predict from and the mispredictions cost more than the tier saves. Random placement outperforms the predictor on that application.
• A large share of the benefit is not the predictor. Random placement into host memory already yields 1.24 times against the two-tier baseline, and reuse prediction takes that to 1.5. Cascading, the obvious policy, manages only 1.07. So the ordering is meaningful but the headline is carried substantially by using the tier at all.
• The advantage shrinks as datasets grow relative to memory. Doubling the oversubscription factor drops the speedup from 1.5 to 1.23. That is the direction the motivating trend is heading, so the benefit is largest in the regime where the problem is least severe.
• It is not purely GPU-orchestrated. The regression that converts the cheap proxy into a reuse estimate runs on a dedicated host thread, and hundreds of thousands of samples are needed before it is usable, with a fallback policy running until then. The claim is about transfers, not about every part of the mechanism.
• The storage configuration is modest. A single consumer-grade SSD on a third-generation interconnect. The baseline this is compared against was designed around saturating much larger arrays, and the relative value of a host memory tier depends directly on how slow the tier below it is.
• The main results use less than half the GPU memory available. GPU memory is capped at 16 gigabytes of the card’s 40, because collecting the characterisation statistics is expensive. A separate experiment at 32 gigabytes shows a 45% speedup, so the finding holds, but the headline configuration is not the full device.
• Only the downward path is managed. Pages fetched from storage still bypass host memory entirely, following the prior design. The argument is that a page can be inserted into the middle tier later without sitting in the critical path, which is reasonable and untested against the alternative.
• Several pieces are hand-set. The middle tier evicts by first-in-first-out, the override threshold is 80%, the transfer mechanism switches strategy above 8 pages and a full warp of threads, and prefetching is not studied at all — everything is demand-driven.
• New costs are real if small. A lookup in the middle tier that misses adds latency in the critical path and burns GPU cycles that could have done application work, and pages now bounce across the interconnect between the top two tiers. The paper puts the total at about 2.41%.

The evaluation

Check yourself

Sources

The paper is “GMT: GPU Orchestrated Memory Tiering for the Big Data Era” by Chia-Hao Chang, Jihoon Han, Anand Sivasubramaniam, Vikram Sharma Mailthody, Zaid Qureshi and Wen-mei Hwu, published at ASPLOS 2024 Volume 3, pages 464-478, DOI 10.1145/3620666.3651353. The authors are at the Pennsylvania State University, NVIDIA and the University of Illinois. The paper carries an artifact appendix.

This explainer is written from the full text, read through a proxy of the ACM Digital Library copy. Everything above comes from the paper’s own sections 1 through 3: the account of how unified virtual memory and its storage extensions place host software in the transfer path, and why that fails under GPU-scale fault concurrency; the description of the prior GPU-orchestrated design that maps storage command queues into GPU memory and bypasses the host in both directions; the fixed design parameters of 64 kilobyte pages, demand-driven movement, clock replacement in the top tier and upward bypass of the middle tier; the three placement policies of cascading, random and reuse-based, with the reuse categories defined by comparison against tier capacities; the virtual timestamp distance proxy and the empirical linear relationship justifying it, along with the sampled least-squares regression offloaded to a host thread and refined every ten thousand samples; the three-state history predictor over prior evictions; the first-in-first-out eviction within the middle tier and the 80% threshold that forces placement there when it would otherwise sit idle; the transfer mechanism study leading to the hybrid scheme that switches to direct thread-driven copies above eight pages with a full warp; the nine applications spanning 1.17% to 93.5% page reuse and their reuse distance distributions; the speedups of 1.5, 1.24 and 1.07 for the three policies, the 12% slowdown on the low-reuse workload, the roughly 2.41% overhead from wasted lookups and extra interconnect traffic, the drop to 1.23 at doubled oversubscription, the 45% result at 32 gigabytes of GPU memory, and the comparison against the host-orchestrated design including the optimistic variant leaving GMT 90.3% ahead.