Scoped Buffered Persistency Model for GPUs

Persistent memory lets a program's data structures survive a crash, but only if you control the order in which writes actually reach it — and the models that give CPUs that control assume a handful of globally-ordered threads, which collapses on a GPU running hundreds of thousands of them; this work from the Indian Institute of Science and the University of Washington defines SBRP, a persistency model that borrows the GPU's own idea of scopes so a programmer orders only the threads that matter, backed by a per-warp hardware buffer that hides the slow persistent-memory writes, speeding PM-aware GPU programs up to 3.45 times over the prior approach.

The argument in one glance

Problem

GPU programs can now use persistent memory, but cannot reason about surviving a crash

Persistent memory is non-volatile memory read and written by ordinary loads and stores at byte granularity, so data structures can live in it and survive a crash. Recent work showed GPU programs benefit greatly from combining the GPU's parallelism with this fine-grain persistence — a GPU-accelerated persistent key-value store runs several times faster than a CPU-only one. But there is a gap: writes to persistent memory sit in volatile caches first and drain to the memory in an arbitrary order, so a crash mid-drain can leave a data structure corrupt. Without a persistency model that pins down the order writes become durable, a programmer cannot reason about whether their GPU program can correctly recover.
What people did

Reuse a CPU persistency model, ordering writes with global barriers

CPUs have a decade of persistency models to borrow from. The natural move, which the prior GPU work implicitly makes, is to reuse the epoch persistency model: place global barriers that divide execution into epochs, so writes within an epoch may become durable in any order but writes in an earlier epoch are all durable before any in a later one. A barrier both flushes the pending writes to persistent memory and enforces the ordering across every thread.
Why it fell short

CPU models assume few, globally-ordered, coherent threads — none true on a GPU

Every assumption behind the CPU model breaks on a GPU. Ordering all persists globally is ruinous across hundreds of thousands of threads and usually unnecessary given the GPU's thread hierarchy. The CPU trick of buffering persists and inferring cross-thread order from cache-coherence messages fails too, because GPU L1 caches are not kept coherent by hardware. And a global barrier flushes and invalidates the L1 cache on every ordering point, forcing slow round-trips to the persistent memory even when only a few nearby threads needed ordering.
This paper's idea

Borrow the GPU's own scopes and acquire/release, and buffer per warp

GPUs already solved the analogous problem for ordinary memory visibility, with scopes — naming the subset of threads that must synchronize — and acquire/release patterns that need no coherence. SBRP applies the same ideas to persistence. It separates ordering a thread's own writes (an ordering fence) from ordering across threads (scoped persist-acquire and persist-release), so a programmer orders only the threads that matter, at the narrowest scope. And it buffers persists in a per-warp hardware structure at each streaming multiprocessor, hiding the slow persistent-memory writes.

Why this problem is worth solving

The heart of the matter is what “durable in the right order” means and why hardware makes it hard, so it is worth being precise.

A persistency model is to durability what a memory consistency model is to visibility. A consistency model defines the order in which one thread’s writes become visible to others; a persistency model defines the order in which writes to persistent memory become durable — the persist memory order. The order matters because a write does not go straight to persistent memory; it lands in a volatile cache and is written back later, and the write-backs can reach the persistent memory in any order the hardware finds convenient. That is fine until a crash. Suppose a program inserts a node into a persistent linked list: it writes the node’s data, then writes the pointer that links it in. If the hardware happens to make the pointer durable before the data, and the power fails in between, recovery finds a pointer into garbage — a corrupt structure. To prevent this, the programmer must be able to insist that the data persists before the pointer, and reason that no crash can violate it. That guarantee is exactly what a persistency model provides, and without a specified one, a persistent-memory program’s recoverability is simply unarguable. The model must say both how to express an ordering within a thread and how to express one across threads, since real data structures are updated concurrently.

The reason the CPU playbook cannot simply be carried over is that a GPU is a different machine at every level the model touches.

CPU persistency models were designed for a handful of threads, and they lean on three things a GPU does not provide. First, they order persists globally: an ordering constraint applies across all threads of the program. On a CPU with a few cores that is reasonable; on a GPU with hundreds of thousands of concurrent threads it is catastrophic, forcing a global rendezvous where usually only a few neighboring threads actually had a dependency. It is also unnecessary, because GPU programs are structured as a hierarchy — threads grouped into warps, warps into thread blocks, blocks into a grid — and most orderings live within one level. Second, CPU models buffer pending persists in per-core structures and figure out the cross-thread order by snooping cache-coherence traffic. GPUs deliberately do not keep their L1 caches coherent in hardware, because coherence across that many cores is impractical, so there is no coherence traffic to snoop. Third, the CPU epoch model realizes its ordering with a global barrier that flushes and invalidates the cache, which on a GPU means every ordering point pays a full, slow trip out to the persistent memory and back, discarding cache contents that nearby threads could have reused. So a straightforward port is not just slow — its very mechanism is absent. Getting persistence right on a GPU needs a model built for the GPU’s hierarchy and its coherence-free caches, which is what makes the work worthwhile.

What people did before, and where it broke

Reuse the CPU epoch persistency model with global barriers (the prior GPU work’s implicit model). Divide execution into epochs separated by a system-wide barrier that flushes writes and orders across all threads. Where it broke. A global barrier over hundreds of thousands of threads is a ruinous rendezvous, and it flushes and invalidates the L1 cache each time, so every ordering point pays a slow persistent-memory round-trip even when only nearby threads needed ordering.
Buffer persists per core and infer cross-thread order from cache coherence (the CPU technique). Hold pending persists in per-core buffers and use coherence messages to establish inter-thread ordering. Where it broke. GPU L1 caches are not hardware-coherent, so there are no coherence messages to snoop, and there is no scalable way to make them coherent across so many cores.
Naively scale a CPU persist buffer to the GPU. Share one buffer across a streaming multiprocessor, or give each thread its own. Where it broke. A shared buffer serializes unrelated persists artificially, while a per-thread buffer costs enormous hardware given the thread count and still cannot detect cross-thread ordering efficiently.
Order only within a thread block, or only globally. Pick a single fixed granularity for cross-thread ordering. Where it broke. Block-only ordering cannot express the cross-block dependencies real programs need (like a final reduction step), while global-only ordering is the expensive rendezvous again — neither matches the varying scope of real dependencies.
The gap. Every prior option either ordered too much, at global scope, or relied on coherence the GPU lacks, or picked one rigid granularity. What was missing was a model that lets a programmer order exactly the subset of threads a dependency spans, without coherence, and buffer persists cheaply at GPU scale — which is what a real GPU persistency model needs.

How SBRP works

Ordering persistent-memory writes at the scope that actually matters

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Enforce ordering only among the threads a dependency actually spans, not globally. The whole inefficiency of the CPU model on a GPU is that it orders every persist against every other, when the GPU’s own hierarchy means most dependencies are local — within a warp or a thread block — and only a few reach across the whole device. SBRP borrows the GPU consistency model’s scopes so a programmer names the narrowest set of threads that must be ordered, and uses acquire/release to declare the dependency explicitly, so a block-local ordering never triggers a device-wide rendezvous. Enforcing exactly the ordering that correctness requires and no more, exploiting the relaxations the model legitimately permits rather than conservatively over-ordering, is the same discipline the weak-memory binary-translation work covered elsewhere on this site applies to consistency, inserting only the minimal barriers a program’s semantics demand instead of a blunt fence everywhere.

Buffer the slow persistent-memory writes so ordering does not mean waiting on them. Persistent memory is much slower and lower-bandwidth than the GPU’s own memory, so a model that drains to it on every ordering point spends its life stalled on the persistent domain. SBRP instead keeps a volatile per-multiprocessor buffer that tracks pending persists and their order, so intra-thread and thread-block orderings resolve in the fast L1 cache and only genuinely cross-block orderings pay the trip out, letting bandwidth-hungry kernels adapt to the memory’s limits. Hiding a slow, bandwidth-limited memory tier behind a buffer so the fast processor keeps working instead of waiting on it is the same move the Mixture-of-Experts inference work covered elsewhere on this site makes, buffering and overlapping slow memory traffic so the compute units stay busy despite the tier’s limited bandwidth.

Track at the granularity that is the sweet spot, not the naive extreme. How finely to track persists is a real design choice: per thread is faithful but costs enormous hardware across so many threads, while per cache line is cheap but falsely orders writes from different threads that merely share a line. SBRP tracks per warp, which is thirty-two times cheaper than per thread, coalesces naturally because a warp’s lock-step writes often fall in one cache line, and introduces little false ordering. Deliberately choosing the middle granularity that fits the hardware’s real constraints, rather than the most precise or the cheapest extreme, is the same right-sizing the in-storage accelerator work covered elsewhere on this site performs when it selects a moderate design that matches the device rather than the largest or smallest possible.

Following a reduction kernel through SBRP

Persisting the partial sums of a parallel reduction

Tracing a reduction shows how intra-block and cross-block orderings differ in cost.

Step 1 of 5

A reduction sums a huge array in parallel, persisting partial sums

The array is split across thread blocks, each summing its chunk over several iterations, and the partial sums are placed in persistent memory so a crash can resume mid-computation rather than restart. Each iteration's partial sum depends on the previous iteration's, so those persists must be ordered.

For all but the final iteration, the dependency stays within a single thread block.

Why it works

Scopes let a programmer order only the threads a dependency spans. The CPU model orders every persist against every other across the whole program → but scopes and acquire/release name the narrowest set of threads that must be ordered → so a thread-block-local dependency is enforced among those threads alone instead of stalling the entire GPU.
Block-scoped and intra-thread orderings stay in the shared cache. A global epoch barrier flushes and invalidates the L1 cache at every ordering point, forcing a slow persistent-memory round-trip → but threads ordered within a block share their multiprocessor’s L1, so SBRP enforces the order from its buffer without invalidating → so the common local orderings avoid the persistent memory entirely.
Acquire and release express cross-thread order without needing coherence. GPUs do not keep L1 caches coherent, so the CPU trick of inferring order from coherence traffic is impossible → but a release followed by a matching acquire declares the dependency explicitly → so inter-thread persist ordering works on hardware that has no coherence messages to snoop.
Per-warp tracking is cheap yet avoids false ordering. Per-thread buffers cost too much across so many threads while per-cache-line tracking falsely orders unrelated writes → but tracking per warp is thirty-two times cheaper and a warp’s lock-step writes coalesce into shared lines → so the buffer is small and orders only what genuinely must be ordered.
A windowed drain policy hides latency while still coalescing. Eager flushing on every allowed ordering kills the chance to merge writes, while lazy flushing bursts and congests the persistent memory → but keeping a fixed small number of persists outstanding streams them steadily → so the buffer hides the memory’s latency and adapts to its bandwidth without sacrificing coalescing.

What it costs (trade-offs and scope)

It requires new GPU hardware, so it runs only in simulation here. SBRP adds a per-multiprocessor persist buffer, three bitmasks, and per-cache-line bits, so unlike the prior work, which ran on real hardware, it is evaluated in a GPU simulator; the storage overhead is about 3 percent of the L1 cache.
The programmer carries more reasoning burden. Instead of a single global barrier, a programmer must choose the right scope and place matching acquire and release operations, and choosing the narrowest correct scope for performance takes care — though this mirrors what GPU consistency models already ask.
The average gains are moderate; the headline numbers combine effects. Over the same system the model improves performance by roughly 14 to 21 percent on average; the up-to-90-percent figure is for reduction specifically, and the up-to-3.45-times figure over the prior work also folds in placing the memory nearer the GPU.
Much of the largest speedup is from the projected hardware, not the model. Placing the persistent memory onboard the GPU rather than across the interconnect accounts for well over a hundred percent improvement on its own; that PM-near design is a future projection, and SBRP’s own contribution over the same system is the more modest figure.
Per-warp tracking still admits some false ordering. It is far less than per-cache-line or per-block tracking, but a warp whose writes straddle an ordering point can still order persists that were independent, so it is a well-chosen compromise rather than a perfect one.
Correctness still rests on using the model correctly. SBRP gives a specification to reason against, but it does not detect a programmer’s missing ordering operations; persistency bugs from omitted fences or acquire/release remain the programmer’s responsibility, and the evaluation covers six applications and Optane-like memory parameters.

The evaluation

The CPU epoch model versus SBRP on a GPU

Property	Epoch model (prior GPU work)	SBRP
Cross-thread ordering
L1 cache at an ordering point
Persist buffering
Relies on cache coherence
Performance	Baseline

The CPU-style epoch persistency model versus SBRP, evaluated in the GPGPU-Sim simulator on six PM-aware GPU applications (a persistent key-value store, a hashmap, image denoising, reduction, a multiqueue, and scan) across two systems — persistent memory across the interconnect and onboard the GPU. SBRP outperforms the epoch model by 14 to 21 percent on average and up to 90 percent (reduction), where scopes contribute about 77 percent of the gain for cross-thread-ordered kernels, and reaches up to 3.45 times over the prior work when combined with placing the memory near the GPU — because it orders only the threads that matter and keeps local orderings in the cache instead of flushing to persistent memory.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

SBRP is the first persistency model built for the GPU rather than ported from the CPU, and its core insight is elegant: the GPU already solved the analogous scaling problem for ordinary memory visibility with scopes and acquire/release, so persistence should reuse those ideas. By letting a programmer order only the subset of threads a dependency spans, keeping intra-thread and block-scoped orderings in the shared L1 cache instead of flushing to persistent memory, and buffering persists per warp with three bitmasks that stall only the involved warps, it speeds PM-aware GPU kernels by 14 to 21 percent on average and up to 90 percent from scopes, or up to 3.45 times over the prior work with the memory placed onboard. Read it with its scope. It needs new GPU hardware, so it is evaluated in simulation rather than on a real chip, unlike the prior work; the largest speedups combine the model with a projected near-GPU memory design whose benefit is separate from the model’s own 14-to-21-percent contribution; the programmer must reason about scopes and place acquire/release correctly, and the model specifies behavior but does not catch missing operations; and per-warp tracking is a good compromise, not a perfect one. The durable lesson is that persistence, like consistency before it, has to be redesigned around the GPU’s hierarchy and its coherence-free caches — the same relaxation that makes GPUs fast is what a correct, efficient persistency model must embrace rather than fight.

Background

Why durable, crash-consistent memory keeps needing formal models

It helps to see this work as part of a long pattern: every time hardware gives software a new form of nondeterminism, correctness demands a formal model to reason against, because intuition cannot survey all the ways things can go wrong. Persistent memory is a sharp case. A program’s writes drain from volatile caches to durable memory in an order the hardware chooses and the programmer cannot see, and a crash can strike at any point in that invisible schedule, so the set of states a recoverable program must tolerate is enormous and impossible to test exhaustively — a bug may surface only for one crash timing on one drain order. This is why the field does not just build faster persistence but insists on specifying persistency models, precise statements of which orderings are guaranteed, so that recoverability becomes something one can prove rather than hope for. The correctness-logic-for-nondeterministic-quantum-programs work covered elsewhere on this site confronts the same shape of challenge in another domain, building a rigorous logic to reason about programs whose outcomes are a distribution rather than a single value, because eyeballing nondeterministic behavior does not work. The broad lesson is that as memory becomes durable and massively parallel at once, the useful foundations are formal models of the allowed orderings — without one, a crash-consistent program’s correctness is not merely unproven but unstatable.

If you remember only three things

1. Persistent memory lets GPU data structures survive a crash, but writes drain from volatile caches to the memory in an arbitrary order, so a crash mid-drain can corrupt a structure (a pointer durable before its data); a persistency model specifies the order writes become durable (the persist memory order) so recoverability can be reasoned about. CPU persistency models fail on GPUs because they order all persists globally (ruinous across hundreds of thousands of threads), buffer per-core and infer cross-thread order from cache coherence (which GPUs lack), and use global barriers that flush and invalidate the L1 cache on every ordering point, forcing slow persistent-memory round-trips even for local dependencies.
2. SBRP (Scoped Buffered Release Persistency) reuses the GPU consistency model's own scaling ideas: it separates intra-thread ordering (an ordering fence) from cross-thread ordering (scoped persist-acquire and persist-release at the narrowest scope covering the involved threads), needing no cache coherence, and it buffers persists in a per-streaming-multiprocessor volatile structure so intra-thread and block-scoped orderings resolve in the L1 cache without flushing to persistent memory. The buffer tracks persists at warp granularity (32 times cheaper than per-thread, and coalescing-friendly, versus false ordering from per-cache-line tracking), uses three bitmasks so only genuinely-involved warps stall, and drains with a windowed policy that balances write coalescing against the memory's limited bandwidth, at about 3 percent L1 storage overhead.
3. In GPGPU-Sim across six PM-aware GPU applications and two systems (persistent memory across the interconnect and onboard the GPU), SBRP beats the CPU-style epoch model by 14 to 21 percent on average and up to 90 percent (reduction), with scopes contributing about 77 percent of the gain for cross-thread-ordered kernels, and reaches up to 3.45 times over the prior work when combined with near-GPU memory. Its limits: it needs new GPU hardware so it is simulated not run on real silicon, the largest speedups fold in a projected near-GPU memory design separate from the model's own contribution, the programmer must reason about scopes and place acquire/release correctly, per-warp tracking still admits some false ordering, and the model specifies behavior but does not catch missing ordering operations.

Check yourself

Check your understanding

1. Why does a persistent-memory program need a persistency model to be crash-recoverable?
2. Why do CPU persistency models fail when applied to a GPU?
3. How does SBRP let a programmer avoid ordering all persists globally?
4. Which statement best reflects the scope and limits of SBRP?

Sources

The paper is “Scoped Buffered Persistency Model for GPUs” by Shweta Pandey, Aditya K Kamath, and Arkaprava Basu, published at ASPLOS 2023, DOI 10.1145/3575693.3575749. The authors are at the Indian Institute of Science, Bangalore, and the University of Washington.

This explainer is written from the full 14-page text. Everything above comes from the paper’s own sections: the introduction and background motivating persistency models for GPUs (the persist memory order, the epoch persistency model, the GPU execution hierarchy of threads, warps, thread blocks, and grids, and GPU scoped synchronization with acquire/release); the two system designs (PM-far with the memory across PCIe and PM-near with it onboard the GPU); the argument that CPU persistency models fall short for GPUs because global ordering does not scale, CPU persist buffers do not extend, and GPU L1 caches are not coherent, with the reduction kernel as the motivating example of varying-scope dependencies; the SBRP model with its oFence for intra-thread order, scoped persist-acquire and persist-release for inter-thread order, and durability fence; the hardware implementation with the per-streaming-multiprocessor FIFO persist buffer tracking persists at warp granularity via a warp bitmask, the order-delay, eviction-delay, and flush-status masks, the acknowledgement counter, the roughly 3 percent L1 storage overhead, and the windowed drain policy; and the evaluation in GPGPU-Sim 4.0 on six applications (gpKVS, hashmap, SRAD, reduction, multiqueue, scan), reporting SBRP outperforming the epoch model by 14 to 21 percent on average and up to 90 percent, scopes contributing about 77 percent of the speedup for intra-threadblock kernels, up to 3.45 times over the prior GPM work, and the sensitivity studies on buffer size, NVM bandwidth, and window size.