GPM: leveraging persistent memory from a GPU

Persistent memory (like Intel Optane) is a kind of RAM that keeps its contents through a power cut yet is byte-addressable at almost DRAM speed, so a program can save its data structures directly and, after a crash, resume instead of starting over — but only CPU programs could use it; GPUs, despite their massive parallelism, had no way to persist data while computing, so today they ship results back to the CPU to save them (which loses the GPU's parallelism, bottlenecks on the CPU, and wastes bandwidth persisting data that didn't even change); this paper (IISc) builds GPM, which lets GPU kernels read and persist to persistent memory DIRECTLY, from inside the kernel, at byte granularity and with NO hardware changes — by mapping the memory into the kernel's address space and cleverly repurposing a GPU synchronization fence (plus selectively turning off a CPU caching feature) to guarantee the data actually reaches persistent memory — plus a GPU-optimized logging scheme that lets hundreds of thousands of threads write the log lock-free; it runs 2.7 to 5.8x faster than a CPU using the same persistent memory and up to 85x faster than shipping results to the CPU.

The argument in one glance

Problem

Persistent memory lets programs survive crashes — but only CPU programs could use it

Persistent Memory (PM), such as Intel Optane, blurs the line between memory and storage: it keeps its contents through a power failure like a disk, but it is byte-addressable and accessed with ordinary loads and stores at near-DRAM speed. That combination created a powerful new capability for CPU programs — fine-grained RECOVERABILITY. A program can persist its data structures directly as it runs, so after a crash or power loss it resumes from a consistent state rather than starting over. Meanwhile, GPUs are a dominant computing platform whose massive parallelism many applications crave, and plenty of them would love BOTH: the GPU's speed AND persistent memory's durability — a persistent key-value store, a checkpointing training job, a recoverable graph search. But GPU kernels simply had no way to reach persistent memory.
What people did

Have the GPU compute, then let the CPU persist the results

The workaround, which the paper calls CPU-Assisted Persistence (CAP), is to keep the persistence on the CPU. The GPU runs its kernel and computes results in its own memory; those results are then transferred to the CPU's memory; and finally CPU threads persist them to PM, flushing their caches to make the data durable. This works with today's hardware — the CPU has the instructions to flush to PM and the OS knows how to manage it — and it is how a GPU-accelerated application would persist its output today.
Why it fell short

Routing persistence through the CPU is slow, coarse, and wasteful

CAP fails on three fronts. First, it cannot guarantee persistence at FINE granularity from within a GPU kernel, so a programmer cannot naturally build recoverable data structures that persist as the kernel computes. Second, it LOSES the GPU's parallelism: the whole point of a GPU is hundreds of thousands of threads, but persistence is bottlenecked on the CPU, which must flush data one 64-byte cache line at a time and cannot hide the latency the way a GPU can. Third, it suffers WRITE-AMPLIFICATION: often a GPU kernel updates only a FRACTION of a data structure, but which fraction is known only after the kernel computes it, so CAP conservatively transfers and persists far more data than changed — up to 39 times more for a persistent key-value store. The root cause is that GPUs have no persistent memory onboard, no instruction to flush their caches to PM, and non-coherent caches, so they cannot persist results while computing.
This paper's idea

Let the GPU kernel persist directly, with no new hardware

This paper (Indian Institute of Science — Pandey, Kamath, Basu) builds GPM (GPU with Persistent Memory), which lets a GPU kernel access and persist to PM-resident data structures DIRECTLY, at byte granularity, from inside the kernel, WITHOUT the CPU or OS and WITHOUT any hardware modification. It maps the persistent memory (Optane, sitting beside the CPU's DRAM) into the GPU kernel's virtual address space using Unified Virtual Addressing, so the kernel reads and writes PM with ordinary loads and stores. To guarantee durability it cleverly REPURPOSES two existing mechanisms: a system-scoped fence (built for CPU-GPU synchronization) already guarantees a GPU's writes reach system memory — where the NVM lives — and GPM selectively disables Intel's DDIO feature (which would otherwise leave those writes sitting in the CPU's volatile cache) so the fence completes only when data truly reaches PM. On top, libGPM offers logging and checkpointing primitives, including Hierarchical Coalesced Logging, which mirrors the GPU's thread hierarchy so hundreds of thousands of threads log lock-free at pre-computed offsets. GPM runs 2.7 to 5.8x faster than a CPU with the same PM, up to 85x faster than CPU-assisted persistence, with about 14 lines of code changed.

Why this problem is worth solving

Recoverability — surviving a crash without losing work or corrupting data — is one of the most valuable properties a system can have, and persistent memory made it cheap and fine-grained for the first time. Understanding that capability comes first. A

PERSISTENT MEMORY (PM), a form of non-volatile memory like Intel Optane, keeps its contents across a power failure yet is byte-addressable and accessed with ordinary loads and stores at near-DRAM latency (roughly 3 to 10 times DRAM). This lets a program keep its actual data structures — a hash table, a list, a key-value store — directly in memory that survives a crash, and update them in place. The catch is that a mere write is not durable: modern CPUs cache writes in volatile caches, so the order data reaches the cache can differ from the order it reaches PM, and a crash mid-update could leave a half-written, corrupt structure. A PERSIST operation makes a write durable by flushing the relevant cache lines to PM and waiting for them to land, and techniques like logging use persists to make a group of updates atomic — all-or-nothing across a crash. With persists and logging, a program recovers to a consistent state and resumes, instead of restarting from scratch

is what PM unlocked. It matters because a GPU-accelerated service that could persist directly would get durability at the speed of memory, not disk — but GPUs were locked out.

The reason they were locked out, and what GPM has to overcome, is that a GPU cannot do the one thing PM requires. A

the persist operation needs to flush volatile caches to PM and confirm the data landed, and GPUs are simply not built for it: there is no persistent memory on the GPU, no instruction to flush a GPU cache to PM, and GPU L1 caches are not even kept coherent. So a GPU cannot persist its results while it computes them. The fallback, CPU-Assisted Persistence (CAP), keeps the GPU out of the persistence loop entirely: the GPU computes, the results are copied to the CPU’s memory, and CPU threads flush them to PM. This works but is triply wasteful — it cannot persist at fine granularity from the kernel (blocking natural recoverable data structures), it discards the GPU’s parallelism because the CPU becomes the serial bottleneck, and it moves and persists EXTRANEOUS data, because the kernel updates only part of a structure and the CPU, not knowing which part, conservatively persists much more. Fixing this requires giving the GPU a way to persist directly

is the core obstacle. The prize is a new class of applications; the barrier is that GPUs have no persist instruction and no onboard NVM.

What people did before, and where it broke

CPU-with-PM applications. Persist data structures directly from the CPU with loads, stores, and cache flushes. Powerful and recoverable, but it leaves the GPU’s parallelism entirely unused for persistence-and-compute workloads.
CPU-Assisted Persistence, filesystem (CAP-fs). The GPU computes, and the CPU writes results to a PM-backed file via the filesystem. Works, but pays OS and filesystem overhead on top of losing GPU parallelism.
CPU-Assisted Persistence, memory-mapped (CAP-mm). Same idea but memory-map the PM file to skip some OS overhead (about 2x faster than CAP-fs). Still CPU-bound, still coarse-grained, still write-amplified.
GPUfs (GPU filesystem calls). Let GPU kernels invoke filesystem system calls. But every thread in a block must participate behind barriers, so fine-grain persistence deadlocks, and repeated syscalls plus the filesystem are slow; it also caps file sizes and cannot do byte-grain in-kernel persistence.
The gap. No approach let a GPU kernel persist to PM directly, at byte granularity, in-kernel, using the GPU’s parallelism and without new hardware. That is the gap GPM fills.

How GPM persists from a GPU kernel

Map PM into the kernel, repurpose a fence to persist, and log with the GPU's parallelism

The gap: PM persistence is beyond the GPU

GPUs have no way to flush their caches to persistent memory, so they ship results to the CPU to persist (CAP), losing the GPU's parallelism, bottlenecking on the CPU, and persisting extraneous data because the CPU does not know which fraction the kernel changed.
↓ →
Map persistent memory into the kernel address space new

GPM uses NVIDIA Unified Virtual Addressing to map the CPU-side Optane persistent memory into a GPU kernel's virtual address space. The kernel then reads and writes PM-resident data structures with ordinary loads and stores, at byte granularity, without involving the CPU or OS.
↓ →
Repurpose the system-scoped fence to persist new

GPUs have no persist instruction, but a system-scoped fence (built to synchronize the CPU and GPU) already guarantees a thread's writes reach SYSTEM memory before it completes — and the NVM lives in system memory alongside DRAM. GPM repurposes that fence to provide persist semantics, no new hardware needed.
↓ →
Selectively disable DDIO so writes reach PM new

Intel's DDIO feature caches writes from I/O devices (including the GPU) in the CPU's volatile last-level cache by default, so the fence would complete while data still sits in volatile cache. GPM selectively disables DDIO when persistence is needed, so the fence completes only once the data truly reaches PM.
↓ →
Persist only what changed, in-kernel new

Because the GPU now persists at byte granularity as it computes, it persists ONLY the entries it actually updated — whose identities are known only at compute time — instead of conservatively persisting a whole structure. This avoids the write-amplification that CAP suffers, moving far less data.
↓ →
Log lock-free with Hierarchical Coalesced Logging new

For atomic transactions, libGPM's Hierarchical Coalesced Logging mirrors the GPU's warp/threadblock/grid hierarchy: each thread logs to a UNIQUE pre-computed offset (no locks, since entry sizes and thread counts are known at launch), and log entries are striped so the GPU's hardware coalescer merges a warp's writes into a single cache-line store — faster and gentler on NVM endurance.

Implemented with no hardware changes on a stock server: Intel Xeon Gold CPUs, Intel Optane NVDIMMs, and an NVIDIA Titan RTX GPU over PCIe. libGPM adds logging, checkpointing, and primitives; GPMbench provides nine GPU workloads across transactional, checkpointing, and native-persistence classes.

The idea

Repurpose an existing mechanism for persistence — no new hardware. The cleverest part of GPM is that it invents no new instruction and modifies no hardware; it notices that mechanisms built for OTHER purposes already provide what persistence needs. A GPU has no flush-to-PM instruction, but its system-scoped fence, built to synchronize the CPU and GPU, already guarantees that writes reach system memory before completing — and since the NVM sits in system memory, that fence IS a persist. The one gap, that Intel’s DDIO caches device writes in the volatile last-level cache, GPM closes by selectively turning DDIO off when persistence is needed. Achieving a new capability by repurposing existing hardware primitives, rather than demanding new silicon, is a move the site rewards repeatedly — the attention-overlap page reads an SM-scheduling register to force co-location, the microsecond-BFT page turns RDMA single-writer regions into an anti-lying primitive, and the virtual-domain page uses address-space IDs to switch domains without a flush. Each finds a lever already in the hardware and bends it to a purpose it was not designed for.

Cut the CPU out of the loop and persist at the source. The whole inefficiency of CAP is that the CPU stands between the GPU and PM: the data must travel to the CPU, and the CPU, serial and latency-bound, must persist it. GPM removes the CPU from the persistence path so the GPU persists its own results directly, at byte granularity, using its own parallelism — and, crucially, persists ONLY the parts it changed, since it knows them at compute time. Removing an intermediary that had become the bottleneck, and moving only the data that matters, is the same cut-out-the-middleman, avoid-the-extraneous-copy discipline the site’s serverless-data-transfer page applies when it lets functions exchange data directly instead of through the controller, and the coherent-NIC page applies when it bypasses the host CPU. The lesson repeats: when a helper on the critical path becomes the bottleneck, the fix is often to let the source do the work itself.

Design the data structure to match the hardware’s parallelism. Logging is normally a serial, locked append — poison for a GPU’s hundreds of thousands of threads. Hierarchical Coalesced Logging makes logging as parallel as the GPU by mirroring the GPU’s own execution hierarchy: because each thread’s log-entry size and the total thread count are known at kernel launch, every thread’s log offset can be computed in advance, so all threads insert concurrently with NO locking; and by striping entries across cache-line units, a warp’s writes merge in the hardware coalescer into a single store. Shaping a data structure so that a massively parallel machine can operate on it lock-free, by exploiting statically-known structure and the hardware’s own coalescing, is the same exploit-the-parallel-structure discipline the site’s graph-processing and concurrency pages use when they compute conflict-free indices or exploit the execution hierarchy. The log is fast because it was built in the shape of the machine.

Persisting from inside a GPU kernel

From a CPU detour to direct, parallel persistence

Enabling fine-grain persistence from a GPU kernel. Numbers from the paper's Xeon plus Optane plus Titan RTX evaluation.

Step 1 of 5

Why the CPU detour is bad

To persist, a GPU computes, ships results to the CPU, and the CPU flushes them to PM one 64-byte cache line at a time. This loses the GPU's parallelism, bottlenecks on the serial CPU, and persists extraneous data — because the CPU does not know which fraction the kernel changed, it conservatively persists much more.

write-amplification reaches 39 times for a persistent key-value store, where only a few of millions of entries change
the changed entries are known only after the kernel computes them, so the CPU cannot persist selectively

Why it works

GPUs have no instruction to flush caches to PM → a system-scoped fence already guarantees writes reach system memory, where the NVM lives → repurposing it (plus selectively disabling DDIO) provides persist semantics with no new hardware.
CAP routes persistence through the serial, latency-bound CPU → GPM persists directly from the kernel using the GPU’s hundreds of thousands of threads → persistence runs at the GPU’s parallelism instead of the CPU’s bottleneck.
The kernel updates only a fraction of a structure, known only at compute time → in-kernel byte-granular persistence writes only the changed entries → the extraneous data movement (up to 39 times) that CAP suffers is eliminated.
Serial locked logging cannot use a GPU’s parallelism → Hierarchical Coalesced Logging computes each thread’s log offset in advance and stripes entries for hardware coalescing → hundreds of thousands of threads log lock-free, up to 6.1 times faster.
Direct byte-granular access makes recovery cheap → a crashed kernel can read the persisted state and resume partial work → restoration costs at most 19% of operation time, versus restarting from scratch.

What it costs (trade-offs and scope)

It leans on specific stock-hardware behaviors. GPM depends on NVIDIA UVA mapping PM over PCIe, on the system-scoped fence reaching system memory, and on the ability to selectively disable Intel DDIO — a particular Xeon-plus-Optane-plus-NVIDIA combination; other platforms may lack these exact levers.
Disabling DDIO is a workaround with a cost. DDIO normally speeds device I/O by caching in the last-level cache, so turning it off during persistence trades some performance; the future eADR feature removes the need (and adds up to 13x), so part of GPM’s benefit is transitional.
Optane’s quirks bottleneck the persist path. Writes cross PCIe to CPU-side Optane, which reaches full bandwidth only for sequential, 256-byte-aligned writes (about 13 GB/s dropping to 0.72 GB/s for random or unaligned access), so workloads with sparse, unaligned updates get low bandwidth — GPM helps most when the access pattern suits Optane.
Recoverable programming is still work. Building recoverable structures needs logging or checkpointing (libGPM helps, about 14 lines), pointer-based structures cannot be checkpointed, and read-only data must be kept in fast GPU memory rather than slow PM — the programmer must decide what belongs on PM.
The biggest wins are workload-dependent. Apps with high write-amplification under CAP or many small persists gain the most (up to 85x); apps that persist a whole structure equally (checkpointing, no write-amplification) gain from parallelism (11 to 18x) but not from avoiding extraneous data, and a workload with little to persist gains little.

The evaluation

GPM on a stock server (four Intel Xeon Gold 6242 CPUs, an NVIDIA Titan RTX with 72 SMs, eight 128 GB Intel Optane NVDIMMs, PCIe 3.0), on nine GPMbench workloads across three classes (transactional key-value store and database, iterative checkpointing for DNN training and simulations, and native-persistence graph and array kernels), against CPU-with-PM, CPU-assisted persistence (filesystem and memory-mapped), and GPUfs, plus fault-injection recovery testing.

How a GPU program uses persistent memory

Property	CPU with PM (no GPU)	CPU-assisted persistence (CAP)	GPM (this work)
Who persists
Granularity
Write-amplification
Parallelism in persisting
Result

Persisting from the CPU wastes the GPU's parallelism; CPU-assisted persistence adds a serial CPU bottleneck and moves up to 39x extraneous data. GPM lets the GPU kernel persist directly at byte granularity, using its own parallelism and persisting only what changed — 2.7 to 5.8x over a CPU with the same PM, up to 85x over CPU-assisted persistence, with about 14 lines of code and no hardware changes.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

Read the scope precisely. GPM leans on specific stock-hardware behaviors — NVIDIA UVA mapping Optane over PCIe, a system-scoped fence that reaches system memory, and the ability to selectively disable Intel DDIO — so it targets a particular Xeon-plus-Optane-plus-NVIDIA combination and may not transfer to platforms lacking those levers. Disabling DDIO to guarantee persistence trades some I/O performance, and the future eADR feature would remove that need (and add up to 13x), so part of the benefit is transitional. The persist path crosses PCIe to CPU-side Optane, which reaches full bandwidth only for sequential, 256-byte-aligned writes (dropping from about 13 to under 1 GB/s for random or unaligned access), so workloads with sparse, unaligned updates see low bandwidth. Recoverable programming still takes work — logging or checkpointing (about 14 lines with libGPM), no pointer-based checkpoints, and keeping read-only data off slow PM. And the largest wins are workload-dependent: apps with high write-amplification under CAP gain the most (up to 85x), while equal-data checkpointing gains from parallelism (11 to 18x) but not from avoiding extraneous data. Within that scope the lessons are sharp and general: achieve a new capability by repurposing existing hardware primitives rather than demanding new silicon; cut the intermediary that became the bottleneck out of the critical path and persist at the source; move only the data that actually changed; and shape a data structure to match the machine’s parallelism so it can be operated on lock-free.

Background

Repurpose the hardware you have, go direct, and shape the log to the machine

This paper sits in the site’s storage and GPU clusters and rhymes with several ideas covered elsewhere on this site. Its cleverest move is to achieve a new capability — persistence from a GPU — by REPURPOSING existing hardware primitives rather than adding silicon: a system-scoped fence built for synchronization becomes a persist, and selectively disabling DDIO makes it reach PM. That is the same find-a-lever-already-in-the-hardware discipline the attention-overlap page uses when it reads an SM register to force co-location, the microsecond-BFT page uses when it turns an RDMA single-writer region into an anti-equivocation primitive, and the virtual-domain page uses when address-space IDs enable a flush-free switch. Its structural move is to cut the CPU out of the persistence loop and let the GPU persist directly at the source, moving only the data that changed — the same cut-out-the-middleman, avoid-the-extraneous-copy discipline the serverless-direct-transfer page applies when functions exchange data without the controller and the coherent-NIC page applies when it bypasses the host CPU, and the same persist-or-move-only-what-matters instinct behind the sparsity and value-profiler pages. And its logging library shapes the data structure to the machine: Hierarchical Coalesced Logging computes conflict-free offsets from the statically-known thread hierarchy and exploits the hardware coalescer, the same exploit-the-parallel- structure, lock-free-by-construction move the graph-processing and concurrency pages make. It is also candid about the substrate’s real cost model — Optane’s alignment and sequentiality quirks — the same know-your-hardware discipline the interconnect-contention and coherent-NIC pages insist on. Together they make GPM a case study in unlocking a new design point with the hardware already on the server.

If you remember only three things

1. Persistent memory (like Intel Optane) keeps its contents across a power failure but is byte-addressable at near-DRAM speed, so a program can persist its data structures directly and recover from a crash instead of restarting — a capability CPU programs gained but GPUs could not use. GPUs have no persistent memory onboard, no instruction to flush their caches to PM, and non-coherent caches, so they cannot persist while computing. The workaround, CPU-Assisted Persistence, ships GPU results to the CPU to persist — which loses the GPU's parallelism, bottlenecks on the serial CPU, and persists up to 39 times more data than changed, because the CPU does not know which fraction the kernel updated.
2. GPM lets a GPU kernel access and persist to PM DIRECTLY, at byte granularity, in-kernel, with NO hardware changes. It maps the PM into the kernel's address space via Unified Virtual Addressing so the kernel uses loads and stores, and it guarantees durability by REPURPOSING a system-scoped fence (which already ensures writes reach system memory, where the NVM lives) and selectively disabling Intel's DDIO (so the fence completes only when data truly reaches PM). Because it persists in-kernel, it writes only the entries that changed, avoiding write-amplification; and libGPM's Hierarchical Coalesced Logging lets hundreds of thousands of threads log lock-free at pre-computed offsets, coalesced into single cache-line writes.
3. GPM runs 2.7 to 5.8x faster than a CPU with the same persistent memory and up to 85x faster than CPU-assisted persistence, with HCL logging up to 6.1x faster and about 14 lines of code, recovering correctly from injected crashes. The transferable lessons: achieve a new capability by repurposing existing hardware primitives instead of demanding new silicon; cut the intermediary that became the bottleneck out of the critical path and act at the source; move or persist only the data that actually changed; and shape a data structure to match the machine's parallelism so it can be operated on lock-free.

Check yourself

Check your understanding

1. GPUs have no instruction to flush their caches to persistent memory, yet GPM makes GPU writes durable with no hardware changes. Predict how.
2. CPU-Assisted Persistence often persists far more data than the GPU kernel actually changed (up to 39x). Predict why, and how GPM avoids it.
3. Logging for transactions is normally a serial, locked append, which would cripple a GPU's hundreds of thousands of threads. Predict how Hierarchical Coalesced Logging makes it parallel.
4. GPM lets the GPU persist directly instead of routing results through the CPU. Beyond avoiding extraneous data, predict the other major benefit of removing the CPU from the persistence path.

Sources

GPM: Leveraging Persistent Memory from a GPU (author PDF, IISc) — ASPLOS 2022 (full text read).
ACM DL: GPM (DOI 10.1145/3503222.3507758) — published version.
GPM-ASPLOS22 artifact (source and GPMbench) — implementation and workloads.