MemSnap μCheckpoints: A Data Single Level Store for Fearless Persistence

A single level store lets a program persist its data just by keeping it in memory, with the OS checkpointing that memory to disk — no files, no read/write calls — but prior single level stores checkpoint the whole process at once, too coarse for a database to durably commit one small transaction, so they end up bolting a write-ahead log back on and betraying the design; this paper (University of Waterloo — Emil Tsalapatis, Ryan Hancock, Rakeeb Hossain, Ali Jose Mashtizadeh) presents MemSnap, whose micro-checkpoints persist a single thread's write transaction concurrently with everyone else's, using per-thread dirty-page tracking and copy-on-write to snapshot in-flight data without blocking — cutting persistence latency 4.5 to 30 times below file random I/O and speeding SQLite 5 times and RocksDB 4 times over the prior single level store Aurora.

The argument in one glance

Problem

A store that persists your memory directly, if only it could commit one transaction

The dream of a single level store is to erase the boundary between memory and storage: you keep your data in ordinary in-memory data structures, and the operating system quietly makes it durable by checkpointing your memory to disk, so there are no files to open, no read and write calls, no serialization. This is a genuinely appealing way to persist application state, and systems like EROS, Aurora, and TreeSLS have revived it. But there is a catch that shows up the moment you want database-grade durability. These systems checkpoint at a coarse granularity — the whole process address space at once, periodically. A database transaction, though, needs to durably commit a single, specific update on demand, and know it survived a crash. A coarse, whole-program checkpoint cannot express "make just this transaction durable, now," so the single level store, by itself, cannot provide transactional persistence.
What people did

Bolt a write-ahead log back onto the single level store

To get the fine-grained, on-demand durability a transaction needs, prior single level stores combine their coarse checkpoints with a conventional file-based persistence mechanism — most often a write-ahead log, where each transaction appends its change to a log file that is flushed to disk before the transaction is acknowledged. The coarse checkpoint handles the bulk state, and the log fills the transactional gap.
Why it fell short

Adding the log betrays the whole point of the single level store

The write-ahead log reintroduces exactly what the single level store was supposed to abolish. You are back to a file API — appending to a log, flushing files, managing on-disk formats — and the application must now maintain both its in-memory structures and their logged shadow, which is the double bookkeeping and complexity the single level store promised to remove. So the design undermines itself: the elegant "just persist my memory" model is bolted to the very file-based machinery it set out to replace, and you get the costs of both worlds. What is missing is a way for the store itself to make a single transaction's memory updates durable, on demand, without falling back on files.
This paper's idea

Persist one transaction's dirty memory concurrently, with micro-checkpoints

MemSnap completes the single level store vision by eliminating the need for the write-ahead-log API. Its core construct is the micro-checkpoint: a persist operation that durably writes to disk just the memory a single write transaction changed, on demand, and concurrently with other threads still running. To know precisely what to persist, MemSnap adds a per-thread dirty-set tracker in the kernel that records, page by page, which memory each thread modified — using the hardware page-fault mechanism so tracking is cheap. To let a transaction persist without stopping everyone, it uses copy-on-write in virtual memory: while a micro-checkpoint's pages are being written to disk, any thread that tries to modify one gets a private copy transparently, so the in-flight snapshot stays consistent and no one blocks. Applications keep their normal locking for concurrency; MemSnap handles the atomicity of persistence. The result is persistence 4.5 to 30 times faster than file random I/O, and SQLite and RocksDB run 5 and 4 times faster than with files and the prior store Aurora, while keeping full ACID.

Why this problem is worth solving

The single level store is a decades-old idea worth reviving because it removes a whole category of programmer burden.

Normally, persisting application data means translating between two very different worlds: your program’s in-memory data structures — objects, trees, hash tables, pointers — and a file’s flat byte stream on disk. You serialize structures to bytes to save them and parse bytes back to structures to load them, you call read and write, you manage file formats and layouts, and you keep the two representations in sync. This translation is pervasive, error-prone, and slow. A single level store abolishes it by making memory itself durable: the operating system treats your address space as the persistent store and checkpoints it to disk, so the data structures you already have in memory are the data structures that survive a crash, with no files and no serialization in between. That is why the idea keeps resurfacing — it promises to make persistence as simple as keeping data in memory. The obstacle to realizing it fully has always been granularity and control: making the whole address space durable periodically is easy, but making a specific, small update durable exactly when the application demands it, which is what real transactional systems require, is hard. Closing that gap is what would turn the single level store from an appealing idea into a practical replacement for file-based persistence, which is the prize MemSnap is after.

The gap that keeps single level stores from being self-sufficient is precisely transactional durability.

Databases and other transactional systems have a strict requirement: when a transaction commits, its effect must be durable — guaranteed to survive a crash — before the commit is acknowledged, and this must happen for each individual transaction, on demand, without waiting. A single level store that checkpoints the entire process address space cannot meet this. A whole-address-space checkpoint is coarse and periodic: it captures everything at once, at intervals, so it cannot say “make just the bytes this one transaction touched durable, right now,” and a crash between checkpoints loses all the transactions since the last one. To recover the missing fine-grained, on-demand durability, prior single level stores fall back on a write-ahead log — a file to which each transaction appends its change and flushes before committing — so the log provides per-transaction durability while the coarse checkpoint provides the rest. But the log is a file-based mechanism, so it drags the entire file API back into a system designed to eliminate it: the application again serializes changes into a log format, issues file flushes, and maintains its data both in memory and in the log. The single level store’s central promise — persist your memory, forget about files — is broken by the log it needs to be correct. The real fix is not to add a log but to give the store itself the ability to durably persist one transaction’s memory updates on demand, which is the capability MemSnap builds.

What people did before, and where it broke

File-based persistence with serialization. Persist data by serializing in-memory structures to files and reading them back. Where it broke. The constant translation between memory objects and flat file bytes is complex, error-prone, and slow, and it is exactly the burden the single level store aims to remove.
Single level stores with coarse checkpoints (EROS, Aurora, TreeSLS). Persist by checkpointing the whole process address space to disk periodically. Where it broke. Coarse, periodic checkpoints cannot durably commit an individual transaction on demand, so they lose recent transactions on a crash and are insufficient for transactional workloads by themselves.
Single level store plus a write-ahead log. Combine the coarse checkpoint with a file-based log for per-transaction durability. Where it broke. The log reintroduces the file API and its serialization and flushing, plus double bookkeeping of data in memory and in the log — undermining the single level store’s whole reason for existing.
Atomic msync and similar page-flush primitives. Use the OS to flush mapped memory atomically. Where it broke. These require contiguous regions, do not track a per-thread dirty set, and are not microsecond-scale, so they cannot isolate one thread’s transaction or persist exactly what it changed quickly enough.
The gap. No system let the store itself persist just the memory an individual write transaction dirtied, on demand and concurrently with other threads, so a single level store could provide transactional durability without a write-ahead log or any file API.

How MemSnap works

Persisting one transaction's dirty memory on demand, without blocking others

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Shrink the checkpoint from the whole program to a single transaction. The defining move of MemSnap is to make the unit of persistence tiny and on-demand instead of huge and periodic. Prior single level stores checkpoint the entire address space, which is why they cannot commit one transaction and must borrow a write-ahead log; MemSnap’s micro-checkpoint instead persists only the memory a single write transaction dirtied, exactly when the transaction commits. This directly fills the transactional gap: a commit is a persist of just that transaction’s changed pages, durable before it returns, with a crash losing nothing that was persisted. By making the checkpoint fine-grained and demand-driven, MemSnap gives the single level store the one capability it was missing, so the store no longer needs any file-based mechanism to be correct. Turning persistence into small, frequent, concurrent checkpoint operations that individual actors trigger, rather than a coarse global snapshot, is the same reframing the persistent concurrent-checkpointing work covered elsewhere on this site is built on, making a checkpoint something granular and overlapping with ongoing work rather than a stop-the-world event. MemSnap applies that reframing at the transaction level to complete the single level store.

Use copy-on-write in virtual memory to persist without pausing anyone. A per-transaction checkpoint is only useful if it does not stall the rest of the program while its data is written to slow disk, and the danger is that a thread modifies a page that is mid-flight, corrupting the snapshot being persisted. MemSnap resolves this with a classic virtual-memory trick applied precisely: while a micro-checkpoint’s pages are being written out, they are protected, and any thread that writes to such a page is transparently given a private copy through copy-on-write, so the version on its way to disk stays frozen and consistent while the writer proceeds on its own copy. Neither side waits: the persistence sees a stable snapshot, and the application never blocks on the I/O. Correctly using copy-on-write so that a snapshot can be read or written while the live data keeps changing underneath, without either corrupting the other, is the same delicate virtual-memory discipline the work on correct copy-on-write covered elsewhere on this site is built on, getting the page-table and protection interactions exactly right so a shared page can safely diverge under concurrent access. MemSnap’s unified mechanism bridges the VM and storage subsystems to make this transparent to the application.

Track dirty memory cheaply and per-thread, so each transaction knows just what to persist. To persist exactly one transaction’s changes, the store must know exactly which pages that transaction touched, and it must learn this without slowing the common case of ordinary writes. MemSnap does it with the hardware the processor already provides: it keeps region pages read-only so the first write to a page traps as a lightweight minor page fault, and its custom handler appends that page to the writing thread’s own dirty list before making the page writable. Because the list is per-thread, one transaction’s dirty set is isolated from another’s, so a commit persists precisely and only that transaction’s pages; and because it rides the existing fault path, tracking adds little overhead and does not disturb swapping or the normal fault handler. This transparent, hardware-assisted dirty tracking is what makes fine-grained per-transaction persistence practical rather than merely possible. Leaning on the memory-management hardware and the operating system’s page-fault machinery to observe and manage persistence at page granularity is the same systems foundation the persistent-memory management work covered elsewhere on this site is built on, using the virtual-memory layer as the place to track and control what must be made durable. MemSnap’s per-thread twist is what turns that tracking into transaction-level isolation.

Following a transaction through MemSnap

Committing one database transaction as a micro-checkpoint

A trace, drawn from the paper's design, of how MemSnap durably persists a single write transaction concurrently with others.

Step 1 of 5

A thread modifies data in a persistent region

A database thread performs a write transaction — updating some records held as in-memory structures inside a MemSnap persistent region. It takes its normal page-granularity locks on the nodes it touches, just as it would to avoid data races with other threads.

The application's locking is only for concurrency; MemSnap will handle making the changes durable.

Why it works

Per-transaction micro-checkpoints fill the durability gap directly. Coarse whole-address-space checkpoints cannot commit one transaction on demand → but persisting just a transaction’s dirty pages when it commits is exactly transactional durability → so the store needs no write-ahead log to be correct.
Per-thread dirty tracking isolates each transaction’s changes. Persisting a transaction requires knowing precisely what it touched → but a per-thread dirty list records each thread’s modified pages separately → so a commit persists exactly and only that transaction’s pages.
Hardware page faults make tracking nearly free. Recording every write in software would be slow → but keeping pages read-only and catching the first write as a minor fault rides the existing hardware path → so dirty tracking adds little overhead and does not disturb swapping.
Copy-on-write lets persistence and execution overlap safely. A thread modifying a page being flushed would corrupt the snapshot or must block → but copy-on-write hands the writer a private copy transparently → so the in-flight checkpoint stays consistent while no thread waits.
Async persistence overlaps I/O with useful work. Waiting for a slow disk write to finish before continuing would kill throughput → but returning once the I/O is initiated and releasing locks lets other transactions run → so the system keeps concurrency high and beats file logging by several times.

What it costs (trade-offs and scope)

It requires kernel changes and a persistent-region programming model. MemSnap is not a library; it adds custom page-fault handling, a copy-on-write object store, and per-thread dirty tracking to the operating system, and applications must place their data in MemSnap persistent regions and use its persist API. Adopting it means integrating with the modified kernel, not just linking code.
Applications must still get their own concurrency right. MemSnap handles the atomicity of persistence, but the application is responsible for using page-granularity locking (or equivalent) so that a data-structure node is not simultaneously modified by multiple threads; the system’s guarantees rest on the application respecting these layout and isolation constraints.
Persistence granularity is the page. Dirty tracking and copy-on-write operate at page granularity, so a transaction that touches a few bytes still persists whole pages; the micro-checkpoints can be as small as a single page, but very small updates scattered across many pages could persist more than the bytes strictly changed.
It is faster than files but not free versus raw disk. MemSnap’s persistence is within about two times of direct disk I/O latency, so it retains the fundamental cost of writing to storage; its advantage is over file-based random I/O and logging, not over the physical device, and the copy-on-write and fault machinery add some overhead of their own.
Results are on specific databases and workloads. The 4.5-to-30-times latency and the 5-times-SQLite and 4-times-RocksDB throughput figures are for particular integrations and benchmarks against file APIs and Aurora; gains depend on the workload’s transaction sizes and access patterns, and other applications or storage devices could differ.

The evaluation

Persisting individual transactions from in-memory data

Property	Single level store + WAL	File-based random I/O	MemSnap
Persists one transaction on demand
Concurrent with other threads
Persists only the changed memory
Keeps the single-level-store model
Reported performance

A comparison drawn from the paper. MemSnap is evaluated against file-based random I/O, direct disk I/O, and the prior single level store Aurora, integrated with SQLite, RocksDB, and PostgreSQL; the 4.5-to-30-times latency, 5-times-SQLite, and 4-times-RocksDB figures are the paper's reported results, with full ACID retained.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The full paper text was available (an author copy from the University of Waterloo group site, retrieved via the Internet Archive when the host was unreachable), so this account of the mechanism — single level stores and why coarse checkpoints force a write-ahead log, MemSnap’s micro-checkpoints that persist a single transaction’s dirty memory on demand, the per-thread hardware-assisted dirty-set tracking via read-protected pages and minor page faults, the copy-on-write snapshotting of in-flight checkpoints, and the persist API with asynchronous mode — follows the paper’s own description. The headline numbers are the paper’s reported results: persistence 4.5 to 30 times faster than file-based random I/O and within about 2 times of direct disk I/O latency, a 5 times SQLite throughput improvement over file APIs, and a 4 times RocksDB throughput improvement over the prior single level store Aurora, with full ACID retained across SQLite, RocksDB, and PostgreSQL integrations. Those figures are for specific databases, workloads, and baselines; results vary with transaction sizes and access patterns, and MemSnap requires kernel support and a persistent-region programming model. Exact per-benchmark numbers, the object-store and page-handling details, and the application isolation constraints should be read from the paper itself.

Background

Why the memory-storage divide is so costly to bridge

It is worth appreciating why persisting data has always been more painful than it seems. A program’s natural habitat is memory, where data lives as rich, pointer-linked structures the CPU manipulates directly; storage, by contrast, is a flat expanse of bytes with no notion of objects or pointers. Every time a program saves or loads data, it crosses this divide, flattening structures into bytes and rebuilding them, an impedance mismatch that pervades software and accounts for a large share of both bugs and slowness in data-intensive systems. Decades of work — databases, serialization libraries, object stores, persistent memory — are in one way or another attempts to soften this divide, and the single level store is the most radical: it proposes to erase it entirely by making memory itself durable, so the structures never have to be flattened at all. The reason this is hard, and why it took a system like MemSnap to make it practical for transactions, is that memory and storage differ not just in format but in speed, granularity, and failure semantics, so bridging them means reconciling a fast, byte-addressable, volatile medium with a slow, block-oriented, durable one while preserving transactional guarantees. That effort to reconcile the memory and storage worlds so applications can treat persistence as a natural extension of memory is the same ambition the persistent-memory-for-databases work covered elsewhere on this site pursues, closing the gap between where data lives and where it durably rests. MemSnap’s contribution is to close it finely enough that a single transaction can cross it, alone, on demand.

If you remember only three things

1. A single level store persists application data by checkpointing the program's memory directly, eliminating file APIs and the serialization between in-memory structures and on-disk bytes. But existing single level stores (EROS, Aurora, TreeSLS) checkpoint the whole address space at a coarse, periodic granularity, which cannot durably commit an individual transaction on demand, so they bolt on a file-based write-ahead log for transactional durability — reintroducing the file API, serialization, and double bookkeeping the single level store was meant to abolish, and undermining the design.
2. MemSnap completes the single level store by eliminating the write-ahead-log API with micro-checkpoints: a persist operation that durably writes just the memory a single write transaction changed, on demand and concurrently with other threads. It tracks each thread's dirty pages cheaply using read-protected pages and minor page faults (giving per-transaction isolation of exactly the changed pages), and it uses copy-on-write in virtual memory so that a thread modifying a page currently being flushed gets a transparent private copy — keeping the in-flight snapshot consistent while no thread blocks. Applications keep their own locking for concurrency; MemSnap handles the atomicity of persistence, and asynchronous persist overlaps the disk I/O with continued execution.
3. MemSnap's persistence is 4.5 to 30 times faster than file-based random I/O and within about 2 times of direct disk I/O latency; integrated into SQLite, RocksDB, and PostgreSQL it retains full ACID while running 5 times faster than file APIs (SQLite) and 4 times faster than the prior single level store Aurora (RocksDB). It requires kernel support (custom page-fault handling, a copy-on-write object store, per-thread dirty tracking) and a persistent-region programming model, persists at page granularity, and its gains depend on transaction sizes, access patterns, and the specific databases and baselines evaluated.

Check yourself

Check your understanding

1. Why do prior single level stores end up needing a write-ahead log, and why is that a problem?
2. What is a MemSnap micro-checkpoint?
3. How does MemSnap persist a transaction's memory without blocking other threads?
4. What are MemSnap's main results and requirements?

Sources

The paper is “MemSnap microCheckpoints: A Data Single Level Store for Fearless Persistence” by Emil Tsalapatis, Ryan Hancock, Rakeeb Hossain, and Ali Jose Mashtizadeh (RCS Lab, University of Waterloo), published at ASPLOS 2024 (Volume 3), DOI 10.1145/3620666.3651334.

This explainer is based on the paper’s full text, obtained from an author copy on the University of Waterloo Reliable Computer Systems group site (rcs.uwaterloo.ca), retrieved via an Internet Archive snapshot when the host was unreachable. The account of the mechanism — single level stores that persist application data by checkpointing memory rather than through file APIs, and why prior systems (EROS, Aurora, TreeSLS) checkpoint at a coarse granularity that forces them to add file-based write-ahead logs; MemSnap’s micro-checkpoints that persist an individual write transaction’s dirty memory on demand and concurrently with other threads via the persist operation (with per-region and per-thread scoping and an asynchronous mode); the hardware-assisted per-thread dirty-set tracking that keeps region pages read-only and records the first write to each page as a minor page fault onto a thread-local list; the unified page-granularity copy-on-write mechanism that transparently gives a private copy to any thread modifying a page currently being flushed, so in-flight checkpoints stay consistent without blocking; and the fast copy-on-write object store providing the persisted objects — follows the paper’s own description. The reported results cited here are the paper’s own: persistence 4.5 to 30 times faster than file-based random I/O and within about 2 times of direct disk I/O latency, micro-checkpoints as small as a single page, a 5 times SQLite throughput improvement over file APIs, and a 4 times RocksDB throughput improvement over Aurora, with full ACID retained across SQLite, RocksDB, and PostgreSQL. Exact per-benchmark numbers, the page-handling and object-store details, and the application isolation constraints should be read from the paper itself; an artifact is available (github.com/etsal/memsnap-artifact).

MemSnap &#956;Checkpoints: A Data Single Level Store for Fearless Persistence