Cloud databases like Aurora and PolarDB split compute nodes from storage, and persistent memory (byte-addressable, durable, dense) would make a perfect fast remote buffer tier — except PM's write bandwidth is scarce and, worse, if the shared PM node has to run a CPU to apply the database's logs it saturates long before the memory fills, defeating the whole point of consolidating PM across many tenants; PilotDB (USTC and Alibaba) makes the compute nodes computation-heavy but DATA-LIGHT and pushes all the thinking onto them — each commit ships PHYSICAL log entries that already carry the exact remote address to write, so the PM node just does dumb one-sided-RDMA memory copies with no parsing, and reads are optimistic and self-verified with a rare CPU-free fix-up — so a PM node saturates at just ONE core instead of 8 to 16, cutting PM write bandwidth up to 55%, matching a monolithic all-local database to within a few percent, recovering a crashed compute node 168x faster, and serving 91% more throughput per dollar.
The argument in one glance
-
Problem
Persistent memory should slot perfectly into a disaggregated database — but it does not
Cloud-native databases like Amazon Aurora and Alibaba PolarDB are DISAGGREGATED: the compute nodes that run SQL and transactions are separated from the storage nodes that hold the data, so each can be scaled and shared independently. Persistent memory (Optane) looks like an ideal new tier for this world — it is byte-addressable, survives crashes, denser than DRAM (a 512 GB PM DIMM where DRAM offers 128 GB), and reachable with ultra-low latency over RDMA, sitting in cost between DRAM and SSD. It should make a superb fast, durable remote buffer pool. Yet persistent memory had not actually been incorporated into the disaggregated database architecture, because two stubborn costs get in the way.
-
What people did
Either let compute nodes drive the PM, or let the PM node apply the logs
Two natural designs exist. ONLOAD (as in pDPM): let the compute nodes manage the remote PM directly, writing their dirty pages and logs straight to it. OFFLOAD (as in AsymNVM and LegoPM): put a CPU on the PM node and have IT apply the database's logs, updating pages in place. One puts the compute nodes in charge; the other puts the memory node in charge. Both are reasonable ways to keep a disaggregated database's data durable in remote persistent memory.
-
Why it fell short
Both waste the two scarce PM resources — bandwidth and the shared node's CPU
The onload design wastes PM's scarce WRITE BANDWIDTH: compute nodes flood the PM with whole dirty pages and logs, and PM bandwidth is far below DRAM's, so it becomes the bottleneck. The offload design wastes the PM node's scarce CPU: making the memory node parse logical logs, apply them, and manage concurrency saturates its cores (8 to 16 of them) long before the PM capacity is filled — and the entire point of persistent memory is to CONSOLIDATE one big shared PM node across many tenants, which only works if per-tenant operations are cheap. On top of this, the fastest way to touch remote memory, ONE-SIDED RDMA (which uses no remote CPU at all and is 65.6% faster than an RPC), cannot be used by these designs, because they need the remote CPU to interpret logical logs and enforce consistency. So the tier that should have been a clean win is blocked by bandwidth waste and CPU saturation.
-
This paper's idea
Keep the shared PM node dumb, and push all the thinking onto the compute nodes
PilotDB (USTC and Alibaba) makes the compute nodes computation-heavy but DATA-LIGHT (small local DRAM, all data living in remote PM) and moves every bit of intelligence onto them, so the shared PM node stays nearly CPU-free. Its central mechanism, COMPUTE-NODE-DRIVEN LOGGING (CDLog), has each compute node track the exact remote PM layout and, at commit, ship PHYSICAL log entries that already carry the target PM address of each changed 256-byte mini-page — via two one-sided RDMA writes. The PM node then just does dumb DMA memory COPIES to the addresses the logs name, no parsing, on a single core. Reads are OPTIMISTIC: a compute node reads a page and its applied-log markers in one RDMA, checks freshness locally, and on the rare stale case (about 0.25%) pulls and applies the missing logs itself. The PM node saturates at ONE core instead of 8 to 16, PM write bandwidth drops up to 55%, throughput matches a monolithic all-local database within a few percent, and a crashed compute node recovers 168x faster.
Why this problem is worth solving
Disaggregation is the organizing principle of modern cloud
databases, and the reason it works is that it lets the cloud
provider scale and share each resource independently. The
architecture is a
a database split
across tiers connected by fast network. COMPUTE nodes run the SQL
parser, the query engine, and the transaction logic, but keep only
a small local buffer of pages. STORAGE nodes hold the durable data
and are shared. Newer designs add a middle tier of MEMORY nodes
that offer a large remote buffer pool, so a compute node with
little local DRAM can still keep a big working set close by over
the network. The payoff is elasticity and consolidation: you can
add compute without adding storage, pack many tenants onto shared
memory and storage nodes, and reuse the same infrastructure across
databases — which is exactly the economics that make cloud
databases affordable
The obstacle is that persistent memory has two scarce resources
that the obvious designs squander, and the scarcest is not the one
you would guess. PM’s write bandwidth is limited, but the deeper
trap is the
the fact that
the economic case for persistent memory rests on CONSOLIDATION —
one large, shared PM node serving many database tenants at once, so
its cost is amortized. That only pays off if each tenant’s
operations demand almost nothing from the PM node’s CPU. But the
straightforward way to keep remote data consistent is to run a CPU
on the memory node that receives each tenant’s logs, parses them,
and applies them to the right pages — and that CPU work does not
scale: it saturates the PM node’s cores long before its large PM
capacity is used up, so a node that could have held the data of
dozens of tenants instead runs out of CPU serving a handful. The
paper measures alternatives needing 8 to 16 cores at peak, and
shows multi-tenant throughput collapsing by half when the PM node’s
CPU saturates
What people did before, and where it broke
- Onload designs (pDPM). Compute nodes manage the remote PM directly, writing dirty pages and logs to it. Simple and CPU-free on the PM side, but it floods the scarce PM write bandwidth with whole pages, so bandwidth becomes the bottleneck.
- Offload designs (AsymNVM, LegoPM). A CPU on the PM node applies the database’s logical logs, updating pages in place. This cuts bandwidth by sending logs not pages, but it saturates the PM node’s CPU (about 8 cores at peak) — and under multi-tenant load that CPU saturation costs roughly 50% of throughput, wrecking consolidation.
- RPC-based coordination (PilotDB-RPC baseline). Coordinate remote PM access through a request-response protocol to the PM node. Correct, but every operation needs the remote CPU, so it takes 16 cores to reach peak throughput and pays 65.6% higher latency than one-sided RDMA — the paper builds this to show the cost of involving the PM node’s CPU at all.
- Remote-DRAM memory tiers (LegoBase). Use ordinary remote DRAM as the middle tier. Fast, but DRAM is not persistent (so it cannot also be the durable log) and far less dense and more expensive than PM, so it needs much more hardware for the same buffer.
- The gap. No design put persistent memory into a disaggregated database while keeping the shared PM node nearly CPU-free AND sparing its write bandwidth — exploiting one-sided RDMA end-to-end. PilotDB does, by making the compute nodes drive everything and the PM node do only dumb memory copies.
How PilotDB works
Three tiers, all intelligence on the compute nodes, dumb copies on the PM node
-
Three tiers: computation-heavy but data-light compute nodes new
Compute nodes get ample CPU but only a small local DRAM buffer (about 5% of the database); shared PM nodes hold a large remote buffer pool (about 50%) in persistent memory; storage nodes keep the rest on SSD/HDD. The compute nodes do all the work but hold almost no data locally, which slashes DRAM provisioning and — because their state lives in remote PM — lets them recover and migrate almost instantly.
-
Keep the shared PM node CPU-free with one-sided RDMA new
Because one PM node must serve many tenants, it must do almost no per-tenant CPU work. PilotDB routes as much as possible through one-sided RDMA — where the requester's network card reads or writes the remote memory directly with no involvement from the remote CPU — which is also 65.6% faster than an RPC to that CPU.
-
Compute-node-driven physical logging (CDLog) new
Each compute node keeps a page metadata table mapping every buffered page to its exact PM address, so it always knows the remote layout. At commit it inspects a dirty-bit vector to find the changed 256-byte mini-pages and writes PHYSICAL log entries: metadata (the target PM address and the log sequence number) plus the new mini-page content, shipped in two batched one-sided RDMA writes.
-
The PM node just copies to the address the log names new
A background log-merger daemon on the PM node polls the incoming log ring buffers and, for each entry, RDMA-copies the new mini-page content to the exact PM address the log already carries — no parsing of logical operations, just dumb DMA copies — updating an applied-LSN marker per mini-page. With doorbell batching it runs on a single core.
-
Optimistic, self-verified reads (log-pull) new
A compute node cannot blindly read a page (the PM node may not have applied recent logs yet), but waiting on the PM node is slow. So it reads the page AND its applied-LSN vector in one RDMA, compares against its own record of what should be applied, and only on a mismatch — a rare 0.25% of reads — pulls the missing log entries via one-sided RDMA and applies them itself. The read path never touches the PM node's CPU.
-
Lock-free coordination and an inclusive buffer new
Ring buffers with strictly-incrementing log sequence numbers plus a global applied-LSN marker let compute nodes coordinate without locks (the fast producer never overruns the slower consumer, and checksums catch partial RDMA failures). Every page cached in a compute node also has a copy in PM (cheap, since PM is inexpensive), which makes compute-node crash recovery nearly instant.
The idea
Put the intelligence where the CPU is cheap, and keep the shared tier dumb. The economic engine of persistent memory is consolidation — one big PM node amortized across many tenants — which collapses the moment that shared node has to think per tenant. Compute nodes, by contrast, are replicated per tenant and CPU-rich, so PilotDB makes an asymmetric bargain: move ALL the intelligence (knowing the PM layout, deciding what changed, resolving target addresses, applying stale logs) onto the compute nodes, and leave the PM node doing only dumb DMA copies. The lever that makes this possible is one-sided RDMA, which lets a compute node write or read remote PM with zero involvement from the PM node’s CPU. This is the do-the-work-where-it-is-cheap discipline the site keeps rewarding — the noisy-quantum page pushing correction onto abundant classical compute, the serverless page draining memory with idle-time work — applied to a two-tier asymmetry: spend the cheap, plentiful compute-node CPU lavishly to save the scarce, shared PM-node CPU and bandwidth.
Send a fully-resolved instruction, not one that needs interpreting. Traditional databases log LOGICAL operations, and an offload design makes the PM node parse and apply them — which is exactly the CPU work that saturates it. PilotDB’s CDLog instead has the compute node, which already knows the remote layout through its page metadata table, emit PHYSICAL log entries that carry the exact target PM address of each changed mini-page. The receiver does not need to understand anything; it copies the bytes to the address the message names. This is the same pre-resolve-the-address, move-the- name-with-the-payload move the site meets in the persistent-memory KV-store page (merge by writing to known pointers) and the address-indirection family — by doing the address resolution on the CPU-rich side, the dumb side is reduced to a memcpy. Physical, self-describing messages are what let a one-sided RDMA copy replace a CPU-bound log apply.
Be optimistic, verify cheaply, and pay only for the rare exception. A compute node reading a remote page risks reading a stale one, because the PM node may not have applied the latest logs — and the safe-but-slow answer is to ask the PM node to confirm freshness, dragging its CPU back into the read path. PilotDB refuses that: it reads the page and its applied-log markers together in one RDMA, checks freshness itself, and only when it finds a stale page (just 0.25% of reads) does it pull and apply the missing logs on its own. The common case is a single CPU-free RDMA read; the rare case is handled by the abundant compute-node CPU. This is the optimistic-common-case-fast discipline the site sees whenever a system bets on the likely case and keeps a cheap check plus a correct fallback for the unlikely one — here it is what keeps the PM node’s CPU entirely off the read path.
A commit and a read, without waking the PM node’s CPU
Where offload burns CPU, and how PilotDB avoids it
Persisting and reading data in remote PM. Numbers from PilotDB's evaluation.
Step 1 of 5
The offload trap
The tempting design gives the PM node a CPU that receives each tenant's logs, parses the logical operations, and applies them to pages. It saves bandwidth (logs, not pages) but the parsing and applying saturate the PM node's cores — about 8 for one tenant at peak, and under many tenants the CPU runs out and throughput halves. The point of one big shared PM node was to serve many tenants cheaply, and this defeats it.
an RPC-coordinated version needs 16 cores and pays 65.6% higher latency than one-sided RDMA PM capacity goes unused because the CPU gives out first — consolidation fails
Why it works
- PM’s scarcest resource under consolidation is the shared node’s CPU, not its capacity → PilotDB moves all intelligence to the CPU-rich compute nodes and leaves the PM node doing dumb copies → the PM node saturates at one core, so it can actually serve many tenants.
- The compute node already knows the remote PM layout via its page metadata table → it emits physical log entries carrying the exact target address → the PM node applies a log as a memcpy, with no parsing, over one-sided RDMA.
- Sending whole dirty pages floods PM’s limited write bandwidth → CDLog sends only the changed 256-byte mini-pages as logs → PM write bandwidth falls 40.8% to 55.5% versus an offload design.
- Confirming a read is fresh would drag the PM node’s CPU onto the read path → optimistic log-pull reads page and applied-markers in one RDMA and self-verifies, fixing the rare 0.25% stale case itself → the read path never touches the PM node’s CPU.
- Compute nodes hold almost no data locally because it all lives in durable PM → a crashed compute node has little to reload → it recovers 168x faster and migrates agilely.
What it costs
- The PM node still needs a core, and replicas add more. Even the background log-merger needs one core, and providing availability means running PM-node replicas, whose log replication and takeover add CPU and coordination overhead beyond the single-core ideal.
- The inclusive buffer has a write cost. Requiring every compute-node-cached page to also exist in PM means an extra PM write whenever a page is brought into the remote buffer; the paper argues this is acceptable because PM is cheap and misses are infrequent, but on write-heavy, high-miss workloads it accumulates.
- It leans on specific PM hardware. The design assumes byte-addressable, RDMA-reachable, low-latency PM, and the 256-byte mini-page is tied to Optane’s media unit — and Optane has since been discontinued, with CXL-based substitutes discussed but not evaluated.
- The buffer-ratio advantage shrinks on huge databases. With a 1 TB database and only 100 GB of PM buffer, throughput falls to 86.9-91.5% of the monolithic ideal (versus 90-97% at 200 GB) as more requests miss to slow storage — the win depends on the PM buffer covering enough of the working set.
- Narrow evaluation and no prefetching. Results center on a single compute node with a skewed access pattern, page fetches are reactive on miss with no proactive prefetching, and some prior systems could not be compared for lack of a compatible build.
The evaluation
A MySQL-based PilotDB prototype on Sysbench, TPC-C, and a production Alibaba workload (200 GB and 1 TB databases), against a monolithic MySQL-ideal (100 GB local DRAM plus 512 GB local Optane), remote-DRAM LegoBase, PM-offload LegoPM, an RPC variant, Aurora, and PolarDB, measuring throughput, PM bandwidth, PM-node CPU, latency, recovery, multi-tenant density, and cost.
Putting persistent memory into a disaggregated database
| Property | Onload (pDPM): compute nodes drive PM | Offload (LegoPM): PM node applies logs | PilotDB (CDLog) |
|---|---|---|---|
| Who does the log work | |||
| PM node CPU at peak | |||
| PM write bandwidth | |||
| RDMA style | |||
| Multi-tenant density | Bandwidth-limited |
Check yourself
Check your understanding
1. Persistent memory has limited write bandwidth, yet the paper argues the harder obstacle to using it in a disaggregated database is the PM node's CPU. Predict why the CPU, not the capacity or bandwidth, is the binding constraint.
2. PilotDB has compute nodes emit PHYSICAL log entries (carrying the exact target PM address) rather than logical ones, so the PM node just copies bytes to the named address. Predict why this shift is what lets a one-sided RDMA copy replace a CPU-bound log apply.
3. On reads, PilotDB fetches a page and its applied-log markers in one RDMA and checks freshness itself, only fixing the rare 0.25% of stale reads on the compute node. Predict why this optimistic approach beats asking the PM node to guarantee freshness.
4. PilotDB's compute nodes are deliberately "data-light" — holding almost no data locally, with everything in remote PM — and this yields 168x faster recovery of a crashed compute node. Predict the connection.
Sources
- ACM DL: Persistent Memory Disaggregation for Cloud-Native Relational Databases — ASPLOS 2023 (the PilotDB system; full text read via the ACM PDF; no arXiv preprint).