CTXNL: A Software-Hardware Co-designed Solution for Efficient CXL-Based Transaction Processing

Zhao Wang, Yiqi Chen, Cong Li, Yijin Guan, Dimin Niu, Tianchan Guan and 3 others

Databases that span many machines are slow because every access to remote data crawls through the network stack; a fast new interconnect called CXL lets machines share memory directly with hardware cache coherence, but that coherence turns out to be so expensive it erases the win; this work from Peking University and Alibaba builds CtXnL, which notices that a database's actual record data only needs to be made consistent at the moment a transaction commits — not on every read and write — so it lets software decide when to synchronize records while keeping strict coherence only for the locks and indexes, cutting over 95 percent of cross-node coherence traffic and roughly doubling throughput.

The argument in one glance

Problem

Databases spanning many machines are slow because remote data crawls through the network

A transaction database that outgrows one machine is split across many, so a transaction often needs data living on another node. Fetching it goes over the network, whose protocol stack is heavy — much of the time is spent just copying bytes between buffers — and because message passing gives no shared memory, systems cannot even cache remote data, so every remote access pays the full network cost again. As the fraction of remote accesses rises, throughput collapses.
What people did

Share memory directly over CXL, with hardware cache coherence

A fast new interconnect, Compute Express Link, offers a way out: it lets many machines share a pool of disaggregated memory directly, with ordinary loads and stores, at sub-microsecond latency — an order of magnitude faster than the network. Better still, its newest version keeps the shared memory cache-coherent in hardware, so a value written on one node is automatically kept consistent on every other, just like memory within a single machine.
Why it fell short

That cross-node coherence is so expensive it erases the win

Maintaining strict coherence across machines is costly. Every shared load or store may have to signal other nodes' caches — invalidate or fetch a line — and each such signal takes multiple roundtrips over the fabric, making a coherent access nearly twice as slow as a plain memory read. Worse, a single centralized filter must track every cached line across all nodes, and it does not scale: for a modest rack it would need billions of entries. Naively run on CXL, transaction systems slow down by more than two-fold on average.
This paper's idea

Synchronize record data only at commit, and keep strict coherence just for metadata

CtXnL notices that a database's memory splits into two kinds. Metadata — locks, indexes, timestamps — genuinely needs strict coherence. But records, the actual data, follow transactional rules under which a write only becomes final at commit, so in-flight transactions can tolerate temporary incoherence and aborted ones need none at all. CtXnL gives records a loose protocol: writes stay node-private until the transaction calls a synchronize operation at commit, at which point the hardware makes them coherent. This cuts over 95 percent of cross-node coherence traffic and roughly doubles throughput.

Why this problem is worth solving

The promise and the trap both live in what CXL adds on top of a plain memory pool, so it is worth being precise about it.

Compute Express Link (CXL) is an interconnect that lets machines attach to a shared pool of memory disaggregated from any one processor, and access it with normal loads and stores at cacheline granularity and sub-microsecond latency — far faster than a network. Its newest version adds hardware cache coherence: when several nodes cache the same shared line, the hardware keeps them consistent using a protocol like the one inside a multicore chip, with a back-invalidation channel that lets the memory pool invalidate a stale copy in some node’s cache. This is convenient — software sees ordinary coherent shared memory — but it is expensive across machines. Enforcing a single global order means that a store cannot complete until every other node has observed it, and each such notification takes multiple roundtrips over the fabric, so a coherent access runs nearly twice as slow as a raw memory read. And to know which nodes cache which lines, the pool keeps a centralized snoop filter that must be inclusive of every processor’s caches — a structure that grows with the cluster and quickly becomes a scaling bottleneck.

The reason this cost is avoidable is a mismatch between what the hardware guarantees and what a database actually needs.

A transaction system’s memory accesses fall into two classes. Metadata — the locks, timestamps, indexes, and latches that coordinate concurrency — must be strictly coherent, because the correctness of synchronization depends on every node agreeing on their exact state. But record accesses, the reads and writes of the actual data values, are wrapped inside transactions that commit all-or-nothing. Under standard transactional consistency models such as strict serializability, a record’s writes are only finalized at the commit point; while a transaction is still running, other transactions are permitted to see temporarily incoherent values, and a transaction that ultimately aborts never needs its writes made visible at all. The concurrency-control algorithm, not the memory hardware, is responsible for producing the correct order. So for the large fraction of accesses that touch records, strict hardware coherence on every load and store is simply doing work the database did not ask for — the coherence only truly matters at commit. The catch is that hardware cannot tell a record access from a metadata access without the application telling it.

What people did before, and where it broke

Partition the data and use the network (RDMA). Shard the database across nodes and fetch remote records with fast network primitives. Where it broke. Even optimized network stacks spend most of their time copying payloads between buffers, and with no coherent caching every remote access pays that cost anew — throughput drops by more than half as remote accesses rise.
Adopt CXL memory sharing as-is (CXL-vanilla). Put the shared data structures on the CXL pool and let the hardware keep them coherent. Where it broke. Strict cross-node coherence signals other caches on nearly every access and leans on a centralized snoop filter, slowing transaction systems by more than two-fold on average and failing to scale past a rack.
Rebuild the network RPC layer on CXL (CXL-RPC). Keep the partitioned design but replace network RPCs with faster CXL ones, so existing systems migrate with little change. Where it broke. It still carries RPC software overhead and cache-flushing costs, running several times slower than raw memory, and it only works when the workload partitions cleanly — cross-partition transactions wreck it.
Optimize the concurrency-control algorithm in software (SILO-style). Cut unnecessary coherence traffic from metadata writes by rethinking the algorithm. Where it broke. It helps with metadata but does nothing for the signaling cost of the record accesses themselves, and being software-only it cannot touch the snoop-filter scaling problem baked into the hardware.
Use transactional-memory hardware. Let the processor track speculative reads and writes and resolve conflicts in hardware. Where it broke. These schemes are confined to a single socket and demand deep changes to processor architecture, making them impractical for a multi-node disaggregated pool.
The gap. Every approach either kept paying the network cost, paid CXL’s full coherence cost, fixed only the software half, or required unshippable hardware. What was missing was a way to keep strict coherence where it is truly needed and skip it where transactional consistency already suffices — without changing processors or the CXL standard — which is exactly what CXL-based transaction processing needs.

How CtXnL works

Decoupling record coherence from memory access, synchronizing only at commit

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Strict coherence is a guarantee the records never asked for, so pay it only at commit. The whole design rests on seeing that vanilla CXL enforces a single global order on every access, while a database’s records only need to become consistent when a transaction commits — in between, transactional consistency positively permits other transactions to see stale values, and an aborted transaction’s writes should vanish entirely. So CtXnL gives records a loose protocol whose writes stay node-private and are made coherent only when the application signals commit, leaving strict coherence to the metadata that genuinely needs it. Recognizing that a strong, expensive guarantee is overkill and enforcing it only at the moments a real conflict or boundary demands is the same insight the key-value store concurrency work covered elsewhere on this site is built on, which finds static write serialization overly conservative and grants a thread exclusive access only during the instant a genuine write conflict is in flight.

Move the whether-and-when of coherence out of hardware and into software. Hardware cannot tell a record access from a metadata access, and teaching it to would mean changing processors or the CXL standard — both off the table. CtXnL’s answer is to keep using CXL’s native coherence datapath as the mechanism, but hand the decision of when to fire it to software, exposed as a handful of operations the transaction code calls at exactly the right moments. The database, which already knows when a transaction commits or aborts, is far better placed than the hardware to say when records must be synchronized. Letting software own the timing while the hardware merely provides the mechanism is the same division of labor the distributed-training communication work covered elsewhere on this site adopts, where a compiler decides when each transfer should happen and the hardware simply carries it out.

Replace the one central bottleneck with a per-node structure, and filter the common case. CXL’s coherence needs a snoop filter that tracks every cached line across all nodes, centralized at the memory pool, which balloons and throttles as the cluster grows. CtXnL instead gives each node its own back-filter that tracks only its own caches, so the tracking work is distributed and scales with the cluster, and it fronts the whole translation path with a small on-chip filter that, because stores are rare and private views usually stay in cache, reports “nothing here” on nearly every read — letting the common-case access skip the expensive lookup entirely. Distributing a structure that used to be a single shared chokepoint so it scales with the machine is the same move the NUMA page-table work covered elsewhere on this site makes, replicating a per-node copy of a hot shared structure to escape the contention of one central bottleneck.

Following a transaction through CtXnL

Running one transaction on shared CXL memory

Tracing a transaction shows how records and metadata are treated differently.

Step 1 of 5

A transaction touches records and a lock

A transaction on one node reads and writes several data records and acquires a lock to coordinate. In vanilla CXL, every one of these accesses — the lock and the records alike — pays strict cross-node coherence, signaling other caches on each.

Records are the bulk of the accesses, so most of the coherence cost is spent on them.

Why it works

Records only need to be consistent at commit, not on every access. Vanilla CXL enforces a single global order on every shared load and store → but transactional consistency finalizes a record’s writes only at commit, tolerating temporary incoherence in flight and none at all for aborts → so making records coherent only at commit skips the vast majority of cross-node signaling with no loss of correctness.
A node-private view lets a write cost nothing across nodes. A coherent store cannot complete until it has signaled every other node’s cache → but writing into a private per-node copy, invisible elsewhere, needs no signal at all → so a record write becomes a purely local operation until the transaction chooses to publish it at commit.
A cheap filter keeps the common-case read fast. Checking for a private view on every read would add a memory lookup to the critical path → but stores are rare and views usually sit in cache, so a small on-chip filter reports “none” almost always → so reads skip the lookup and go straight to shared memory in the overwhelming common case.
A distributed per-node filter replaces the unscalable central one. CXL’s single centralized snoop filter must track every node’s cached lines and becomes a bottleneck as nodes multiply → but giving each node its own filter that tracks only its own caches spreads the work → so tracking scales with the cluster instead of choking on it.
Leaving the decision to software keeps the hardware unchanged. Teaching hardware to tell records from metadata would need new processor or CXL features → but exposing loose coherence as a small set of software-invoked operations over CXL’s existing datapath needs neither → so the whole scheme runs on standard hosts and an unmodified CXL standard.

What it costs (trade-offs and scope)

It is not measured end-to-end on real CXL hardware. An FPGA prototype supplies the latencies, but a channel-dependency deadlock prevented full-system runs, so the headline results come from a simulator calibrated to that prototype plus a modeled ideal chip; CXL 3.0 memory sharing is not broadly available yet.
The prototype flatters the gain; on ideal hardware it shrinks. The roughly-doubled throughput is on a slow FPGA whose coherence signaling is inflated; on an ideal ASIC CXL the improvement is about 1.36 times, because the win scales with how expensive coherence is — so as CXL matures the relative benefit falls.
Correctness now rests on the software. The application must classify each allocation and call the synchronize operation at commit and the withdraw at abort; a misplaced or missing call breaks consistency. Coherence responsibility moves from the hardware to the developer.
It adds hardware and memory overhead. A new agent at the memory node, a cuckoo-hash translation table and its buffers in memory, two per-node filters, and a threefold expansion of the physical address space; the translation table can fail to insert and trigger a blocking resize.
The benefit is largest for memory-bound, store-light workloads. Gains are bigger on the memory-insensitive benchmark (93 percent) than the compute-heavy one (77 percent), and the whole design assumes stores are rare so the private-view structures stay sparse; a store-heavy record workload would stress them.
Metadata still pays full coherence. Only records are decoupled; locks, indexes, and timestamps keep strict CXL coherence with all its signaling and central-filter costs.
It is shown in one research database framework. The store is built on a single OLTP research codebase; whether the approach carries over to production databases is untested.

The evaluation

Sharing data for distributed transaction processing

Property	Network (RDMA)	CXL-vanilla	CXL-RPC	CtXnL
Data sharing model		Shared memory, strict coherence
Record coherence cost	None, but slow network		RPC software overhead
Tracking structure	Not applicable		Not applicable
Cross-node coherence traffic	Not applicable	High	Not applicable	Cut over 95 percent
Relative throughput	CtXnL 6.47 times higher

Ways of sharing data across nodes for transaction processing, evaluated on a DBx1000-based key-value store over TPC-C and YCSB with six concurrency-control algorithms, using an FPGA CXL prototype for latencies and a calibrated simulator for full-system runs. CtXnL cuts over 95 percent of cross-node coherence traffic and beats vanilla CXL memory sharing by up to 2.08 times (about 1.36 times on ideal-ASIC CXL), network-based systems by 6.47 times, and CXL-RPC by up to 7.3 times on poorly-partitioned workloads.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

CtXnL is a clean idea, well matched to its problem: it observes that CXL’s strict hardware coherence is overkill for the record data of a transaction database, which only needs to be consistent at commit, and so decouples record coherence from memory access — writes stay node-private until a software-invoked synchronize at commit — while keeping strict coherence for the metadata that truly needs it, cutting over 95 percent of cross-node coherence traffic and roughly doubling throughput without changing processors or the CXL standard. Read the numbers with their frame. Nothing is measured end-to-end on real CXL hardware — an FPGA prototype supplies latencies and a calibrated simulator supplies the results — and the headline 2.08 times shrinks to about 1.36 times on an ideal future chip, because the win scales with how costly coherence is. Correctness now depends on the developer classifying allocations and placing the synchronize and withdraw calls correctly; the design adds a hardware agent, translation tables, and address-space expansion; the gains are largest on memory-bound, store-light workloads; and metadata still pays full coherence. The durable idea is the reframing: coherence is a service the hardware was providing whether or not the database wanted it, and once software can say when records actually need to be consistent, most of the cost simply disappears.

Background

Why a new memory technology needs new systems software

It helps to see CtXnL as one instance of a recurring pattern. A new memory technology arrives promising to break an old bottleneck — here, CXL promises to replace the slow network between machines with fast, coherent shared memory — but the raw hardware, used as it ships, rarely delivers the promise, because its default behavior is tuned for a general case that a specific application does not need. CXL keeps everything strictly coherent because that is the safe, simple default; a database knows most of its accesses do not require it. The gains come not from the hardware alone but from co-designing a thin software layer that tells the hardware what the application actually needs, so the hardware stops doing unnecessary work. The compute-in-memory compiler work covered elsewhere on this site tells the same story from a different corner: a powerful new class of memory-centric chips that is nearly useless until a co-designed software stack lets programs express what should run where. The broader lesson is that emerging memory and interconnect technologies are only as good as the systems software built to steer them, and the interesting research is increasingly in that steering.

If you remember only three things

1. Distributed transaction databases are slow because fetching remote data over the network is dominated by protocol-stack overhead, and message passing offers no coherent caching; the CXL interconnect promises a fix by letting machines share disaggregated memory directly with hardware cache coherence at sub-microsecond latency, but maintaining that strict cross-node coherence is so expensive — multiple fabric roundtrips to signal other caches on nearly every access, plus a centralized snoop filter that cannot scale — that naively using CXL slows transaction systems by more than two-fold on average.
2. CtXnL observes that a database's memory splits into metadata (locks, indexes, timestamps) that genuinely needs strict coherence and records (the data values) that, under transactional consistency, only need to be consistent at commit — in-flight transactions tolerate temporary incoherence and aborted ones need none — so it gives records a loose hybrid primitive: a record write becomes a Local-Store to a node-private "view" that signals no one, reads are served from the view or the shared copy (fronted by a cheap on-chip filter that skips the lookup in the common case), and a software-invoked Global Synchronize at commit merges the private writes and invalidates stale copies, with Withdraw discarding them on abort, all over CXL's native datapath with no processor or specification changes and a distributed per-node filter replacing the centralized one.
3. Built on a DBx1000 key-value store and evaluated over TPC-C and YCSB with six concurrency-control algorithms (via an FPGA CXL prototype for latencies and a calibrated simulator for full-system runs), CtXnL cuts over 95 percent of cross-node coherence traffic and improves throughput up to 2.08 times over vanilla CXL memory sharing (about 1.36 times on an ideal ASIC), 6.47 times over network-based systems, and up to 7.3 times over CXL-RPC on poorly-partitioned workloads — but it is not measured end-to-end on real hardware, its gain shrinks as CXL matures, it shifts coherence correctness onto the developer, adds hardware and memory overhead, helps most on memory-bound store-light workloads, and still pays full coherence for metadata.

Check yourself

Check your understanding

1. Why does naively adopting CXL memory sharing perform far worse than expected for transaction processing?
2. What is the key insight that lets CtXnL avoid most of that coherence cost?
3. How does a record write avoid cross-node coherence traffic in CtXnL?
4. Which statement best reflects the limits of CtXnL?

Sources

The paper is “CtXnL: A Software-Hardware Co-designed Solution for Efficient CXL-Based Transaction Processing” by Zhao Wang, Yiqi Chen, Cong Li, Yijin Guan, Dimin Niu, Tianchan Guan, Zhaoyang Du, Xingda Wei, and Guangyu Sun, published at ASPLOS 2025, DOI 10.1145/3676641.3716244. The authors are at Peking University, the DAMO Academy of Alibaba Group (Hupan Lab), and Shanghai Jiao Tong University.

This explainer is written from the full 18-page text. Everything above comes from the paper’s own sections: the introduction and background on network-induced overheads in distributed transactions and on CXL memory sharing (Global Fabric Attached Memory, the back-invalidation coherence channel, and the snoop filter); the motivation quantifying vanilla CXL’s slowdown (2.51 times on average, with remote cache signaling nearly twice as slow as raw access at over 850 nanoseconds on the prototype, and the centralized snoop filter needing on the order of a billion entries at rack scale) and the insight that record accesses tolerate temporary incoherence while metadata needs strict coherence; the CtXnL primitive design (Local-Load, Local-Store, Global Synchronize, and Withdraw) with its litmus-test semantics; the architectural supports (the CtXnL hardware agent with its view memory shim, cuckoo-hash view address table, and the VMS filter and back filter that distribute what the central snoop filter did, plus the user library for per-allocation primitive selection and the runtime thread for table resizing); and the evaluation on a DBx1000 key-value store over TPC-C and YCSB, reporting the reduction of over 95 percent of cross-node coherence traffic and throughput improvements of up to 2.08 times over vanilla CXL (1.36 times on ideal ASIC), 6.47 times over network-based systems, and up to 7.3 times over CXL-RPC on cross-partition-heavy workloads.