Cxlalloc: Safe and Efficient Memory Allocation for a CXL Pod

A CXL pod is a small group of machines wired to one shared pool of memory they can all read and write, and an ordinary memory allocator breaks there for three reasons — the shared memory may not be kept cache-coherent by hardware, it is touched by several separate processes at once, and with more machines a crash mid-operation is more likely; this paper (UT Austin, UIUC, NVIDIA — Newton Ni, Yan Sun, Zhiting Zhu, Emmett Witchel) presents cxlalloc, the first allocator that handles all three, keeping coherence in software (with a memory-based compare-and-swap for hardware that has none), keeping pointers meaningful across processes, and recovering from partial failures without blocking.

The argument in one glance

Problem

Sharing one pool of memory across machines breaks the humble memory allocator

Compute Express Link (CXL) is a way for a processor to read and write memory that lives across a fast link, most often over PCIe. It is becoming a practical way to build cheaper, disaggregated memory in data centers. A newer capability is letting several machines connect to a single CXL memory module and share it at cacheline granularity; a small group of hosts (say 8 to 16) sharing one such module is called a CXL pod. Applications that want to allocate and share memory dynamically in a pod need a memory allocator — the familiar piece of software behind malloc and free that hands out chunks of memory and takes them back. But an allocator that works fine on one machine breaks in a CXL pod, for three reasons. First, the shared memory may not be kept cache-coherent by hardware across hosts, so the atomic operations allocators rely on to coordinate (like compare-and-swap) may not work between machines. Second, the shared memory is touched by several separate processes on different hosts at once, and allocators normally assume their bookkeeping lives inside one process with one address space. Third, with more machines involved, it becomes more likely that some thread or process crashes in the middle of an allocation, and that must not freeze everyone else. So the mundane act of allocating memory becomes genuinely hard in a pod.
What people did

Reuse shared-memory, persistent-memory, or existing CXL allocators

There are existing allocators to draw on. Shared-memory allocators let multiple processes share a heap. Persistent-memory allocators already use offset pointers and can replay a log to recover their metadata after a crash. And some recent allocators target CXL memory specifically, using lock-free data structures and reference counting to tolerate a thread dying.
Why it fell short

Each existing allocator misses at least one of the pod's three demands

None of these fully fits a CXL pod. Traditional allocators assume full hardware cache coherence, and adapting them to limited or absent coherence is hard because they interleave thread-local and global bookkeeping, so the metadata that needs coherence cannot be cleanly separated out. They also use ordinary absolute addresses as pointers and lean on the operating system to keep memory mappings non-overlapping and instantly visible — guarantees the OS can no longer give once different processes on different hosts share the memory, so a pointer made in one process may point somewhere else, or fault, in another. Persistent-memory allocators use offset pointers but still rely on the OS for those mapping guarantees, and many assume a total-failure model with a quiet recovery pause and lock-based or garbage-collected recovery that blocks live threads — unacceptable when a single crashed thread must not stall a highly-available system. Prior CXL allocators tolerate partial failure with reference counting, but reference counts need coherence, sit inside every allocation, and cause contention even on read-heavy workloads. What was missing was one allocator that handles limited coherence, cross-process sharing, and partial failure together.
This paper's idea

Separate the coherent metadata, keep pointers portable, and recover without blocking

Cxlalloc is a user-space memory allocator designed for CXL pods, and it addresses all three challenges at once through careful layout and new protocols. For coherence, it separates the metadata that truly needs hardware cache coherence into its own contiguous region and keeps the rest coherent in software, so it works whether the device offers full coherence, coherence in a small region, or none at all; for a device with no coherence, it introduces a memory-based compare-and-swap (mCAS) implemented in near-memory processing logic on an FPGA, which serializes atomic operations at the device so hosts can still synchronize. For cross-process sharing, it guarantees pointer consistency: offset pointers plus placing heap metadata and data at the same offsets in every process give spatial consistency (a pointer names the same physical memory everywhere), and a signal handler that installs memory mappings on demand plus a hazard-pointer protocol for safely reclaiming mappings give temporal consistency (a pointer made in one process can be dereferenced immediately in another). For partial failure, it uses lock-free data structures and a lightweight recovery protocol: each allocator operation begins by atomically updating eight bytes of state in place, like a single-element redo log, which is enough to idempotently redo an interrupted operation on recovery, so a crashed thread never blocks the live ones. The result is the first allocator appropriate for a CXL pod, with competitive performance — reaching up to 80 percent of its maximum allocation throughput even with no hardware coherence, using mCAS.

Why this problem is worth solving

CXL pods are a near-term way to pool memory across machines, and pooled memory is useless to applications without an allocator that works there.

Data centers waste enormous amounts of memory that is stranded on machines not using it, and CXL is emerging as a practical way to pool memory so it can be shared and provisioned flexibly. Hardware that lets several hosts share one CXL module — a pod — is arriving, and applications that run across a pod will want to allocate and free shared memory dynamically, just as any program allocates memory today. But allocation is foundational: almost every non-trivial application does it constantly, and if the allocator is slow, unsafe, or fragile in a pod, then nothing built on top can be fast, safe, or reliable there. So providing a correct, efficient allocator is a precondition for the whole idea of pod-shared memory being usable, not a side detail. The hard part is that a pod violates three assumptions the ordinary allocator was built on all at once, and they interact.

The three challenges are not independent tweaks; each one undermines a guarantee that single-machine allocators simply assume.

A single-machine allocator quietly relies on three things. It relies on hardware cache coherence, so that when one thread updates a shared counter or list with an atomic compare-and-swap, every other thread sees it; without coherence across hosts, those atomics do not work, and the allocator has no safe way to coordinate. It relies on the operating system to manage the address space within one process, so that memory mappings never overlap and become visible to every thread instantly; once different processes on different hosts share the memory, the OS can no longer promise this, and a pointer valid in one process may be garbage in another. And it relies, usually, on a total-failure model where if something crashes, everything restarts together, so recovery can pause the world; in a pod a single thread can die while everything else keeps running, and the allocator must not block those survivors or leave its structures inconsistent. Because these three broken assumptions pull in different directions — coherence wants metadata in special memory, sharing wants portable pointers, failure wants non-blocking recovery — solving them together, without one fix breaking another, is what makes pod allocation genuinely hard and worth a new design.

What people did before, and where it broke

Use a traditional shared-memory allocator. Let multiple processes share a heap with ordinary pointers and atomics. Where it broke. It assumes full hardware coherence and a single-process address space, so it fails when coherence is limited and when pointers must cross processes on different hosts.
Adapt a persistent-memory allocator. Reuse offset pointers and log-replay recovery from persistent memory. Where it broke. It still relies on the OS to keep memory mappings non-overlapping and visible, and typically assumes total failure with a blocking, quiescent recovery that stalls live threads.
Use a prior CXL allocator with reference counting. Tolerate a dead thread by reference-counting allocations to reclaim their memory. Where it broke. Reference counts require coherence, are embedded in every allocation, and cause contention even under read-heavy access, and they are hard to adapt to different coherence models.
Assume hardware coherence will just arrive with CXL 3. Wait for the specification’s inter-host coherence to be widely implemented. Where it broke. Given its cost and complexity, it is unclear that full inter-host coherence will be common; current sharing prototypes support none at all, so an allocator cannot depend on it.
The gap. No memory allocator satisfied all three pod constraints — limited coherence, cross-process sharing, and partial failure — together, without trade-offs like a fixed heap size or a capped allocation size.

How cxlalloc works

Making one shared pool of memory safely allocatable across hosts

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Synchronize on memory that hardware does not keep coherent. Cxlalloc’s coherence strategy is to stop assuming the hardware will make shared updates visible everywhere, and instead to shrink the coherence-dependent state and, where there is none, build a synchronization primitive at the memory itself. Coordinating threads over memory-centric hardware that lacks ordinary cache coherence, by putting the synchronization logic in or near the memory, is the same undertaking the work on software transactional memory for processing-in-memory systems (PIM-STM) covered elsewhere on this site pursues, providing correct concurrent access where the usual coherent-atomics assumption does not hold. Cxlalloc does this with a memory-based compare-and-swap in near-memory logic, so hosts can still coordinate even when the CXL device offers no coherence at all.

Free shared memory only when no one can still be using it. A recurring hazard in cross-process shared memory is reclaiming a mapping or object while another process still holds a pointer to it, which would turn that pointer into a dangling reference; cxlalloc’s hazard-pointer protocol exists precisely to prevent that. Guarding against freeing memory that is still reachable, so that a later dereference can never hit reclaimed storage, is the same principle the work on practical mitigation of temporal memory-safety violations (Vik) covered elsewhere on this site is built on, closing the window where a use-after-free could occur. Cxlalloc applies it to memory mappings across processes, publishing a hazard pointer before use so a mapping is only reclaimed once every process has released it.

Keep serving even when a participant dies. Cxlalloc’s failure design treats a crashed thread as a normal event a highly-available system must absorb, not a reason to stop the world: lock-free structures and a tiny in-place redo record let the rest of the pod continue and let recovery finish the dead thread’s work. Building a system that degrades around a partial failure rather than collapsing, so a single crash does not take everyone down, is the same idea the work on cooperative graceful degradation in containerized clouds covered elsewhere on this site develops, keeping the healthy majority running when one part fails. Cxlalloc guarantees that a thread crashing inside an allocator function never blocks the live application threads.

Following an allocation through cxlalloc

Allocating and sharing a block in a CXL pod

A trace, drawn from the paper, of how cxlalloc serves and shares an allocation across processes.

Step 1 of 5

The operation records its intent in eight bytes

A thread begins an allocation by atomically writing eight bytes of state in place, functioning as a single-element redo log. This record captures enough about the operation to be finished later if the thread crashes partway through.

This tiny in-place log is what makes recovery idempotent and non-blocking.

Why it works

Separating coherence-critical metadata spans all coherence models. Interleaving thread-local and global metadata forces the whole layout to need coherence → but cxlalloc pulls the coherence-dependent state into its own region → so the same allocator is correct under full, limited, or no hardware coherence.
mCAS synchronizes hosts with no coherence at all. Ordinary atomics need hardware coherence to be seen across hosts → but a memory-based compare-and-swap serialized in near-memory logic sidesteps that → so hosts can coordinate even on a device that offers no coherence.
Offset pointers make a pointer portable across processes. Absolute addresses mean different things in different address spaces → but a pointer as an offset into consistently-placed heap regions → names the same physical memory in every process, giving spatial consistency without the OS.
On-demand mappings plus hazard pointers give safe temporal consistency. A pointer from another process may reference memory not yet mapped, or freed → but a fault-driven signal handler installs the mapping and hazard pointers block premature reclamation → so the pointer can be dereferenced immediately and never dangles.
A tiny in-place redo log makes recovery non-blocking. Lock-based or garbage-collected recovery stalls live threads → but eight bytes of idempotent state plus lock-free structures let an interrupted operation be redone → so a crashed thread is cleaned up without freezing the survivors.

What it costs (trade-offs and scope)

The no-coherence mode needs special hardware and is slower. mCAS relies on near-memory processing logic (implemented on an FPGA) and an uncacheable memory region, and reaches about 80 percent of the maximum allocation throughput rather than matching cached compare-and-swap — a real, if modest, penalty for running without any hardware coherence.
It assumes the CXL device itself is reliable. Cxlalloc’s recovery depends on the shared memory keeping its contents across process crashes and OS reboots, which requires a reliable device (for example, backed by an independent power supply or batteries); it tolerates partial software failure, not failure of the memory device.
Cross-process consistency adds machinery and layout constraints. Providing pointer consistency requires offset pointers, a signal handler in every process, and a hazard-pointer reclamation protocol, and it constrains the layout (data regions must be contiguous in virtual address space) — complexity a single-process allocator avoids.
The mCAS region cannot be cached by the CPU. Because the near-memory logic cannot track or invalidate CPU caches, the device-biased region used for mCAS must be marked uncacheable, so accesses to it forgo caching — a performance constraint on exactly the region used for cross-host synchronization.
It targets small pods sharing a multi-headed module. The design is built for a CXL pod of roughly 8 to 16 hosts sharing a single memory module at cacheline granularity; much larger scales, different sharing topologies, or unreliable devices are outside its evaluated scope.

The evaluation

Allocating shared memory in a CXL pod

Property	Shared/PM allocators	Prior CXL allocator (ref-counting)	Cxlalloc
Works without full hardware coherence
Pointer valid across processes
Partial-failure tolerance
Recovery behavior
Throughput with no coherence

A comparison of memory allocators for a CXL pod. Cxlalloc is the first to satisfy all three pod constraints — limited coherence, cross-process sharing, and partial failure — using separated coherence metadata with mCAS for no-coherence devices, offset pointers with fault-installed mappings and hazard pointers for cross-process pointer consistency, and lock-free structures with a single-element redo log for non-blocking recovery, reaching up to 80 percent of maximum allocation throughput with no hardware coherence.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ copy hosted at UT Austin, of the ASPLOS 2026 paper). The system described is the paper’s: cxlalloc is a user-space memory allocator for a CXL pod (a group of hosts sharing CXL-attached memory) that addresses three challenges — limited inter-host hardware cache coherence, cross-process sharing, and partial failure. It separates the metadata needing hardware coherence into its own region and maintains the rest coherent in software; for devices with no coherence it uses a memory-based compare-and-swap (mCAS) implemented in near-memory processing logic on an FPGA. It provides pointer consistency across processes — spatial (a pointer names the same physical memory everywhere) via offset pointers and consistent offsets, and temporal (a pointer made in one process can be dereferenced immediately in another) via a signal handler that installs mappings and a hazard-pointer reclamation protocol. It tolerates partial failure with lock-free data structures and a recovery protocol where each operation atomically updates eight bytes of in-place state, like a single-element redo log, enabling idempotent redo without blocking live threads. The paper reports that cxlalloc is the first allocator to satisfy all these constraints, has best-in-class time and space performance on YCSB and memcached-trace workloads, and achieves up to 80 percent of its maximum allocation throughput with no hardware coherence using mCAS on a commercial CXL device. The allocator is open source. Exact layouts, protocols, the mCAS implementation, and full results should be read from the paper.

Background

Why sharing memory across machines keeps forcing systems to be redesigned

It helps to see cxlalloc within a long pattern: every time memory is shared across a new boundary, the software built for the old boundary has to be rethought. Decades of allocators, data structures, and recovery schemes quietly assume a single machine with one coherent memory, one address space managed by one operating system, and a failure model where the whole thing stops together. Shared memory across machines dissolves all three assumptions, and CXL pods are only the latest arena to do so — the same tensions appeared with distributed shared memory, with persistent memory, and with RDMA-based disaggregation. What recurs is that hardware offers a tantalizingly simple interface (here, load and store to remote memory) but withholds the invisible guarantees software leaned on (coherence, address-space management, atomic all-or-nothing failure), so systems must rebuild those guarantees themselves, in software or with a little help from new hardware primitives. The broad effort to make CXL-based shared memory efficient by co-designing the software with the hardware is the same terrain the work on a software-hardware co-designed solution for efficient CXL-based memory (CTXNL) covered elsewhere on this site occupies, meeting the pod’s missing guarantees with a mix of both. Cxlalloc’s contribution is to show that even the most basic service, memory allocation, can be made correct and efficient across a pod by carefully separating what needs coherence, keeping pointers portable, and recovering without ever stopping the world.

If you remember only three things

1. A CXL pod is a small group of hosts (roughly 8 to 16) sharing one CXL-attached memory module at cacheline granularity, and applications running across it need an allocator. Ordinary allocators break there for three reasons: the shared memory may not be kept cache-coherent by hardware (so cross-host atomics fail), it is accessed by several separate processes (so absolute-address pointers are not valid across them), and with more hosts a thread is more likely to crash inside the allocator (and must not block the others).
2. Cxlalloc handles all three. For coherence, it separates the metadata that needs hardware coherence into its own region and keeps the rest coherent in software, and for devices with none it adds mCAS, a memory-based compare-and-swap done in near-memory FPGA logic on an uncacheable region. For cross-process sharing, it gives pointer consistency: offset pointers at consistent offsets for spatial consistency, and a fault-driven signal handler plus hazard-pointer reclamation for temporal consistency. For partial failure, it uses lock-free structures and begins each operation by atomically writing eight bytes of in-place state (a single-element redo log), so an interrupted operation can be idempotently redone on recovery without blocking live threads.
3. Cxlalloc is the first allocator appropriate for a CXL pod, with best-in-class time and space on YCSB and memcached-trace workloads and up to 80 percent of maximum allocation throughput with no hardware coherence (using mCAS). The trade-offs: the no-coherence mode needs special near-memory hardware and an uncacheable region and is somewhat slower, it assumes the CXL device itself is reliable (keeps memory across crashes), the cross-process machinery adds complexity and a contiguous-layout constraint, and it targets small pods rather than large-scale sharing.

Check yourself

Check your understanding

1. Why does an ordinary memory allocator fail to work in a CXL pod?
2. What is mCAS, and why does cxlalloc need it?
3. How does cxlalloc make a pointer created in one process usable in another process on a different host?
4. Predict: a thread crashes (say, killed by the OS out-of-memory killer) right in the middle of a cxlalloc allocation. What happens to the other hosts, and how is the allocator's state fixed?

Sources

The paper is “Cxlalloc: Safe and Efficient Memory Allocation for a CXL Pod” by Newton Ni, Yan Sun, Zhiting Zhu, and Emmett Witchel (The University of Texas at Austin, the University of Illinois Urbana-Champaign, and NVIDIA), published at ASPLOS 2026, DOI 10.1145/3779212.3790149. It is licensed CC BY, with an author copy hosted at UT Austin and the allocator open-sourced at github.com/nwtnni/cxlalloc.

This explainer is written from the paper’s full text (the authors’ hosted copy). Cxlalloc is a user-space memory allocator for a CXL pod that addresses three novel challenges: limited inter-host hardware cache coherence, cross-process sharing, and partial failure. It separates coherence-critical metadata into its own contiguous region and maintains the rest coherent in software, and for devices with no hardware coherence it introduces a memory-based compare-and-swap (mCAS) implemented in near-memory processing logic on an FPGA. It provides pointer consistency across processes — spatial consistency via offset pointers placed at consistent offsets, and temporal consistency via a signal handler that installs memory mappings on demand plus a hazard-pointer reclamation protocol. It tolerates partial failure using lock-free data structures and a recovery protocol in which each operation atomically updates eight bytes of in-place state (like a single-element redo log) to allow idempotent, non-blocking recovery. The paper reports that cxlalloc is the first allocator to satisfy all these constraints, has best-in-class time and space performance on YCSB and memcached-trace workloads, and achieves up to 80 percent of its maximum allocation throughput with no hardware coherence (using mCAS) on a commercial CXL device. Numbers are the paper’s own; exact layouts, protocols, and full results should be confirmed in the paper.