PIPM: Partial and Incremental Page Migration for Multi-host CXL Disaggregated Shared Memory

New CXL hardware lets several servers share one pool of disaggregated memory with the caches kept coherent across all of them, but that shared memory is slower to reach than a server's own local DRAM — and the usual trick of migrating hot pages into fast local memory was built for a single machine and works badly when many hosts share the same pages; this paper (UC Santa Cruz — Gangqi Huang, Heiner Litz, Yuanchao Xu) presents PIPM, hardware that migrates only the needed parts of a page, a little at a time, while staying coherent, for up to 2.54 times higher performance.

The argument in one glance

Problem

Shared CXL memory is slower than a server's own local DRAM

Compute Express Link (CXL) is a new interconnect that lets memory be disaggregated — pulled out of individual servers into a shared pool that multiple hosts can access — and, in its multi-host form, kept cache-coherent across all those hosts. This CXL disaggregated shared memory (CXL-DSM) is attractive because it lets many servers flexibly share a large memory pool instead of each over-provisioning its own DRAM. But there is a fundamental catch: reaching memory across the CXL link is slower than reaching a server's own local DRAM. So when a host repeatedly touches data that lives in the shared pool, it pays that higher latency again and again, and performance suffers. The classic remedy in tiered and NUMA memory systems is page migration: detect the data a host uses heavily and move it into that host's fast local memory, so subsequent accesses are cheap. The trouble is that the page-migration machinery everyone has is built for a single host managing its own memory tiers — and a multi-host, cache-coherent shared pool is a very different setting, where the same pages may be shared among hosts and every movement has to respect coherence. Applying single-host migration here works poorly.
What people did

Reuse single-host page migration on the shared pool

The natural first move is to take the page-migration approaches that already work for single-host tiered memory and apply them to CXL-DSM: monitor which pages a host accesses often, and migrate those whole pages from the slow shared pool into the host's fast local DRAM, the same way a single machine promotes hot pages from a slow tier to a fast one. Each hot page is copied, wholesale, into local memory, and future accesses go to the local copy instead of across the CXL link.
Why it fell short

Whole-page, single-host migration fights the multi-host setting

Single-host migration methods are inefficient in a multi-host CXL-DSM system for several linked reasons. They move data at the granularity of a whole page, so even if a host only touches a small part of a page frequently, the entire page is copied across the link — wasting bandwidth and time on data that is not actually hot. Worse, in a coherent multi-host pool the same page can be shared and accessed by several hosts at once, so migrating a whole page and redirecting accesses to a local copy has to be reconciled with the multi-host coherence protocol; a migration scheme that was never designed with that protocol in mind either breaks coherence or forces heavy, coarse synchronization. And the monitoring and decision logic inherited from single-host systems does not account for the shared, cross-host access patterns that determine what is worth moving here. The result is that naive migration in CXL-DSM moves too much data, too coarsely, and at odds with coherence — often costing more than it saves. What was missing was a migration mechanism built for the multi-host coherent setting from the start: one that can move just the needed portions of data, incrementally, while keeping everything coherent, and that decides what to move based on the real cross-host access behavior.
This paper's idea

Migrate only the hot parts, a little at a time, coherently, in hardware

PIPM (Partial and Incremental Page Migration) is a hardware-based migration mechanism designed for multi-host CXL-DSM. Rather than copying whole pages, it supports fine-grained, partial migration: it moves only the portions of a page that a host actually uses, and does so incrementally over time rather than all at once. Crucially, PIPM is co-designed with the CXL multi-host coherence protocol, so the data it moves into a host's fast local DRAM remains coherently accessible — a host can read and write its locally-migrated data with the same coherence guarantees as the shared pool, without the migration breaking the cross-host consistency the protocol provides. Because migration decisions matter, PIPM builds in hardware-based monitoring and decision-making: it tracks access behavior and chooses what to place where in hardware, transparently, without needing application or software changes. In effect, PIPM transparently harvests each host's local memory as a fast cache for the shared pool, moving in just the right sub-page data at the right time while preserving coherence. The paper reports that this delivers performance improvements of up to 2.54 times, and 1.86 times on average, over the default multi-host CXL-DSM configuration.

Why this problem is worth solving

CXL disaggregated shared memory is a major direction for datacenter memory, and its whole value proposition depends on making the slow shared pool feel fast.

Datacenters waste enormous amounts of memory because each server must provision for its own peak, leaving much of it stranded and idle elsewhere. Disaggregating memory into a shared pool that many hosts draw from — which CXL now makes possible with hardware cache coherence across hosts — directly attacks that waste, letting capacity be allocated where it is needed. But the pool only pays off if using it does not cripple performance, and the unavoidable fact is that pool memory, reached across the CXL link, is slower than local DRAM. So the entire benefit of CXL-DSM hinges on bridging that latency gap: on making the data a host uses heavily behave as if it were local. Page migration is the established way to do that, which is why getting it right in this new setting is so consequential. If migration is inefficient — moving too much, fighting coherence, guessing wrong — the shared pool stays slow and the disaggregation dividend evaporates. A migration mechanism that makes shared-pool data cheap to access is thus what turns CXL-DSM from a capacity convenience into something you can run demanding workloads on, which is why an up-to-2.54-times improvement over the default configuration is significant.

The core realization is that in a coherent multi-host pool, migration must be fine-grained and coherence-aware, not the coarse whole-page copy inherited from single machines.

Single-host page migration makes two assumptions that quietly break in CXL-DSM. First, it assumes moving a whole page is fine, because on one machine the page is the natural unit and the copy is cheap and local. In a shared pool reached over a link, copying a whole page when only a fraction is hot wastes the scarce, slower link bandwidth — so the right granularity is partial, moving just the used portions, and incremental, spreading the work over time rather than paying a big upfront cost. Second, it assumes the mover owns the data, so redirecting accesses to a local copy is straightforward. In a multi-host coherent pool the data is shared, and the coherence protocol is what keeps every host’s view consistent; a migration that ignores the protocol either corrupts that consistency or has to fall back on coarse, expensive synchronization. The insight behind PIPM is that migration in this setting must be co-designed with the coherence protocol, so that a host’s local copy is a first-class, coherent participant rather than an off-to-the-side duplicate. Once migration is both fine-grained and coherence-native, you can safely and cheaply pull just the right data into local DRAM — and doing the monitoring and decisions in hardware keeps it transparent and fast. Getting the granularity and the coherence right together is what makes local memory usable as a fast cache for the pool.

What people did before, and where it broke

Apply single-host tiered-memory migration to CXL-DSM. Monitor hot pages and promote them from the slow pool to local DRAM, as on one machine. Where it broke. It was designed for a single host owning its tiers, and does not fit a shared, coherent, multi-host pool.
Migrate whole pages. Move data at page granularity when a page looks hot. Where it broke. If only part of a page is hot, copying the whole page wastes scarce CXL-link bandwidth on cold data.
Migrate all at once. Copy the hot data in one upfront operation. Where it broke. A large synchronous copy is costly and can stall access, rather than spreading the work incrementally.
Ignore the multi-host coherence protocol. Treat the migrated local copy as if the host owned it. Where it broke. In a coherent shared pool the same page can be used by several hosts, so migration must respect the protocol or it breaks consistency or forces heavy synchronization.
The gap. No migration mechanism was co-designed with the CXL multi-host coherence protocol to move just the hot sub-page data, incrementally and coherently, guided by cross-host access behavior.

How PIPM works

From coarse single-host migration to partial, coherent, incremental movement

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Move only the data that is actually used, on demand and incrementally. PIPM’s first principle is to stop copying whole pages and instead migrate the fine-grained portions a host really touches, spread over time, so scarce CXL bandwidth is spent only on hot data. Moving data between a slow, large memory and a fast, near one at a fine granularity and on demand, rather than in coarse bulk transfers, is the same discipline the work on tensor migration and prefetching in unified memory (DeepUM) covered elsewhere on this site applies, migrating just what the computation needs when it needs it. PIPM brings that fine-grained, incremental philosophy to page migration in a shared CXL pool, where over-moving is especially costly.

Co-design the movement with the coherence hardware so local copies stay consistent. Because the pool is cache-coherent across hosts, PIPM cannot treat a migrated local copy as private; it co-designs migration with the CXL multi-host coherence protocol so that data placed in local DRAM remains a coherent participant. Building the data-movement mechanism as a hardware-software co-design tightly integrated with the memory system’s coherence and access path, rather than bolting it on top, is the same approach the work on a hardware-software co-designed disaggregated memory system (Clio) covered elsewhere on this site takes, making remote memory efficient by designing the hardware for it. PIPM’s co-design is what lets a host read and write its migrated data at local-DRAM speed without violating the shared, coherent view.

Give each host fast local access to shared-pool data, transparently. The payoff is that PIPM turns a host’s own DRAM into a fast cache for the disaggregated pool, placing the right data locally so accesses avoid the CXL link — all in hardware, without changing applications. Transparently placing data across a fast local tier and a slower CXL-attached tier so that hot data ends up close to the compute is the same enterprise the work on transparent page placement for CXL-enabled tiered memory (TPP) covered elsewhere on this site pursues, hiding the tiering from software. PIPM extends that idea to the harder multi-host, coherent, shared-pool case with partial and incremental migration.

Following hot data through PIPM

Serving a host's repeated access to shared-pool data

An illustration, drawn from the paper's abstract, of how PIPM migrates partially and coherently. Internal mechanism details are the paper's.

Step 1 of 5

A host keeps accessing data in the shared pool

A server in the multi-host CXL-DSM system repeatedly reads and writes some data that lives in the disaggregated shared memory pool, paying the higher CXL-link latency on every access.

Only a portion of each page is actually hot for this host.

Why it works

Partial migration spends bandwidth only on hot data. Copying a whole page moves cold bytes too → but migrating only the sub-page portions a host uses → so the scarce CXL-link bandwidth is not wasted on data that is never touched.
Incremental movement avoids a costly upfront copy. A single large migration stalls and front-loads cost → but moving the data gradually over time → so the work is spread out and overlaps with useful execution.
Coherence co-design makes the local copy usable directly. A migrated copy that ignores the protocol breaks consistency → but co-designing migration with the CXL multi-host coherence protocol → so the local data is a coherent participant a host can read and write safely.
Hardware monitoring keeps it transparent and fast. Software-driven decisions add overhead and need app changes → but doing monitoring and placement decisions in hardware → so migration is transparent to applications and reacts quickly to access behavior.
Local DRAM becomes a fast cache for the pool. Repeated pool accesses each pay CXL latency → but placing the hot data in the host’s local memory → so subsequent accesses hit fast local DRAM instead of crossing the link.

What it costs (trade-offs and scope)

It requires hardware support co-designed with the coherence protocol. PIPM is a hardware-based mechanism integrated with the CXL multi-host coherence protocol, so realizing it means building that hardware and protocol support, not a software-only change.
Fine-grained, incremental migration adds bookkeeping. Tracking and moving sub-page portions, and maintaining their coherence, is more complex machinery than a coarse whole-page copy, which is the cost of avoiding wasted movement.
Benefits depend on access patterns. The gains come from hosts repeatedly using localizable portions of shared data; workloads whose accesses are scattered or truly shared-write-heavy across hosts may benefit less from moving data local.
It targets multi-host CXL-DSM specifically. PIPM addresses the shared, coherent, disaggregated setting; its advantages over single-host migration are tied to that context rather than being a general tiered-memory result.
Reported figures are against the default configuration. The up-to-2.54-times and 1.86-times-average gains are measured over the default multi-host CXL-DSM setup under the paper’s workloads and conditions; results elsewhere may differ.

The comparison

Migrating shared-pool data in multi-host CXL-DSM

Property	Default CXL-DSM (no migration)	Single-host page migration	PIPM
Where hot data lives
Migration granularity
Respects multi-host coherence
Monitoring and decisions
Performance

A comparison of ways to handle hot shared-pool data in multi-host CXL disaggregated shared memory. Without migration, every access crosses the slower CXL link; single-host migration moves whole pages coarsely and fights the multi-host coherence protocol. PIPM migrates only the hot sub-page portions, incrementally, into each host's local DRAM while staying coherent via co-design with the coherence protocol, and decides placement in hardware — delivering up to 2.54 times (1.86 times on average) higher performance than the default configuration. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s abstract and public metadata, not its full text, which was not available at the time of writing. The high-level claims are the authors’: PIPM (Partial and Incremental Page Migration) is a hardware-based solution for multi-host CXL cache-coherent disaggregated shared memory (CXL-DSM) that transparently leverages host-side local memory. It is co-designed with the CXL multi-host coherence protocol so that data migrated into local DRAM remains coherently accessible; it supports fine-grained (partial and incremental) data migration rather than whole-page movement; and it integrates hardware-based monitoring and decision-making to optimize data placement. The reported result is a performance improvement of up to 2.54 times (1.86 times on average) over the default multi-host CXL-DSM configuration. The background above on CXL, disaggregated shared memory, cache coherence across hosts, page migration, and the local-versus-pool latency gap is standard context provided to make the approach intuitive. The Flow, the step-by-step walkthrough, and the “Why it works” reasoning are illustrations built from the abstract’s description of the mechanism, not detailed accounts from the paper — the specifics of how partial and incremental migration is implemented, how it integrates with the coherence protocol, and how the hardware monitoring and decisions work should be read from the paper. Do not treat specific procedural details here as established beyond what the abstract states.

Background

Why disaggregated memory keeps turning into a data-placement problem

It helps to see PIPM within the broader movement toward disaggregated and pooled memory. The economic pull is strong: servers waste memory by provisioning for their own peaks, and pooling that memory so many machines can share it reclaims the waste — which is exactly what CXL, with hardware coherence, now enables across hosts. But every form of disaggregation trades that flexibility for distance: the pooled memory is farther away and slower to reach than local DRAM. So the recurring problem, in one form after another, becomes data placement and movement — deciding what to keep close, what to leave in the pool, and how to move data between them cheaply and correctly. The field keeps rediscovering that coarse, naive movement squanders the benefit, and that the movement mechanism has to be designed together with the memory system’s coherence and access path rather than layered on top. PIPM is a clean instance for the multi-host coherent case: it makes migration fine-grained and coherence-native so local memory can serve as a fast cache for the pool. The broader platform it serves — turning CXL-based pooled memory into something cloud systems can actually deploy and rely on — is the same terrain the work on CXL-based memory pooling systems for cloud platforms (Pond) covered elsewhere on this site occupies, making disaggregated memory practical. PIPM’s particular contribution is to show that partial, incremental, coherence-co-designed migration substantially closes the latency gap that would otherwise undermine multi-host CXL shared memory.

If you remember only three things

1. Compute Express Link (CXL) enables multi-host cache-coherent disaggregated shared memory (CXL-DSM), where several servers share a memory pool with hardware coherence across them — reclaiming the memory that servers otherwise waste by provisioning for their own peaks. But pool memory reached over the CXL link is slower than a host's local DRAM, so a host that repeatedly touches shared-pool data keeps paying that latency. The classic fix is page migration — move hot data into fast local memory — but the existing methods were designed for a single host managing its own tiers. In a shared, coherent, multi-host pool they are inefficient: they move whole pages (wasting link bandwidth on cold data), do it coarsely, and clash with the multi-host coherence protocol.
2. PIPM (Partial and Incremental Page Migration) is a hardware-based migration mechanism built for this setting. Instead of copying whole pages, it migrates only the sub-page portions a host actually uses (partial) and does so gradually over time (incremental), so scarce CXL bandwidth is spent only on hot data. It is co-designed with the CXL multi-host coherence protocol, so data moved into a host's local DRAM stays a coherent participant — the host reads and writes it at local speed without breaking the consistent cross-host view. And it integrates hardware-based monitoring and decision-making to choose what to place where, transparently, with no application changes. In effect PIPM turns each host's local memory into a fast, coherent cache for the shared pool.
3. The paper reports PIPM delivers up to 2.54 times (1.86 times on average) higher performance than the default multi-host CXL-DSM configuration. The trade-offs and scope: it requires hardware support co-designed with the coherence protocol (not a software-only change), fine-grained incremental migration adds bookkeeping over a coarse copy, the gains depend on access patterns being localizable rather than scattered or heavily shared-write across hosts, and it targets the multi-host CXL-DSM case specifically. This explainer is based on the paper's abstract; the implementation of partial and incremental migration, its integration with the coherence protocol, and the full evaluation should be confirmed in the paper.

Check yourself

Check your understanding

1. Why is data placement a problem in multi-host CXL disaggregated shared memory?
2. Why do single-host page-migration methods work poorly in multi-host CXL-DSM?
3. What are PIPM's key design choices?
4. What does PIPM achieve, and what are the limits of the approach?

Sources

The paper is “PIPM: Partial and Incremental Page Migration for Multi-host CXL Disaggregated Shared Memory” by Gangqi Huang, Heiner Litz, and Yuanchao Xu (University of California, Santa Cruz), published at ASPLOS 2026, DOI 10.1145/3779212.3790203.

This explainer is written from the paper’s abstract and public metadata; the full text was not available at the time of writing, so the page is marked a draft. As reported there, PIPM is a hardware-based migration mechanism for multi-host CXL cache-coherent disaggregated shared memory that transparently leverages host-side local memory. It is co-designed with the CXL multi-host coherence protocol so migrated local data stays coherent, supports fine-grained partial and incremental migration instead of whole-page movement, and integrates hardware-based monitoring and decision-making to optimize placement. The paper reports performance improvements of up to 2.54 times (1.86 times on average) over the default multi-host CXL-DSM configuration. Background on CXL, disaggregated shared memory, cache coherence, and page migration is standard context. The specific numbers are the paper’s own; the mechanism details and full evaluation should be confirmed in the paper.