M5: Mastering Page Migration and Memory Management for CXL-based Tiered Memory Systems

Yan Sun, Jongyul Kim, Zeduo Yu, Jiyuan Zhang, Siyuan Chai, Michael Jaemin Kim and 8 others

CXL memory cheaply expands a server's RAM but runs 2-3x slower than local DDR, so you want hot pages on fast DDR and cold ones on slow CXL — which means constantly figuring out WHICH pages are hot and migrating them, and today's OS methods do this on the CPU by scanning page tables or catching faults, which is both inaccurate (a boolean "accessed" bit can't tell a warm page from a scorching one) and so expensive it burns up to 700% more kernel CPU and actually makes Redis slower; M5 (Illinois, Seoul National, and Intel) instead puts a real access COUNTER inside the CXL controller right next to the memory, so it counts every access precisely for almost free, uses a compact streaming sketch to keep the top-K hottest pages AND words (so it can skip migrating pages where only a few bytes are hot), and drives migration with a feedback policy that keeps moving pages only while bandwidth keeps improving — identifying pages 3.4x hotter than the best CPU method and running memory-heavy workloads up to 96% faster.

The argument in one glance

Problem

CXL memory is cheap and roomy, but slow — so you must keep the hot data close

CXL (Compute Express Link) lets a server attach extra DRAM over the PCIe bus, expanding capacity and bandwidth far more cost-effectively than adding DDR slots (the same bandwidth as DDR5 with a third of the pins). The catch is latency: CXL DRAM is 2-3x slower to access than local DDR, roughly 140-170 nanoseconds of extra delay per access. That makes a CXL server a TIERED memory system — a small fast tier (local DDR) and a large slow tier (CXL DRAM) — and performance depends on keeping the frequently-accessed HOT pages on the fast tier and the rarely-touched COLD pages on the slow tier. Since access patterns shift over time, the system must continuously identify which pages are hot and MIGRATE them between tiers. Do that badly and the whole benefit of the cheap capacity evaporates behind slow accesses.
What people did

Track page hotness on the CPU, the way the OS already does

The OS already has machinery for this from the NUMA world, so the natural move is to reuse it. AutoNUMA balancing scans page-table entries and reads each page's ACCESS BIT to see if it was touched. TPP periodically resets pages and catches "hinting" faults on the next access. DAMON scans PTEs on a schedule. Memtis samples cache-miss addresses with the CPU's performance-monitoring hardware (PEBS). Each way, the CPU periodically inspects memory activity, decides which pages are hot, and migrates them up to fast DDR — the established recipe for tiered memory.
Why it fell short

The CPU sees hotness poorly, and pays dearly to see it at all

CPU-driven hotness tracking is both inaccurate and expensive, and on CXL that is disqualifying. INACCURATE: the page-table access bit is a single BOOLEAN — it records whether a page was touched in an epoch, not how many times — so a page touched once looks identical to one touched a million times, and warm pages masquerade as hot. Measured against a precise reference, the "hot" pages AutoNUMA and DAMON pick receive only 21% and 29% of the accesses that the truly-hottest pages get. They also migrate whole 4KB pages even when only a few 64-byte words are actually used (86% of Redis pages touch 16 or fewer of their 64 words), wasting the scarce fast tier. EXPENSIVE: scanning PTEs, invalidating TLBs, and handling faults raise kernel CPU cycles by up to 487% and 733%, and raise Redis 99th-percentile latency by 34-39% — DAMON actually makes Redis SLOWER than doing no migration at all. The tool meant to help ends up hurting.
This paper's idea

Count accesses in the CXL controller, next to the memory

M5 (University of Illinois, Seoul National University, and Intel — Sun, Kim, Yu, and colleagues) moves hotness tracking off the CPU and INTO the CXL controller, which sits right next to the memory and can watch every access. A hardware Page Access Counter (PAC) and Word Access Counter (WAC) count accesses to every 4KB page and every 64-byte word precisely, with zero CPU involvement. Because tracking every address at scale is infeasible, an online tracker uses a compact CountMin-Sketch (an SRAM streaming structure borrowed from Row-Hammer defenses) plus a top-K list to keep the hottest pages and words — costing just 0.01% of a DRAM module's die area. A software policy (Monitor, Nominator, Elector, Promoter) then migrates pages driven by measured bandwidth, skips pages that are only sparsely hot, and migrates only while it keeps helping. The result identifies pages 3.4x hotter than the best CPU method and runs memory-intensive workloads up to 96% faster, with essentially no CPU overhead.

Why this problem is worth solving

CXL is the industry’s answer to memory that is both too small and too expensive, and it only pays off if the software can hide its latency by keeping the right data on the fast tier — which turns the whole thing into a hotness-tracking problem. The reason that is hard starts with what the OS has to measure, the

how frequently each page of memory is being accessed, so the system can rank pages and put the hottest ones on fast DDR. The natural signal the OS already exposes is each page’s ACCESS BIT — a single bit the hardware sets when the page is touched. But a boolean is a terrible hotness meter: it says whether a page was accessed at least once in an epoch, not how MANY times, so a page read a million times and a page read once look exactly the same. To get any resolution, the OS must accumulate access-bit observations over many epochs, each of which requires walking the page tables — and even then it is counting epochs-touched, not accesses. So the fundamental signal is coarse, and squeezing information out of it is what makes the tracking both inaccurate and slow

. When the reference the paper builds measures the truth, the pages CPU methods flag as hot turn out to receive only a fifth to a third of the accesses the genuinely hottest pages get — the system is migrating the wrong pages.

The second reason it is hard is that the act of measuring is expensive enough to defeat its own purpose, a classic

the situation where the overhead of monitoring a system degrades the very system it is trying to optimize. To track hotness, CPU-driven methods must periodically scan page-table entries, reset present bits, invalidate TLB entries across cores (TLB shootdowns), and handle the resulting faults — all of which consume CPU cycles that would otherwise run the application, and all of which are especially damaging to latency-sensitive services. The paper measures this monitoring raising kernel CPU cycles by up to 733% and inflating Redis tail latency by nearly 40%, to the point where DAMON’s page migration leaves Redis SLOWER than doing nothing. The cure is worse than the disease

. So the tiered-memory problem is squeezed from both sides: the CPU sees hotness badly, and it pays so much to see it that the seeing itself hurts. The way out is to stop asking the CPU to do the measuring at all — to put the accounting where the data is, on the CXL device, where every access can be counted exactly and for free. That reframing is the heart of M5.

What people did before, and where it broke

AutoNUMA balancing / hinting faults (ANB, TPP). Reset pages and catch faults on the next access to sample hotness. Burns CPU on PTE access, TLB invalidation, and fault handling — badly degrading latency-sensitive apps — and its “hot” pages get only 21% of the accesses the truly-hottest pages receive.
PTE scanning (DAMON). Periodically scan valid page-table entries and read access bits. The boolean access bit cannot rank hotness, so it needs many epochs (each expensive) to accumulate a weak signal — raising kernel cycles up to 733% and making Redis 16% SLOWER than no migration at all.
Cache-miss sampling (Memtis, PEBS). Sample last-level-cache miss addresses with performance-monitoring hardware. It misses most addresses (imprecise), and crucially Intel PEBS currently cannot even sample LLC misses that go to CXL memory — the exact accesses that matter.
Migrating whole pages regardless of density. All of the above migrate a full 4KB page when it is deemed hot, even if only a handful of its 64-byte words are used — and most pages in Redis, Memcached, and CacheLib are sparse — wasting the scarce fast tier and polluting caches.
The gap. No method measured page hotness accurately AND cheaply AND at word granularity. M5 does, by counting accesses in the CXL controller instead of inferring them on the CPU, then migrating with a policy that respects density and bandwidth.

How M5 works

Count in the controller, sketch the top-K, respect density, and migrate on feedback

Put access counters in the CXL controller new

The CXL device has near-memory logic (an Accelerated Function Unit), so M5 adds a Page Access Counter that snoops every memory access and increments a per-4KB-page count, and a Word Access Counter that counts per-64-byte word. Because the counting happens at the memory, it is precise (every access, not a sample) and costs the CPU nothing.
↓ →
Track the top-K hot pages and words with a sketch new

Keeping an exact counter for every page across a large memory is infeasible, so the online tracker uses a CountMin-Sketch — a compact SRAM structure (borrowed from Row-Hammer defenses) that estimates any address's access count by hashing it into a few rows and taking the minimum — feeding a sorted top-K list. For 32K entries it recovers the hottest pages with 0.97 accuracy, at 0.01% of a DRAM module's die area.
↓ →
Distinguish sparse hot pages from dense ones new

The word counters reveal that many "hot" pages actually touch only a few of their 64 words. M5's Nominator uses the hot-word data to mark, per page, which words are hot (a 64-bit mask) — so it can avoid promoting sparse pages that would occupy fast DDR while leaving most of their space cold, and instead prioritize genuinely dense hot pages.
↓ →
Monitor per-tier bandwidth density new

A software Monitor reads host performance counters for pages and read-bandwidth on each tier, and computes BANDWIDTH DENSITY — bandwidth per allocated page. If CXL's density exceeds DDR's, hot pages are stuck on the slow tier and should move up aggressively; the actual bandwidth signals whether migrations are paying off.
↓ →
Migrate under a feedback policy new

The Elector scales migration frequency by the CXL-to-DDR density ratio and keeps migrating only while DDR bandwidth keeps increasing — the signal that recent promotions actually helped. It invokes the in-kernel Promoter, which calls the Linux migrate_pages() to copy chosen pages up to DDR, update page tables, and flush TLBs, after safety checks.
↓ →
Demote cold pages with MGLRU

As hot pages are promoted to the limited DDR tier, cold pages must be evicted down to CXL to make room. M5 reuses the kernel's Multi-Generation LRU (MGLRU), a proven, precise, low-cost mechanism, for demotion — so M5's new machinery focuses on the hard part (accurate, cheap promotion) and leans on existing infrastructure for the rest.

Prototyped on an Intel Agilex FPGA CXL controller (with a Device Coherence Engine for host-CXL coherence), evaluated against AutoNUMA balancing (ANB) and DAMON on Redis, Memcached, CacheLib, Liblinear, SPEC CPU 2017, and GAP graph benchmarks. Hardware trackers add 0.01% die area; software policy runs the Monitor/Nominator/Elector/Promoter loop.

The idea

Put the accounting where the data is, not on the CPU. The deepest move is a relocation: instead of asking the CPU to infer hotness by scanning page tables and reading a coarse access bit, M5 counts accesses directly in the CXL controller, which sees every access to the memory it fronts. That single change fixes both failures at once — the count is exact rather than a boolean guess, and it costs the CPU nothing rather than hundreds of percent of kernel cycles. This is the same build-the-true-metric discipline the site just met in the datacenter-IO-control page, which invents an accurate device-occupancy metric because bytes and IOPS lie — and here M5 invents an accurate hotness metric because the access bit lies, in both cases by profiling the real thing instead of trusting a cheap proxy. It is also near-data processing: the counter lives next to the memory because that is the one place every access is naturally visible, so the measurement is both free and complete.

Do not let the measurement cost more than what it measures. CPU-driven tracking is a cautionary tale in the observer effect — the monitoring is so heavy (up to 733% more kernel cycles, Redis tail latency up by nearly 40%) that DAMON’s migration leaves Redis slower than no migration at all. The overhead of finding hot pages swamped the benefit of moving them. M5’s hardware counters run pipelined with memory accesses and touch no CPU, so the measurement is essentially free and the benefit is kept. This is the off-critical-path discipline the site rewards — the IO-control page splitting a cheap per-request path from an occasional expensive one, the persistent-memory database keeping its shared node dumb — here taken to the limit by moving the measurement entirely off the CPU and onto dedicated hardware that watches the data for free.

Track the top-K approximately, and move at the right granularity. Two refinements make the idea practical. Exact per-page counters for a large memory are too expensive, so M5 uses a CountMin-Sketch — a bounded-size probabilistic summary that estimates any address’s count with tiny memory, borrowed from Row-Hammer defense — to feed a top-K list, recovering the hottest pages at 0.01% area. And because word counters reveal that most “hot” pages are actually SPARSE (only a few words used), M5 refuses to migrate a whole 4KB page for a few hot bytes, promoting dense hot pages instead. These are the approximate-summary-of-a-huge-stream and match-the-granularity-to-the-real-access disciplines the site sees elsewhere — a sketch where an exact structure is infeasible, and finer measurement to avoid moving what will not be used — and together they turn an in-hardware counter into a practical migration engine.

Why the CPU is the wrong place to find hot pages

From a coarse, costly CPU scan to a free, precise controller counter

Finding and migrating hot pages in a CXL tiered-memory system. Numbers from M5's evaluation.

Step 1 of 5

The tiered setup

The server has a small fast tier (local DDR) and a large slow tier (CXL DRAM, 2-3x higher latency). Performance depends on keeping the hottest pages on DDR and cold ones on CXL, and since access patterns shift, the system must keep re-identifying hot pages and migrating them up. The quality of that hot-page identification decides everything.

CXL adds ~140-170 ns of extra latency per access versus local DDR
keep the wrong pages on DDR and the cheap capacity is buried behind slow accesses

Why it works

The page-table access bit is a boolean that cannot rank hotness → a hardware counter in the CXL controller counts every access exactly → M5’s hot pages get 3.4x more real accesses than the CPU methods’ (0.72 vs 0.21).
CPU-driven scanning, TLB shootdowns, and fault handling burn up to 733% more kernel cycles → counting in the controller is pipelined with accesses and touches no CPU → the measurement is free and no longer degrades the workload (DAMON made Redis slower).
Exact per-page counters for large memory are infeasible → a CountMin-Sketch approximates any address’s count in tiny SRAM, feeding a top-K list → precise hot-page tracking fits in 0.01% of a DRAM module’s die area.
Most “hot” pages are sparse (few words used) → word counters let M5 promote dense hot pages and leave sparse ones on CXL → the scarce fast tier holds data that is actually accessed.
Migration has a cost that only pays off if it helps → a feedback policy migrates only while DDR bandwidth keeps rising → M5 aggressively promotes on skewed workloads (up to 96%) and holds back on uniform ones where migration would not amortize.

What it costs

A limited time window, not global truth. M5 ranks pages by accesses within a recent window, so a page hot NOW but not overall can be migrated suboptimally; its accuracy is 0.72 against the 1.0 of a full-execution offline reference.
Accuracy fades with a very large footprint. The sketch’s precision degrades gracefully as the tracked memory grows; at 64x the tested footprint (8 GB to 512 GB) it noticeably drops, needing more sketch entries or multiple tracker instances per rank.
Uniform-hotness workloads gain little. When all pages are about equally hot, migration cost (roughly 54 microseconds, needing over 300 accesses to amortize) rarely pays off, so M5’s benefit shrinks on graph workloads like GAP.
The policy needs hand-tuning. The Elector’s frequency-scaling function and default parameters are chosen per workload by trying a few reasonable values; there is no automated adaptation.
It requires modern CXL hardware. M5 needs a CXL Type-2/3 device with an Accelerated Function Unit and device-to-host memory access; older CXL.mem devices and non-CXL tiers (HBM, older persistent memory) cannot host the counters, and huge-page tracking needs extra work not evaluated here.

The evaluation

An Intel Agilex FPGA CXL controller with the PAC/WAC counters and CountMin-Sketch trackers, plus the M5-manager software, evaluated on Redis, Memcached, CacheLib, Liblinear, SPEC CPU 2017, and GAP graph benchmarks, against AutoNUMA balancing (ANB) and DAMON, measuring hot-page accuracy, end-to-end performance, and CPU overhead.

Finding hot pages for CXL tiered memory

Property	PTE scanning (DAMON, AutoNUMA)	Cache-miss sampling (Memtis, PEBS)	M5 (counters in the CXL controller)
Where hotness is measured
What it measures
CPU overhead
Accuracy (access-count ratio)
Sparse-page aware		No

By counting accesses where they happen instead of inferring them on the CPU, M5 gets exact, word-granular hotness for free, identifying pages 3.4x hotter and running memory-intensive workloads 14-20% faster on average (up to 96% on skewed workloads) — while DAMON's CPU-side monitoring is so heavy it makes Redis slower than doing no migration at all.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

M5 ranks hotness within a recent time window rather than over the whole execution, so it can occasionally migrate a page hot only for the moment (its 0.72 accuracy versus a 1.0 offline oracle reflects this), and the CountMin-Sketch’s precision degrades gracefully as the tracked footprint grows very large, needing more entries or multiple trackers past roughly a 64x scale-up. Its benefit is real only where hotness is skewed and migration cost (about 54 microseconds, over 300 accesses to amortize) is repaid; on uniform-hotness workloads it wisely gains little. The migration policy still needs per-workload hand-tuning with no automated adaptation, and the whole approach requires a modern CXL Type-2/3 device with near-memory logic — older CXL.mem parts, HBM, and persistent memory cannot host the counters, and huge pages need extra handling. None of this dulls the core insight — that CPU-driven hotness tracking is doomed on CXL because the access bit is too coarse to rank hotness and the scanning is so heavy it degrades the very workloads it means to help, so the fix is to count accesses where they actually happen, in the CXL controller, precisely and for free; to track the top-K with a compact sketch; and to migrate at the right granularity, skipping sparse pages, under a feedback policy that moves data only while it keeps helping.

Background

Measure where the data is, do not let monitoring cost more than it saves

M5 sits in the site’s memory-and-storage-systems cluster, alongside the persistent-memory KV-store, the disaggregated database, the serverless-memory, and the datacenter-IO-control pages, and rhymes with several ideas covered elsewhere on this site. Its founding move is the build-the-true-metric discipline the IO-control page shares: just as bytes and IOPS lie about device occupancy so that page had to profile the real cost, the page-table access bit lies about hotness (a boolean cannot rank a million accesses against one), so M5 builds an accurate hotness metric by counting accesses in hardware — and it does so via near-data processing, putting the counter in the CXL controller because that is the one place every access is naturally visible and free to observe. Its second move is a stark lesson in the observer effect: CPU-driven tracking’s monitoring is so heavy that it degrades the workload it optimizes (DAMON makes Redis slower than no migration), so M5 moves the measurement entirely off the critical path onto dedicated hardware — the same keep-the-hot-path-cheap discipline the IO-control page uses with its fast/slow split and the persistent-memory database uses when it keeps its shared node dumb. Its refinements are site regulars too: a CountMin-Sketch is the approximate-summary-of-a-huge-stream move (and a borrow-a-solved-model one, lifted from Row-Hammer defense, like the pages that recast a problem as max-flow or Bayesian optimization); word-granular tracking is the match-the-granularity-to-the-real-access discipline the graph-mining and persistent-memory pages share, refusing to move what will not be used; and the migrate-only-while-it-helps policy is the feedback-driven, predict-then-correct control the IO-control page uses to adjust its rate. Even its precise offline PAC counter is a ground-truth reference that exposes how far the proxies fall short — the same compute-the-optimum-to-measure-the-gap move the compiler-inlining page makes.

If you remember only three things

1. CXL cheaply expands server memory but runs 2-3x slower than local DDR, forming a tiered system that needs hot pages on fast DDR and cold ones on slow CXL — requiring continuous hot-page identification and migration; but the OS does this on the CPU, and it is both inaccurate (the page-table access bit is a boolean that cannot tell a page touched once from one touched a million times, so the "hot" pages CPU methods pick get only 21-29% of the accesses the truly-hottest pages get) and crushingly expensive (up to 733% more kernel cycles, and DAMON's monitoring makes Redis slower than doing no migration at all).
2. M5 moves hotness tracking into the CXL controller, next to the memory: hardware Page and Word Access Counters count every access to each 4KB page and 64-byte word exactly and for free, a CountMin-Sketch (borrowed from Row-Hammer defense) plus a top-K list keeps the hottest pages in 0.01% of a DRAM module's die area, and a software policy (Monitor, Nominator, Elector, Promoter) migrates dense hot pages driven by measured bandwidth density — skipping sparse pages where only a few words are used, and migrating only while DDR bandwidth keeps rising.
3. The result identifies pages 3.4x hotter than the best CPU method (0.72 vs 0.21 access-count ratio) and runs memory-intensive workloads 14-20% faster on average, up to 96% on a highly skewed workload, with essentially no CPU overhead — the honest limits being a recent-window (not global) view, graceful accuracy loss at very large footprints, little benefit on uniform-hotness workloads, per-workload policy tuning, and a requirement for modern CXL hardware. The durable lessons: measure the true quantity where the data is rather than inferring it from a coarse proxy on the CPU, never let monitoring cost more than it saves, and migrate at the right granularity only while it helps.

Check yourself

Check your understanding

1. CPU-driven tiered-memory systems rank page hotness using the page-table access bit. Predict why this signal is fundamentally too weak for good migration decisions.
2. The paper reports that DAMON's page migration actually makes Redis 16% SLOWER than doing no migration at all. Predict what this reveals about CPU-driven hotness tracking.
3. M5 tracks per-64-byte-WORD access counts, not just per-page, and finds that most "hot" pages in Redis touch 16 or fewer of their 64 words. Predict why this word-granularity matters for migration.
4. M5 delivers up to 96% speedup on a highly skewed workload (roms_r) but almost nothing on uniform-hotness graph workloads (GAP). Predict what this contrast reveals about when hot-page migration helps.

Sources

ACM DL: M5: Mastering Page Migration and Memory Management for CXL-based Tiered Memory Systems — ASPLOS 2025 (full text read via the ACM PDF; no arXiv preprint).
Artifact (GitHub, ece-fast-lab/ASPLOS-2025-M5) — open-source artifacts.