TPP: Transparent Page Placement for CXL-Enabled Tiered-Memory

A server can now hold a second pool of memory that is cheaper but slower, and TPP changes the Linux kernel so that rarely read pages move down into it while frequently read pages return to fast memory, all without the application knowing.

Why memory placement became a money problem

Start with a cost figure, because that is what motivated the work. TPP comes from Meta, and Meta measured memory as a share of what one server rack costs to buy and to power. Across six of their hardware generations, that share grew from about 15% to more than 33%. Memory went from a minor line in the budget to a third of it.

Now look at what that memory is actually doing. The authors profiled four families of services that run across a large part of their fleet: a web service, a distributed caching service, a data warehouse engine, and an advertising service. The result is the number the whole paper rests on:

55–80% of the memory an application holds is not read at all within any two-minute interval.

Say that as a picture. A server holds 100 GB for a service. In two minutes the service reads only 20 GB to 45 GB of it. The rest is allocated, paid for, powered, and untouched.

That memory does not need to be fast. It only needs to exist. And that is the opening: if the cold part could sit in cheaper memory, a server could hold more data for the same money.

The hardware that made a second tier realistic

CXL stands for Compute Express Link. It is an open industry standard built on top of the PCI Express interface, the connector normally used for graphics cards and network cards.

Two properties matter here.

First, CXL memory is reached with ordinary load and store instructions, at cache-line granularity. There is no driver call and no page fault. The hardware keeps the caches coherent.

Second, the operating system sees CXL memory as a NUMA node with memory but no CPUs. Linux already knows how to allocate from such a node, so nothing about the interface is new to it.

The cost is latency. The paper places the tiers on one scale:

CXL therefore adds roughly 50–100 ns over a local memory access. Compare that to the two alternatives on either side. Memory over a network is about ten times worse. A solid-state disk is about a hundred times worse. CXL memory is slower than local memory, but it is the same kind of thing, and that is what makes a tier out of it rather than a storage device.

What people did before, and where it broke

Building a slow memory tier is not a new idea. Three earlier approaches were reasonable, and each one runs into a specific wall.

Approach one: let the application decide. Earlier tiering work on non-volatile memory asked the programmer to say which data is hot and which is cold. This works when there is one application and one team. It does not work in a hyperscale fleet, where thousands of rapidly changing services share the same kernel. Nobody can tune them all.

Approach two: treat the slow tier as swap space. Meta already ships a system called TMO, short for Transparent Memory Offloading, which detects cold pages and writes them to a compressed swap pool. Pointing that at CXL memory would be easy. It would also throw away the reason CXL is interesting. With swap, every access to an offloaded page causes a major page fault, and the kernel must read the whole page back. The effective access time rises far above 200 ns, and the cache-line load and store behaviour is gone.

Approach three: use Linux NUMA Balancing. Linux already migrates pages between NUMA nodes. A kernel task samples a subset of a process’s pages, marks them, and catches the next access as a minor page fault. It then moves the page to the node the accessing processor sits on. This is close to what is needed. The paper shows two problems with it, measured in the next section.

And approach zero: change nothing. Give Linux a CXL node and let it run. This fails in a way that is worth understanding, because it explains TPP’s whole design.

Here is the failure, on Meta’s web service. When the service starts, it reads its program files from disk. Those reads create file pages — copies of disk content held in memory. They fill the local memory node. Then the service begins handling requests, which needs anonymous pages — the program’s stack, heap, and memory it maps directly. The local node is full, so Linux starts freeing space by writing pages to swap.

Swapping is slow. Measured against TPP, default Linux frees the local node 44 times slower. While the kernel is slowly freeing space, new allocations cannot wait, so they go to the CXL node. And once an anonymous page is on the CXL node, plain Linux never brings it back.

The outcome: only 22% of that service’s memory accesses were served from the local node, and throughput fell by 16.5% against a machine with all memory local.

The idea, one piece at a time

TPP is not one mechanism. It is three changes to Linux memory management, plus one optional fourth. Each one fixes a specific part of the failure above.

Piece 1 — demote by copying, not by swapping

When the local node runs low, Linux picks victim pages and writes them to the swap device. TPP intercepts that moment. Instead of writing the victim to disk, it puts the victim on a demotion list and copies it to the CXL node in the background. The paper’s phrase is direct: migrating a page to another NUMA node is orders of magnitude faster than swapping it.

Which pages get picked? TPP reuses a structure Linux already maintains. For each node and each page type, Linux keeps two lists in LRU order — Least Recently Used, meaning the page touched longest ago sits at the end. One list is called active, the other inactive. A page on the inactive list has not been touched recently.

TPP takes its demotion victims from the inactive lists. Because demoted pages are still in memory and still readable, TPP can scan inactive anonymous pages as well as inactive file pages, which plain swap-based reclamation avoids doing aggressively. If a migration fails — for example the CXL node itself is full — TPP falls back to normal swapping for that one page.

Piece 2 — separate “stop reclaiming” from “start allocating”

This is the subtlest change and the one with the largest measured effect.

Linux keeps watermarks on each node. When free pages fall below the low watermark, reclamation starts. New allocations to that node then halt until reclamation has freed enough pages to reach the high watermark. One threshold controls both actions.

That coupling is the bug on a tiered machine. Under a burst of allocation, reclamation cannot keep up, allocations halt on the local node, and the burst lands on the CXL node instead. Those burst pages are exactly the wrong ones to exile: they are created while handling a request, so they are short-lived and hot.

TPP splits the single threshold into two. Reclamation keeps running until free memory reaches the demotion watermark. New allocations may proceed as soon as free memory is above the lower allocation watermark. The demotion watermark is always set higher. The gap between them is permanent free space in the local node.

That free space buys two things. New request-handling allocations land locally. And there is somewhere to put pages promoted back up from the CXL node.

How aggressive should this be? A knob controls it, exposed at /proc/sys/vm/demote_scale_factor. Its default value is 2%, meaning reclamation begins once only 2% of the local node’s capacity is still free.

Piece 3 — promote a page only after it proves itself twice

Pages on the CXL node can turn hot. A demoted page may be read again, and a page allocated there during a burst may be read constantly. Without a way back up, those pages are trapped, and every access pays the extra 50–100 ns forever.

TPP reuses NUMA Balancing to find them, with two modifications.

The first modification is about cost. Plain NUMA Balancing samples pages on every node. Sampling a local page is pointless here — a hot local page is already in the right place, and the sampling only generates page faults for nothing. So TPP samples only the CXL node. Since the CXL node is supposed to hold cold pages, faults there are rare, and the detection cost stays near zero.

The second modification is about accuracy, and it is the clever part. Plain NUMA Balancing promotes a page the moment a fault touches it. One access is enough. That creates a ping-pong: a page read once travels up, fills local space, gets picked as a demotion victim shortly after, and travels back down. The traffic is pure waste, and it consumes the very free space that promotion needs.

TPP adds memory to the decision, using the LRU lists again:

• If the faulting page is on the inactive list, do not promote it. Mark it as accessed and move it to the active list.
• If the faulting page is already on the active list, promote it.

So a page must be touched, and then touched again while still counted as active, before it moves. One access no longer earns a trip. This is hysteresis: the system requires more evidence to change state than to stay in it.

When TPP does promote, it deliberately ignores the allocation watermark on the target local node. Squeezing the local node is intentional. It triggers reclamation of colder local pages, which is the correct response to a hot page arriving.

Piece 4 — the optional page-type policy

The three pieces above look at behaviour only, never at page type. TPP adds one optional policy for operators who want it: allocate new file pages on the CXL node from the start, while leaving anonymous pages under the normal local-first policy.

The reasoning follows the profiling. Services generate large amounts of file cache during startup that is then rarely read. Under the default policy that cache occupies the local node until TPP demotes it, and meanwhile pushes anonymous pages onto the CXL node that must later be promoted back. Starting file pages low removes both migrations. A file page that does turn out to be hot is promoted normally.

local node:  [file][file][file][file][file]
CXL node:    [ empty ]
free pages:  falling

Why it works

The chain is worth writing out link by link, because each step depends on the one before it.

Because CXL memory appears as an ordinary memory node, the kernel can read and write it with load and store instructions. → Because there are no page faults on a normal access, the slow tier can hold live data instead of only swapped-out data. → Because copying a page between nodes is far faster than writing it to disk, the local node can be freed quickly instead of 44 times more slowly. → Because the local node is freed quickly, and because a separate allocation watermark keeps free space in reserve, bursts of new request pages land in fast memory instead of slow memory. → Because promotion requires a page to be hot twice, the promotion traffic is small enough that it does not consume that reserved free space. → Because promotion traffic is small and mostly correct, promotions succeed, and hot pages that were stuck in the slow tier come back. → Therefore the fast tier ends up holding the pages that are actually read, and the application runs at nearly the speed of a machine where all memory is local.

Notice how the pieces support each other. Fast demotion is useless if allocation still halts. Reserved free space is wasted if promotion floods it with pages read once. The paper shows this directly: with the watermarks coupled together, promotion nearly stops, trapped pages generate 55% of the memory traffic, and throughput falls 12%.

What it costs

Every design gives something up. This one gives up five things.

• Copying work. TPP is always moving pages. In steady state the migration rate is 4–16 MB/s, or 1,000 to 4,000 pages per second. The paper argues this is far below the CXL link bandwidth and causes no meaningful processor cost. That claim holds for Meta’s configurations; a machine with a very small local tier would move more.
• Slower convergence. Requiring two accesses before promotion delays the steady state. Reaching peak local traffic takes five minutes longer than promoting on the first access.
• A tuning knob that can be set wrong. The default reclamation trigger is 2% of local capacity. The paper is explicit about the failure mode: if the hot working set is larger than the local node, aggressive reclamation makes pages thrash between the two nodes.
• A small loss of transparency. The page-type policy is optional precisely because it is a guess about the application, not a measurement of it.
• Nothing for bandwidth-bound work. TPP only sends cold pages to the CXL node, and cold pages draw almost no bandwidth. A workload limited by memory bandwidth rather than capacity gains nothing. The paper lists this as future work.

What the measurements showed

The setup matters, so state it first. TPP was built into Linux kernel v5.12. The functional tests ran on pre-production x86 processors with an FPGA-based CXL memory card supporting the CXL 1.1 specification, which the operating system saw as a memory node with no CPUs.

Those FPGA cards were about 250 ns slower than the eventual target hardware. So the performance numbers were mostly gathered a second way: on dual-socket servers configured to imitate a CXL machine, with all processor cores and one memory node on one socket, and a second memory node with no cores. Two processor vendors confirmed that CXL latency should resemble cross-socket latency.

The baseline is an ideal machine: one memory node holding the entire working set, all of it local. That baseline is 100%. Every number below is throughput as a percentage of it. Two capacity ratios were tested — 2:1, matching Meta’s production configuration, and 1:4, a stress test where the local node is a small fraction of the memory.

Summarising the table the way the paper does: TPP stays within 1 percentage point of the ideal all-local machine in five of the six cases. It is up to 18% better than default Linux, and 5–17 percentage points better than NUMA Balancing and AutoTiering.

One row deserves a second look. On the cache service at 1:4, the local node holds only 20% of the working set, and TPP still reaches 99.5% of all-local throughput. That is the case the whole design targets: a server built with a small amount of expensive fast memory and a large amount of cheap slow memory.

What each piece contributes

The authors turned the pieces off one at a time, using the cache service at the 1:4 ratio.

Separating the watermarks is what makes the rest possible. Without it, the local allocation rate is limited by how fast reclamation runs, and promotion nearly halts, because new allocations consume freed pages the instant they appear. Trapped pages then generate 55% of memory traffic and throughput falls 12%. With the separation, the local allocation rate rises 1.6× at the 95th percentile, and promotion holds a steady 50 KB/s, rising to 1.2 MB/s at the 99th percentile during surges.

Requiring two accesses before promotion is a smaller but clear win:

Preferring the CXL node for file pages gave the third result. With this policy on, the web service at 2:1 served 97% of traffic locally and reached 99.5% of baseline throughput. The two cache services at 1:4 served 85% and 72% locally, reaching 99.8% and 98.5%. Across all three, the loss against the ideal machine was 0.2 to 2.5 percentage points.

How TPP relates to swap-based offloading

The paper also compares against TMO, Meta’s swap-based memory offloading system, and finds the two are complementary rather than competing.

Used alone against CXL memory, TMO cannot fill the tier. Because a swap pool is only written during reclamation and never allocated into directly, TMO used just 45%, 61% and 7% of the CXL memory capacity on the three 2:1 workloads. TPP used 83%, 92% and 87% of the same capacity, because it both allocates there and demotes there.

Run together, each improves the other. TMO frees system-wide memory, which gives TPP room to move pages: migration failures fell from 20 to 5 pages per second. TPP in turn converts TMO’s single swap step into demote-then-swap, which gives a half-warm page a second chance. Process stalls in TMO dropped by 30%, and because TMO throttles itself using stall measurements, its memory saving rose from 13.5% to 16.5% of total capacity — 2 GB in absolute terms.