WASP: Workload-Aware Self-Replicating Page-Tables for NUMA Servers

On a big multi-socket server, a program's page tables can end up in far-away memory, so every address translation pays a remote-access penalty — and a recent fix, replicating the page tables so each socket has a local copy, sometimes doubles performance but sometimes makes it worse, and only a human could decide which, per application, which is impossible on a shared cloud server; this work from the Chinese Academy of Sciences and Huawei builds WASP, which watches three cheap hardware signals and automatically turns replication on or off, and even picks a remote copy when a contended local one would be slower, matching or beating the best hand-tuned choice.

The argument in one glance

Problem

On a NUMA server, page tables in remote memory slow every translation

Big servers are built from multiple sockets, each with its own local memory, and reaching another socket's memory is markedly slower — up to two and a half times the latency. Every memory access first needs its virtual address translated, and on a translation-buffer miss the hardware walks the page table, which itself lives in memory. If those page tables sit in a remote socket's memory — which happens when the OS spreads them out, or migrates a process without moving its page tables — every walk pays the remote penalty. As memory outgrows the translation buffer, these page-table walks multiply, and this page-table NUMA effect becomes a real drag.
What people did

Replicate the page tables so every socket has a local copy

A recent fix, page-table self-replication, copies an application's page tables to every NUMA node, so each socket has an identical local replica. Then every page-table access is served from local memory, and the remote-walk penalty disappears. It is transparent — applications need no changes — and prior systems like Mitosis showed it can nearly erase the page-table NUMA effect.
Why it fell short

Replication sometimes backfires, and only a human could tell

But replication is not always a win. Forcing every page-table access to local memory can overload the local memory controller when another application is co-located on the same node — so heavily that a local access becomes slower than a remote one, and performance drops below not replicating at all, by more than 1.6 times in one case. Whether replication helps or hurts depends on the application and whichever neighbors happen to share its node, which no user can foresee. Yet the systems that offer it expose only a manual on/off switch.
This paper's idea

Watch cheap hardware signals and flip replication automatically

WASP turns the manual switch into an automatic, workload-aware decision. Once a second it reads three lightweight hardware indicators — how memory-intensive the app is, how often it misses the translation buffer, and how long a page-table access actually takes — and runs them through a short hierarchy of checks to enable or disable replication for that application. Crucially, when it does replicate, it measures the real access latency to every socket and steers the app to the fastest replica, even a remote one, so a contended local copy never traps it.

Why this problem is worth solving

The performance drag this work attacks comes from a collision between two familiar facts about big machines.

A large server is built from several sockets, each with its own bank of memory, forming what is called a NUMA machine — non-uniform memory access — because a core reaches its own socket’s memory quickly but another socket’s memory slowly, sometimes more than twice as slowly. Separately, every program uses virtual memory, and turning a virtual address into a physical one requires consulting a page table, a multi-level tree the hardware must walk when the small on-chip translation cache misses. That walk reads memory several times. The two facts collide badly: the page tables themselves are stored in memory, so if they sit on a remote socket, every walk pays the remote-access penalty. As modern workloads grow far larger than the translation cache can cover, walks become frequent, and this page-table-caused NUMA effect turns into a first-order performance problem, on par with the more familiar cost of remote data accesses.

The fix this work automates is elegant, but it is precisely the fact that it is not always safe that creates the whole problem.

Page-table self-replication is a technique for erasing the remote-page-table penalty: instead of keeping one copy of an application’s page tables somewhere in the machine, it makes a separate identical copy on every socket. Because each socket has its own local replica, whenever the hardware walks the page table it reads from nearby memory, and the remote-access penalty vanishes. The appeal is that it is entirely transparent — the application is not modified and does not even know it is happening — and prior systems demonstrated that, when it applies, it can almost completely remove the page-table NUMA effect. The catch, and the reason this is not simply always on, is that keeping several synchronized copies costs some memory and, more importantly, redirects a burst of traffic onto local memory that may already be under pressure, so it is a tool that must be aimed, not applied blindly.

What people did before, and where it broke

Let the OS place page tables wherever. Under default policies, page tables land where allocation and migration put them. Where it broke. They routinely end up remote — an even-spreading policy can leave three-quarters of them on other sockets, and migrating a process strands all of its page tables remotely — so walks are slow.
Replicate page tables to every socket (Mitosis). Keep a local copy on each node and always read the local one. Where it broke. It is a manual switch, and forcing every access local can overload a local memory controller when another app shares the node — making local slower than remote, over 1.6 times worse than not replicating — and it cannot back off during a run.
Replicate system-wide for all processes. Turn replication on for the whole machine at once. Where it broke. It helps some applications and hurts others, and a blanket switch cannot tell them apart, so it degrades exactly the workloads that suffer local contention.
Decide with the translation-buffer miss rate. Use the obvious single signal to guess whether replication is worthwhile. Where it broke. A translation-buffer miss does not imply a remote page-table walk, and it says nothing about memory-controller contention, so one signal is far from enough to make the call.
The gap. Replication is powerful but double-edged, and whether it wins turns on the interaction between an application’s characteristics and the unpredictable neighbors it is co-located with — knowledge no user has, especially on a shared cloud server. What was missing was a way to make the enable-or-disable decision automatically, per application, from cheap signals, and to handle the contention case where even a local replica is the wrong choice.

How WASP works

Turning a manual replication switch into an automatic decision

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Never apply the optimization when it would actually hurt — and measure to know. The central realization is that replication, which sounds like a pure win, can quietly make things worse: pushing every page-table access onto local memory can overload a local controller already busy with a co-tenant, until the local copy is slower than a remote one. WASP therefore refuses to replicate blindly. It gates the decision on cheap indicators, and even after enabling replication it directly measures the true access latency to every socket and only uses a copy if it is genuinely the fastest. Cheaply checking whether a seemingly-beneficial operation would in fact improve the outcome, and dropping it when it would not, is the same guard the post-deduplication compression work covered elsewhere on this site builds in, filtering out matches that look useful but would enlarge the result rather than shrink it.

Replace a knob a human cannot set with an automatic, workload-aware controller. The prior systems made replication a switch the user had to flip, but the right setting depends on the application and the ever-changing set of neighbors it shares a socket with, which no operator can know on a busy cloud server. WASP dissolves the knob into a controller that reads a handful of live hardware signals every second and decides for itself, adapting the machine to the workload rather than asking the workload’s owner to configure the machine. Turning a manual, expert-only configuration into a system that measures what a workload is doing and automatically fits itself to it is the same shift the cooperative-embodied-agent work covered elsewhere on this site makes, replacing hand-set choices with a runtime that profiles the workload and adapts across the stack to match it.

Trust the real measured cost, not a convenient proxy for it. WASP deliberately does not decide from proxies like the fraction of accesses that are remote or how imbalanced the controllers look, because those describe where traffic goes without capturing what it costs once contention sets in. Instead it times an actual page-table access to each socket, so the number driving its choice is the real latency the application will experience, contention and all. Grounding a decision in the true behaviour measured directly rather than in a first-order approximation of it is the same discipline the virtual-memory-simulation work covered elsewhere on this site is built on, insisting that the honest way to model an operating-system cost is to run its real code rather than assume a fixed number for it.

Following the reasoning through the contention trap

The case where naive replication makes things worse

The paper's own worst case shows why measuring latency, not just enabling replication, is the crux.

Step 1 of 5

Remote page tables, a co-tenant on the local node

GUPS, a memory-thrashing benchmark, runs after a migration has left its page tables remote, while a bandwidth-heavy neighbour, Stream, runs on the same local node. This is the configuration where the obvious fix backfires — the two applications will fight over the local memory controller.

GUPS is exactly the kind of memory-intensive, translation-miss-heavy workload replication is meant to help.

Why it works

Measuring the real page-table latency catches contention that a proxy would miss. A remote-access ratio or a controller-imbalance number tells you where accesses go, but not what they actually cost → but directly timing a page-table access to each socket reveals when a busy local controller has made local memory slower than remote → so WASP sees the exact case that silently sabotages naive replication and would be invisible to a coarser signal.
A short hierarchy of checks filters out the apps that cannot benefit first. Blindly replicating for every process helps some and hurts others → but gating replication behind a memory-intensity check and then a translation-miss check screens out apps that walk the page table rarely → so replication is only ever attempted where it could plausibly pay off, and left off elsewhere.
Choosing the fastest replica, even a remote one, beats the always-local rule. Prior replication forces every access to the local copy and cannot reconsider → but WASP, having timed all the copies, sends the app to whichever replica is fastest right now → so when the local controller is overloaded it uses a remote replica and outruns manual replication by more than a quarter on average.
Reading a few standard counters cheaply makes the whole decision nearly free. A decision mechanism that itself cost real performance would defeat the purpose → but WASP samples only a handful of hardware performance events once a second → so its overhead is under one percent, small enough to leave running continuously.
Re-deciding gradually lets it track a changing mix of neighbours without regressions. A one-time choice cannot follow a workload whose co-tenants come and go → but WASP re-reads its indicators every second and can enable or disable replication as conditions shift → so it converges on the right setting for the current situation and, across every application tested, never made performance worse.

What it costs (trade-offs and scope)

Thresholds are hand-tuned per server. The decision hinges on two thresholds — a memory-access-rate and a translation-miss-rate cutoff — fixed by experiment on the authors’ machines; a different server needs re-tuning, and the authors defer an online-prediction method to future work, so it is not fully turnkey across hardware.
It does not yet coordinate with AutoNUMA. Running WASP alongside Linux’s standard data-page migration is usually slower than WASP alone, because the two mechanisms are uncoordinated; the headline results disable AutoNUMA, and fixing the interaction is left as future work.
The gains are modest and machine-dependent. Average speedups are about 1.37 times on a four-node ARM server and 1.22 times on a two-node one, and less on x86; the dramatic 6.69-times figure is the migration worst case, not typical. The clearest new win is the 15-to-27 percent by which it beats manual replication through shortest-latency selection.
It builds on existing replication machinery. WASP is the automatic decision plus the remote-replica insight layered on top of page-table self-replication from prior work; the replication itself, and its costs, are inherited rather than new.
Memory-management operations slow down. Because replicas must be kept in sync, calls like changing page protections can cost up to about 2.5 times more over large ranges; this matches prior replication systems, and the authors argue it is negligible because such calls are infrequent, but it is real for allocation-heavy code.
Replication uses extra memory. Keeping a page-table copy per socket adds memory — small, under one percent on a four-socket machine by prior measurement, but growing with the socket count — a cost paid whenever replication is on.
Decisions lag fast phase changes. Indicators are sampled once a second, so a workload that shifts phase or co-tenant faster than that could be briefly mis-set; the gradual re-decision limits the harm but does not eliminate the lag.

The evaluation

Managing page-table placement on a NUMA server

Property	Default Linux	System-wide replication	Manual per-process (Mitosis)	WASP
Where page tables can sit
When replication is used
Handles local-controller contention	Not applicable
User effort and knowledge
Result	Baseline (slow remote walks)

Page-table placement strategies on NUMA servers, with WASP evaluated on a four-node ARM and a two-node x86 machine across 11 workloads using three lightweight indicators (memory access rate, translation-buffer miss rate, page-table access latency). WASP automatically matches or beats hand-tuned replication — by 15 to 27 percent through selecting the shortest-latency replica — never degrades any application, improves performance about 1.37 times on average on the four-node server, and adds under one percent overhead; per-server thresholds and coordination with AutoNUMA remain limitations.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

WASP is a clean, practical improvement: it turns page-table self-replication from a manual switch that a user cannot set correctly into an automatic, workload-aware decision driven by three cheap hardware signals, and it adds the genuinely new insight that under local-controller contention the right move is a remote replica, not a local one — so it matches or beats hand-tuned replication, never degrades an application, and costs under one percent. Read it with its boundaries. The two thresholds are tuned by experiment on the authors’ servers and need re-tuning elsewhere; it does not yet coordinate with Linux’s AutoNUMA, and its headline results disable it. The gains are modest and machine-dependent — about 1.37 times on a four-node ARM server, less on smaller or x86 machines — with the striking 6.69-times number being a migration worst case; the durable differentiator is the 15-to-27 percent it beats manual replication by. It builds on prior replication machinery, inherits its memory and memory-operation overheads, and samples once a second, so it lags fast phase changes. The lasting idea is the reframing: measure the true latency, and never apply the optimization where it would actually hurt.

Background

Why co-located workloads make shared servers hard

It is worth stepping back to see why a decision this small — whether to keep an extra copy of some page tables — is so hard to get right. The reason is co-location: modern servers, especially in the cloud, pack many mutually indifferent workloads onto the same hardware to keep expensive machines busy, and the instant two of them share a socket, each quietly shapes the other’s performance through the resources they both draw on — the memory controllers, the interconnect, the caches. A choice that is clearly good for one application in isolation can become bad the moment a bandwidth-hungry neighbour lands beside it, and neither the application nor its owner can see the neighbour coming. That is what turns a simple optimization into a contextual gamble, and it is why WASP has to measure the live situation rather than decide once. The same truth — that sharing hardware for efficiency creates hidden, workload-dependent interference that has to be reasoned about rather than assumed away — drives the shared-resource-partitioning work covered elsewhere on this site, which quantifies exactly how one process’s use of a shared structure bleeds into another’s. The broader pattern is that as we pile more independent work onto shared machines, performance stops being a property of a program alone and becomes a property of the whole noisy neighbourhood it runs in.

If you remember only three things

1. On a NUMA server, page tables can land in remote memory — because the OS spreads them out, or migrates a process without moving them — so every translation-triggered page-table walk pays a remote-access penalty, a growing drag as workloads outgrow the translation buffer; page-table self-replication fixes this by giving every socket a local copy, but it can also backfire, because forcing all page-table accesses local can overload a co-located local memory controller until local memory is slower than remote, dropping performance below not replicating at all, and today it is only a manual on/off switch a user cannot set correctly.
2. WASP makes the decision automatic and workload-aware: once a second it reads three cheap hardware indicators — memory access rate, data-translation-buffer miss rate, and actual page-table access latency to each socket — and runs a short hierarchy that disables replication for apps that are not memory-intensive or rarely walk the page table, enables it for those that are, and then, crucially, times the real latency to every replica and steers the application to the fastest one, even a remote replica when a contended local one is slower.
3. On four-node ARM and two-node x86 servers across 11 workloads, WASP matches or beats hand-tuned replication — by 15 to 27 percent via shortest-latency selection — never degrades any application, improves performance about 1.37 times on average on the four-node server (up to 6.69 times in the migration worst case), and adds under one percent overhead; but its two thresholds are hand-tuned per server, it does not yet coordinate with Linux's AutoNUMA (and disables it for the headline results), its gains are modest and machine-dependent, and it inherits the memory and memory-operation overheads of the underlying replication.

Check yourself

Check your understanding

1. Why does replicating page tables to every socket sometimes hurt performance?
2. What does WASP do that a manual replication switch cannot?
3. Why does WASP measure actual page-table latency instead of a proxy like the remote-access ratio?
4. Which statement best reflects the limits of WASP?

Sources

The paper is “WASP: Workload-Aware Self-Replicating Page-Tables for NUMA Servers” by Hongliang Qu and Zhibin Yu, published at ASPLOS 2024, pages 1232 to 1248, DOI 10.1145/3620665.3640369. The authors are at the Shenzhen Institute of Advanced Technology of the Chinese Academy of Sciences, the University of Chinese Academy of Sciences, and Huawei Cloud’s Shuhai Lab.

This explainer is written from the full 17-page text. Everything above comes from the paper’s own sections: the background on NUMA architecture and its non-uniform latencies, on multi-level page tables and the cost of page-table walks, and on the page-table-caused NUMA effect arising from interleave placement and process migration; the motivation showing with GUPS that page-table self-replication can nearly erase the effect in one case (RPI-LD) yet degrade performance 1.61 times in another (RP-LDI) when a co-located application overloads the local memory controller, with the Stream-thread experiment demonstrating local access becoming slower than remote; the characterization of replication across memory allocation policies (first-touch versus interleave) and NUMA traffic metrics (memory access rate, remote access ratio, memory-controller imbalance, interconnect imbalance); the WASP design with its three chosen indicators (memory access rate, data-translation-buffer miss rate, and page-table access latency, chosen over remote-access-ratio and controller-imbalance proxies), the hierarchical and gradual decision mechanism with its per-second sampling and shortest-latency-replica selection, the ARM64 and x86_64 implementations with their performance counters and the hand-set thresholds; and the evaluation on four-node ARM and two-node x86 servers reporting that WASP at least matches manual replication and beats it by 15 to 27 percent through shortest-latency selection, never degrades any application, improves performance 1.37 times on average on the four-node server (up to 6.69 times in migration), adds under one percent overhead, and the discussions of its uncoordinated interaction with AutoNUMA, memory-management-operation overheads, and real co-location with Redis and MySQL.