TMO: transparent memory offloading in datacenters

Meta taught the Linux kernel to measure how much work an application loses to memory shortage, then used that one signal to automatically push each container's cold memory out to compressed RAM or SSD — saving 20–32% of DRAM across millions of servers.

Why this problem is worth solving

Every server in a large fleet carries DRAM sized for its worst moment, and most of that DRAM spends most of its time holding data nobody reads. The paper measures both halves of this waste precisely.

The cost half: across Meta’s hardware generations, DRAM is projected to reach 33% of server cost and its power draw 38% of server infrastructure power. The alternatives are dramatically cheaper. A compressed-memory pool (cold pages compressed in RAM) costs about a third of raw DRAM for typical 3× compressibility. An NVMe SSD — solid-state drive attached over the fast NVMe interface — contributes under 3% of server cost, and per byte of capacity it is about ten times cheaper than even compressed memory.

The opportunity half: the authors measured how recently seven large applications touched their memory. On average, 35% of allocated memory had not been touched in the last five minutes — with enormous variance, from 19% cold (a cache service, almost everything hot) to 62% cold (the Web application). Cold memory in DRAM is money parked in the wrong asset.

There is also a second, less obvious pool of savings. The paper names the memory tax: memory consumed not by the application itself but by the machinery of running applications in a datacenter — logging, profiling, service discovery (the datacenter tax, 13% of server memory), plus the routing and proxy sidecars of microservice architectures (the microservice tax, 7% more). One fifth of fleet memory pays this tax, and most of it has looser performance requirements than application memory — an ideal first target.

Offloading means moving cold pages out of DRAM into a cheaper backend and faulting them back on access. Doing this transparently — in the kernel, with no application changes — is the only approach that scales to thousands of diverse workloads. The question the paper answers is the control question: how much can each container be squeezed, on which backend, before the application suffers?

What people did before, and where it broke

The closest prior system is Google’s fleet-wide zswap deployment (the paper calls it g-swap). It compresses cold pages into a RAM pool and controls aggressiveness with a target promotion rate: keep pages-swapped-back-per- second below a threshold chosen by offline profiling plus machine learning. It was the proof that kernel-driven offloading works at datacenter scale. Three limits remained.

One backend. g-swap supports only compressed memory. But compression is hostage to the data: Meta’s advertising models store quantized byte-encoded values that compress at just 1.3–1.4×, making a compressed pool nearly pointless — while an SSD stores them at a tenth of the cost. A real fleet needs heterogeneous backends, and the backends differ wildly: the 90th- percentile read from compressed memory takes about 40 microseconds, while SSD read latency across Meta’s generations spans 470 microseconds to 9.3 milliseconds — a 20× spread within one fleet.

Offline profiling. Per-application profiling does not scale to thousands of services that change daily.

The proxy metric lies. A promotion rate counts events; it does not measure harm. The same rate that is catastrophic on a slow SSD is harmless on a fast one. Worse, the paper demonstrates the metric inverting: under TMO, a host with a faster SSD sustained a higher swap-in rate — and processed more requests per second, because aggressive offloading freed memory the application put to use. A threshold on promotion rate would have forbidden exactly the behavior that improved performance.

There was also a problem inside Linux itself. The kernel’s reclaim algorithm, shaped in the era of spinning disks, treated swap as an emergency valve and skewed heavily toward evicting file cache first — it would throw away a workload’s actively-used file pages long before touching cold anonymous memory. Offloading needed reclaim to be rebalanced, not just triggered.

The idea

PSI: measure the harm itself

Pressure stall information (PSI) is the paper’s foundational contribution: a kernel mechanism that reports, for each process, container, or the whole machine, the share of potential work lost while stalled waiting for a resource — computed separately for memory, CPU, and I/O.

For memory, PSI counts time in exactly three situations that only occur under shortage: a process forced to reclaim pages while allocating; a process waiting on I/O for a refault — a fault on a page that was resident recently and got evicted; and a process blocked reading a page back from swap. Two aggregates are kept: some (at least one process in the container is stalled — a latency signal) and full (all are stalled — a productivity-collapse signal). TMO’s control loop uses some.

The metric’s power is what it automatically accounts for. A slow SSD makes each stall longer, so pressure rises — the device’s performance is priced in. An application that touches offloaded pages often stalls often — its sensitivity is priced in. An application with internal parallelism hides stalls — priced in. None of this needs profiling, models, or device tables. The cost is a little bookkeeping at context switches — negligible in production — and PSI now ships enabled by default in major Linux distributions, including Android.

Senpai: hold pressure at “barely measurable”

Senpai, a userspace agent, turns the signal into action. Every six seconds, for every container, it computes how much memory to ask the kernel to reclaim:

reclaim = current_memory × 0.0005 × max(0, 1 − PSI_some / 0.1%)

Read it from the right. The threshold is 0.1% — one thousandth of runtime lost to memory stalls. If a container’s pressure exceeds that, reclaim stops entirely; the workload gets room. Far below it, Senpai trims 0.05% of the container’s footprint per cycle, capped at 1% per period. As pressure approaches the threshold, the steps shrink toward zero. The equilibrium is a container running with the minimum memory it needs, plus a whisper of deliberate, harmless pressure to prove it.

Two design details carry production weight. Squeezing is gradual (full contraction takes minutes) but expansion is instant — a growing workload is never blocked; Senpai originally adjusted cgroup memory limits, but a limit can block a rapidly growing container, so the authors added a stateless memory.reclaim file to the kernel instead: reclaim exactly this much, hold nothing. And Senpai watches I/O pressure too, because offloading’s harm can arrive sideways — through a busier SSD — rather than as memory stalls. It also throttles offload writes to about 1 megabyte per second per host, protecting SSD write endurance. The reclaim work costs 0.05% of fleet CPU cycles.

What to offload: fix the kernel’s old bias

Senpai decides how much; the kernel’s reclaim algorithm decides which pages, using its existing least-recently-used lists — cheap and battle-tested. But the historic file-cache bias had to go. The authors added non-resident cache tracking: when a file page is evicted, a per-container counter value is stored in a shadow entry; on a later fault, the reuse distance tells the kernel whether this was a refault of recently-resident data (working set!) or a first touch. Reclaim now takes file cache alone until refaults appear, then balances file cache against anonymous memory by comparing the refault rate with the swap-in rate — minimizing total paging. These changes were upstreamed into Linux.

The backend itself is per-application, chosen by data compressibility: zswap (compression with zstd) for compressible workloads, SSD swap for the rest. Senpai needs no reconfiguration either way — a slower backend produces more pressure per offloaded page, and the loop backs off by itself.

footprint 40 GB | PSI 0.00% (threshold 0.1%)
reclaim = 40 GB × 0.0005 × (1 − 0) = 20 MB

Why it works

Because PSI counts only stall time from qualifying memory events, it is a direct measurement of harm — not a proxy like swap-in counts that must be reinterpreted per device and per application. → Because harm is measured where it happens, one global threshold (0.1%) transfers across thousands of applications and a 20×-latency spread of backends: slow hardware raises pressure faster and the loop backs off by itself. → Because the squeeze is proactive but gentle — trims of 0.05% every six seconds, shrinking near the threshold, with instant release on growth — the system discovers each container’s true working set without ever gating an allocation. → Because reclaim balances file cache against anonymous memory using refault evidence rather than the old file-first bias, both kinds of cold memory are offloaded and total paging is minimized. → Therefore the fleet runs with DRAM sized near the sum of true working sets instead of allocated footprints — 20–32% smaller — while user-facing services hold their latency targets.

What it costs

• A thousandth of runtime, on purpose. Steady-state operation keeps every container slightly memory-starved; that whisper of stall time is the price of knowing the working set. The reclaim machinery itself costs 0.05% of CPU cycles.
• SSD wear is a real budget. Offload writes are throttled to about 1 MB/s per host — a hard constraint that caps how much anonymous memory can churn through an SSD backend.
• Backend choice is still manual. Whether an application gets zswap or SSD swap is decided by humans looking at compressibility. The authors name the fix — a kernel-managed hierarchy (warm pages compressed, cold pages to SSD, NVM and CXL tiers later) — as future work.
• The knobs were tuned empirically. The global threshold and ratio came from iterative production experiments; the paper shows a slightly more aggressive configuration saving more memory but visibly cutting Web’s request rate — through file-cache evictions of application bytecode, an I/O-side effect a memory-only view would miss.
• Reaction to contraction is slow. Squeezing takes minutes by design; TMO harvests steady-state waste, it does not respond to emergencies — that remains the job of out-of-memory killers.

The evaluation

TMO’s evaluation is a production report, not a testbed: more than a year running across millions of servers, with A/B load tests on tens of production hosts for the detailed studies.

Two deployment stories are worth keeping. The rollout order was tax-memory first (relaxed SLAs), then file-cache-only mode fleet-wide, then full swap for the largest applications — each step de-risking the next. And the observability was worth money on its own: PSI-driven accounting exposed one application repeatedly re-extracting a self-extracting binary into file cache; extracting it once ahead of time cut that application’s memory by 70% — a finding conventional accounting missed because it discounts file cache entirely.