Towards an Adaptable Systems Architecture for Memory Tiering at Warehouse-Scale

Google replaced about a quarter of its datacenter DRAM with slower, cheaper memory and kept the median performance loss near 2% — by demoting pages that stay cold for minutes, pulling them back within a second of reuse, and never taking a page fault on the way.

Why this problem is worth solving

A warehouse-scale computer is one building-sized system running hundreds of thousands of jobs. Its main memory is

The waste is equally well measured. Between 28% and 42% of allocated memory in the studied address spaces had not been touched for two minutes. That memory still occupies DRAM priced for nanosecond access. Technologies like Intel Optane (and, in the future, CXL-attached memory) offer roughly 3× the capacity per dollar at roughly 3× the latency — the paper’s tier2 DIMMs show about 325 ns idle read latency, versus roughly 100 ns for DRAM. If the cold pages could live there, a quarter of the DRAM bill disappears.

The catch is the blast radius. This is not one appliance running one database; the change lands under 100,000+ different production jobs, from batch analytics to user-facing services with strict tail-latency targets. The paper’s own bar: replace 25% of DRAM while keeping performance degradation below 5% for the latency-sensitive class. A design that works for the average job but stalls the tail is a failed design at this scale.

What people did before, and where it broke

Hardware-managed caching (Intel “memory mode”). The memory controller treats all DRAM as a giant cache in front of Optane. Applications see one flat memory. It requires zero software change — and that is its flaw at fleet scale: the operating system cannot see hit rates, cannot steer specific jobs, cannot cap how much slow memory a latency-critical service touches, and gives up all of DRAM as cache even for jobs that need none. Control and observability are exactly what a fleet operator sells SLOs with. (An SLO, service level objective, is a promised performance target.)

Fault-based OS tiering (swap, zswap). The kernel compresses or moves cold pages out of DRAM. The page disappears from the page table, so the next access traps into the kernel — a page fault — and the application waits while the page is decompressed or migrated back. Faults happen exactly at the worst moment: the instant the application needs the data. Google runs zswap fleet-wide and knows the cost; sub-millisecond stalls are acceptable for truly-cold data but not as the recovery path for a tier holding a quarter of all memory.

Academic tiering systems (Nimble, HeMem, and others). These pioneered better migration mechanisms and hot-page tracking, but were evaluated on single machines with benchmark suites. The paper’s argument is that the hard problems only appear at scale: policy must hold across thousands of diverse, co-located, constantly-changing services, and it must be adaptable in days (userspace rollout) rather than months (kernel rollout).

The idea

TMTS stands on three design commitments.

First: the slow tier is memory, not a swap device. Tier2 is mapped like any other memory. When an application touches a demoted page, the load completes at tier2 latency — about three times slower than DRAM, but with no page fault and no kernel entry. A tiering mistake costs nanoseconds, not milliseconds. This single property removes the failure mode that makes swap-based tiering dangerous for latency-sensitive services.

Second: manage by two observable ratios. The system steers with a pair of fleet-wide metrics.

Third: demote patiently, promote urgently. The two directions are asymmetric. A page cold for eight minutes will likely stay cold, so demotion can rely on slow, cheap page-table scans. But a tier2 page that gets touched once is likely to be touched again immediately — every further access pays tier2 latency. So promotion uses

page P: tier1
rule: no direct tier2 allocation

Why it works

The causal chain from design to fleet-safe savings:

• Tier2 is directly addressable → a mistimed demotion costs extra nanoseconds per access, not a millisecond fault → the worst case is mild, so policies can be aggressive enough to actually fill tier2.
• Demotion waits for minutes of proven coldness → pages in tier2 are overwhelmingly ones nobody touches → STRR rises to ~25% while STAR stays near 0.3% → performance impact stays inside the 5% budget.
• Promotion is driven by hardware samples of actual tier2 loads → reuse is detected in well under a second, not at the next scan → a re-activated page pays tier2 latency only briefly → hot working sets migrate back before they accumulate damage.
• Policy lives in a userspace daemon reading kernel telemetry → Google tuned thresholds, sampling, and scheduling integration in days per iteration across 2 years → the deployed policy is the product of fleet-scale A/B measurement, not simulation.
• The cluster scheduler sees per-job tier behavior → tier-unfriendly, latency-sensitive jobs land on machines with tier2 headroom → the tail, not just the median, is protected.

What it costs

• Performance. Median IPC (instructions per cycle) impact of 2.3% versus identical control machines; a later 1000-machine study measured 3.5% aggregate. The distribution is bi-modal with a tail: at P5 the impact reaches about 7%.
• Hugepage erosion. Tiering fragments memory, cutting hugepage coverage from 45% to 30% and raising TLB misses by 7% — worth about 1.5% of the total impact. Two fixes (migrating hugepages intact, more aggressive compaction) each bought back ~0.5%.
• Hardware interference. When the Optane DIMM controller’s buffer fills, it stalls the memory channel it shares with DRAM — about 1.5% impact from increased DRAM latency. Newer Optane and CXL memory reduce this coupling.
• Migration and scan overhead. Background scans, sampling, and the daemon consume CPU, and migrations consume bandwidth — though 80% of tier2 bandwidth is applications, only 20% migrations (two thirds of that demotions).
• Blind spots. PEBS samples loads only; store-only pages wait for the slower scan backstop. Demotion stays on the local socket because memory-controller QoS cannot throttle cross-socket bursts, limiting placement freedom.

The evaluation

The evaluation is unusual for an architecture paper: a two-year A/B experiment on live production, with 2,200 machines running two-tier memory against 1,000 identically-scheduled one-tier control machines, across 6 clusters and over 100,000 jobs (about a quarter of them latency-sensitive HILS). Headline results: tier2 residency held at the ~25% target; memory utilization stayed identical to control (median 77%, P95 91%), meaning the cheap tier genuinely substituted for DRAM rather than idling; cold-memory coverage averaged 75%; and median STAR was 0.3% with P95 under 1% — inside the “below 5% degradation” budget.

The policy studies each ran on ~300-machine slices for 2 weeks. Demotion: a class-aware 8-minute/2-minute cold age cut HILS median STAR by 46% (0.35% → 0.19%) versus a flat 2-minute policy. Promotion, the study that matters most, compared three detectors:

Two adaptive extensions close the paper. Making the Borg scheduler tier-aware (about 20% of jobs are tier-friendly, 17% unfriendly) cut STAR by 30% and improved latency-sensitive job performance 4% on a 600-machine trial. And letting the TCMalloc allocator hint likely-cold allocations raised Spanner’s cold-memory identification from 10% to 42% while reducing its query latency 2%.