Pond: CXL-Based Memory Pooling Systems for Cloud Platforms

Cloud servers waste memory that nobody can rent, so Pond puts a small shared memory pool next to a few servers and uses machine learning to decide which parts of a virtual machine can safely live there.

Why anyone should care about leftover memory

Think about one server in a cloud datacenter. It has 64 CPU cores and 512 GB of memory. Customers rent

Suppose customers rent virtual machines that each want 8 cores and 32 GB. After eight of them, all 64 cores are gone. But only 256 GB of the 512 GB of memory has been used. The other 256 GB is free, working, powered — and completely unrentable, because there is no core left to attach it to.

This is called memory stranding. The memory is not broken and it is not full. It is simply stuck on the wrong side of a machine boundary.

The reason this matters is money. In a modern server, DRAM (

A datacenter operator is therefore paying for a large amount of memory that earns nothing.

What people tried before, and why it was not enough

The obvious fix is to stop putting all the memory inside individual servers. Instead, put some of it in a shared pool that several servers can borrow from. A server that runs short takes a slice; a server with spare capacity returns it. This idea is called memory disaggregation, and it is not new.

The problem was always speed. Earlier pooling systems reached the pooled memory over a network link. That turns a memory access from something that takes tens of nanoseconds into something that takes microseconds — roughly a hundred times longer. To hide that, those systems had to trap accesses with

For a public cloud provider, both are unacceptable, for a reason that is easy to state and hard to work around: the provider cannot see inside the customer’s virtual machine, and must not. The virtual machine is opaque. The provider does not know whether it is running a database, a game server or a compiler, and is not allowed to inspect it or modify it.

So the requirement is strict. Pooled memory must be:

1. As fast as ordinary local memory, or close to it.
2. Invisible to the customer — no guest changes, no new drivers.
3. Compatible with virtualization accelerators, which need memory to be allocated once, up front, and never moved.

The piece of hardware that changed the situation

CXL stands for Compute Express Link. It is an industry standard that runs over the same physical wiring as PCIe, the connector normally used for graphics cards and network cards. The important property is this: CXL lets a processor reach memory on another device using ordinary load and store instructions, the same instructions it uses for its own memory.

This is the change that makes the whole paper possible. There is no page fault, no driver call, no network packet. The memory simply appears in the address space, and reading it costs nanoseconds rather than microseconds.

CXL memory is still slower than memory attached directly to the processor. The question the paper has to answer is: how much slower, and does that matter?

Insight one: keep the pool small

The natural instinct is to build the biggest pool possible. A pool shared by a whole rack of servers gives the most flexibility, because any spare memory anywhere can be used anywhere.

Pond’s authors measured what actually happens, and found the instinct is wrong. Most of the savings arrive very early: a pool shared by 8 to 16 processor sockets already captures most of the benefit. Going bigger adds little.

That matters because pool size and speed pull in opposite directions. A larger pool needs CXL switches to connect everything, and each switch adds at least 70 ns, with estimates above 100 ns. Pond avoids switches. It uses a single external memory controller — one chip with several CXL ports that presents itself to each of 8–16 sockets as a memory device.

So the first insight is a refusal: do not build the ambitious version. The small pool is nearly as good and much faster.

Insight two: a lot of memory is paid for and never read

The second insight comes from looking at what virtual machines actually do with the memory they ask for.

Customers ask for a size. They rarely use all of it. Measuring across Azure’s production fleet, the authors found that the median virtual machine never touches 50% of the memory it was given. Half the memory is bought, allocated, and never read or written even once.

Now the two insights combine into the paper’s idea:

Untouched memory does not care how fast it is. If a virtual machine is never going to read a page, it costs nothing to put that page on slower pooled memory.

The difficulty is that the provider cannot look inside the virtual machine to find out which memory is untouched. It has to predict it, before the virtual machine starts.

The mechanism: a NUMA node with no cores

Pond’s trick for steering a guest operating system away from pool memory is elegant, and it requires no cooperation from the guest at all.

Recall that operating systems already prefer memory on their own NUMA node. Pond exploits this by showing the virtual machine a NUMA node that contains memory but no CPUs. The paper calls this a zNUMA node — a zero-core NUMA node.

Because the guest operating system sees a node with no processors on it, its normal memory allocation policy fills the local node first, and only spills into the zNUMA node when local memory runs out. If the zNUMA node is sized to match the amount of memory the virtual machine was never going to touch, then in practice the guest never reads it.

The guest needs no modification. It is simply following the NUMA rules it already follows.

request: 8 cores, 32 GB
known:   customer id, VM size, past history
unknown: everything running inside

Why this works

The causal chain is worth writing out, because each link depends on the one before it.

Because CXL carries ordinary loads and stores, pooled memory can be reached without page faults. → Because there are no page faults, memory can stay statically allocated, which is what virtualization accelerators require. → Because the pool is limited to 8–16 sockets and uses no switches, the extra cost is only 70–90 ns rather than 180 ns or more. → Because half of a typical virtual machine’s memory is never read, most of the pooled memory is never actually accessed. → Because a NUMA node with no cores is filled last by any normal operating system, the untouched memory lands in the pool without changing the guest. → Therefore the provider recovers stranded memory while the customer sees performance within a few percent of ordinary local memory.

Notice that the last step depends on a prediction being right. That is why the monitoring and mitigation stage exists: it converts a wrong prediction from a broken promise into a one-time 50 ms-per-gigabyte repair.

What it costs

Every design gives something up, and this one gives up three things.

• New hardware. Pond needs an external memory controller ASIC with several CXL ports and its own DDR5 channels. This is a real chip that must be built, not a software change.
• Prediction risk. Two machine-learning models sit in the path of every placement decision. When they are wrong, a customer’s workload runs slower until the mitigation completes.
• A latency floor that never goes away. Pooled memory is permanently 70–90 ns further away. The design works by making sure that distance is rarely paid, not by removing it.

What the measurements showed

The evaluation ran 158 cloud workloads — in-memory databases and key-value stores (Redis, VoltDB, TPC-H on MySQL), data and graph processing (Spark, GAPBS), high-performance computing (SPLASH2x), CPU benchmarks (SPEC CPU, PARSEC), and a set of Azure’s own internal workloads. With seven different zNUMA sizes and at least three repetitions each, this came to more than 3,500 experiments and 10,000 machine hours.

Servers were production Azure machines: two Intel Skylake 8157M sockets with 384 GB each, or two AMD EPYC 7452 sockets with 512 GB each. On the Intel machines, local memory latency measured 78 ns and remote-socket latency 142 ns. The DRAM savings themselves were estimated by simulation, driven by the production traces from 100 clusters.

A 7% reduction in DRAM cost may sound modest. Put it next to the earlier number: DRAM is roughly 50% of server cost, and a hyperscale provider buys millions of servers. A few percent of that number is very large.