Memory-harvesting VMs in cloud platforms

Alexander Fuerst, Stanko Novakovic, Iñigo Goiri, Gohar Irfan Chaudhry, Prateek Sharma, Kapil Arya and 4 others

Memory is roughly half a cloud server's cost, yet while cores run above 80% allocated, most Azure servers leave 30–50% of memory idle — and harvesting it naively would break the platform's real promises, adding tens of seconds to VM creation and poisoning NUMA locality and large pages for paying customers; MHVMs grow and shrink around regular VMs using a reserved buffer that keeps reclamation off the creation critical path, NUMA-aware and 2MB-contiguous-only reclamation, and elastic Hadoop/FaaS frameworks that hide the resizing — harvesting ~20% of unallocated memory with zero measured impact and cutting user costs up to 93%.

The argument in one glance

Problem

The most expensive resource is the least allocated

Cloud platforms must keep spare capacity so customers can scale and stay available — but idle capacity is unmonetized capital. The paper's three-week Azure study across thousands of servers shows the asymmetry: more than half the servers have over 80% of CORES allocated, while MEMORY allocation sits at 50–70% for most. Memory is roughly 50% of server cost. That gap — 30–50% of the fleet's most expensive component sitting idle — is a very large check nobody was cashing.
What people did

Spot VMs, then core-harvesting VMs

The classic monetization is Spot VMs: fixed-size, discounted, evicted when a regular VM needs the space. Harvest VMs improved on this — they keep a minimum size but dynamically grow and shrink their CPU allocation with the server's unallocated cores, yielding fewer evictions and lower cost than Spot. But their memory is fixed.
Why it fell short

Memory is stateful — flexing it breaks production promises

Fixed memory wastes the monetization opportunity AND caps core harvesting (many harvested cores are useless without memory to feed them). But memory, unlike cores, holds state: reclaiming it means the guest must free pages and return them. At the measured 2–3 GB/s, shrinking puts TENS OF SECONDS on the critical path of creating a paying customer's VM. Harvested memory squats on NUMA nodes, forcing new VMs to span nodes (up to 20% median, 40% worst-case degradation on production-derived workloads). And 4KB-granular reclamation fragments host memory, denying future VMs the large pages that several workloads measurably need. Prior harvesting proposals addressed none of these.
This paper's idea

Harvest with a shock absorber, at 2MB granularity, NUMA-aware

Memory-Harvesting VMs (MHVMs) extend Harvest VMs with dynamic memory managed entirely by the per-server agent. A reserved unallocated-memory BUFFER serves new regular VMs instantly (reclamation replenishes it off the critical path); reclamation runs at 6.2 GB/s via 1GB batches and background pre-reclamation, returns ONLY contiguous 2MB pages (a guest memory partition makes that cheap), respects NUMA via exposed vNUMA topology; and misbehaving MHVMs are evicted within 30 seconds. Elastic Hadoop and FaaS frameworks hide the resizing from users. Production replay: zero added creation time, zero fragmentation, zero extra spanning — while harvesting ~20% of unallocated memory and cutting user costs up to 93%.

Why this problem is worth solving

This is cloud economics with a systems core. A hyperscaler’s fleet must hold slack — for scale-out bursts, failover, and placement flexibility — and the art is monetizing that slack without ever compromising the paying tenants it exists for.

discounted, evictable VMs sold from spare capacity: fixed resources, evicted whenever a regular (full-price) VM needs the space. The standard slack-monetization product on AWS, Azure, and GCP

were the first answer;

Azure’s improvement (Ambati et al.): an evictable VM with a guaranteed minimum that dynamically grows and shrinks its CORE allocation to match whatever is unallocated on its server — fewer evictions and better utilization than Spot, but with memory size fixed

the second. Both leave memory — half the server bill — untouched, and the characterization shows memory is precisely where the slack lives: cores are the packing bottleneck, so servers fill on cores while 30–50% of memory idles.

What makes this a research paper rather than a pricing memo is that memory resists harvesting in ways cores don’t. A core is time-sliced statelessly; taking it back is instant. Memory holds pages that must be given back — by a guest OS, at finite speed, from particular physical addresses. Every one of the platform’s quiet quality guarantees — VMs create in seconds, VM memory is NUMA-local (only 2% of Azure VMs span nodes today), VM memory is backed by performance-critical

2MB (rather than 4KB) mappings in the hypervisor’s page tables, cutting TLB pressure and page-walk depth; Azure backs VMs with contiguous large pages at creation, and the paper measures severe degradation (worst on SQL) when mappings fragment to 4KB

— is a constraint that naive memory reclamation violates. The paper’s contribution is noticing that these three constraints, not the resizing mechanism, are the actual problem — and engineering them away with production-grade care, on real Azure servers, from the team that ships this stack.

What people did before, and where it broke

Spot VMs: fixed size means they can neither shrink to avoid eviction nor grow to soak up newly freed capacity — so they’re evicted more and monetize less.
Harvest VMs / Elastic VMs / Burstable VMs: all flex CPU only. Core-only harvesting either demands a large fixed memory reservation (to be useful when many cores appear) or confines harvesting VMs to CPU-bound, small-footprint jobs.
Deflatable VMs and hypervisor memory-trading research: proposed multi-level memory reclamation but considered none of the production challenges — creation-time impact, NUMA spanning, fragmentation. Working-set estimation and performance-metric-driven resizing policies are considered too invasive for opaque customer VMs in a public cloud.
The mechanisms exist; the system didn’t.

the two standard guest memory-return mechanisms: a balloon driver allocates guest pages so the host can repurpose them (the guest’s size appears unchanged); hot-remove additionally shrinks the guest physical address space, so users see their true size. Hot-ADD grows it. Baseline throughput: ~4.4–4.6 GB/s to reclaim, over 40 GB/s to add

are in every modern OS/hypervisor — but at baseline speeds, with 4KB granularity and no NUMA awareness, using them for harvesting would visibly damage the paying side of the platform.

How MHVMs work

MHVMs: the server agent does everything

Placement & pricing

An MHVM deploys like any VM with a minimum size (charged as a standard evictable VM) and a maximum (e.g., 4× minimum); harvested resources beyond the minimum are billed at a deep discount (α ≈ 0.1). Only the per-server agent knows harvesting exists — no scheduler or control-plane changes. Multiple MHVMs per server share harvest proportionally to minimum size and reclaim in parallel (throughput scales linearly).
↓ →
The unallocated-memory buffer new

A reserved pool (e.g., 192 GB on a 512 GB server) that MHVMs may NOT harvest, spread evenly across NUMA nodes. New regular VMs draw from the buffer instantly; the agent then replenishes it by reclaiming from MHVMs OFF the critical path. The buffer is the shock absorber that decouples customer-visible creation time from guest-speed reclamation.
↓ →
Fast reclamation: batching + pre-reclamation + notification new

Hot-remove upgraded three ways: multi-GB request batches (host asks for 1 GB at a time, not small chunks); a guest driver that pre-reclaims 128 MB chunks in the background during a resize (returning them if unused for 1 s); and a paravirtual notification telling the MHVM its TARGET size up front, so applications release memory once rather than incrementally. Combined: 4.4 → 6.2 GB/s, and in production 8.5–10.6 GB/s.
↓ →
2MB-contiguous reclamation only new

MHVMs must return contiguous, aligned 2 MB pages — never loose 4KB pages — so host memory stays large-page-clean for future regular VMs. A special guest memory partition, created at boot, isolates the harvesting framework's memory from OS daemons, making large contiguous ranges cheap to free without page-trading. A guest that cannot comply is evicted and its memory freed wholesale.
↓ →
NUMA-aware sizing new

MHVMs receive a virtual NUMA topology, and the agent assigns and reclaims memory per-node; the buffer's even spread across nodes maximizes the chance a new regular VM fits entirely in one node. Fleet analysis: with vNUMA plus a 128 GB buffer, harvesting adds ZERO NUMA spanning versus the no-harvesting baseline (buffer alone cannot achieve this).
↓ →
Expedited evictions + elastic frameworks new

Normal eviction gives 30 s warning when the minimum is needed. The agent immediately evicts an MHVM that returns too little, too slowly, non-contiguously, or from the wrong node. Above it all, MH-Hadoop and MH-FaaS absorb resizing: the Hadoop Resource Manager drains or kills containers to fit the shrinking budget; the FaaS controller re-routes invocations toward MHVMs with headroom and treats evictions as server failures.

From Microsoft Azure/Research and Indiana University (ASPLOS 2022) — the production Harvest-VM lineage (Bianchini's group), with Fuerst (of the FaaS keep-alive line) on the academic side. Implemented for Hyper-V with Windows guests; mechanisms validated on KVM and Linux guests too.

The idea

Separate the customer’s clock from the guest’s clock. The deepest obstacle is a timescale mismatch: regular-VM creation is a seconds-scale SLA, while guest-cooperative reclamation is inherently tens-of-seconds work. The buffer resolves it by pre-paying: keep enough unallocated memory on hand that arrivals are served from stock, then restock from the MHVMs asynchronously. The characterization is what makes this cheap — 82% of server-hours see no VM events at all, 60% of arrivals need ≤16 GB, and 90% of changes are under 32 GB — so a 192 GB buffer absorbs essentially everything (production replay: zero added creation time across 144 regular VM creations), while still leaving ~20% of unallocated memory harvestable. The buffer size is the explicit knob trading harvest yield against safety, and the paper charts the whole curve.

Make the invariants constraints on the mechanism, not hopes about behavior. NUMA locality and large-page contiguity are protected structurally: reclamation is only accepted in aligned 2MB chunks and only from the right node, with the guest memory partition arranged at boot so compliance is cheap — and an MHVM that can’t comply is simply evicted (within 30 s, demonstrated against an adversarial guest). This inverts the usual best-effort posture: the paying tenants’ guarantees are enforced by construction, and all slack risk lands on the evictable tenant, which is exactly who signed up for it.

Sell the elasticity through frameworks, not raw VMs. A VM whose memory yo-yos between 32 and 128 GB is a terrible raw product but an excellent substrate: MH-Hadoop’s Resource Manager just runs fewer or more containers, MH-FaaS’s controller routes invocations to wherever headroom currently is, and the end user sees only cheap capacity (the vanilla-FaaS comparison shows what happens otherwise — the guest OS kills containers mid-shrink and invocations fail). This is the same move Harvest VMs made for cores, extended with the memory-specific machinery (proactive target-size notifications so frameworks release memory ahead of the reclaim), and it’s why the harvested capacity is deliverable as SaaS/PaaS/FaaS rather than as a footgun.

An hour in the life of an MHVM

One production server, replayed (512 GB, 192 GB buffer, MHVM min 32 GB)

Regular VM = full-price customer VM with strict SLAs. Buffer = reserved unallocated memory MHVMs cannot touch. Hot-add/hot-remove = guest address-space grow/shrink. Numbers from the paper's mechanism benchmarks and the 10-server production replay (144 regular VMs, 34 MHVMs).

Step 1 of 7

An MHVM lands in the cracks

The server holds regular VMs plus unallocated space. An MHVM with a 32 GB minimum (4 cores) is placed and immediately grows — hot-add runs at over 40 GB/s — to absorb the unallocated memory beyond the protected buffer, up to its maximum.

growth is nearly free; shrinking is the engineering problem
MHVM pricing: minimum at evictable rates + harvested at ~90% off

Why it works

Reclamation is slow (guest-cooperative, GB/s) but arrivals are small and rare (60% ≤16 GB; 82% of server-hours event-free) → a modest reserved buffer converts almost every arrival into an instant allocation, moving reclamation off the SLA path → zero measured creation-time impact while ~20% of unallocated memory is still harvested.
The dangerous side effects (fragmentation, spanning) are properties of HOW memory is returned → constraining the mechanism — 2MB-aligned chunks only, per-NUMA-node requests, guest partition for cheap contiguity — prevents them by construction → regular VMs keep large pages and locality without any monitoring or remediation.
Every compliance failure has the same cheap remedy: evict the MHVM (its tenants signed up for evictions) → the platform never needs to trust the guest → adversarial or broken guests cost at most 30 seconds.
Applications built for elasticity (container schedulers, FaaS controllers) already know how to run at variable scale → a harvesting manager plus target-size notifications lets them resize proactively, in bulk, at large-page granularity → loaded MHVMs reclaim as fast as idle ones, and end users see only cheap SaaS/PaaS/FaaS.
Memory and cores harvested together unlock each other → harvested cores get memory to feed them, and memory-hungry workloads (exactly the ones Spot/Harvest served worst) gain most — MHVMs beat Harvest VMs by 21–45% on cost.

What it costs

Harvest yield is deliberately sacrificed for safety. The buffer holds back capacity (larger buffer, less harvest — the paper’s curves make the trade explicit), and the zero-impact operating point harvests ~20% of unallocated memory, not all of it.
Only elastic, eviction-tolerant workloads apply. Strict performance requirements, limited parallelism, or placement-sensitive memory usage disqualify a workload; the intended delivery is via frameworks, which had to be modified (MH-Hadoop, MH-FaaS).
Guest and framework modifications. A resizing driver, memory partition, and harvesting manager in the guest (implemented for Windows; portable in principle), plus framework extensions — this is provider-integrated machinery, not a bolt-on.
Allocated-but-idle memory stays off the table. MHVMs harvest only UNALLOCATED memory; harvesting memory that regular VMs hold but don’t use would risk their SLAs and is left as future work.
MHVM workloads absorb the variability. Halving memory cost jobs 52% longer runtimes in the replay; the deal is explicit — variability and evictions in exchange for up to 93% discounts.
Information leakage is bounded, not zero. MHVM resizing reveals co-located VM arrivals/departures on that server; regional quotas are the proposed mitigation.

The evaluation

Three layers, increasingly real. Mechanisms: reclamation throughput across VM sizes, guests (Windows/Ubuntu), hypervisors (Hyper-V/KVM), and optimizations — 4.4 GB/s baseline to 6.2 GB/s with batching and pre-reclamation; hot-add above 40 GB/s; parallel MHVMs scale linearly. Fleet analysis: replaying three weeks of traces from thousands of Azure servers to chart harvest yield, creation-time impact, and NUMA spanning against buffer size and MHVM minimums. Production replay: one hour of real arrival/departure traces re-executed on 10 reserved production servers — 144 regular VMs running production-derived workloads, 34 MHVMs running MH-Hadoop/MH-FaaS.

Monetizing spare cloud capacity: three generations

Property	Spot VMs	Harvest VMs (cores only)
What flexes
Protecting the paying tenants	Eviction only	Eviction + core give-back (instant — cores are stateless)
Reclamation performance	N/A	N/A for memory
Value to users (same workload)	~80% off regular; most evictions
Value to the platform	Monetizes spare capacity coarsely, with churn	Better utilization of spare cores

Characterization: thousands of Azure servers, 6 regions, 3 weeks, 4 server generations. Mechanisms: 2-NUMA-node production servers (80 logical cores, 512 GB), Hyper-V (KVM validated), Windows and Ubuntu guests. Production replay: 10 servers, 1 hour of real traces, 144 regular VMs running production-derived workloads (SQL, Search, ML, etc.), 34 MHVMs (min 4 cores/32 GB, max 2×), 192 GB buffer. Costs: public Azure prices, Spot discount 80%, harvested-resource discount α=90–100%.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The headline “up to 93%” is a pricing-model projection (α = 0.9–1.0 discounts on harvested resources, plausible but chosen by the authors), not a shipped price sheet. The production replay is one hour on ten hand-selected servers — carefully chosen to be interesting, but small; the fleet-scale numbers come from trace-driven analysis. Guest-side machinery was fully implemented for Windows guests (Linux argued portable), and the elasticity story depends on framework integration — raw MHVMs would be a hazardous product, as the vanilla-FaaS container-killing comparison shows. Harvesting reaches only UNALLOCATED memory; the bigger pool of allocated-but-idle memory stays future work. And the study predates today’s memory-tiering alternatives (CXL pooling attacks the same stranded-memory economics from the hardware side).

Background

The supply side of cheap compute

This paper is the producer-side counterpart to several consumer-side papers on this site: SpotServe builds LLM serving atop evictable instances, and the serverless cluster (IceBreaker, RainbowCake, and kin — including work by this paper’s own first author) optimizes what runs inside exactly the kind of cheap, variable capacity MHVMs manufacture. It also completes an arc within Azure’s own research line: Harvest VMs flexed cores, this work flexes memory, and the follow-on question it names — harvesting allocated-but-idle memory — is the one CXL-era memory pooling now pursues in hardware. The transferable design lesson is the buffer pattern: when a slow, cooperative reclamation process must coexist with a fast SLA, don’t speed up reclamation to SLA speed — decouple the clocks with a pre-paid reserve, and let the characterization data size it. Add the enforcement stance (invariants guaranteed by mechanism, all residual risk assigned to the party being paid to bear it) and you have a template for monetizing slack in any multi-tenant system.

If you remember only three things

1. Cloud servers fill up on cores while 30–50% of memory — roughly half of server cost — sits unallocated, because Spot VMs are rigid and Harvest VMs flex only CPUs; but memory is stateful, so harvesting it collides with three production invariants prior work ignored: seconds-scale VM creation (guest reclamation runs at GB/s), NUMA locality (spanning costs up to 40%), and large-page contiguity (4KB reclamation fragments the host for every future tenant).
2. MHVMs close each hazard by mechanism rather than monitoring: a reserved buffer serves new regular VMs instantly and is replenished off the critical path (zero added creation time in production replay); reclamation accepts only NUMA-targeted, aligned 2MB chunks from an isolated guest partition, at 6.2–10.6 GB/s via batching, pre-reclamation, and proactive target-size notifications; and any non-compliant MHVM is evicted within 30 seconds — while elastic MH-Hadoop and MH-FaaS hide the resizing entirely from users.
3. At the zero-impact operating point Azure can harvest ~20% of its unallocated memory, and users running memory-hungry elastic workloads save up to 93% versus regular VMs (21–45% even versus core-only Harvest VMs, with FaaS gaining most) — the honest boundaries being that yield is deliberately traded for safety via the buffer, only unallocated (not idle-allocated) memory is touched, and the discount prices in real variability: halve an MHVM's memory and its Hadoop jobs run 52% longer.

Check yourself

Check your understanding

1. Harvest VMs already grew and shrank CPU allocations dynamically in production. Predict why extending the same idea to memory required a research paper's worth of new machinery.
2. The unallocated-memory buffer means Azure deliberately harvests LESS memory than it could. What does the buffer buy, and why does the fleet data say it's cheap?
3. MHVMs are forbidden from returning anything but contiguous, aligned 2MB chunks — and a guest that can't comply is evicted. Why such a hard line on granularity?
4. Why do the authors deliver MHVMs to users through modified frameworks (MH-Hadoop, MH-FaaS) rather than selling them as raw VMs?

Sources

ACM DL: Memory-harvesting VMs in cloud platforms — ASPLOS 2022