Coach: Exploiting Temporal Patterns for All-Resource Oversubscription in Cloud Platforms

Cloud servers sit half idle, and everyone knows the processor is the most wasted resource — but Azure's own measurements show that fixing the processor just moves the shortage somewhere else, with memory becoming the binding constraint on 49 percent of servers instead of 29; Coach (Microsoft and Illinois) therefore oversubscribes everything at once, and does it by noticing that some machines peak at noon and others at midnight, so a virtual machine's allocation should be a schedule rather than a number.

Why this problem is worth solving

The economics here are simple and large.

One category of waste is less obvious than the rest.

What people did before, and where it broke

• Sell held-back capacity cheaply as interruptible instances. Spot and burstable offerings, widely deployed. Where it broke. They address only the capacity deliberately kept in reserve, not the capacity already sold to someone who is not using it, which the measurements here show is the larger pool.
• Pool memory across servers so unsellable memory becomes sellable. The established answer to the stranding problem, and a good one. Where it broke. It works for memory and is unavailable for processors or network. Once processors are oversubscribed, network becomes the binding constraint on 18 percent of servers, and no pooling exists for it.
• Let special instances borrow what their neighbours are not using. Harvest-style virtual machines, deployed in production. Where it broke. The customer must modify their workload to cope with being evicted and with an allocation that changes underneath them. That rules out the ordinary unmodified applications which are most of the fleet.
• Oversubscribe processors, the most underutilized resource. The obvious target, and where most published work concentrates. Where it broke. It relocates the problem. Processor stranding rises from 8 to 25 percent as cores that were merely unused become unsellable, and the binding constraint moves to memory and network — so the gain is much smaller than the processor numbers suggest.
• Give each machine one oversubscription rate for its lifetime. How existing oversubscription works, predicting a single ratio per resource. Where it broke. It has to assume the peak, so a machine that runs at 30 percent for sixteen hours and 75 percent for eight is treated as needing 75 percent all day. Measured across a cluster, that leaves roughly half the achievable savings on the table.
• The gap. Three separate causes of waste with three separate fixes, when the causes are coupled through the server. And a scheduling model that describes a machine’s needs with one number when its behaviour is a repeating daily shape.

How Coach recovers capacity

The idea

The three causes of low utilization are coupled, so fixing the worst one relocates the problem rather than solving it. This is the measurement that justifies the paper existing, and it is unusually clean. Across a million virtual machines in ten Azure clusters, processors are the least stranded resource at 8 percent while memory, network and storage strand at 18, 29 and 54 percent — and processors are the binding constraint on new placements 69 percent of the time. So the obvious move is to oversubscribe processors, which is what most prior work does. Doing it in simulation shows exactly what goes wrong: processor stranding rises to 25 percent, because cores that were merely allocated-and-unused become unallocated-but-unsellable, and the binding constraint moves to memory at 49 percent and network at 18. The waste does not disappear, it migrates. And the existing fix for stranding, pooling memory across servers, is available for memory and not for processors or network. The conclusion is that partial solutions have a ceiling set by whichever resource they push the shortage onto. Attacking stranded capacity by pooling it is the approach of the memory pooling work covered elsewhere on this site, which this paper positions as necessary and insufficient.

A machine’s resource needs are a daily shape, not a number, and the shapes are complementary. The second observation is what makes all-resource oversubscription tractable rather than just necessary. Machines have consistent rhythms — the measurements show that with four six-hour windows, 80 percent of machines vary by at most 20 percent in processor and 5 percent in memory between the same window on consecutive days. Since some peak at noon and others at night, a scheduler that knows the shapes can pair complementary machines on one server instead of having each reserve its own peak. Quantified across a cluster, a single daily window recovers about 8 percent of both processor and memory; four six-hour windows recover about 20 percent of processor and 15 percent of memory; and multiplexing at five-minute granularity, the unattainable ideal, recovers 34 and 18. Most of what is available comes from a handful of coarse windows, and savings plateau at around six four-hour ones. The scheduling change this implies is modest — compare per-window vectors instead of single numbers, adding one dimension per window per resource, which the paper measures as under a millisecond per placement. Predicting behaviour from a customer’s own history using features the platform already collects is the same infrastructure as the on-node learning work covered elsewhere on this site.

The design is built around not trusting the prediction, and says so explicitly. What makes this deployable rather than merely clever is how much savings it deliberately declines. Every resource splits into a guaranteed portion that is always physically present and an oversubscribed portion supplied on demand — and for memory the guaranteed portion uses statically mapped physical pages that cannot be resized while the machine runs, so it is sized by the highest percentile across all windows rather than per window, giving up the very savings the time-window insight identified. The oversubscribed portion is exposed to the guest as a memory node with no processors attached, which makes the guest operating system naturally prefer the fast portion without any modification. The paper states that it oversubscribes conservatively despite opportunities to save more, that it avoids techniques requiring knowledge of what runs inside the machine, and that over-allocating is acceptable because performance is prioritised. It even reports abandoning more sophisticated learning approaches because the gains were marginal and the decisions were too hard for operators to interpret. Keeping the customer’s workload unmodified, rather than requiring it to handle a shifting allocation, is precisely what separates this from the memory harvesting work covered elsewhere on this site.

Placing two machines on one server

The model is a random forest predicting in five percent buckets, chosen over gradient-boosted alternatives specifically because it overfits less — which matters here because an underprediction causes contention while an overprediction only costs savings.

Why it works

• The binding constraint moves when you relieve it. Processors are the scarce resource on 69 percent of servers, so oversubscribing them helps → but that pushes the constraint onto memory at 49 percent and network at 18, where the existing pooling fix does not apply → so any partial approach has a ceiling set by whatever it makes scarce next.
• Daily rhythms are stable enough to schedule against. With four six-hour windows, 80 percent of machines vary by at most 20 percent in processor and 5 percent in memory between the same window on consecutive days → so a forecast made once holds for days → and because some peak at noon and others at night, pairing them recovers roughly twice what a lifetime peak allows.
• Coarse windows capture most of the available benefit. A single daily window recovers about 8 percent of each resource while the five-minute ideal recovers 34 and 18 → four six-hour windows already reach 20 and 15, plateauing around six → so the scheduling cost stays small while most of the opportunity is taken.
• Splitting each resource decouples safety from savings. The guaranteed portion is statically mapped and fast, the borrowed portion is allocated on demand and can fault → so the split point becomes a dial between performance and density → and exposing the borrowed portion as a memory node without processors makes an unmodified guest prefer the fast half by itself.
• Overprediction is much cheaper than underprediction, so the design leans that way. An overprediction costs only savings while an underprediction costs a customer’s performance → so the forecaster is chosen for resistance to overfitting, the immovable portion is sized by the worst window rather than each window → and only an underprediction that lowers the maximum across all windows actually causes an under-allocation.

What it costs (trade-offs and scope)

• The headline capacity gain is mostly not from the new idea. Existing single-rate oversubscription already delivers 22 percent more capacity over no oversubscription. Time-window scheduling adds 16 percent on top of that, and an aggressive variant a further 9. The temporal-pattern contribution is the middle number.
• The design gives up savings on purpose, repeatedly. The immovable memory portion is sized by the worst window rather than each window. Over-allocation is described as acceptable because performance is prioritised. The paper says outright that it oversubscribes conservatively despite opportunities to save more.
• Some workloads still lose. Fine-tuning a language model runs 1.24 times slower even under the conservative configuration, because its large working set and per-iteration allocation churn stress on-demand paging. A fully oversubscribed configuration degrades tail latency by up to 2.35 times, and an under-allocated one by 1.8.
• The measured underutilization is a lower bound and the paper says so. Telemetry samples the maximum over five-minute intervals, so finer-grained idleness is invisible. The ideal five-minute multiplexing figure is itself computed from that same coarse data.
• Transparency has an exception. Machines using device pass-through or direct memory access can fault on unbacked regions, which requires either address translation hardware most devices lack, or paravirtualised cooperation from the guest declaring which ranges it uses for input and output.
• Cluster-scale results are simulated. The simulator runs the real production scheduler code on real traces and is validated against the real scheduler’s constraints, and contention is estimated by replaying utilization onto the resulting placements rather than measured.
• Simplicity is a documented retreat. An earlier version used more sophisticated learning; the paper reports it gave marginal improvements with decisions too opaque for operators to troubleshoot, so they moved back to interpretable higher-level concepts.
• It is not deployed as evaluated. The rollout plan starts with fungible resources at low rates and expands as confidence in the mitigations builds, and the new instance type may be opt-in and discounted to compensate customers for accepting degradation risk.
• The monitoring costs real hardware. Tracking accesses for page reclamation needs two hyper-threaded cores per server, about 1.25 percent, so the 16 percent processor saving nets out at 15.

The evaluation

Check yourself

Sources

The paper is “Coach: Exploiting Temporal Patterns for All-Resource Oversubscription in Cloud Platforms” by Benjamin Reidys, Pantea Zardoshti, Inigo Goiri, Celine Irvene, Daniel S. Berger, Haoran Ma, Kapil Arya, Eli Cortez, Taylor Stark, Eugene Bak, Mehmet Iyigun, Stanko Novakovic, Lisa Hsu, Karel Trueba, Abhisek Pan, Chetan Bansal, Saravan Rajmohan, Jian Huang and Ricardo Bianchini, published at ASPLOS 2025 Volume 1, pages 164-181, DOI 10.1145/3669940.3707226. Most authors are at Microsoft; two interned there. It is open access under a Creative Commons Attribution licence.

This explainer is written from the full text, read from the authors’ copy on arXiv (2501.11179). Everything above comes from the paper’s own sections 2 through 5: the characterization of over a million virtual machines across ten Azure clusters, including the finding that machines lasting more than a day are 28 percent of machines but 96 percent of resource hours, the stranding figures for each resource type, the bottleneck distribution and its variation across clusters, and the simulated effect of oversubscribing processors and then memory on both stranding and the binding constraint; the utilization study showing processor variation reaching 60 percent while half of machines vary within 10 percent on memory, the day-to-day consistency of peaks and valleys under different window lengths, the savings achievable at different window granularities including the five-minute ideal, and the analysis of predicting a new machine from a customer’s previous ones; the section 3 design of the new instance type with its physically and virtually addressed memory portions, the memory node without processors used to funnel accesses, the working-set trade-off study, and the treatment of direct device access through address translation hardware or guest cooperation; the scheduling formulation with its per-window vectors, the separate check for the static portion, the multiplexing of borrowed demand, and the discussion of percentile and window-length choice; the monitoring, two-level contention prediction and escalating mitigation policies; the design philosophy section describing the retreat from more sophisticated learning, the staged rollout plan and the possibility of discounted opt-in instances; and the section 4 evaluation covering workload performance across four instance configurations, prediction accuracy, packing and contention across four scheduling policies, the mitigation comparison, and the platform overheads.