SOL: safe on-node learning in cloud platforms

Every server in a cloud runs dozens of small management programs that decide things — how many cores to lend out, how fast to run the processor, which memory pages are cold — and they all decide by fixed rules, because the machine learning that would decide better has to run beside paying customers where a bad model does real damage; this paper (Microsoft and Stanford) surveys the 77 such programs on an Azure node, finds that a third could benefit from learning, and builds a framework whose central idea is that the part that learns and the part that acts must be separated, so the acting half keeps making safe decisions while the learning half is starved, wrong, or dead.

Why this problem is worth solving

The setting is unfamiliar enough to be worth describing before the mechanism.

The second question is why the learning has to happen in that awkward place.

What people did before, and where it broke

• Static heuristics. What production agents use today. A threshold, a fixed frequency, a fixed sampling rate. They are predictable, cheap, and they never surprise anyone, which is exactly why they survive in a system where surprises are expensive.

• The gap. They cannot adapt. A fixed overclocking policy raises the frequency for a disk-bound workload that gains nothing, and a fixed memory scanning rate either wastes effort on quiet memory or misses the activity that matters.

• Centralised prediction services. Train offline on fleet-wide data, serve answers over a network interface. This is the established production pattern and it works for many cloud decisions.

• The gap. Bounded by network latency and bandwidth in both directions. Neither the update rate nor the answer rate can reach the millisecond decisions these agents need, and the input telemetry is too voluminous to send at all.

• One-off learned agents. A line of research has already built individual on-node learning agents — harvesting idle cores, managing power, routing storage requests — and demonstrated that each beats its static policy.

• The gap. Each is built for its own purpose, and each treats the model as the contribution. None addresses what happens when the model is wrong, when the data is corrupt, or when the agent is starved, which is what stands between a research result and something running next to customers.

• Careful offline vetting before deployment. The standard discipline for putting a model into production — evaluate it thoroughly, and only ship it once it behaves.

• The gap. Not available here. The model is trained on this node, on a workload inside an opaque virtual machine, after deployment. There is no earlier moment at which it could have been checked, so the protection has to be built into the runtime instead.

How a learning agent is structured so that failure is survivable

The idea

Start by counting what is actually there. The paper opens with a survey rather than a technique, and it is the part that gives the rest its authority. Of the 77 agents on an Azure node, 25 configure the machine, 23 run node services, 18 monitor and log, 7 watch for failures, 2 control resources, and 2 provide operator access. The authors then ask which classes would gain from learning and answer three of the six — resource control, because the best allocation depends entirely on the current workload; monitoring, because it currently treats every sample as equally valuable and so oversamples quiet periods and undersamples busy ones; and watchdogs, because they can only check conditions simple enough to be certain, which means they detect problems after customers already feel them. Those three classes are 35 percent of the agents, and notably the two smallest classes by count are among them, because what matters is decision frequency rather than population. Establishing the opportunity by measuring the fleet before proposing anything is the same discipline as the memory-harvesting work covered elsewhere on this site, which begins by showing how much memory sits unallocated before arguing anyone should reclaim it.

The learner and the actor must not share a fate. The central design decision is that the expensive half and the safety-critical half run independently. Collecting telemetry and training a model may take substantial computation, and this is a process that can be throttled without warning by a hypervisor busy with something more important. Taking the control action is cheap — set a frequency, return some cores, migrate some pages. If those live in one loop, then whenever the learner is starved the actor is starved too, and whatever decision was last in force stays in force. Splitting them means the actor keeps running to its own schedule and acts on whatever it has, which may be a fresh prediction, an expired one it must discard, or nothing at all. The measurement that justifies this is direct — inject a 30-second delay into the learning thread just as a workload finishes a burst, and a design that waits for its prediction keeps the processor overclocked for 30 seconds into an idle phase and burns 36 percent more power, while a design that gives up after five seconds and reverts to the nominal frequency burns 3 percent. Deciding what a system does when its clever component is unavailable is exactly the discipline that the kernel machine-learning work covered elsewhere on this site had to build for itself, where a policy chooses between the accelerator and the processor while the system runs.

Four safeguards, ordered by how early each can see trouble. The framework does not supply the safety logic, because what counts as harm differs per agent. It supplies the obligation — a developer cannot implement an agent without filling in all four. Validating each data point is the earliest, catching corrupt readings before they are learned; without it, feeding just 5 percent invalid readings to the overclocking agent costs 17 percent of the workload’s performance. Assessing the model catches a learner that is training on uninterpretable data or on a workload it cannot model; without it, a broken policy that always overclocks raises the power draw of a disk-bound workload by 268 percent, against 18 percent with the check in place. Prediction expiry catches staleness. And the end-to-end check on the outcome is the last line, explicitly a lagging indicator — by the time customer latency has moved, the harm has happened — which is why the earlier three exist. Watching a proxy for customer pain and backing off when it moves is precisely how the memory offloading work covered elsewhere on this site decides how aggressively to reclaim.

Four chances to stop a bad decision

This matters more than it sounds. Telemetry across millions of machines fails in mundane ways, from misconfigured drivers to counters whose meaning changed between processor generations. Feeding 5 percent bad readings to this agent without the check costs 17 percent of the workload's performance.

Why it works

• Independent loops mean starvation degrades one half only. Running the expensive learning and the cheap actuation on separate schedules → a throttled learner cannot block a control decision → the agent falls back to a safe action instead of leaving a stale one in force, which is worth 36 percent of power against 3 percent in the injected-delay test.
• Expiry times make staleness a detectable condition. Attaching a validity window to every prediction rather than passing a bare value → the consumer can distinguish a current answer from an old one → a delayed agent takes the conservative path instead of confidently acting on a description of a workload that has moved on.
• Blocking predictions while still training allows recovery. Intercepting the model’s output during a bad period, but continuing to update it → the agent stops causing harm without being permanently disabled → it can resume once accuracy returns, which matters because the cause is often a transient workload phase rather than a broken model.
• Discarding uninterpretable samples prevents a systematic bias. In the core-harvesting agent, throwing away data from moments when the customer used every core it had → the model never learns from samples that cannot distinguish “needed exactly this many” from “needed more and could not get it” → it stops systematically underpredicting demand, which was costing 40 percent of customer tail latency.
• The obligation costs almost nothing to meet. Reimplementing a previously published agent inside the framework → 1,990 lines against the original 1,900 → the full set of safeguards is about 90 lines of difference, which is the paper’s strongest argument that the discipline is affordable.

What it costs (trade-offs and scope)

• The framework supplies structure, not safety. Every safeguard is a function the developer writes. What counts as bad data, an inaccurate model, or unacceptable customer impact is agent-specific, so an agent with badly chosen thresholds is unsafe in a well-structured way.
• Safety is bought with efficiency. Every fallback is conservative by construction — nominal frequency, all cores returned, pages left in fast memory. An agent that trips its safeguards often is one that has given up most of the benefit that motivated it.
• The end-to-end check can only react. It measures a proxy for customer harm, so it fires after that harm has begun. On an adversarial memory workload it took minutes to fully resolve a sustained violation, which is why the earlier checks carry most of the weight.
• Proxies stand in for what cannot be seen. A virtual machine is opaque, so the agents reason about instruction throughput, processor stall counters, and virtual-core wait times rather than the customer’s own latency. The overclocking agent assumes higher instruction throughput means the workload benefits, which the authors note is imperfect.
• The evidence is one node with injected faults. Agents on different nodes are independent, so the authors argue a single machine is the right unit — but the failure conditions are deliberately induced rather than observed in production, and there is no fleet-scale deployment result.

The evaluation

Check yourself

Sources

SOL: Safe On-Node Learning in Cloud Platforms. Yawen Wang, Daniel Crankshaw, Neeraja J. Yadwadkar, Daniel Berger, Christos Kozyrakis, Ricardo Bianchini. ASPLOS 2022, pages 622 to 634. DOI 10.1145/3503222.3507704. Full text read from the authors’ preprint, arXiv 2201.10477.