When a large-scale failure crunches a cloud's capacity, the operator and the applications each try to cope on their own — the operator treats every app as a black box and can accidentally kill the parts that matter most, while the apps shed load blind to how bad the outage really is; this paper (UC Irvine — Kapil Agrawal and Sangeetha Abdu Jyothi) makes the two cooperate in a public cloud without exposing app internals, by letting apps put simple criticality tags on their containers and having a Phoenix planner-and-scheduler turn off only the non-critical containers to keep the critical services alive, improving critical-service availability by up to 2 times and planning a 100,000-node recovery in under 10 seconds.
The argument in one glance
-
Problem
When a disaster crunches capacity, something has to be turned off — but who decides what?
Large data centers fail in big ways: weather, human error, hardware faults, and load spikes can knock out a large fraction of a cluster at once, and a large share of production incidents trace back to such infrastructure failures. When that happens there is suddenly not enough capacity to run everything, so parts of the running applications must be scaled back or turned off. The question is who decides what gets cut. The cloud operator sees the whole cluster and can move fast, but in a public cloud it cannot see inside a tenant application — it is a black box. The application knows which of its own parts matter, but it cannot see how bad the outage is. Today these two act separately, and that separation is what makes failures hurt more than they should.
-
What people did
Let the operator and the apps each degrade on their own
In practice the operator runs its own resilience heuristics — inferring which virtual machines look user-facing from their access patterns, preempting low-priority pods, assuming one long outage is better than many short ones — while each application runs its own graceful degradation, shedding load, dimming optional features, or tripping circuit breakers. Both are trying to keep things alive, but neither tells the other what it knows.
-
Why it fell short
Acting blind, the operator kills critical parts and the apps degrade obliviously
Because the operator treats each app as a black box, its heuristics can inadvertently terminate exactly the components an application considers critical, while sparing ones the app would happily give up. Meanwhile the apps degrade obliviously — they cannot see the extent of the infrastructure failure, so they cannot prioritize sensibly against it. The one cooperative approach that exists, Meta Defcon, works only in a first-party cloud because it requires developers to add in-application knobs; a public cloud provider cannot modify tenant code, so that door is closed.
-
This paper's idea
Make them cooperate through the container — tag criticality, turn off only the non-critical
Phoenix has applications attach simple criticality tags to their containers, which tells the operator what matters without revealing any business logic — the container is exactly the right granularity, since microservice apps are already built as tagged containers. Armed with those tags, a central planner ranks every container across all apps by criticality and the operator objective, and a scheduler turns off the non-critical containers to free capacity for the critical ones. This diagonal scaling keeps critical services alive during a crunch, raising their availability by up to 2 times, and plans a 100,000-node recovery in under 10 seconds.
Why this problem is worth solving
The reason failures hurt is not just the failure itself but the loss of coordination exactly when coordination matters most. Graceful degradation is the practice of keeping a system useful, at reduced performance or with some features off, instead of letting it fail completely when resources run short. During a large-scale failure — a disaster that takes out a big fraction of a cluster — the surviving machines cannot run everything that was running before, so a capacity crunch forces choices about what to keep. Applications already know how to degrade themselves: they shed excess load, dim optional features (a technique called brownout), or trip circuit breakers to protect the core. Infrastructure operators also degrade: they preempt low-priority pods and reclaim capacity. The trouble is that both degrade in isolation. The app cannot see how widespread the outage is, so it degrades obliviously, and the operator cannot see inside the app, so it degrades blindly. Coordination is most valuable precisely under a crunch, and today it is missing, which is why the same outage causes far more damage to the services users actually care about than it needs to.
What makes the problem hard, rather than merely unsolved, is the black-box wall between provider and tenant. In a public cloud the provider runs applications it did not write and cannot inspect: for privacy, security, and simple practicality, each tenant application is a black box whose internal logic the operator never sees. This is why the operator’s resilience heuristics have to guess — inferring which virtual machines are user-facing from how they are accessed, for example — and why those guesses can go badly wrong, terminating a component the app treated as critical because from the outside it looked idle. The one system that made the operator and apps cooperate, Meta’s Defcon, sidesteps the black box by having developers add explicit degradation knobs to their own code, which is fine inside a company that owns both the apps and the cloud, but impossible in a public cloud where the provider cannot modify tenant code and tenants will not expose their business logic. So the real question is not just whether cooperation helps, but whether it can be achieved without ever piercing the black box. That constraint is what rules out the obvious answers and forces a design that carries just enough information across the wall to be useful and nothing more.
What people did before, and where it broke
- Operator black-box heuristics. The cloud provider infers criticality from the outside — which VMs look user-facing from their access patterns, which pods have low priority — and preempts accordingly to reclaim capacity. Where it broke. The guesses are wrong often enough to matter: without knowing the app’s own view, the operator can terminate the components the app considers critical while sparing ones it would gladly drop, so a failure hits the services users care about hardest.
- Application-level graceful degradation. The app sheds load, dims optional features via brownout, or trips circuit breakers to protect its core when it senses pressure. Where it broke. The app degrades obliviously, unaware of how extensive the infrastructure failure is, so it cannot prioritize against the real capacity available, and it has no way to make the operator spare its critical parts.
- Defcon-style cooperative degradation in a first-party cloud. Meta’s Defcon lets developers add in-application knobs that expose degradable features to the infrastructure, so operator and app cooperate. Where it broke. It requires modifying the applications and relies on the provider owning both the apps and the cloud; in a public cloud the provider cannot change tenant code and tenants will not reveal their logic, so this approach does not transfer.
- Priority-based pod preemption without app cooperation. Schedulers preempt lower-priority pods to make room, using priorities set independently of any coordinated, application-aware policy. Where it broke. Priorities are set per-app in isolation and are not reconciled against a site-wide objective, so during a real crunch there is no coherent, cluster-wide notion of what to keep alive across all tenants.
- The gap. No mechanism let a public-cloud operator perform graceful degradation that respects each application’s own criticality — carrying just enough information across the black-box wall to know what to keep — while reallocating the freed capacity toward the critical services under a cluster-wide objective.
How Phoenix works
Turning independent, blind degradation into cooperative, criticality-aware degradation
-
-
new
-
new
-
new
-
new
-
new
The idea
Carry criticality across the black-box wall, and nothing else. The whole design turns on a single, narrow piece of information: for each container, how much the application cares about it. That is enough for the operator to make good choices, and it reveals nothing about what the container actually does. Criticality tags express this as a small set of levels — C1 for the parts that drive the business down to C5 for good-to-have extras — so Overleaf can say its chat service is more expendable than its editor without disclosing how either works. Choosing the container as the unit is what makes this clean: microservice applications are already decomposed into tagged containers, the resources freed by turning one off are trivially readable from its spec, and the tag rides on tagging machinery the cluster management framework already provides. Deciding to move only the minimum useful signal across a trust boundary, rather than exposing internals, is the same efficiency-under-a-black-box discipline the SLA-guaranteed resource management for shared microservices covered elsewhere on this site is built on, allocating to services by their externally-observed requirements instead of prying into their code.
Diagonal scaling makes “turn something off” a first-class resource move. Classic scaling has two axes: horizontal (add or drop replicas) and vertical (grow or shrink a container). Phoenix adds a third, orthogonal one — diagonal scaling — that prunes an application by switching off its non-critical microservices entirely during a crunch. This reframes resilience itself: instead of an app being simply up or down, it now has many valid intermediate states where only a chosen subset of components runs, so a resilience target like the Recovery Time Objective becomes a range rather than a single number — stringent for critical sub-services, lenient for auxiliary ones. Reclaiming capacity from where it is least needed and steering it to where it is needed most, rather than treating capacity as fixed, is the same reclamation instinct the memory-harvesting work for cloud platforms covered elsewhere on this site relies on, gathering idle resources and putting them to productive use when demand appears.
Rank locally, then reconcile globally against one objective. The planner deliberately splits the decision into two clean stages, because the two questions are different. Within an application, the only thing that matters is the app’s own structure — you order its containers by walking the dependency graph from the entry points inward, using criticality to break ties, so that no microservice is switched on before at least one of its predecessors and the ordering is both topology-aware and criticality-aware. Across applications, the question becomes the operator’s: whose critical container should win the next scarce unit of capacity, decided by a cluster-wide objective such as max-min fairness or revenue. A priority queue interleaves the per-app lists under that objective into one global order. Separating a per-tenant preference from a site-wide reconciliation objective, so both are honored without one overriding the other, is the same layered-objective structure the uncertainty-aware multi-stage resource management covered elsewhere on this site is organized around, optimizing each stage’s local needs under one coherent end-to-end goal.
Following a large-scale failure through Phoenix
From a datacenter outage to a critical-services-first recovery
A trace, drawn from the paper's mechanism, of how Phoenix responds when a large-scale failure suddenly removes much of a cluster's capacity.
Step 1 of 5
A large failure removes much of the cluster
A disaster — say a power or network event — takes out a big fraction of the servers at once. The surviving capacity cannot host all the containers that were running, so without intervention the cluster scheduler will drop whatever it happens to drop, possibly including critical services.
Phoenix targets stateless workloads, which the paper notes make up more than 60 percent of resource utilization in large data centers; stateful services are left for future work.
Why it works
- App-aware information beats guessing. The operator’s black-box heuristics can only infer what matters and sometimes kill critical parts → but a criticality tag states it directly, at container granularity, without exposing logic → so the operator keeps the right services alive, which is what lifts critical-service availability by up to 2 times over non-cooperative baselines.
- Diagonal scaling frees capacity where it is cheapest to give up. Turning off a non-critical container releases resources that are easy to estimate from its spec → but those resources are exactly what the critical services need under a crunch → so pruning the least-valued work funds the most-valued work instead of failing everything uniformly.
- Splitting per-app and cross-app ranking honors both views. A dependency-graph traversal captures what each app needs internally → but a separate global objective decides who wins across tenants → so neither a single app nor a blind cluster policy dominates, and the freed capacity is allocated coherently site-wide.
- A heuristic scheduler stays fast enough to run mid-disaster. Optimal container-to-server packing is NP-hard and too slow for a live outage → but best-fit plus migration plus last-resort deletion approximates it cheaply → so the planner clears a 100,000-node cluster in under 10 seconds instead of stalling when speed matters most.
- Container-level cooperation reuses what the cloud already has. Microservice apps are already built as independently deployed, taggable containers → but that means criticality tags need no new abstraction and freed capacity is readable from container specs → so cooperation is achievable in today’s public clouds without modifying tenant applications.
What it costs (trade-offs and scope)
- Scope is stateless workloads. Phoenix targets stateless microservices, which the paper says account for more than 60 percent of resource utilization in large data centers; turning off and restarting stateful services safely is harder and is explicitly left as future work, so the availability gains do not automatically extend to databases and other stateful components.
- Cooperation depends on honest, sensible tagging. The whole benefit rests on applications attaching accurate criticality tags. The paper offers help — stub-microservice-based and frequency-based tagging methods, and a chaos-testing framework to check them — but manual tagging is tedious for large apps built by many teams, and a mis-tagged critical service could still be turned off.
- Turning off components is a real loss of function. Diagonal scaling keeps critical services alive by shutting non-critical ones down, so users of the pruned features see them disappear during a crunch; the design assumes apps can meaningfully declare which features are safe to lose, which not every application can do cleanly.
- The scheduler is a heuristic, not an optimum. Because packing is NP-hard, Phoenix approximates it with best-fit, migration, and deletion; this is what keeps it fast, but it means the placement is not guaranteed optimal, and the last-resort deletion of low-priority containers adds churn during an already-stressful event.
- Evidence is one real cluster plus simulation. The strongest real-system numbers come from a 200-CPU Kubernetes cluster running two applications (Overleaf and a hotel-reservation service); the very large scales — up to 100,000 nodes — are shown through the AdaptLab simulator replaying Alibaba traces, so behavior on other real large production clusters is inferred rather than directly measured.
The evaluation
Responding to a large-scale failure in a public cloud
| Property | Default Kubernetes | Priority (tags, non-cooperative) | Phoenix (cooperative) |
|---|---|---|---|
| App criticality visible to operator | |||
| How capacity is reclaimed | |||
| Cluster-wide objective | |||
| Planning at 100,000 nodes | |||
| Critical-service availability under failure |
Check yourself
Check your understanding
1. Why do independent resilience solutions hurt availability during a large-scale failure in a public cloud?
2. What is diagonal scaling, and how does it differ from horizontal and vertical scaling?
3. How does the Phoenix planner decide what to keep running, and why is it split into two steps?
4. What are the main results and the stated scope of Phoenix?
Sources
The paper is “Cooperative Graceful Degradation in Containerized Clouds” by Kapil Agrawal and Sangeetha Abdu Jyothi (University of California, Irvine), published at ASPLOS 2025, DOI 10.1145/3669940.3707244.
This explainer is based on the paper’s full text, obtained from the authors’ arXiv preprint (arXiv:2312.12809). The account of the mechanism — the motivating problem (independent operator and application resilience solutions during a capacity crunch, and the public-cloud black-box constraint that rules out Defcon-style app modification), criticality tags on containers (levels C1 through C5), diagonal scaling as a third scaling axis, the two-step Phoenix planner (a priority estimator that walks each application’s dependency graph and criticality tags, and a global ranking module that reconciles across applications under a fairness or revenue objective), the bin-packing scheduler with its best-fit, migration, and deletion heuristic, and the agent that enacts the plan on Kubernetes — follows the paper’s own description. The reported results cited here are the paper’s own: critical-service availability up to 2 times higher than non-cooperative baselines, planning a 100,000-node cluster in under 10 seconds via the AdaptLab simulator (replaying Alibaba traces), and full recovery in under 4 minutes on a real 200-CPU Kubernetes cluster running Overleaf and a hotel-reservation application. The paper’s scope is stateless workloads. Exact per-experiment numbers, the appendix algorithms, and the linear-program formulations should be read from the paper itself.