Embracing Imbalance: Dynamic Load Shifting among Microservice Containers in Shared Clusters

A microservice usually runs as many identical container replicas, and the standard practice is to spread requests evenly across them — but in a big shared cluster those replicas are not really equal, since some land on busier or slower machines, so balancing forces every replica to be provisioned for the worst case and wastes a lot of resources; this paper (University of Macau and collaborators) argues you should instead embrace the imbalance, building Imbres, a system that dynamically shifts load toward the replicas that can handle it and scales resources in step, steering entirely by live latency gradients with no offline profiling, and reports up to 62 percent less resource use and up to 82 percent fewer SLA violations.

The argument in one glance

Problem

The replicas of one microservice are not really equal, but we pretend they are

A microservice is usually run as many identical container replicas so it can handle load and tolerate failures. In a small deployment those replicas really are interchangeable. But in a large shared cluster, where many services are packed together on the same machines, the replicas of one microservice end up in very different situations: some land on machines that are busy with other tenants, some on quieter ones, and the contention shifts over time. So the replicas experience both spatial imbalance (they differ from each other right now) and temporal imbalance (each one differs from itself over time). The standard way to run such a service ignores this and spreads requests evenly across the replicas, as if they were equal. They are not, and treating them as equal is exactly what makes the service either miss its performance targets or waste resources to avoid missing them.
What people did

Load-balance evenly, and provision for the worst-off replica

The common approach is to distribute requests evenly across a microservice replicas and then, to guarantee the service-level agreement, give every replica enough resources that even the unluckiest, most-contended one still meets its latency target. Scaling adds more replicas when the aggregate is under pressure.
Why it fell short

Balancing forces worst-case provisioning, which wastes resources

If every replica gets an equal share of the load but the replicas are unequal in capability, then the slowest replica sets the bar: to keep it within the SLA you must over-provision the whole service. The result is substantial resource wastage, because most replicas are given headroom they do not need just so the worst one is covered. Load balancing spends resources to hide the imbalance instead of exploiting it, and it does nothing to move work away from a replica that is momentarily struggling.
This paper's idea

Embrace the imbalance — shift load to the capable replicas and scale in step

Imbres inverts the goal: instead of forcing balance, it deliberately makes the load uneven, shifting requests toward the replicas that can handle them and away from the ones that cannot, moment to moment. It co-optimizes three things that must move together — how load is shifted, how upstream and downstream containers are connected (they are only partially connected), and how resources are scaled — and it steers entirely from online gradients of latency, needing no offline profiling. On real microservice benchmarks it reports cutting resource allocation by up to 62 percent and SLA-violation probability by up to 82 percent.

Why this problem is worth solving

The whole design of a microservice deployment assumes its replicas are fungible, and that assumption quietly breaks in a shared cluster.

A microservice is a single function of an application — a catalog service, a recommendation service — run as a set of identical container replicas behind a load balancer, so requests can be spread across them for capacity and resilience. In a shared cluster, many such services from many applications are co-located on the same physical machines to raise utilization, under a unified resource scheduler. The consequence is that two replicas of the same service, meant to be interchangeable, actually run in different micro-environments: one may share its host with noisy, resource-hungry neighbors while another sits on a lightly loaded host, and those conditions change minute to minute as other tenants come and go. So the replicas are unequal in the latency they can deliver for the same work, and that inequality is both spatial (across replicas at one instant) and temporal (within one replica over time). Any scheme that assumes equal replicas is fighting the reality of the cluster it runs in, which is why imbalance is the right thing to design around rather than to suppress.

Because the target is a latency promise, not raw throughput, the way an operator hits it determines the bill.

A service level agreement, or SLA, is a promise about the quality of service, most often stated as a bound on latency — for example, that the 99th-percentile response time stays under some threshold. Meeting an SLA is what operators are ultimately measured on, and it is a tail property, so it is dominated by the worst-behaving requests and replicas rather than the average. The usual way to guarantee a tail latency target under even load balancing is to add resources until even the slowest replica clears the bar, which is safe but expensive, because it sizes the whole service to its worst case. The alternative Imbres pursues is to change where the load goes so that the tail is shaped by routing rather than paid for with capacity: keep struggling replicas lightly loaded so they still meet the target, and let capable replicas absorb more. Framing the problem as hitting a latency SLA at least cost, rather than balancing load for its own sake, is what makes deliberately embracing imbalance the economical choice.

What people did before, and where it broke

Even load balancing across replicas. Spread requests uniformly so no single replica is favored, the default in most microservice deployments. Where it broke. When replicas are actually unequal, even load makes the weakest replica the bottleneck, forcing worst-case provisioning of the entire service and wasting resources on replicas that did not need the headroom.
Autoscaling on aggregate load. Add or remove replicas based on the service overall utilization or request rate. Where it broke. Aggregate signals miss the per-replica imbalance; scaling out to relieve a few contended replicas over-provisions the healthy ones, and scaling reacts to totals rather than steering work away from the specific replicas that are struggling.
Profile-based placement and tuning. Characterize each service offline to predict its resource needs and place it well. Where it broke. Offline profiles go stale in a shared cluster where interference shifts continuously; a static characterization cannot track the temporal imbalance that appears as neighbors change, so it is slow to respond to the conditions that actually cause SLA misses.
Load balancing with weighted routing. Send more requests to faster replicas using fixed or slowly-updated weights. Where it broke. Weighting is a step in the right direction, but done in isolation it does not coordinate with resource scaling or with the constraint that upstream containers connect to only a subset of downstream containers, so it cannot jointly decide how much to shift, whom to connect, and how much to scale.
The gap. No system deliberately exploited replica imbalance by shifting load, managing partial upstream-to-downstream connectivity, and scaling resources all at once, driven by a fast online signal rather than offline profiles — which is what it takes to meet SLAs while actually reducing, rather than padding, resource allocation.

How Imbres works

Turning replica imbalance into lower cost, driven by live latency gradients

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Stop hiding the imbalance and start using it. The conventional stance treats replica imbalance as noise to be suppressed by spreading load evenly, which forces the whole service to be sized for its worst replica. Imbres takes the opposite stance: the imbalance is real and persistent, so the efficient move is to route around it. It deliberately makes the load uneven — pushing requests toward replicas with latency headroom and pulling them off replicas that are contended — so that struggling replicas stay within the SLA because they are lightly loaded, not because they were handed extra resources. The saving comes precisely from not provisioning every replica for the worst case, since the load, rather than the capacity, is what now absorbs the imbalance. Recognizing that a system does not need every unit to be equal, and that steering work toward the parts best able to do it beats forcing uniformity, is the same efficiency insight the microservice resource-management work covered elsewhere on this site is built on, allocating to shared microservices according to their real, differing behavior rather than a one-size guarantee.

Shifting, connecting, and scaling are one problem, so solve them together. The reason weighted routing on its own underdelivers is that load shifting is entangled with two other decisions: how upstream containers are connected to the downstream containers they can reach, and how much resource the service is scaled to. Change one without the others and you either cannot realize the shift you wanted, or you shift load only to over-provision elsewhere. Imbres co-optimizes all three in tandem — the amount of load shifted, the connection management between partially-connected upstream and downstream containers, and the resource scaling — so the decisions reinforce rather than fight each other. Treating the interacting knobs of a multi-tier service as a single joint optimization instead of independent controllers is the same discipline the uncertainty-aware multi-stage resource-management work covered elsewhere on this site relies on, managing a pipeline of dependent stages as one coordinated problem rather than tuning each stage alone.

Steer by the live gradient of latency, not by a profile. The imbalance Imbres exploits is temporal — it changes as neighbors on the shared hosts come and go — so any controller built on an offline profile would always be reacting to conditions that have already passed. Imbres instead relies solely on the online gradient of latency: it watches how latency responds to changes in load and resources in the moment, and follows that gradient to decide where to shift and how to scale. This makes it rapidly responsive and free of any offline characterization step, which matters because the whole opportunity lives in transient conditions a static model cannot see. Driving allocation from a fast, live feedback signal rather than a precomputed model is the same control philosophy the container IO-control work covered elsewhere on this site is built on, adjusting resource shares from continuous runtime measurement instead of a fixed profile so the system tracks conditions as they actually change.

Following a request surge through Imbres

Absorbing a contention spike without over-provisioning

An illustrative trace, based on the paper's abstract, of how Imbres responds when some replicas of a microservice slow down under interference. Internal algorithmic specifics beyond the abstract were not available for this draft.

Step 1 of 5

Some replicas slow down as neighbors get noisy

On a few of the shared hosts, other tenants start consuming resources, so several replicas of the microservice see their latency climb while others, on quieter hosts, remain fast. Under even load balancing this would push the slow replicas toward an SLA violation.

This is the temporal, spatial imbalance the paper targets: the replicas are momentarily unequal, and which ones are slow keeps changing.

Why it works

Uneven load spares you from worst-case provisioning. Even load makes the weakest replica set the resource bar for the whole service → but shifting work off the strugglers keeps them within the SLA without extra capacity → so the service is sized closer to its real need, cutting allocation by up to 62 percent in the paper.
Joint optimization realizes shifts that isolated controllers cannot. Weighted routing alone cannot honor connectivity limits or coordinate with scaling → but optimizing shifting, connection management, and scaling together makes each decision feasible and mutually reinforcing → so the intended load movement actually happens and does not merely relocate the waste.
Managing partial connections makes the topology cooperate. An upstream container reaches only some downstream containers, so a naive shift can be blocked → but adjusting which upstream containers connect to which downstream ones routes load to where the headroom is → so the shift is physically realizable through the call graph.
Online latency gradients track imbalance as it moves. Offline profiles are always stale against shifting interference → but following the live gradient of latency registers new contention immediately → so Imbres is rapidly responsive and needs no profiling step, which is what lets it exploit transient headroom.
Shaping the tail by routing lowers violations directly. SLA misses are a tail phenomenon driven by the worst replicas → but steering load away from those replicas shapes the tail without buying capacity → so the paper reports SLA-violation probability falling by up to 82 percent even as resources shrink.

What it costs (trade-offs and scope)

This is a draft written without the paper’s main text. The full paper was behind a publisher access wall for this draft (no arXiv version, no author preprint on the authors’ lab page, and no web-archive snapshot), so this account is built from the paper’s abstract. The reported figures, up to 62 percent lower resource allocation and up to 82 percent lower SLA-violation probability, are the abstract’s own; the optimization formulation, the exact gradient-based control algorithm, the connection-management mechanism, the benchmarks, and the baselines are not reproduced here and should be read from the paper.
Embracing imbalance concentrates load on the capable replicas. By design, load is pushed toward the replicas with headroom, so the approach depends on there being capable replicas to absorb it and on the gradients correctly identifying them; how it behaves when most replicas are contended at once, or when the gradient signal is noisy, is a question for the full evaluation.
Gradient-driven control assumes a usable live signal. Relying solely on online latency gradients is what makes Imbres fast and profiling-free, but it also ties correctness to the quality and stability of that signal; latency measurements in shared clusters can be noisy, and the abstract does not characterize how the controller handles measurement noise or avoids oscillation.
Connection management adds moving parts. Reconnecting upstream and downstream containers to realize shifts is powerful but introduces churn and coordination that the abstract does not quantify; the net benefit assumes this overhead stays small relative to the resources saved.
Evidence is benchmark-based against particular baselines. The results come from real microservice benchmarks compared with state-of-the-art systems; the up-to figures are best cases, and how the gains vary across applications, cluster loads, and SLA targets is detail this draft cannot report.

The evaluation

Handling unequal microservice replicas in a shared cluster

Property	Even load balancing	Autoscale on aggregate load	Imbres
How load is distributed
How the SLA is met
Coordinates shifting, connections, scaling
Signal it relies on	Load counts
Reported outcome

A qualitative comparison based on the paper's abstract; the paper's main body, including its optimization formulation, control algorithm, connection-management mechanism, benchmarks, and specific baselines, was not available for this draft. The up-to-62-percent and up-to-82-percent figures are the abstract's reported results.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This is an honest draft assembled from the paper’s abstract; the full text was not accessible, so the internal design — the joint optimization formulation, the online-gradient control algorithm, how partial upstream-to-downstream connections are managed, and how load shifting is coordinated with scaling — is not reproduced, and no numbers beyond the abstract’s own have been introduced. The reported results are up to 62 percent lower resource allocation and up to 82 percent lower SLA-violation probability against state-of-the-art systems on real microservice benchmarks. What is solid is the core idea: in a shared cluster the replicas of a microservice are genuinely unequal and their inequality shifts over time, so rather than balancing load evenly and provisioning for the worst replica, Imbres deliberately shifts load toward capable replicas, manages the partial connectivity between upstream and downstream containers so the shift is realizable, scales resources in tandem, and steers the whole thing from live latency gradients with no offline profiling. Read the mechanism, the control details, and the evaluation against the paper itself; treat this page as a faithful account of the approach at the level of its abstract, not of its implementation or full results.

Background

Why shared-cluster microservices waste so much

It helps to see why over-provisioning is the default that Imbres is fighting. Cloud operators pack many services onto shared machines to raise utilization, but that packing is exactly what makes any one service’s performance unpredictable: a replica’s latency depends not just on its own load but on whichever noisy neighbors happen to share its host at that moment. Faced with a hard latency SLA and this unpredictability, the safe engineering move is to give every replica generous headroom so that even a badly-interfered one stays compliant — and since operators cannot know in advance which replicas will be unlucky, they pad all of them. Multiplied across thousands of services, that padding is enormous waste, and it grows precisely because the cluster is shared. That tension, between the SLA an operator must guarantee and the resource efficiency they are pressed to achieve on contended shared hardware, is the same pressure the quality-of-service-aware multi-stage serving work covered elsewhere on this site operates under, meeting latency targets for a pipeline of services while trying not to squander the capacity that sharing was meant to reclaim. Imbres bet is that the way out is not more padding but smarter routing: let the load be uneven so the capacity does not have to be.

If you remember only three things

1. In a large shared cluster, the identical container replicas of a microservice are not actually equal — they suffer temporal and spatial performance imbalance because they sit on machines with different, time-varying contention — so balancing requests evenly across them forces operators to over-provision every replica to cover the worst one, wasting substantial resources just to meet the latency SLA.
2. Imbres embraces the imbalance instead of suppressing it: it dynamically shifts load toward the replicas with headroom and away from the struggling ones, co-optimizes this shifting with connection management (upstream containers are only partially connected to downstream ones) and with resource scaling, and drives the whole system from online gradients of latency, needing no offline profiling, which makes it rapidly responsive to shifting interference.
3. On real microservice benchmarks the paper reports Imbres cutting resource allocation by up to 62 percent and SLA-violation probability by up to 82 percent versus state-of-the-art systems, by shaping the latency tail through routing rather than paying for it with capacity. This explainer is a draft from the abstract without the paper body, so the optimization formulation, control algorithm, connection mechanism, and benchmarks should be confirmed against the paper.

Check yourself

Check your understanding

1. Why does evenly load-balancing a microservice waste resources in a large shared cluster?
2. What does it mean to "embrace imbalance," and how does Imbres do it?
3. Why does Imbres rely on online latency gradients instead of offline profiling?
4. What should a reader keep in mind about this particular explainer?

Sources

The paper is “Embracing Imbalance: Dynamic Load Shifting among Microservice Containers in Shared Clusters” by Shutian Luo, Jianxiong Liao, Chenyu Lin, Huanle Xu, Zhi Zhou, and Chengzhong Xu (University of Macau and collaborators), published at ASPLOS 2025 (Volume 2), DOI 10.1145/3676641.3716255.

This explainer is a draft. The paper’s main text was not accessible at the time of writing (the ACM Digital Library PDF returned a bot-verification challenge, there is no arXiv version, no author preprint was found on the authors’ lab publication page, and no web-archive snapshot exists), so it is written from the paper’s abstract. The abstract supplies the problem framing (containers within the same microservice encounter temporal and spatial performance imbalance in large-scale shared clusters, so the common load-balancing approach wastes resources through over-provisioning to meet SLAs), the core idea (leverage load imbalance by dynamically shifting load across microservice containers with imbalance awareness), the identified challenge (integrating load shifting with resource scaling while accommodating the partial connection between upstream and downstream containers), the system (Imbres, which optimizes load shifting, connection management, and resource scaling in tandem and relies solely on online gradients of latency, eliminating offline profiling), and the reported results (reducing resource allocation by up to 62 percent and decreasing SLA-violation probability by up to 82 percent compared to state-of-the-art systems on real microservice benchmarks). The paper’s optimization formulation, its gradient-based control algorithm, its connection-management mechanism, the specific benchmarks, and the baselines it compares against are not reproduced here and should be read from the paper; no numbers beyond the abstract’s have been introduced.