Erms: Efficient Resource Management for Shared Microservices with SLA Guarantees

In production clusters 40% of microservices serve more than 100 different products at once, so one product's surge ruins another's latency — Erms profiles each microservice's piecewise-linear latency curve, splits every SLA into per-microservice targets in closed form, and turns sharing into an advantage by letting the tightest-deadline service go first.

Why this problem is worth solving

A modern online product is a

Two production realities sharpen the problem. First, interference: microservice containers share hosts with batch jobs (colocation is how clusters reach acceptable utilization — the same tension VELTAIR attacks for DL serving, covered elsewhere on this site), and interference visibly changes how latency responds to load. Second — the paper’s signature observation — sharing: the Alibaba traces show 40% of microservices serve more than 100 distinct online services, each with its own SLA and workload pattern. A surge in one product’s traffic queues everyone at the shared node; conversely, the shared node is exactly where global coordination can pay. The paper’s motivating experiment shows that simply prioritizing services correctly at one shared microservice saves over 40% of resources — an opportunity every prior per-service manager leaves untouched.

What people did before, and where it broke

Statistical decomposition. GrandSLAm splits the SLA proportionally to each microservice’s average latency; Rhythm weights contributions by mean, variance, and correlation with end-to-end latency. Both produce fixed per-microservice targets — but the right division depends on the current workload, because latency-vs-load curves are nonlinear and each microservice sits at a different point on its curve at any moment.

Learning-based managers. Firm labels critical microservices and tunes them with reinforcement learning — labeling is costly, and independently boosting several critical microservices in one service lands in sub-optimal corners (Firm shows the longest resource tail in the evaluation, 3× Erms’s containers in the extreme). Configuration-searching deep-learning approaches must evaluate many candidate allocations per decision — a non-starter at 1,000+ microservices per service.

The missing curve. All prior work models microservice latency by summary statistics. Measured reality (benchmarks and Alibaba traces alike) is a piecewise-linear curve with a knee: latency grows slowly with load until the container’s thread pool saturates, then climbs steeply as queueing takes over. Interference moves both parameters — on a heavily-loaded host the post-knee slope is 5× steeper and the knee arrives earlier. A manager that cannot see the knee cannot know which microservice is about to blow the budget.

The idea

Profile the curve, not the moments. The piecewise-linear model L = a·(γ/n) + b — latency as slope times per-container load plus intercept, valid per segment, per interference level — is the load-bearing abstraction. It is invertible: given a latency target, it tells you the container count; given interference, it tells you the slope. Its simplicity is a feature twice over: it needs little training data (production microservices change often), and it makes everything downstream solvable rather than searchable.

Turn the graph into algebra. Splitting an end-to-end budget across a DAG looks combinatorial. Erms dissolves it with two observations. Parallel branches must receive equal targets at the optimum (otherwise the looser branch donates slack to the tighter one and saves containers). Sequential neighbors combine into a single virtual microservice whose effective slope has a closed form — the square-root-weighted composition that also answers how to split a budget between two stages optimally. Applying these two merges recursively, any graph becomes a chain, and the chain’s optimal budget split is a textbook convex problem with an explicit solution. No learning, no enumeration — which is why Erms scales to 1,000+-microservice services and dynamic graphs.

Make sharing pay. At a shared microservice, mixing all services’ traffic means the tightest deadline dictates provisioning for everyone’s load — the worst of all worlds. Erms orders services by their computed latency targets: tightest first. The intuition: a low target signals a service built from many latency-sensitive pieces; serving it first lets its target be met with few containers, while looser services absorb queueing where their budgets have slack. The modified-workload model makes this tractable — priority-k traffic sees the cumulative load of priorities 1..k, so each service’s targets can be recomputed independently instead of solving the coupled n!-ordering problem. Strict priority risks starvation, so priorities are enforced probabilistically. This scheduling alone cuts container counts by up to another 50%.

Place against interference, globally. Scaling uses average interference for tractability; placement then repairs the variance — assigning containers to hosts to minimize aggregate latency, accounting for both existing batch-job pressure and the interference the new containers will themselves create. Balancing raw utilization (the earlier Erms-IPM approach, kept as a baseline) ignores that different microservices degrade differently under the same pressure.

model: L = a·(γ/n) + b, per segment, per interference
knee + slope shift with host pressure

Why it works

• Latency curves have knees and interference-dependent slopes → summary statistics miss exactly the regime changes that cause violations → an invertible piecewise-linear model turns “how many containers?” into arithmetic.
• Parallel-equal and sequential-fusion merges are provably optimal local moves → any DAG collapses to a chain whose budget split has a closed form → SLA decomposition scales to 1,000+ nodes and runtime-dynamic graphs where search-based managers time out.
• At shared nodes, deadlines and volumes are mismatched across tenants → serving tightest-deadline traffic first lets each target be met at its own price → sharing becomes the largest single source of savings (40–50%) instead of the largest hazard.
• The cumulative-workload trick decouples the joint scheduling problem → per-service recomputation replaces n! orderings → priority scheduling stays cheap enough to run online.
• Simple models are data-efficient and interpretable → profiles stay fresh as microservices evolve, and operators can read why a target moved — properties RL and DL managers lack at this scale.

What it costs

• Profiling infrastructure. Continuous tracing (Jaeger-class), per-microservice/per-interference fitting, and the knee detection must stay current as code and traffic evolve.
• Model risk at the edges. Piecewise linearity is an empirical fit; workloads sitting exactly at a knee, or interference outside profiled levels, degrade the closed forms’ accuracy.
• Priority machinery. Request tagging and probabilistic priority queues at shared microservices touch the data plane; starvation is mitigated, not impossible.
• Average-interference scaling. The scaling model sees mean interference; placement repairs variance afterward — a deliberate tractability split, not a guarantee.
• Evaluation scale. The real cluster is 20 hosts with DeathStarBench/TrainTicket; production-scale evidence (1,000+-microservice Taobao graphs, 20,000-microservice traces) comes from profiling validation and trace-driven simulation.

The evaluation

Deployed on a 20-host Kubernetes cluster (32 cores / 64 GB each) running DeathStarBench (Social Network, Media, Hotel Reservation — 36/38/15 microservices, with shared microservices across services) and TrainTicket (~10 services, 23 shared microservices, dependency graphs that change shape per request), driven by static workloads from 600 to 100,000 requests/minute and dynamic Alibaba-trace replays; SLAs at P95 from 50 to 200 ms. Baselines: GrandSLAm, Rhythm, Firm, and Erms-IPM (the utilization-balancing placement variant).