Performance tuners pick the best configuration for an application by timing candidate settings and keeping the fastest, but in a shared cloud a configuration's runtime swings wildly with whoever else is on the machine, so an unlucky good setting looks bad and the tuner is fooled; this paper (Northeastern, MIT, University of Utah — Rohan Basu Roy, Vijay Gadepally, Devesh Tiwari) introduces DarwinGame, which stops trying to measure absolute runtime and instead co-locates rival configurations so they suffer identical noise and runs a multi-phase tournament (Swiss, double-elimination, then barrage) to rank them by who beats whom, landing within 4.2 percent of the true optimum where the best prior tuner is over 40 percent off, and cutting execution time more than 27 percent with under 0.5 percent variability.
The argument in one glance
-
Problem
Tuning picks a setting by timing it, but in the cloud the clock lies
Software has many knobs — application-level and system-level parameters — and the right combination can make a program run several times faster than a bad one. A performance tuner searches this vast space by running the program under candidate configurations, timing each, and keeping the fastest. That works when you have a dedicated machine, because the measured time is the configuration true time. But developers increasingly tune on shared cloud VMs, where other tenants on the same hardware inject interference that comes and goes unpredictably. Now the clock lies: the same configuration can finish in 440 seconds one run and far longer the next, varying by more than 40 percent, purely because of who else was on the machine. A genuinely excellent setting that happened to run during a noisy moment looks terrible, and the tuner throws it away.
-
What people did
Just run an existing tuner in the cloud, or brute-force every setting there
The obvious response is to take a state-of-the-art tuner — OpenTuner, ActiveHarmony, a Bayesian tuner like BLISS — and let it do its usual search directly in the shared cloud environment, trusting the times it measures. If you distrust the search, you can go further and exhaustively run every configuration in the cloud, one by one, and keep whichever came out fastest.
-
Why it fell short
Both are fooled by noise, landing far from the real optimum
Every prior tuner is designed for interference-free measurement, so feeding it noisy times makes it misjudge which configurations are good; in the paper the best prior tuner lands more than 40 percent worse than the true optimum, and repeated tuning runs pick different winners each time. Exhaustive search fails for the same reason: each configuration is timed under whatever interference happened during its single run, so the "winner" is often just the configuration that got lucky, not the best one. Absolute runtime, measured once in a noisy environment, is simply not a trustworthy signal.
-
This paper's idea
Stop measuring absolute time — rank configurations by head-to-head games under identical noise
DarwinGame abandons the idea of measuring any configuration true time. Instead it co-locates several rival configurations of the same application on the same node at the same time, so they all experience identical interference, and lets them race: whoever does more work is the relative winner of that game, and the shared noise cancels out of the comparison. Because a single game can still be a fluke, DarwinGame runs a multi-phase tournament — a Swiss-style regional round to filter a huge field, a double-elimination global round that forgives one bad game, then a barrage-style playoff — judging configurations on both speed and consistency. It lands within 4.2 percent of the true optimum, cuts execution time more than 27 percent over prior tuners, and holds variability under 0.5 percent.
Why this problem is worth solving
The entire premise of tuning breaks when the measurement it depends on becomes unreliable. A program exposes many configurable parameters — buffer sizes, thread counts, algorithm choices, kernel settings — and each combination is one point in a tuning search space whose size is the product of every parameter’s options. For Redis with application- and system-level knobs, that space is 7.8 million configurations. The execution time varies enormously across them: in the paper’s data, more than 93 percent of configurations run at least twice as slow as the best one, a spread of more than three times. A tuner’s job is to find the fast point without exhaustively trying all of them, and the only way it knows whether one configuration beats another is to run each and time it. That single dependency — comparing configurations by their measured execution time — is the load-bearing assumption of every tuner. It holds perfectly in a dedicated, interference-free machine, where the measured time is the true time. The moment tuning moves to an environment where the measured time is contaminated by things outside the program, the search is optimizing against a corrupted signal, and being good at search no longer helps.
And the contaminated environment is exactly where tuning increasingly happens. Developers tune on shared cloud VMs for concrete reasons: a specific accelerator may only be rentable on the cloud, pay-as-you-go sharing avoids the upfront cost of a dedicated cluster, and an application destined to run in the cloud should be tuned there so its sensitivity to interference is captured. But on shared hardware, other tenants co-located on the same node contend for caches, memory bandwidth, and cores, and that contention is neither observable nor controllable by the user — it rises and falls as neighboring workloads come and go. The effect on a measurement is large and time-varying: the paper shows the same configuration’s runtime swinging by more than 40 percent across runs, and different tuning sessions at different times selecting different winners. Crucially, the interference is not a fixed offset you could calibrate away; it is a moving disturbance that lands differently on whichever configuration happens to be running when it strikes. This is why an effective cloud tuner has to want two things at once — a configuration that is genuinely fast, and one whose performance is stable under noise — because a fast-but-fragile setting is worthless if its speed evaporates whenever a neighbor gets busy. Making tuning robust to an adversary you cannot see or measure is what this problem demands.
What people did before, and where it broke
- Run an existing tuner in the cloud as-is. Take OpenTuner, ActiveHarmony, or a Bayesian tuner like BLISS and let it search the space using the runtimes it measures on the shared VM. Where it broke. These tuners assume interference-free measurement, so noisy times make them misinterpret which configurations are good; the best of them lands more than 40 percent worse than the true optimum, and repeated sessions disagree on the winner.
- Exhaustive search in the cloud. Skip the smart search and just run every configuration once in the shared environment, keeping the fastest observed. Where it broke. Each configuration is timed under whatever interference happened during its one run, so the recorded winner is frequently the configuration that got a quiet moment rather than the one that is actually best; it also costs a prohibitive number of runs.
- Bayesian optimization over measured runtimes. Model the space probabilistically and sample promising points to converge on the optimum. Where it broke. The model is only as trustworthy as its observations; feeding it high-variance, interference-corrupted runtimes injects variance into the model itself, so it converges unstably and its choices differ from one noisy run to the next.
- Individually stress-testing configurations under controlled interference. Evaluate each configuration separately against injected or background noise to estimate its robustness. Where it broke. Because the configurations are exposed to noise at different times, they never face the same disturbance, so the comparison is still apples-to-oranges; the paper finds co-locating rivals under identical noise beats this by more than 10 percent.
- The gap. No tuner gave up on measuring any configuration’s absolute runtime and instead compared configurations relatively under provably identical noise, then structured those noisy comparisons so that the genuinely best-and-most-consistent configuration reliably survives — which is what tuning in an interference-prone cloud actually requires.
How DarwinGame works
Turning unreliable absolute timings into robust relative rankings through a tournament
-
-
new
-
new
-
new
-
new
-
new
The idea
Give up on measuring truth; measure who wins. The deepest move in DarwinGame is a refusal. It refuses to try to recover any configuration’s true, interference-free runtime from a noisy cloud, because that number is not recoverable without a prohibitive number of runs. Instead it asks a strictly easier and more robust question: of two configurations running side by side right now, which is doing more work? By co-locating rival configurations of the same application on the same node at the same time, DarwinGame guarantees they all face identical interference, so the shared noise cancels out and what remains is a fair relative comparison — a game with a clear winner. It never needs to know how fast anyone “really” is; it only needs to know who beat whom under the same conditions. Turning an unanswerable absolute measurement into an answerable relative one, precisely because the disturbance is uncontrollable and unmeasurable, is the same acceptance-of-uncertainty stance the QoS-and-uncertainty-aware resource-management work covered elsewhere on this site is built on, designing around the fact that performance in a shared setting cannot be pinned to a single clean number and must be reasoned about probabilistically.
One game is a fluke, so build a tournament to squeeze out the luck. A single co-located game still has randomness — the field of opponents, the particular moment. So DarwinGame does not trust any one result; it stages a multi-phase tournament that makes the good-and-stable configurations prove themselves repeatedly against different opponents before winning. Each phase uses the tournament format suited to its job: a Swiss-style regional phase is best at surfacing the strongest players from an enormous pool, pairing high scorers against each other so the field narrows fast across ten thousand regions; a double-elimination global phase gives a proven configuration a second life in a loser bracket, so it is not knocked out by a single bad game caused by an interference spike; and a barrage-style playoff runs direct knockouts among the finalists. Structuring a search so that a large candidate field is aggressively winnowed and only repeatedly-validated survivors reach the decision is the same discipline the Bayesian compiler-optimization work covered elsewhere on this site applies to navigating a vast tuning search space, spending evaluation effort where the promising configurations concentrate rather than spreading it thin.
Judge on consistency too, and stop early to keep tuning cheap. Because a fast-but-fragile setting is useless in a shared cloud, DarwinGame explicitly scores configurations on two axes: an execution score for how much work they do, and a consistency score that rewards a configuration for winning across multiple rounds rather than once. A setting that wins its first game but folds later scores worse than one that keeps winning, so the tournament naturally selects for stability under noise — which is why the chosen configuration varies by under 0.5 percent and is picked in 93 of 100 sessions. And because tuning time is itself a cost, DarwinGame terminates a game the instant the leader is far enough ahead, rather than running every game to completion, and ends a region as soon as one configuration wins consistently. Spending measurement effort only until the ranking is clear, instead of exhaustively evaluating every candidate to the end, is the same economy the cost-model-guided program-tuning work covered elsewhere on this site pursues, cutting off expensive full evaluations once cheaper evidence already reveals the winner.
Following a tuning session through DarwinGame
Finding a robust Redis configuration in a noisy AWS VM
A trace, drawn from the paper's design, of how DarwinGame tunes an application whose runtime is corrupted by cloud interference.
Step 1 of 5
Millions of configurations, no trustworthy clock
Redis exposes a 7.8-million-configuration space, and the developer can only tune on a shared AWS VM where interference from co-located tenants makes any single runtime swing by more than 40 percent. Timing configurations one at a time would rank them by luck, so DarwinGame will not do that.
More than 93 percent of Redis configurations run at least twice as slow as the best, so the payoff for finding the right one is large — if the noise can be beaten.
Why it works
- Identical shared noise cancels out of a relative comparison. Any single absolute runtime in the cloud is corrupted by whatever interference struck during it → but co-locating rivals on one node subjects them to the same disturbance at the same time → so the difference in their work done reflects the configurations, not the noise, giving a fair winner.
- A tournament converts flukey games into reliable rankings. One co-located game still has randomness from opponents and timing → but making configurations win repeatedly against varied opponents across multiple phases averages out that randomness → so the genuinely strong settings survive and the lucky ones are exposed, landing DarwinGame within 4.2 percent of optimal.
- Double elimination protects strong settings from one bad day. An interference spike can make an excellent configuration lose a single game → but a loser bracket keeps it alive until a second loss → so a proven configuration is not discarded on weak, one-game evidence, which is exactly the failure mode that fools absolute-time tuners.
- Scoring consistency selects for noise robustness directly. A configuration that is fast once but fragile is useless in a shared cloud → but rewarding configurations that win across many rounds folds stability into the selection criterion → so the winner varies under 0.5 percent and is reproducibly chosen in 93 of 100 sessions.
- Early termination keeps the robustness affordable. Playing every game to completion and every region to exhaustion would make tuning slow and costly → but ending a game once the leader is clearly ahead and a region once a configuration wins consistently cuts the work → so DarwinGame gets its robustness without an exhaustive number of runs.
What it costs (trade-offs and scope)
- It needs to run rivals in parallel, which consumes resources. Playing a game means co-locating multiple copies of the application on a node simultaneously, so DarwinGame spends cores and memory to run several configurations at once; the approach assumes you can afford this parallel co-location, and its game size is tied to the number of available cores.
- It finds relative winners, not absolute performance numbers. By design DarwinGame never estimates any configuration’s true interference-free runtime — it only ranks configurations against each other. If a user actually needs an absolute performance prediction rather than the best-of-the-field, that is outside what the tournament provides.
- The design has several tuned knobs. The number of regions (set to ten thousand), the work-done-deviation threshold for early termination, and the minimum-work fraction before a game can end are parameters; the paper reports the outcome is robust to reasonable ranges (for example, results change by under 3.7 percent as regions vary from five thousand to fifteen thousand, and under 2.7 percent as the deviation varies from 5 to 15 percent), but the robustness is empirical.
- Dynamic-parameter tuning is left out on purpose. The tournament could be extended with feedback loops to tune dynamically adjustable parameters like thread count, but the paper found this raised tuning time and resource use by over 10 percent for under 5 percent performance gain, so it is not enabled — meaning some parameter types are not addressed.
- Evidence is specific applications on one cloud. The results come from Redis, GROMACS, FFmpeg, and LAMMPS on AWS m5 VMs; the more-than-27-percent improvement and 4.2-percent-from-optimal figures are for these workloads and this platform, and behavior under very different interference regimes or on other providers is inferred rather than exhaustively shown.
The evaluation
Tuning an application in a shared, interference-prone cloud VM
| Property | Existing tuner in cloud | Exhaustive search in cloud | DarwinGame |
|---|---|---|---|
| What signal it trusts | |||
| Handles interference | |||
| Distance from true optimum | |||
| Result stability across sessions | |||
| Reported outcome |
Check yourself
Check your understanding
1. Why do existing performance tuners fail when used in a shared cloud environment?
2. What is the core idea that lets DarwinGame tune reliably despite noise?
3. Why does DarwinGame use a multi-phase tournament instead of a single co-located game?
4. What are DarwinGame's main results, and what are its limitations?
Sources
The paper is “DarwinGame: Playing Tournaments for Tuning Applications in Noisy Cloud Environments” by Rohan Basu Roy (University of Utah), Vijay Gadepally (Massachusetts Institute of Technology), and Devesh Tiwari (Northeastern University), published at ASPLOS 2025, DOI 10.1145/3669940.3707259. It is licensed CC BY, and the artifact is available at Zenodo (10.5281/zenodo.14014918).
This explainer is based on the paper’s full text, obtained from the authors’ arXiv preprint (arXiv:2509.25090). The account of the mechanism — the failure of absolute-runtime measurement under cloud interference, the co-located “games” that expose rival configurations to identical noise, the execution score and consistency score, early game termination by a work-done-deviation threshold, and the four-phase tournament (Swiss-style regional filtering across ten thousand regions, a double-elimination global phase with a loser bracket, barrage-style playoffs, and a final) — follows the paper’s own description. The reported results cited here are the paper’s own: more than 27 percent reduction in execution time over existing solutions, less than 0.5 percent performance variability, within 4.2 percent of the dedicated-environment optimum versus more than 40 percent for the best prior tuner (BLISS), the same configuration selected 93 of 100 tuning runs, and about 15 percent improvement when integrated with existing tuners (ActiveHarmony and BLISS). Experiments use Redis, GROMACS, FFmpeg, and LAMMPS on AWS m5 VMs. Exact per-application numbers, the precise scoring formulas, and the tournament parameters and their sensitivity should be read from the paper itself.