AQUATOPE: QoS-and-Uncertainty-Aware Resource Management for Multi-stage Serverless Workflows

Serverless workflows miss deadlines for two tangled reasons — cold starts and wrong per-function resources — so Aquatope untangles them with two Bayesian models, one predicting how many warm containers each minute needs, the other searching for the cheapest resource recipe that still meets the deadline.

Why this problem is worth solving

Serverless platforms run functions — short-lived pieces of code in containers — and increasingly compose them into multi-stage workflows: an ML inference pipeline where object detection fans out into vehicle and human recognition; a video service that decodes, watermarks and re-encodes chunks in parallel; a social network of a dozen cooperating functions. Workflows inherit serverless’s virtues (fine-grained scaling, modular deployment, pay-per-use) and amplify its two chronic diseases.

Cold starts. When an invocation arrives and no container for that function is loaded, the platform must create one: launch the container, set up the runtime, fetch libraries — often longer than the function’s own execution. In a workflow, cold starts cascade: each stage’s cold start delays the trigger of the next. And a cold start is not just slow — it also runs hungrier, because the invocation cannot reuse cached execution context (database connections, loaded ML models) and must run initialization code. Cold and warm invocations of the same function have different performance and different resource needs.

Wrong-sized resources. Major platforms make users pick a memory limit and allocate CPU proportionally to it — a compute-heavy, memory-light function must buy memory it never touches to get the CPU it needs. In a workflow, every stage has its own right size, the configuration space grows exponentially with stages, and the composition style (chains, fan-outs, callbacks) adds unpredictability.

Underneath both sits uncertainty. Serverless functions are short and densely colocated, so interference is high; arrival patterns fluctuate wildly (in the Azure production trace used here, over 40% of invocation streams have a coefficient of variation above 2). Any manager that treats its measurements as truth — one profiling run, one point prediction — is building on sand. The paper’s target is stated as quality of service (QoS): an end-to-end latency the workflow must meet, at minimum cost.

What people did before, and where it broke

Cold-start mitigation has a rich lineage: fixed keep-alive windows (hold containers 10 minutes after use — 51% of invocations still start cold in the evaluation), utilization-driven autoscaling (44%), histogram policies built from inter-arrival times, and IceBreaker’s Fourier-based prediction — these last two eliminate 13–17% more cold starts. But histograms and frequency decompositions capture periodicity, not arbitrary temporal structure; they use no external context (time of day, trigger type); and they produce point estimates — no notion of confidence — so fluctuating loads whipsaw them. All were also designed for single functions, not cascading stages.

Resource management likewise: autoscalers react to utilization, random search burns budget, and CLITE — the state-of-the-art Bayesian- optimization tuner from the cluster world — encodes QoS as a hand-crafted penalty term and assumes noiseless measurements. On a noisy FaaS platform, outlier samples mislead its model: under injected noise its found configurations cost 37–64% more.

And the halves are coupled. The paper’s ablation makes it concrete: strip Aquatope’s pre-warmed pool away and its resource manager must satisfy QoS across a mixture of cold and warm behaviors, forcing conservative allocations — 64% more CPU time and 28% more memory time. Eliminating cold starts is not just a latency fix; it is what makes per-function resource optimization a well-posed problem.

The idea

Predicting warmth, with confidence intervals

Every minute, Aquatope asks: how many warm containers will each function of each workflow need in the next window? Its predictor is a hybrid Bayesian neural network in three parts. An LSTM encoder-decoder (a recurrent network suited to sequences) is trained on the invocation time series and then used as an automatic feature extractor — its final hidden state is a compact summary of recent history. A small prediction network combines that summary with external features the time series alone cannot see: time of day, day of week, and the function’s trigger type (HTTP, storage event, …). And Monte-Carlo dropout makes the whole thing approximately Bayesian: randomly dropping weights across many forward passes yields a predictive mean and variance — a confidence interval, not a guess.

Workflow structure feeds the predictor too: seeing upstream containers invoked in this window predicts downstream demand in the next — cascading cold starts preempted by construction. Accuracy: 5.7% error (SMAPE) versus 9.5% for a vanilla LSTM, 18.6% for the ARIMA model production systems use, 24.5% for naive keep-alive. The uncertainty matters most when load misbehaves: against IceBreaker on highly-fluctuating traces (coefficient of variation 1–4), Aquatope removes 13–41% more cold starts — precisely because a wide confidence interval buys a few extra warm containers when the future is genuinely unclear.

Optimizing resources, distrusting every sample

With warm starts near-guaranteed, the container resource manager searches for each stage’s CPU, memory and concurrency settings — minimize cost subject to end-to-end latency ≤ QoS — using Bayesian optimization: build a cheap statistical stand-in (a surrogate model) for the expensive black-box function “configuration → performance,” and let an acquisition function pick the most informative next sample. Aquatope rebuilds each piece for a noisy world:

• Two Gaussian processes, not one. Instead of a hand-crafted objective blending cost and QoS penalties (CLITE’s approach), Aquatope models cost and latency separately with fixed-noise Gaussian processes. The latency model proactively filters candidate configurations unlikely to meet QoS; the cost model ranks the survivors. Both converge faster for modeling one thing each.
• Noise-aware acquisition. Classic expected improvement assumes the best observation is exact truth. Aquatope uses constrained noisy expected improvement with quasi-Monte-Carlo integration — improvement weighted by the probability of satisfying QoS, under Gaussian observation noise.
• Anomaly pruning. Non-Gaussian noise — an interference spike, a network hiccup — produces outliers that would warp the models. For each observation, a diagnostic Gaussian process trained on all other points predicts what this one should have been; measurements outside the 95% interval are discarded.
• Batch sampling. Serverless scales sideways, so Aquatope profiles three candidate configurations in parallel per iteration instead of one — faster convergence at no quality loss.

The search lands within 5% of the oracle-optimal cost, and under injected noise where CLITE degrades 37–64%, Aquatope holds near-optimal. The models also watch themselves: when observations start deviating — new input formats, function updates — incremental retraining on a sliding window re-converges with about 20 fresh samples in two minutes.

sample batch: 3 configs, profiled in parallel
latency GP: filters QoS-risky candidates proactively
cost GP: ranks the feasible ones

Why it works

Because the pre-warmed pool makes warm starts the norm (cold starts under 4%), the resource manager faces a single, stable performance regime — warm-start behavior — instead of a cold/warm mixture that would force conservative overprovisioning (the ablation’s +64% CPU time). → Because the demand predictor couples temporal features with calendar context and quantifies its own uncertainty, it buys extra warmth exactly when the future is unclear and trims it when the future is confident — beating point-estimate predictors most where load fluctuates most. → Because the optimizer models cost and latency separately, prices observation noise into its acquisition math, and prunes non-Gaussian outliers with diagnostic models, each expensive profiling run moves the search toward the optimum instead of occasionally away from it. → Because both components are probabilistic, the system degrades gracefully under the platform’s inherent noise rather than being steered by it — and recovers from genuine behavior changes by retraining rather than misallocating. → Therefore end-to-end: QoS violations fall 5× (to under 3%), execution cost falls 34% on average and up to 52% against QoS-meeting alternatives, with all decision-making off the critical path (sub-10 ms predictions, sub-100 ms acquisition steps).

What it costs

• Two models per workflow, continuously maintained. Training is cheap (50 seconds on a week of traces; batches selected in under 100 ms) but the machinery — LSTMs, GPs, anomaly detectors, retraining loops — is a standing operational commitment per application.
• Pre-warming spends memory to save latency. The pool holds containers that may not be used; the bet is that uncertainty-guided sizing wastes less than fixed keep-alive (it provisions 23% less memory time than even IceBreaker) — but the floor is not zero.
• Exploration is paid in real executions. BO profiles candidate configurations on live infrastructure under a time and cost budget; brand-new workflows run suboptimally until the search converges.
• QoS is the user’s promise to keep. The evaluation sets QoS at pre-saturation latency; a workflow with an unattainable target would keep the optimizer searching a feasible region that does not exist.
• Scope: warm-start, per-function knobs on one platform. The implementation is OpenWhisk with CPU/memory decoupling; storage-side configuration, cross-server placement, and non-container FaaS designs are outside the studied surface.

The evaluation

Implemented on Apache OpenWhisk across a seven-server local cluster (40- and 88-core Xeon machines); driven by the Azure Functions production trace (Poisson arrivals within each one-minute interval, scaled to realistic cluster utilization — 43% CPU, 29% memory on average). Applications: two synthetic workflow shapes (chain, fan-out/fan-in) and three end-to-end services — an ML security-camera pipeline, a Sprocket-style video-processing framework, and the DeathStarBench serverless social network.