Concurrency-Informed Orchestration for Serverless Functions

Serverless functions run in containers that take a long time to start cold, so platforms keep recently-used containers warm — but they treat this as plain caching and miss what happens when one function gets flooded with simultaneous requests; this work from UVA, Alibaba, and Amazon builds CIDRE, which notices that a busy warm container can often be reused faster than cold-starting a new one, and rather than trying to predict which is cheaper it races both and takes the winner, while evicting containers in a way that treats each function's fleet as one elastic object — cutting cold starts by up to 75 percent, and now running in production at Alibaba Cloud.

The argument in one glance

Problem

Cold starts are slow, and keeping containers warm is treated as caching

In serverless computing, you upload a function and the platform runs it on demand inside an isolated container. But spinning up a fresh container — downloading its image, starting the language runtime, loading the code — is slow, sometimes a hundred times longer than the function itself runs. This cold start is the main pain point. The standard cure is to keep recently-used containers alive in memory so the next request reuses one, a warm start, and platforms decide which to keep exactly like a cache.
What people did

Model keep-alive as a classic cache: hits, misses, and evictions

State-of-the-art keep-alive policies treat a warm container as a cached object, a warm start as a cache hit, a cold start as a miss, and reclaiming a container as an eviction, then borrow decades of caching theory to maximize hits and minimize misses. It is a clean, well-understood model, and it works reasonably well for functions invoked one request at a time.
Why it fell short

The caching model ignores what concurrency does to a function

Real serverless workloads are intensely bursty — a single function can spike to thousands of simultaneous requests — and the caching model misses two things this causes. First, when all a function's warm containers are busy, the model always cold-starts a new one, ignoring that a busy container might free up after a shorter wait than a cold start takes. Second, a function under load owns many containers that together form one elastic object, but the cache scores each alone and tends to evict a whole function's fleet in careless bulk, throwing away containers about to be reused.
This paper's idea

Orchestrate with concurrency in mind, from scaling to eviction

CIDRE rethinks the whole container lifecycle around concurrency. When no idle container is free, instead of guessing whether reusing a busy one or cold-starting is cheaper — a guess made unreliable by variable execution times — it speculatively does both and serves the request from whichever becomes ready first. And it evicts using a priority that treats all of a function's containers as one elastic set, so evictions stay balanced across functions. The result cuts cold starts by up to 75 percent, and its scaling policy now runs in production at Alibaba Cloud.

Why this problem is worth solving

To see the opportunity, you have to look closely at what a request can do when it arrives.

A serverless function runs inside a container, and a request can be served three ways. A cold start creates a brand-new container — downloading the image, initializing the runtime, loading the code — which is the slow path, sometimes a hundred times the function’s own runtime. A warm start reuses an idle container that a previous request left alive in memory, skipping all that setup; this is the fast path platforms try to maximize. But under concurrency there is a third option the classic model misses: when every warm container of a function is busy serving other requests, a new request can wait in line for one of them to finish rather than cold-starting a fresh container. This is a delayed warm start — it pays a queuing delay instead of a startup delay. The key insight is that the queue is often shorter than a cold start, so waiting beats starting fresh. Analysis of production traces shows the majority of concurrent requests would be served faster by reusing a busy container than by cold-starting, a large and previously untapped opportunity.

The reason exploiting this is hard, and why concurrency also breaks eviction, comes down to structure the cache model cannot see.

Serverless workloads are extremely bursty: a single function routinely receives hundreds to thousands of requests at nearly the same instant, so the platform spins up many containers of that one function to serve them. This has two consequences the traditional caching model handles badly. First, whether a delayed warm start beats a cold start depends on how much execution time is left on a busy container — but function execution times vary widely (most functions vary by more than a quarter), so this cost cannot be reliably predicted. Second, all the warm containers of one function form a single elastic object that grows and shrinks with demand, yet the cache treats each container as an independent item. Because a function’s containers tend to share similar priority, the cache evicts them in a clump — throwing out a whole function’s fleet at once, including containers that are about to be reused. Concurrency, in short, creates both an opportunity in scaling and a hazard in eviction that a per-object cache is blind to.

What people did before, and where it broke

Time-to-live keep-alive. Keep each container alive for a fixed window (say ten minutes) after use, then evict it. Where it broke. A blunt timer ignores how valuable a container is, keeping useless ones and dropping useful ones, and does nothing about concurrency.
LRU and classic caching. Evict the least-recently-used container, treating keep-alive as a textbook cache. Where it broke. It scores containers independently, so under concurrency it bulk-evicts a whole busy function’s containers and never considers reusing a busy one.
GDSF-based keep-alive (FaasCache). Use a richer cache priority combining recency, frequency, provisioning cost, and size. Where it broke. Still a per-object cache that always cold-starts when warm containers are busy, and still evicts a function’s containers as an undifferentiated group.
Layer-wise pre-warming (RainbowCake). Share and pre-warm container layers to cut cold-start cost, predicting future demand. Where it broke. Under high concurrency the shared layers a request needs are often themselves busy, so it must wait or cold-start anyway, and it leans on demand prediction.
Heterogeneity-aware keep-alive and auto-scalers (IceBreaker, CodeCrunch, ENSURE). Exploit differences between servers, or reserve burst-buffer capacity, to lower cold-start cost. Where it broke. They optimize around the cold start rather than avoiding it through busy-container reuse, and reserving spare capacity under high concurrency and tight memory is itself costly.
The gap. Every policy treated keep-alive as a caching or provisioning problem and missed the concurrency-specific structure: that a busy container is a reusable resource, and that a function’s containers are one elastic object. Closing that gap — reusing busy containers when it pays, and evicting in a balanced way — is exactly what serving bursty serverless workloads needs.

How CIDRE works

Orchestrating containers with concurrency in mind

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

When you cannot predict which path is cheaper, race both and take the winner. The central decision under concurrency is whether to reuse a busy container — waiting a queue for it to free up — or to cold-start a new one. Which is faster depends on how much execution time is left on the busy container, and because function runtimes vary so much, any prediction is error-prone. CIDRE’s move is to refuse the bet: it launches the cold start and the speculative wait simultaneously and serves the request from whichever finishes first. This sidesteps cost modeling entirely and comes with a guarantee — the request is never slower than a plain cold start, since that path is always running too. Choosing between two ways of getting the same result by whichever proves cheaper in the moment, rather than committing to one blindly, is the same instinct the multi-turn LLM-serving work covered elsewhere on this site follows, deciding per request whether to fetch a cached result or recompute it based on which will actually be faster.

Only pay for the expensive option when it earns its keep. Racing both paths has a cost: the basic version provisions a new container for every speculative wait, and if that container is never reused it wastes memory and evicts other functions’ useful containers, thrashing the cache. So CIDRE adds a lightweight, hint-based gate. It tracks whether recently cold-started containers actually got reused before sitting idle past the function’s typical runtime; if they did not, it concludes the function already has enough warm containers and disables the cold-start path, re-enabling it only when the queuing delay grows longer than a cold start would take. Spending on an expensive mechanism only where a cheap signal shows it will pay, and switching it off where it will not, is the same discipline the self-replicating page-table work covered elsewhere on this site practices, replicating only where a workload signal says the copy is worth its cost.

Treat a function’s containers as one working set, not a heap of independent items. The eviction failure is that a classic cache scores each container alone, so a busy function’s many containers, sharing similar priority, get swept out together — discarding a working set that is about to be reused. CIDRE’s priority instead folds in function-level concurrency: it divides a container’s score by the number of containers its function holds, so a function hoarding many becomes a natural place to reclaim, while a function down to its last container is protected, and it measures frequency as an invocation rate that decays for functions whose traffic has died. Evictions spread across functions rather than annihilating one. Keeping the working set that will actually be reused resident, instead of thrashing it out in a careless sweep, is the same goal the out-of-core graph-processing work covered elsewhere on this site pursues, holding the right resident set in scarce fast memory rather than evicting what it is about to need.

Following a burst through CIDRE

Serving four concurrent requests to one function

Tracing a burst shows speculative scaling and balanced eviction working together.

Step 1 of 5

Four requests, one idle container

A burst of four requests arrives for one function, but only a single idle warm container exists. CIDRE dispatches the first request to it — a true warm start with almost no overhead. The other three have no idle container waiting.

Real functions routinely see hundreds to thousands of concurrent requests; four is just illustrative.

Why it works

Reusing a busy container can beat cold-starting a new one. A cold start pays full container startup, which can dwarf the function’s runtime → but a busy warm container often frees up after a queue shorter than that startup → so waiting on it turns an expensive cold start into a cheap delayed warm start, which trace analysis shows helps the majority of concurrent requests.
Racing both paths beats predicting which is cheaper. Function execution times vary too much to reliably predict when a busy container will free up → but launching a cold start and a speculative wait together and taking whichever finishes first sidesteps prediction entirely → so CIDRE is guaranteed never to do worse than a plain cold start, with no error-prone cost model.
Gating the cold-start path avoids wasteful over-provisioning. Always provisioning during a speculative wait can create containers that are never reused and thrash the cache → but disabling cold starts for a function once its recent ones sat idle, and re-enabling when queues grow costly, keeps only the containers that pay off → so speculation gets its safety without the waste.
Treating a function’s containers as one elastic object balances evictions. A plain cache scores each container alone and bulk-evicts a whole function’s fleet when they cluster at the bottom → but dividing priority by how many containers a function holds spreads evictions across functions → so no function loses its whole working set to a careless sweep, raising the warm-start rate.
A rate-based, aging frequency adapts to shifting load. A raw reuse count keeps a once-popular function’s containers alive long after its traffic dies → but measuring invocations per minute lets a quiet function’s priority decay as time passes → so stale containers age out and cache space follows the functions actually under load.

What it costs (trade-offs and scope)

Most of the gain comes from speculative scaling, not the smarter eviction. The ablation shows the concurrency-informed eviction alone cuts overhead only about 3.6 percent; the speculative scaling does the heavy lifting, so the eviction refinement is a modest contributor.
The headline numbers are best-cases. The up-to-75-percent cold-start cut and up-to-39-percent overhead reduction are the most favorable configurations; typical reductions are smaller, and the production deployment — which runs only the basic speculative scaling — cut cold starts by 34.5 percent and tail overhead by about 10 percent.
Speculation spends resources to save latency. The basic policy provisions a container for every speculative wait even when it ends up reusing a warm one, which can waste work; the conditional policy curbs this but relies on hint-based heuristics, medians over a fifteen-minute window, that can misjudge.
It gives up on prediction by design. CIDRE assumes execution time is too volatile to predict and races instead, so in the cases where a good predictor would help, it still pays for the speculative provisioning rather than exploiting the forecast.
A delayed warm start still makes the request wait. Reusing a busy container is a bet that the queue is shorter than a cold start; the guarantee is only that it is no worse than a cold start, not that it is optimal, and when busy containers do not free up the request pays the queue before falling back.
The full system is proven mostly in simulation and a small testbed. Only the basic speculative scaling is in production; the complete CIDRE, including the concurrency-informed eviction, is evaluated on a three-node cluster and trace replays.
Some baselines are handicapped in the setup. Policies that rely on server heterogeneity were run on a homogeneous cluster, which the authors note weakens them, tilting those comparisons in CIDRE’s favor.

The evaluation

Keep-alive policies under bursty concurrency

Property	FaasCache	IceBreaker	CIDRE (basic)
When warm containers are busy
Eviction awareness		Server heterogeneity	Per-container
Avg overhead ratio (100GB Azure)	About 43 percent	About 43 percent	About 34 percent
Cold-start reduction vs FaasCache	Baseline	Modest
Deployed in production	No	No

Keep-alive and orchestration policies for serverless functions, evaluated in OpenLambda on a three-node cluster with production traces from Azure Functions and Alibaba Cloud Function Compute. CIDRE cuts the cold-start ratio by up to 75.1 percent and average invocation overhead by up to 39.3 percent versus state-of-the-art keep-alive, approaching the offline ideal; its basic speculative scaling runs in production at Alibaba Cloud, cutting cold starts 34.5 percent there.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

CIDRE is a sharp, practical result: by noticing that concurrency turns a busy container into a reusable resource and that a function’s containers form one elastic object, it reuses busy containers when it pays — racing a cold start against a speculative wait rather than predicting which is cheaper — and evicts in a balanced way, cutting cold starts by up to 75 percent, with its scaling policy already serving 110,000 requests per second in production at Alibaba Cloud. Read the numbers with their conditions. The eye-catching figures are best-cases; the production deployment runs only the basic speculative scaling and delivered a still-real but smaller 34.5 percent cold-start cut. The ablation shows the speculative scaling, not the smarter eviction, does most of the work. Speculation spends resources to buy latency and leans on hint-based heuristics; the design deliberately forgoes prediction; a delayed warm start still makes a request queue; and the full system, eviction included, is mostly evaluated in simulation and on a three-node testbed, with some baselines weakened by a homogeneous cluster. The durable idea is reframing keep-alive from a caching problem into a concurrency-aware orchestration problem — and, when the cost is unpredictable, racing instead of guessing.

Background

Why cold starts are serverless's defining tax

It helps to step back and see why so much serverless research circles the same enemy. The promise of serverless is that you write a function and never think about servers: the platform runs it on demand, scales it to zero when idle, and bills only for what runs. But scaling to zero means that when a request arrives and nothing is warm, the platform must build an execution environment from scratch — and that build can take longer than the function itself, so the very elasticity that makes serverless cheap also makes it slow to respond. Every serverless system is, in some form, fighting this cold-start tax: pre-warming containers, sharing layers, keeping the right ones alive, packing more onto each server. The serverless function-warming work covered elsewhere on this site takes another angle on the same enemy, deciding when and where to keep functions ready to blunt the startup cost. CIDRE’s contribution is to notice that under real bursty load the tax can often be dodged entirely, by reusing a container already running the same function instead of building a new one — turning a cold start into a short wait.

If you remember only three things

1. Serverless functions run in containers whose cold start (building the environment from scratch) can be a hundred times longer than the function runs, so platforms keep recently-used containers warm and treat the choice of which to keep as a classic caching problem — but this model ignores what concurrency does: when a function is flooded with simultaneous requests, a busy warm container can often be reused after a shorter queue than a cold start takes (a delayed warm start), and a function's many containers form one elastic object that plain caching bulk-evicts carelessly.
2. CIDRE reorchestrates the container lifecycle around concurrency with two techniques: speculative scaling, which — rather than predicting whether reusing a busy container or cold-starting is cheaper, a prediction made unreliable by variable execution times — races both and serves the request from whichever is ready first (guaranteeing performance no worse than a cold start), refined by a conditional gate that disables wasteful cold starts once a function has enough warm containers; and a concurrency-informed eviction priority that divides a container's score by how many containers its function holds and uses an aging invocation rate, so evictions balance across functions instead of wiping out one function's working set.
3. Implemented in OpenLambda and evaluated on production Azure and Alibaba traces, CIDRE cuts the cold-start ratio by up to 75.1 percent and average invocation overhead by up to 39.3 percent versus state-of-the-art keep-alive, approaching the offline ideal, and its basic speculative scaling runs in production at Alibaba Cloud (serving 110k requests/sec, cutting cold starts 34.5 percent there) — but the headline figures are best-cases, the ablation shows speculative scaling does most of the work while the smarter eviction adds little, speculation spends resources and uses hint-based heuristics, and the full system is proven mostly in simulation on a small testbed.

Check yourself

Check your understanding

1. What is a "delayed warm start," and why does it matter?
2. Why does CIDRE race a cold start against reusing a busy container instead of predicting which is cheaper?
3. How does CIDRE fix the "imbalanced eviction" problem?
4. Which statement best reflects the limits of CIDRE?

Sources

The paper is “Concurrency-Informed Orchestration for Serverless Functions” by Qichang Liu, Yue Cheng, Haiying Shen, Ao Wang, and Bharathan Balaji, published at ASPLOS 2025, DOI 10.1145/3676641.3716253. The authors are at the University of Virginia, Alibaba Group, and Amazon; the system is open-sourced.

This explainer is written from the full 15-page text. Everything above comes from the paper’s own sections: the introduction framing cold starts, keep-alive as caching, and the two concurrency-caused challenges (delayed warm starts and imbalanced evictions); the background and workload analysis quantifying cold-start overhead (40.4 percent of cold starts exceed the execution time) and function concurrency (90th and 99th percentile of 120 and 4,482 requests per minute); the what-if analyses showing that the majority of concurrent requests would benefit from reusing a busy container and that a simple concurrency-aware eviction already helps, plus the opportunity-space and execution-variance analyses; the CIDRE design describing basic and conditional speculative scaling (with Algorithm 1’s idle-time and queuing-cost gating over a fifteen-minute window) and the concurrency-informed priority that divides by the function’s container count and uses an aging invocation rate; and the evaluation in OpenLambda against TTL, LRU, FaasCache, RainbowCake, IceBreaker, CodeCrunch, Flame, ENSURE, and an offline oracle, reporting up to 75.1 percent cold-start reduction and 39.3 percent overhead reduction, the ablation (FaasCache 44.8 percent overhead down through CIP, BSS, and CSS to CIDRE’s 27.6 percent), and the Alibaba Cloud production deployment serving 110k requests per second with a 34.5 percent cold-start reduction from basic speculative scaling.