Flame: A Centralized Cache Controller for Serverless Computing

Yanan Yang, Laiping Zhao, Yiming Li, Shihao Wu, Yuechan Hao, Yuchi Ma and 1 other

Serverless platforms keep finished function containers warm in memory so the next request avoids a slow start, and that keep-alive memory is expensive — one large cloud provider spends over 20 percent of its serverless cluster memory on it; every platform decides what to keep using a controller on each individual server, and this paper (Tianjin University and Huawei Cloud) shows that design cannot work, because invocations are wildly skewed (in one trace 2 functions out of 384 draw over 90 percent of requests) and the load balancer decides which servers those land on, so a local controller faces a dilemma it has no power to resolve; the fix, borrowed from software-defined networking, is to separate the control plane from the servers entirely.

The argument in one glance

Problem

Keeping functions warm works, and costs a fortune

A serverless function that has no container already running must start one, and that startup can take orders of magnitude longer than the work itself. The standard remedy is to keep finished containers alive in memory, since resuming a paused one takes under half a millisecond. It works — but memory held for a function nobody is calling is memory wasted, and the authors report from their own platform, one of China's largest cloud providers, that keeping cold starts below 1 percent consumes more than 20 percent of the serverless cluster's memory. At their production scale that is roughly 8.8 million dollars a year.
What people did

Put a smart controller on every server

Since not everything can be kept warm, each server runs a controller that decides what to keep and for how long. The simplest version keeps everything for a fixed time. Better ones, such as FaasCache, treat it as a caching problem and prioritise by how popular a function is, how much memory it needs, and how expensive its cold start would be — evicting the least valuable when room runs short. These are sensible cache policies and they measurably beat fixed timeouts.
Why it fell short

The local controller cannot see, or fix, the actual problem

Function popularity is extremely skewed — in one production-derived trace, 2 functions out of 384 draw over 90 percent of all requests — and the LOAD BALANCER decides which servers those land on. Hash-based routing sends a function back to where it ran before, which is good for reuse but piles hot functions onto a few servers: one server here showed a 4.7 percent cold-start rate against 0.04 percent on another, a hundredfold gap. Switching to round-robin spreads the hot functions out and fixes that, but destroys reuse and costs three times the cache memory. Neither is fixable locally, because what determines the outcome is WHERE instances live, and a server-local controller does not choose that. Measured across a cluster, more than 75 percent of cache memory went to functions that were seldom called.
This paper's idea

Separate the control plane, the way networking did

This paper (Tianjin University and Huawei Cloud — Yang, Zhao, Li, Wu, Hao, Ma, Li) takes the move that reshaped network management: lift the decision-making out of the individual boxes into one controller with a global view. A cluster-wide manager identifies which functions are genuinely hot, using invocation counts weighted so that recent behaviour dominates but a momentary burst does not, and places their instances so hot functions are spread rather than piled together. Each server then splits its memory in two: a PROTECTED area for hot functions that nothing may evict, and a TEMPORARY area where everything else is kept on a timer and can be dropped on demand. The result cuts cache memory by 36 percent while reducing cold starts nearly sevenfold.

Why this problem is worth solving

The cost of the standard remedy is easy to underestimate, and the paper gives production figures for it.

A SERVERLESS function has no long-running server behind it — the platform starts a container when a request arrives and tears it down afterwards. When no container is ready, the request pays a COLD START, and initialising the environment can take orders of magnitude longer than running the function. Work on faster startup has got sandbox creation under 2 milliseconds, but the application’s own setup, resetting variables and reading configuration, still takes hundreds of milliseconds, so the problem is reduced rather than removed. The alternative is to avoid starting at all: after a function finishes, PAUSE its container and keep it in memory, since resuming takes under half a millisecond. That works, and it is what platforms do. The bill is memory held for functions nobody is currently calling — on the authors’ own platform, holding cold starts under 1 percent consumes over 20 percent of the serverless cluster’s memory, which at their scale of 40,000 functions and 1.3 billion daily requests is around 8.8 million dollars a year

What makes the standard fix fail is a property of the workload interacting with a component the cache controller does not own.

Serverless invocations are extremely SKEWED: across Azure’s published traces, 90 percent of invocations go to roughly 10 percent of functions, and in individual traces here the concentration is far sharper still — as few as 2 functions out of 384 accounting for over 90 percent of requests. That interacts badly with request routing. HASH-BASED routing sends a function’s requests to the server that ran it last, which maximises reuse of the warm container but means a handful of servers receive nearly all the traffic from the hot functions, so those servers run out of memory while others sit idle. ROUND-ROBIN routing spreads requests evenly, which removes the imbalance but means each function’s requests scatter across servers that each need their own warm copy — costing about three times the cache memory for the same result. There is no local policy that escapes this, because the choice is about PLACEMENT, and a controller that only sees one server has no say in it

What people did before, and where it broke

Keep everything warm for a fixed time. Production platforms hold a finished container for a set period, often around fifteen minutes, then release it. Simple, predictable, and the baseline everything else is measured against.
But it ignores everything about the function. It spends the same memory on a function called constantly and one called twice a day, which is why it consumes the most memory per request of any method measured here.
Treat it as a cache and prioritise. FaasCache scores each function by popularity, memory footprint and cold-start cost, and evicts the least valuable when space runs short. This is a genuine improvement and remains the strongest local method in the comparison.
But independent servers make the same decision independently. The same hot function ends up cached on many servers at once, and no server can tell that a function it sees rarely is being hammered next door. Over 50 percent of cached instances were seldom invoked, and more than 75 percent of cache memory went to functions that were not hot.
Adjust the load balancer to relieve pressure. CH-RLU trades cache locality against server load in its routing decisions, which does reduce contention.
But relieving pressure by spreading breaks reuse. Its cold-start ratio is worse, reaching 16.8 percent on one trace, because requests for a function no longer return to the server holding its warm container.
Predict demand and cache ahead of it. Icebreaker forecasts arrival rates and decides what to keep, cutting keep-alive cost by caching less.
But prediction fails exactly when it matters. Sudden bursts are what create cold starts, and short-term shifts in which functions are hot are the hardest thing to forecast, leaving it under-provisioned and with nearly triple the cold starts.
The gap. Every approach improved the policy while leaving the decision on the individual server. Nobody had asked whether a server-local controller can solve this at all — and once you notice that the deciding factor is placement, the answer is that it cannot.

How lifting the decision changes what is possible

Take the decision off the server, decide globally, and split each server's memory in two

Each server decides what to keep from what it sees

A controller on every machine tracks the functions it has run and keeps the most valuable ones warm, evicting when memory runs short. It cannot see cluster-wide popularity, cannot know the same function is cached elsewhere, and does not choose where instances are created.
↓ →
Score every function globally, with recent behaviour weighted most new

A cluster-wide manager aggregates invocation counts from all servers into one score per function. Counts are collected in fixed periods and combined so the newest period counts fully, the one before it half, and so on — recent behaviour dominates while a single burst cannot flip the ranking.
↓ →
Draw the line at half the total demand new

Sort functions by score and take the smallest set whose scores sum to half of everything. Given how skewed the workload is, that is a very small number of functions. These are the hotspots, and the list is recomputed and pushed to every server as behaviour changes.
↓ →
Place hot instances apart rather than together new

Give every server a score too, the sum of the hot functions it already holds, representing its load pressure. When a hot function needs a new instance, choose among servers with enough free resources the one that will least concentrate hotspots, so no single machine becomes the bottleneck.
↓ →
Split each server memory into protected and temporary new

Hot functions go into a protected area with no expiry that cannot be evicted to make room for anything else. Everything else goes into a temporary area on a timer, evictable on demand. When the hot list changes, instances move between the two areas rather than being destroyed and recreated.
↓ →
Shrink back after the burst passes new

A traffic spike creates many instances of a hot function at once, and nothing would release them until the hot list is next recomputed. The local agent watches how often each cached instance is actually reused and reclaims the surplus, keeping this one decision local to avoid overloading the central controller.

Implemented in about 4,000 lines of Go on OpenFaaS over Kubernetes, evaluated on 8 servers with 256GB each, against four prior methods on 8 traces of 384 functions derived from Azure production data, with function bodies drawn from ServerlessBench and FunctionBench.

The idea

Ask whether the decision is even being made in the right place. Every system in the comparison improves the policy — better scoring, better eviction, better prediction — while leaving the decision on the individual server, and the paper’s contribution is showing that no amount of policy improvement can reach the problem. The evidence is the dilemma the authors construct: with hash-based routing the hot functions concentrate and one server suffers a hundred times the cold-start rate of another; with round-robin they spread out and the cost is three times the cache memory. Both are consequences of WHERE instances live, and a server-local controller has no vote on that. Once stated this way the fix follows from a well-known precedent, which the authors name explicitly: software-defined networking separated the control plane from the individual switches for exactly this reason, because decisions that depend on global state cannot be made from local state however cleverly. Recognising that the limiting factor is the scope of the decision rather than the quality of the algorithm is the same wrong-level-of-abstraction diagnosis covered elsewhere on this site where the constraint turns out to sit in the interface or the placement rather than in the technique everyone was refining.

The savings come from the popular functions, not the unpopular ones. The intuitive story about cache waste is that cold things are being kept alive pointlessly, and it is partly true — over 75 percent of cache memory went to functions that were rarely called. But the measurement that explains the result is different: about 82 percent of the memory this system saves comes from the hot functions, the ones generating more than 90 percent of requests. Two effects account for it. The same hot function was being cached redundantly on many servers at once, because each server independently concluded it was worth keeping. And because platforms create one instance per concurrent request, a traffic spike manufactures a crowd of instances that nothing releases afterwards. A global view fixes the first by knowing where copies already exist, and the second by watching reuse rates and reclaiming the surplus. Finding that the waste is concentrated in the items everyone assumed were well managed, rather than in the obvious suspects, is the same look-again-at-where-the-cost-actually-is discipline covered elsewhere on this site where a small fraction of the workload turns out to carry most of the consequences.

Keep a hedge against your own classifier being wrong. Splitting each server’s memory into a protected area for hot functions and a temporary area for everything else looks like a simple priority scheme, and the second half is doing more work than it appears. If only hot functions were cached, every other function would suffer badly, since the hot ones would take all the memory — the authors call this the second-class citizen effect. But the more interesting reason is that hotspot detection is not instantaneous or infallible: a function that has just become popular has not made the list yet, and one that has just cooled is still on it. The temporary area, filled opportunistically with whatever idle memory exists and evictable at any moment, catches exactly those cases without any explicit mechanism. Building a cheap fallback path that absorbs the errors of your primary classifier, rather than trying to make the classifier perfect, is the same design-for-being-wrong instinct covered elsewhere on this site where a system reserves capacity for what it is confident about and lets everything else compete for the remainder.

One function through a traffic spike

Following a function as it heats up, gets placed, bursts, and cools again

Settings are the paper's chosen configuration: hotspot threshold at half the total demand, temporary keep-alive of 300 seconds.

Step 1 of 5

Counting invocations without over-reacting

Every server reports how often each function was called, and the manager combines the counts into one score per function. The current period counts fully, the previous one half, the one before a quarter. Recent behaviour dominates, but a single burst cannot dislodge a function that has been steadily popular.

the idea is borrowed from how a virtual machine decides which methods are hot enough to be worth compiling
without the decay, the hot list would oscillate with every spike, and moving instances in and out of protection costs more than it saves

Why it works

Invocations are extremely skewed and routing decides which servers receive them → the deciding factor is placement, which no server-local controller chooses → so lifting the decision to a global controller reaches a problem that better local policy cannot.
One controller knows where every instance already lives → the same hot function stops being cached redundantly on many servers → which with burst reclamation produces 82 percent of the memory savings from hot functions alone.
Scores weight recent periods most but decay rather than reset → a momentary spike cannot flip the hot list → avoiding the oscillation that makes local controllers move instances in and out repeatedly.
Hot functions sit in memory nothing may evict, and everything else competes for the rest → popular functions get guaranteed residency without starving the others → giving 0.86 percent cold starts against 1.83 to 6.11 percent for the alternatives.
The temporary area is filled opportunistically and dropped on demand → functions misclassified by a detector that is necessarily slightly behind still get cached → so detection error degrades performance gently instead of causing cold starts.

What it costs (trade-offs and scope)

A central controller is a scaling limit. The authors measure it: at 300,000 requests per second, decision latency exceeds a full second. Their answer is to partition the front end, which brings it to about 200 milliseconds — but that is partially reversing the centralisation the paper argues for.
One decision is handed back to the servers anyway. Burst reclamation is done locally because doing it centrally was too expensive at high concurrency, and the measured cost is real: the system reaches about 86 percent of what a fully centralised design achieves.
The protected area has no ceiling. Nothing caps how much of a server hot functions may occupy. The authors argue that variable workloads make this rare in practice rather than showing it cannot happen.
It reacts rather than predicts. The design is deliberately workload-agnostic, which makes it general, but it cannot pre-warm ahead of a known spike the way a forecasting system attempts to. The authors position it as combinable with prediction rather than superior to it.
The headline saving is an extrapolation. The 5.9 million dollars a year comes from applying measured per-request memory costs to their production cluster’s scale, not from running the system there.

The evaluation

Testing ran on 8 servers of 256GB each with a modified OpenFaaS, against four prior methods: fixed-timeout keep-alive, FaasCache, CH-RLU and Icebreaker. Traces came from Azure’s published serverless data covering more than 50,000 functions over two weeks, sampled down to 8 independent traces of 384 functions each, with real function bodies drawn from two benchmark suites and matched to each trace entry by memory footprint and execution time. Both routing strategies were tested, since which one is in use changes the problem substantially. Three things were measured: how often requests hit a cold start, how much memory caching consumed, and how many requests could not be served at all for lack of cache resources.

Deciding which serverless functions to keep warm

Property	Fixed timeout on each server	Smart cache policy on each server	One controller for the cluster
What it can see
Chooses where instances live
Cold-start rate measured
Requests dropped for lack of cache room
Memory held per request

Against the strongest local method the reduction in cache memory averages 36 percent under hash-based routing and about 40 percent under round-robin, with a cold-start rate nearly seven times lower. Combined cold starts and dropped requests fall from between 2.61 and 7.45 percent to 0.88, and the 99th-percentile request latency improves more than tenfold. Holding total cluster memory fixed while splitting it across more servers makes every local method worse and leaves this one roughly flat.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The most interesting tension is one the authors surface themselves: having argued that decisions must be centralised, they hand one back. Reclaiming the surplus instances a traffic burst creates is done by each server locally, because doing it centrally proved too expensive under concurrency, and they measure what that costs — about 86 percent of the performance an idealised fully centralised design reaches. In the same spirit they measure the scaling ceiling of their own controller, finding decision latency above a full second at 300,000 requests per second, and recommend partitioning the front end to fix it, which again trades some of the global view away. Two smaller gaps are worth noting. The protected memory area has no size limit, so hot functions could in principle consume an entire server, and the argument that this rarely happens rests on workloads being variable rather than on any guarantee. And the design is deliberately reactive rather than predictive, which makes it general and workload-agnostic but means it cannot pre-warm ahead of a spike it could have anticipated — the authors position it as composable with forecasting rather than better than it. Finally, treat the 5.9 million dollars a year as what it is: measured per-request memory costs extrapolated to their production cluster’s scale, not a result from deploying there. The measurements that carry the argument are the cold-start and dropped request rates on the real 8-server testbed.

Background

When better policy cannot reach the problem, because the decision is in the wrong place

This paper sits in the site’s datacenter and memory systems clusters and rhymes with several ideas covered elsewhere on this site. Its diagnosis is that a series of increasingly sophisticated policies all failed because the decision was scoped to one machine while the outcome was determined by placement — the same the-constraint-is-not-where-you-are-looking reasoning the graph-mining page uses when an interface rather than an algorithm turns out to be the obstacle, and the microservice-debugging page uses when a rule validated at one scale stops describing a larger system. The skew it exploits, with two functions out of 384 drawing over 90 percent of requests, is the same power-law concentration the quantum optimisation, recommendation-inference and instruction-prefetching pages each build on, and as on those pages the useful move is to treat the few differently from the many rather than to tune a single policy for both. Its split of each server into memory that hot functions own outright and memory everything else competes for is structurally identical to the reserved-ways cache policy the instruction-prefetching page describes, where lines proven to matter are protected from eviction by lines that are not. And its willingness to hand one decision back to the local agents and then measure exactly what that costs is the same know-what-your-simplification-buys honesty the compiler pages show when they report how far a cheap heuristic lands from an exhaustive search.

If you remember only three things

1. Serverless platforms avoid cold starts by keeping finished containers warm, which works — resuming a paused container takes under half a millisecond — but the memory is expensive, consuming over 20 percent of one large provider serverless cluster to hold cold starts below 1 percent. Every platform makes those decisions with a controller on each individual server. That cannot work, because invocations are extremely skewed and the load balancer determines which servers receive the hot functions: hash routing preserves reuse but concentrates hotspots, producing a hundredfold cold-start gap between servers, while round-robin spreads them and costs three times the memory.
2. Since the deciding factor is WHERE instances live and no server-local controller chooses that, the fix is the one software-defined networking made: lift the control plane out of the machines. A cluster-wide manager scores every function by invocation counts weighted so recent periods dominate but a single burst cannot flip the ranking, takes the smallest set accounting for half the demand as the hot set, and places their instances so hotspots are spread rather than piled. Each server then splits memory into a protected area hot functions own outright and a temporary area everything else competes for — the second acting as a cheap hedge against a detector that is necessarily slightly behind.
3. The result is 0.86 percent cold starts against 1.83 to 6.11 percent for the alternatives, dropped requests falling from around 1 percent to 0.02, cache memory down 36 percent, and 99th-percentile latency more than ten times better. The instructive measurement is where the savings come from: about 82 percent are on the HOT functions, not the cold ones, because independent servers were caching the same popular function redundantly and bursts left behind instances nothing released. Reported honestly, the authors hand burst reclamation back to the servers for practical reasons and measure the cost at roughly 14 percent of an ideal centralised design.

Check yourself

Check your understanding

1. With hash-based routing one server showed a hundred times the cold-start rate of another; switching to round-robin fixed that but tripled cache memory. Predict what this pair of results establishes.
2. Over 75 percent of cache memory went to functions that were rarely called, yet about 82 percent of the memory this system saves comes from the HOT functions. Predict how both can be true.
3. Rather than ranking functions by how often they were called recently, the design weights each period by half of the one after it. Predict what that decay is protecting against.
4. Each server reserves memory that hot functions own outright, and gives the rest to everything else on a timer. Predict why the second area exists rather than caching only hot functions.

Sources

Flame: A Centralized Cache Controller for Serverless Computing (ACM Digital Library, DOI 10.1145/3623278.3624769) — ASPLOS 2023; the full 16-page paper, read in full for this explainer.