Serverless platforms keep finished function containers alive for minutes so the next request skips a slow cold start, but those idle containers still hog RAM — Azure traces show a container's memory sits unused 89% of its life — and the obvious fix, pushing cold pages to a remote memory pool over RDMA, backfires with general tiered-memory tools because they sample blindly and offload pages the next request actually needs, spiking latency up to 14x; FaaSMem (Shanghai Jiao Tong) instead exploits the serverless lifecycle, sorting a container's pages into buckets by ROLE — runtime pages (loaded once, never touched again, offload immediately), init pages (offload lazily over a request window), execution pages (short-lived, leave alone) — and adds a semi-warm period that gradually drains cold pages at a rate tuned to how soon the container is likely reused, cutting local memory 9.9-79.8% and packing 108-218% more containers per node with negligible tail-latency cost.
The argument in one glance
-
Problem
Keep-alive avoids cold starts but leaves memory idle for most of a container's life
Serverless (Function-as-a-Service) platforms have a dilemma. Starting a function fresh is slow — a COLD START must spin up a container, load the language runtime, import packages, and initialize — so to keep latency low, platforms KEEP-ALIVE: after a function finishes, they do not recycle its container but hold it warm and idle for minutes, ready for the next request. The cost is memory. A warm-but-idle container still occupies all the RAM it allocated, and Azure production traces show that a container's memory is inactive for 89.2% of its lifetime under a 10-minute keep-alive. Across a datacenter, this is an enormous amount of RAM held hostage by containers doing nothing, which caps how many functions a node can host.
-
What people did
Offload cold pages to a remote memory pool with tiered-memory tools
Disaggregated memory offers a fix: a remote MEMORY POOL, reachable over RDMA (or, soon, CXL), where a machine can park pages it is not actively using and reclaim its local RAM. The operating system already has tiered-memory machinery for exactly this — tools like DAMON, TMO, and TPP detect cold pages by sampling access patterns and transparently offload them to slower memory, recalling them on access. So the obvious plan is to point these tools at idle serverless containers and let them push the cold pages out to the pool.
-
Why it fell short
General offloading is blind to the serverless lifecycle, so it hurts QoS or barely helps
General tiered-memory tools fail on serverless because they are AGNOSTIC to the container's lifecycle and just sample continuously. Two failure modes result. Too AGGRESSIVE: sampling during keep-alive flags pages as cold that the very next request will need, so when that request arrives it pays the remote-access penalty — DAMON drives response latency up to 14x. Too CONSERVATIVE: feedback-based tools like TMO offload only about 0.05% of memory every 6 seconds, which is far too slow for functions that live for seconds. Neither fits the short, bursty, structured lifetime of a serverless function, where a container loads a runtime, initializes, runs a few brief requests, and sits idle — and where whether a page is truly cold depends entirely on which STAGE the container is in. A stage-blind policy cannot tell the difference.
-
This paper's idea
Sort pages by their lifecycle role, and drain them at a reuse-tuned rate
FaaSMem (Shanghai Jiao Tong University) exploits the serverless lifecycle the general tools ignore. Its first idea, PAGE BUCKET (Pucket), segregates a container's pages into segments by the ROLE they play — Runtime pages (the language runtime, loaded once and hardly touched again), Init pages (user packages, datasets, caches loaded at function init), and Execution pages (short-lived per-request temporaries) — using time barriers in the kernel's LRU list, and applies a DIFFERENT offloading policy to each: offload runtime pages immediately, init pages lazily over a request window, and execution pages not at all. Its second idea, the SEMI-WARM period, gradually drains cold pages to the pool at a rate tuned to the container's 99th-percentile reuse interval, stopping the instant a new request arrives. Together they cut local memory 9.9-79.8% and raise container density to 108-218%, with negligible 95th-percentile latency impact.
Why this problem is worth solving
Serverless economics hinge on packing many functions onto each
machine, and keep-alive, the very mechanism that makes serverless
fast, is what wastes the memory that limits packing. The waste is
structural, rooted in
the serverless trade-off
between latency and memory. A COLD START — creating a container,
loading the language runtime, importing the function’s packages,
and running its initialization — can take hundreds of milliseconds
to seconds, which is unacceptable for latency-sensitive functions.
To avoid paying it on every invocation, platforms KEEP-ALIVE: when
a function finishes, its container is kept warm and idle for a
timeout window (often around 10 minutes) so a subsequent request
can reuse it instantly. The bet is that the function will be called
again soon, and reusing a warm container is far cheaper than a cold
start. But a warm idle container holds all its allocated memory the
whole time, even though it is doing nothing — and Azure traces show
that is most of the time (89.2% of a container’s lifetime its
memory is inactive)
Disaggregated memory should solve this, but the reason off-the-shelf
offloading fails points at the real insight. General tiered-memory
tools treat all of a container’s pages the same and sample their
access patterns blindly, which ignores that a serverless container
moves through
a highly structured
sequence of stages, each with a characteristic memory behavior.
First a runtime-loading stage brings in the language runtime and
libraries — a large block of pages accessed once during loading and
then rarely again. Then an initialization stage loads the user’s
packages, models, and cached data. Then the container serves a
handful of short requests, each allocating temporary working memory
that is freed the moment the request ends. Then it sits idle in
keep-alive, waiting to be reused. Crucially, whether a given page is
“cold” and safe to offload depends entirely on which stage produced
it and which stage the container is in: a runtime page is cold
almost immediately, an execution page is cold as soon as its request
ends, but an init page may be hot across many future requests
What people did before, and where it broke
- Vanilla keep-alive (no offloading). Hold warm containers in local RAM for the whole timeout. Fastest reuse, but memory is idle 89.2% of the time, so the node hosts far fewer functions than its RAM could support — the waste FaaSMem targets.
- Sampling-based tiered memory (DAMON, TPP). Continuously sample page access frequency and offload the coldest to remote memory. Being stage-agnostic, it flags init pages as cold during keep-alive and offloads them, so the next request stalls on remote access — DAMON pushes response latency up to 14x.
- Feedback-based offloading (TMO). Offload conservatively based on system memory pressure feedback. Safe for latency, but far too slow — about 0.05% of memory every 6 seconds — so for functions that live seconds it frees essentially nothing; FaaSMem offloads 45.5x more while holding latency.
- Linux swap / zswap. Swap cold pages to disk or compressed RAM. Coarse-grained and disk-latency-bound, with no notion of the function lifecycle, so it neither targets the right pages nor times the offload to reuse.
- The gap. No prior system tied offloading to the serverless container lifecycle — classifying pages by their role (runtime vs init vs execution) and timing the offload to when the container is likely to be reused. FaaSMem does both, which is what lets it free a lot of memory without paying the remote-access penalty that sinks stage-blind tools.
How FaaSMem works
Sort pages by role, offload each on its own schedule, and drain gently during idle
-
Exploit the lifecycle a general tool ignores new
FaaSMem is built around the serverless container lifecycle — runtime loading, initialization, a few short executions, then idle keep-alive — because whether a page is safe to offload depends on which stage created it. This structure is exactly what stage-agnostic tiered-memory tools throw away by sampling uniformly.
-
Sort pages into buckets by role (Pucket) new
Page Bucket segregates a container's pages into three segments using time barriers inserted into the Linux kernel's multi-generation LRU list: a Runtime-Init barrier and an Init-Execution barrier. Runtime pages (the ~76 MB language runtime), Init pages (user packages, datasets, caches), and Execution pages (per-request temporaries) each land in their own bucket with an independent offloading policy.
-
Runtime pages: offload immediately (reactive) new
The runtime is loaded once at startup and hardly accessed again during execution, so its pages are cold almost immediately. After the first request completes, FaaSMem offloads all inactive runtime pages at once. In experiments this caused ZERO subsequent page recalls across 11 benchmarks — a large, safe, high-value reclaim.
-
Init pages: offload lazily over a request window new
Init pages are risky — nearly 60% of containers serve at most two requests, so a page unused so far may still be needed. FaaSMem watches the descent gradient of remaining Init pages against requests executed; when it flattens (pages have stabilized), it fixes a request-window size and offloads the rest. Web services need a large window (about 20 requests), ML inference just one. A periodic rollback re-checks which pages are truly cold.
-
Execution pages: leave them alone new
Per-request execution memory is short-lived — allocated for a request and freed the instant it ends — so offloading it yields marginal benefit and risks recalls. FaaSMem simply does not offload the Execution bucket, spending its effort only where it pays.
-
Semi-warm period: drain gently, tuned to reuse new
During keep-alive, FaaSMem enters a semi-warm period and gradually offloads the remaining cold pages to the pool at a controlled rate — percentile-based (e.g. 1%/s) or amount-based (e.g. 1 MB/s) — with real-time bandwidth monitoring to avoid saturating the network. The window is set from the container's 99th-percentile reuse interval, and offloading STOPS the instant a new request arrives, so an active container is never drained.
The idea
Do not treat all the pages the same — sort them by the role that determines their coldness. The failure of general tools is a uniform-policy failure: they sample every page the same way, so they cannot distinguish a runtime page (cold seconds after startup) from an init page (potentially hot for many requests) from an execution page (dead the moment its request ends). FaaSMem’s Page Bucket segregates pages by their lifecycle role and gives each bucket its own offloading policy — immediate for runtime, windowed for init, none for execution. This is the segment-the-population-and-tailor- the-policy discipline the site keeps rewarding: the 8-bit-numerics page rescales only the outlier gradients rather than widening every value, the FHE-compiler page keeps some subcircuits scalar while vectorizing others, and here the offloader stops applying one blunt rule to a heterogeneous population and instead matches the treatment to the kind. Coldness is not a property of a page in the abstract; it is a property of the page’s role, and classifying by role is what makes safe, aggressive offloading possible.
Beat a general tool by exploiting the structure it is blind to. DAMON and TMO are general-purpose, which is their strength and their weakness — they work on any workload precisely because they assume nothing about it, and that is why they sample blindly and either over- or under-offload on serverless. FaaSMem wins by knowing something they do not: the serverless container lifecycle, with its predictable stages and the reuse-interval distribution of real functions. It uses that domain knowledge to decide what to offload (by role) and when (by reuse interval). This is the same specialized-beats-general move the site sees when a collectives library exploits the PCIe contention structure a general one cannot see, or a GNN scheduler exploits graph structure a static kernel ignores — generality has a price, and a system that embraces the structure of its domain can pay far less.
Balance two opposing costs at a point set by the actual distribution. Offloading trades memory saved against a remote-access penalty risked, and the two pull in opposite directions — offload everything and you free the most memory but stall the next request; offload nothing and you keep QoS but save nothing. The semi-warm period is FaaSMem’s tuned middle: it drains cold pages GRADUALLY, at a rate the network can absorb, over a window sized to the 99th-percentile reuse interval — so by the time a container is statistically likely to be reused, only the safely-cold pages are gone, and offloading halts the moment a real request arrives. Provisioning the window to a tail percentile of the reuse distribution, rather than a worst-case guess, is the same provision-to-the-actual-distribution discipline the site’s worst-case-provisioning family uses to avoid paying for an extreme that rarely happens.
One idle container, sorted and drained
From a memory-hoarding warm container to a lean one
A kept-alive serverless container. Numbers from FaaSMem's evaluation.
Step 1 of 5
The keep-alive waste
A function finishes and its container is held warm for minutes so the next request skips a cold start. But the container keeps all its allocated RAM — runtime, packages, caches — while doing nothing. Azure traces show this idle memory is inactive 89.2% of the container's lifetime, and across a datacenter it is the main thing limiting how many functions fit per node.
a cold start (spin up container, load runtime, import packages, init) costs hundreds of ms to seconds keep-alive trades that latency for memory held hostage during the idle window
Why it works
- Whether a page is cold depends on the lifecycle stage that created it → FaaSMem sorts pages into role buckets (runtime, init, execution) with time barriers in the LRU list → it can apply the right policy to each kind instead of one blind rule to all.
- Runtime pages are loaded once and hardly reused → offloading them immediately after the first request is a large, safe reclaim → zero subsequent recalls across 11 benchmarks.
- Init pages may stay hot across future requests but eventually stabilize → offloading them only after their usage gradient flattens over a request window → memory is freed without stalling the requests that still need those pages.
- Offloading trades memory against a remote-access penalty → draining gradually over a window sized to the 99th-percentile reuse interval, and stopping on any new request → most cold pages leave before a likely reuse, and active containers are never drained.
- General tools are stage-agnostic by design → FaaSMem exploits the known serverless lifecycle and reuse distribution → it offloads 45.5x more than conservative TMO while avoiding the 14x latency spikes of aggressive DAMON.
What it costs
- Evaluated at small scale. The results come from a 2-node cluster, which may not capture behaviors that only appear in large-scale deployments with many nodes and pools.
- Reuse estimation can misfire. The 99th-percentile reuse-interval estimate does not account for cold-start incidents, so under extreme burstiness the semi-warm timing can be wrong, occasionally draining a container that is about to be reused.
- Load imbalance can strand semi-warm containers. Under severe load imbalance, a node may lack the local memory to call a semi-warm container’s pages back from the pool when a request arrives, forcing rescheduling — which the paper leaves as future work.
- Bursty tails still take a hit. While the 95th-percentile is barely affected, in bursty scenarios tail latency rises 25.0% versus no semi-warm — the design deliberately protects 95% of requests, not the extreme tail.
- Needs RDMA infrastructure. It depends on RDMA-capable networking for the remote pool (about 32 Gbps for a 5,000-container node), and CXL-based memory pools are not yet production-ready, so the deployment surface is constrained today.
The evaluation
Eleven applications — 8 micro-benchmarks (float, matmul, linpack, image, chameleon, pyaes, gzip, json) plus BERT inference, Graph BFS, and an HTML web service — driven by high- and low-load Azure-derived traces, on a 2-node RDMA cluster with a Fastswap remote pool, against vanilla keep-alive, DAMON, and TMO.
Reclaiming idle memory from kept-alive serverless containers
| Property | Vanilla keep-alive (no offload) | General tiered memory (DAMON / TMO) | FaaSMem |
|---|---|---|---|
| Aware of function lifecycle? | |||
| Page treatment | |||
| Offload aggressiveness | |||
| QoS / tail latency | |||
| Memory reclaimed / density | None / baseline |
Check yourself
Check your understanding
1. General tiered-memory tools like DAMON offload cold pages by sampling access patterns, yet they drive serverless response latency up to 14x. Predict why a technique that works fine for general workloads fails badly here.
2. FaaSMem's Page Bucket sorts a container's pages into Runtime, Init, and Execution segments and offloads each on a different schedule. Predict why one policy for all pages cannot match this.
3. FaaSMem's semi-warm period drains cold pages gradually over a window sized to the container's 99th-percentile reuse interval, rather than offloading everything at once or nothing at all. Predict the reasoning behind this design.
4. FaaSMem offloads 45.5x more memory than the conservative TMO tool yet, unlike aggressive DAMON, barely affects 95th-percentile latency. Predict how it escapes the usual trade-off between reclaiming memory and protecting latency.
Sources
- ACM DL: FaaSMem: Improving Memory Efficiency of Serverless Computing with Memory Pool Architecture — ASPLOS 2024 (full text read via the ACM PDF; no arXiv preprint).
- FaaSMem source code (GitHub) — open-source implementation.