When a website suddenly gets far more visitors than usual it needs more machines immediately, and the cloud's fastest-scaling product — where you upload individual functions and the platform runs as many copies as demand requires — can react in well under a second; the difficulty is that this product assumes your code is a small stateless function, while real web applications are large stateful programs where the developer's own code is a rounding error next to the framework holding it together, so the two obvious routes both fail, since uploading the whole application is impossible and rewriting it as functions is impractical; this paper (Shanghai Jiao Tong) takes a third route, leaving the application running as it is and having the JAVA RUNTIME watch it execute, pick out the slow parts, and ship just those fragments off to be run elsewhere — and because the runtime cannot work out in advance exactly what a fragment will need, it deliberately sends an incomplete package and lets the remote copy ask for missing pieces as it discovers them, warming that process up on duplicate requests nobody sees so that real users never pay for the learning.
The argument in one glance
-
Problem
The cloud option that scales fastest is the one web applications cannot use
A website hit by a sudden surge of traffic needs capacity immediately, and the cloud sells several ways to get it, all of which trade badly. Keeping spare machines permanently running reacts instantly but is paid for around the clock whether or not a burst ever comes. Starting machines on demand avoids that waste but takes far too long to help with a surge already in progress. The fastest-reacting option by a wide margin is FUNCTION-AS-A-SERVICE, where you upload individual functions and the platform runs as many copies as demand requires, billing only for what runs. But it is built on assumptions ordinary web applications violate. It expects small stateless functions, while a web application holds sessions and local state, and one measured here packages to 67 megabytes against a platform limit of 50 — it cannot even be uploaded.
-
What people did
Split the application into functions, by hand or by analysis
If the whole application cannot be uploaded, the natural response is to break it into pieces and upload only some. This can be done by hand, refactoring the code into separate deployable functions, which is how applications designed for this model are built in the first place. Or it can be done mechanically, analysing the source ahead of time to slice out self-contained fragments along with everything they depend on.
-
Why it fell short
The application is mostly framework, and the framework is unanalysable
Both routes founder on the same fact. In the forum application measured here, 99.6 percent of the packaged code belongs to the frameworks it is built on rather than to the developer — nearly 25 thousand classes providing authentication, database mapping and request routing. Rewriting by hand therefore means untangling code you did not write and are tightly bound to, which the authors judge impractical. Slicing it automatically ahead of time fails for a different reason: those same frameworks generate classes while the program runs and dispatch calls through general-purpose stubs, so no advance analysis can determine what a given piece of code will actually reach. The application is simultaneously too big to ship whole, too entangled to split by hand, and too dynamic to split by analysis.
-
This paper's idea
Leave it running, watch it, and ship slow fragments out at runtime
This paper (Shanghai Jiao Tong — Zhao, Wu, Tang, Zang, Wang, Chen) proposes SEMI-FAAS, refusing both alternatives. The application keeps running as a normal program holding its own state, and the JAVA RUNTIME — which sees exactly what the program does as it does it, precisely where static analysis is blind — profiles execution, picks time-consuming methods, and ships only those fragments elsewhere to run. Since the runtime cannot determine in advance everything a fragment will touch, it deliberately sends an INCOMPLETE package and lets the remote copy request missing classes and data as it discovers them, each request permanently refining the package so the cost fades. Three mechanisms keep that cost tolerable — packaging the hidden state behind native calls so they need not come home, proxying database connections that cannot be moved, and above all running duplicate requests that nobody sees so the learning happens off the critical path. With warm instances it reaches sub-second scaling, two orders of magnitude faster than the alternatives.
Why this problem is worth solving
The capability being chased here is genuinely valuable and genuinely
inaccessible to the applications that most need it, so it is worth being exact
about both halves.
FUNCTION-AS-A-SERVICE
is a cloud model in which you upload individual functions rather than whole
servers, and the platform runs as many copies as incoming demand requires,
billing only for the time they actually execute. Because the unit is small and
the platform manages the copies, it reacts to a surge far faster than any
approach requiring machines to be started and applications to be launched on
them, and it charges nothing while idle. That combination is exactly what a
website facing unpredictable traffic wants. The model achieves it by assuming
the unit is SMALL and STATELESS — a function receives its inputs, computes, and
returns, keeping nothing that must survive. This is what lets the platform
create and destroy copies freely and place them wherever there is room. A
traditional web application breaks both assumptions. It is one large program
that holds sessions, caches and local files in memory between requests, and it
is physically large — one studied here packages to 67 megabytes against a
platform limit of 50, so uploading it is not merely unwise but impossible
Bridging that gap requires reasoning about the program while it runs, not before.
To run a
fragment of a program somewhere else you must send everything it needs — its
code, and the data it will touch — and that bundle is what this paper calls a
CLOSURE. Computing one requires knowing what the fragment will reach, which for
ordinary code a compiler can work out by inspection. Web frameworks defeat that.
They GENERATE classes while the program is running rather than shipping them in
the source, and they route calls through general-purpose dispatch stubs whose
destination is decided at the moment of the call, so an ahead-of-time analysis
cannot tell what any given fragment actually invokes. What can tell is the
MANAGED RUNTIME — the virtual machine executing the program, which sees every
class actually loaded and every method actually called because it is the thing
doing the loading and calling. It can therefore profile which methods are slow
and observe what they touch. The catch is that observation is a record of what
happened before, not a guarantee about what will happen next, so any closure it
builds may be missing something — which means the design must have an answer for
running into that gap rather than assuming it away
What people did before, and where it broke
- Keep spare machines permanently running. Reacts to a burst instantly because the capacity already exists. But it is paid for continuously whether used or not, and one variant must be committed to for at least a year.
- Start machines on demand. Avoids paying for idle capacity, but provisioning takes far too long — the measurements show severe latency fluctuation while machines start and applications launch on them, which is precisely the window a burst occupies.
- Container auto-scaling services. A middle option that scales container capacity automatically, faster than raw machines but with much coarser granularity for configuration and billing than the function model.
- Run the whole application as a function. Attempted and rejected. The application holds state that the platform may discard when a function finishes, causing data loss, and it exceeds the platform’s upload size limit outright.
- Rewrite the application into functions by hand. Attempted and rejected. Almost all the code belongs to frameworks rather than the developer, so the refactoring means untangling code you did not write and are tightly coupled to.
- Slice the application automatically ahead of time. Attempted and rejected. Frameworks generate classes at run time and dispatch through general stubs, so advance analysis cannot determine what a fragment reaches.
- The gap. Every route treats the choice as between running the monolith on the function platform or converting it into functions, and both are closed. Nobody had asked whether the application could stay a monolith while individual pieces of it are executed elsewhere at run time. That is the gap this paper fills.
How fragments are shipped out while the application keeps running
Profile at run time, send an incomplete package, and let it complete itself
-
The application can be neither uploaded nor split
A web application is too large and too stateful for the function platform, too entangled with framework code to rewrite by hand, and too dynamic for ahead-of-time slicing because frameworks generate classes and dispatch calls while running.
-
Let the runtime choose what to send new
The virtual machine executing the program profiles it, using framework annotations to narrow the candidates and timing to find the slow ones. Because it is the thing loading classes and dispatching calls, it sees what static analysis cannot, and it needs no change to the application source.
-
Send a package that is knowingly incomplete new
For a chosen method the runtime builds a closure of the code and data it expects to be needed and sends it with the request. Since profiling records the past rather than predicting the future, this is accepted as approximate rather than being made safe by over-approximating.
-
Let the remote copy ask for what is missing new
When execution abroad hits a class or value it does not have, it pauses and requests it from the home server, which replies and lets execution resume. Each request permanently enriches the package, so the frequency falls sharply as the same code path is exercised again.
-
Remove the two sources of unavoidable requests new
Calls into native code carry hidden state outside the managed heap and would otherwise come home every time — over 200 thousand such calls per request — so a packaging interface marshals that state into the closure. Database connections cannot move at all, so a proxy on the home server relays them.
-
Do the learning where nobody is waiting new
The first executions are dominated by missing pieces and by the platform starting a fresh runtime. So duplicate copies of real requests are sent out with their effects discarded, purely to refine the package and warm the runtime, while the genuine requests are served at home until the remote side is ready.
The idea
Refuse the two options and find the third. The framing everyone had accepted was a choice between running a web application on the function platform, which fails on state and size, and converting it into functions, which fails on complexity. The paper’s move is to notice that neither the application nor the functions need to be the unit of migration. The application stays exactly where it is, keeping its state and its monolithic structure, and individual time-consuming FRAGMENTS are executed elsewhere while everything around them continues at home. That is a genuinely different arrangement rather than a compromise between the two, and it inherits the elastic property that matters — capacity appears in well under a second — without inheriting the statelessness requirement that made the platform unusable. Declining a dichotomy by finding the axis nobody was varying is a move covered elsewhere on this site wherever a system stops choosing between two established options and mixes at a finer granularity than either assumed.
Send something knowingly wrong, and let being wrong repair it. The most elegant decision is to accept an incomplete package. The runtime cannot know what a fragment will reach, and the safe response would be to over-approximate until correctness is assured, which for a framework-heavy application means sending nearly everything and forfeiting the point. Instead the closure is allowed to be wrong, and the mechanism for being wrong — pausing, asking the home server for the missing class or value, resuming — is also the mechanism that fixes it, because each answer is retained. So the cost is front-loaded and self-liquidating, dropping from hundreds or thousands of interruptions per request during warm-up to single digits afterwards. Starting from an approximation and letting the failures themselves complete it is the same optimistic-execution-with-a-correcting-path discipline covered elsewhere on this site, where a system takes the fast route by default and keeps a slower path that both handles the exceptions and teaches it to avoid them.
Move the expensive phase to where nobody is waiting. Even with a self-repairing closure, the first executions are brutal — the package is at its emptiest, the runtime is cold, and the platform is still constructing an instance. Serving real users through that would produce exactly the latency spike the system exists to prevent. So duplicate copies of genuine requests are sent to the remote side with their effects thrown away, purely so the closure fills in and the runtime warms up, while the real requests continue to be served at home. When the remote side is finally cheap, traffic moves. Paying a necessary cost off the critical path, on work whose results are deliberately discarded, is the same hide-the-warm-up discipline covered elsewhere on this site, and here it is measured as reducing worst-case latency by about six and a half times.
Absorbing a traffic burst
From a burst arriving to capacity in under a second
How a request comes to run elsewhere while the application stays put. Numbers from the paper's measurements.
Step 1 of 5
Normal operation
The web application runs as an ordinary program on its own server, holding sessions and connections and serving requests. Meanwhile the virtual machine underneath is profiling it — using the annotations frameworks already carry to narrow candidates, and timing to identify which methods are worth sending elsewhere.
nothing in the application source is changed, and the platform and operating system are untouched, so commercial function services can be used as they are how much work is sent away is controlled by a single ratio, which is how the system scales up and down
Why it works
- Frameworks generate classes and dispatch calls while running → ahead-of-time analysis cannot determine what a fragment reaches → but the virtual machine can, because it is the component actually loading and dispatching.
- A profile records the past rather than predicting the future → any package it builds may be missing pieces → so the design makes missing pieces a normal event handled by asking home, rather than something to be prevented.
- Every answer to such a request is retained → the same code path needs fewer requests each time it runs → so interruptions fall from over a thousand per request during warm-up to single digits afterwards.
- The expensive phase is unavoidable but its timing is not → running duplicate requests whose effects are discarded pays it while real users are still served at home → cutting worst-case latency by about six and a half times.
- Native calls and database connections would otherwise cause interruptions on every request → packaging their hidden state and proxying the connections removes both categories entirely → which is what makes the residual cost small enough to be worth paying.
What it costs (trade-offs and scope)
- Steady-state performance is worse, and the authors say so. Once stabilised, tail latency is about 15 percent above a dedicated machine on the open platform and 31 percent on the commercial one, from the residual interruptions, the bookkeeping barriers and the connection proxies. This buys elasticity, not speed.
- Network distance dominates the commercial-platform gap. The larger overhead there comes mainly from latency between function instances and the home server even within the same virtual network, and placing the open platform across availability zones pushed its overhead to 23 percent — so physical proximity matters more than the mechanism.
- It costs throughput even when idle. Maintaining the bookkeeping the offloading needs reduces peak throughput by about 7 percent for one application, though it is negligible for the other two and could be recompiled away if offloading is never used.
- The authors state precisely when this does not suit. Requests should take at least milliseconds, interruptions must stay rare, and bursts must be infrequent enough that the platform cost is acceptable — and they note that applications with frequent synchronisation on shared variables may see considerable overhead.
- It is tied to a managed runtime and required library changes. The approach needs a virtual machine that can observe and intervene, was built on one specific Java implementation, and required modifying 15 standard library classes, though those are reusable across applications.
The evaluation
BeeHive was built on a production Java virtual machine and deployed on two function platforms — an open-source one and a commercial one — without modifying either platform or the operating system. It was compared against on-demand machines, always-on burstable machines and a container auto-scaling service, using three applications: a self-built image-processing service, an open-source forum containing nearly 25 thousand classes, and an open-source blogging system, exercised with computation-heavy, mixed and database-heavy requests respectively. Bursts were generated by doubling the workload partway through a three-minute run.
Adding capacity when a burst arrives
| Property | Always-on spare machines | On-demand machines or containers | Semi-FaaS offloading |
|---|---|---|---|
| Reaction time | |||
| Cost when no burst | |||
| Application changes needed | |||
| Steady-state latency | |||
| Suits a burst that is already happening |
Check yourself
Check your understanding
1. The paper reports that slicing the application automatically ahead of time does not work. Predict why, given that automatic slicing works for many programs.
2. The runtime deliberately sends a package it knows may be incomplete. Predict why that is better than sending everything the fragment might need.
3. Duplicate copies of real requests are sent to the function platform with their results thrown away. Predict what this achieves.
4. A single request makes over 200 thousand calls into native code. Predict why that number matters so much to the design.
Sources
- BeeHive: Sub-second Elasticity for Web Services with Semi-FaaS Execution (ACM DL, DOI 10.1145/3575693.3575752) — ASPLOS 2023.
- Author PDF (IPADS, Shanghai Jiao Tong University) — the full 14-page paper, read in full for this explainer.
- ASPLOS 2023 conference talk — the authors’ presentation of the design and results.