Lightweight Fault Isolation: Practical, Efficient, and Secure Software Sandboxing

Running someone else's untrusted code safely normally means a process or a virtual machine, which is cheap while running and brutally expensive to switch between — thousands of cycles — so cloud platforms that juggle thousands of short-lived programs turned to WebAssembly instead and pay 15 to 20 percent to run everything through a compiler they have to trust; this paper (Stanford) revisits the old idea of just checking the machine code, finds that one ARM64 addressing mode makes the bounds check free, and gets full isolation of loads, stores and jumps for 7 percent with 65,000 sandboxes in one address space and a verifier of 300 lines.

The argument in one glance

Problem

Hardware isolation is cheap to run and expensive to leave

The standard way to contain untrusted code is to give it its own address space and let the memory management unit enforce the boundary. While the code runs this costs almost nothing. Crossing the boundary is the problem: a context switch between Linux processes costs thousands of cycles on a modern processor, orders of magnitude more than saving and restoring registers, and even optimised microkernel implementations cost hundreds. That is fine for a long-running program and ruinous for a platform running thousands of short-lived ones, or for any system whose components talk constantly across the line.
What people did

Do the isolation in software instead

Put every sandbox in one address space and enforce the boundary by inserting checks into the untrusted code itself, so that no load, store or jump can reach outside its region. Then switching domains is just changing a register, and the cost of communication collapses. This is software-based fault isolation, an idea from the early nineties, and it is the only approach that gets both cheap execution and cheap switching at the same time.
Why it fell short

It was abandoned as too slow and too painful

Multi-sandbox software systems have historically cost 20 percent or more on ordinary benchmarks, which pushed implementations to compromise — some check only stores and jumps, leaving loads unguarded so sandboxes can read each other, and some achieve low overhead only by supporting very few sandboxes. Implementing it was also unpleasant: modifying compiler backends and assemblers, coping with the irregularity of variable-length instruction encodings, and working without complete machine-readable specifications of the instruction set. The industry moved on to language-based sandboxing.
This paper's idea

The situation changed, so look again

This paper (Stanford — Zachary Yedidia) observes that three things are different now. ARM64 is everywhere, and its fixed-width, regular encoding is far friendlier to this technique than variable-length instruction sets. Address spaces are large enough to hold tens of thousands of sandboxes side by side. And ARM publishes a complete machine-readable specification of the instruction set, which makes a verifier tractable to build. The result gets full isolation of loads, stores and jumps for about 7 percent, with 65,000 sandboxes in one address space and a verifier core of 300 lines.

Why this problem is worth solving

The technique has an unusual structure, and where the trust sits is the whole argument.

SOFTWARE FAULT ISOLATION puts untrusted programs in one address space and confines each to its own region by rewriting its machine code so that every load, store and indirect jump is forced to stay inside. Three pieces are involved: a tool that inserts the checks, a VERIFIER that reads the finished machine code and confirms the checks are all present and correct, and a runtime that mediates access to the outside world. The crucial property is where trust lands. The tool that inserted the checks does not need to be trusted at all, because the verifier examines the output. Language-based sandboxing works the other way — a program in a safe language is type-checked and then compiled, so the guarantee rests on the compiler being correct. That is a real burden: the high-performance option is over two million lines with hundreds of known miscompilation bugs, and the safer alternative at 200,000 lines produces measurably slower code

The whole scheme then turns on one detail of the instruction set.

Give each sandbox a 4 gigabyte region aligned to a 4 gigabyte boundary. Then the base address has zeroes in its low 32 bits, and adding any 32-bit value to it produces an address that is necessarily inside the region — the arithmetic cannot escape, whatever the untrusted program put in that register. ARM64 already has an instruction for exactly this shape: an add that zero-extends a 32-bit register before adding it to a 64-bit one. Used as a standalone guard it costs two cycles. But ARM64 also has an ADDRESSING MODE that performs the same combination inside a load or store, so a load through an untrusted pointer becomes a load from the sandbox base plus the low 32 bits of that pointer, and microarchitectural documentation shows the two forms perform identically. The bounds check stops being an instruction and becomes a property of how the memory instruction was written

What people did before, and where it broke

Give each program its own address space. The classic answer, enforced by hardware, with essentially no cost while the program runs.
But crossing the boundary costs thousands of cycles. Measured here, an empty system call takes 129 nanoseconds on Linux against 22 with software isolation, and passing a byte through a pipe takes 1504 against 46.
Run untrusted code in a virtual machine. Modest CPU overhead and strong isolation, widely deployed.
But cloud platforms are already virtualised, so this needs nesting, which costs more or is simply unavailable — one of the machines used here does not support it. And an extra level of page tables doubles the cost of a translation miss.
Use a container-style handler process. No CPU overhead, and system calls are mediated by another process.
But every mediated call becomes multiple context switches. The mature implementation measured here takes 12 microseconds for an empty system call against 26 nanoseconds.
Compile untrusted code from a safe language. This is what the industry actually does, and WebAssembly powers real edge computing platforms.
But the guarantee now rests on the compiler, which is either two million lines with known miscompilation bugs or a smaller one that generates slower code, and the measured overhead is 15 to 20 percent at best.
The gap. Software fault isolation is the only approach that gets cheap execution and cheap switching together, and it was written off for costs that were partly artefacts of the instruction sets it had been tried on.

How the bounds check disappears

From a two-cycle guard on every access to isolation that is almost free

Isolate with hardware and pay on every crossing

Give each untrusted program its own address space and let the memory management unit enforce the boundary. Nothing is inserted into the code, so execution is at full speed, but every transition between domains costs a context switch measured in thousands of cycles.
↓ →
Size each sandbox to match a register width new

Make every sandbox exactly 4 gigabytes and align it to a 4 gigabyte boundary. Now the base address has zeroes in its low 32 bits, so combining it with any 32-bit value produces an address that is necessarily inside the region. The bound is enforced by arithmetic rather than by comparison, and there is nothing to branch on.
↓ →
Fold the guard into the memory instruction new

ARM64 has an addressing mode that adds a zero-extended 32-bit register to a 64-bit base as part of the load or store itself. Rewriting an untrusted load to use the sandbox base plus the low half of the pointer register makes it safe at no cost, since both forms have identical timing.
↓ →
Reserve a few registers and give them invariants new

One register permanently holds the sandbox base and is never written. Others are declared to always contain a valid address, so anything computed into them must go through a guard. The stack pointer and return address get invariants too, guarded when written rather than when used, so functions that never spill the return address need no guards at all.
↓ →
Check the finished machine code in one linear pass new

A small verifier reads the text segment and confirms three things: every load, store and indirect branch either uses a safe addressing mode or a reserved register, reserved registers are only written by guards, and no instruction outside an allowed list appears. The compiler that produced the code is never trusted.
↓ →
Keep the runtime in the same address space and call it directly new

Put a read-only table of runtime entry points in the sandbox first page, reachable from the base register that is already reserved. A runtime call is then a load followed by an indirect branch, with no trampoline, no mode switch and no page table change — which is why an empty system call costs 22 nanoseconds.

Implemented as a 1,500-line pass over assembly text from any compiler, with a 300-line verifier core, evaluated on 14 SPEC 2017 benchmarks on an Apple M1 and a cloud ARM64 instance.

The idea

Choose the sandbox size so the instruction set does the checking for you. Every design decision here follows from one choice: make each sandbox exactly 4 gigabytes and align it to a 4 gigabyte boundary. That makes the bound a property of arithmetic rather than a comparison — combine a 64-bit base whose low 32 bits are zero with any 32-bit value at all and the result is inside the region, with no branch, no comparison and no way for a hostile value to escape. ARM64 then hands over the rest, because it already has an addressing mode that performs exactly that combination inside a load or store, so the guard costs literally nothing on the common case. The measurements make the significance clear: with only the standalone two-cycle guard the scheme is unattractive, and moving to the folded form is by far the largest single improvement, with everything afterwards delivering diminishing returns. What looks like a clever trick is really a question of fitting the abstraction to a primitive the hardware already had. Selecting a system parameter so that an existing instruction becomes the enforcement mechanism, rather than building enforcement on top, is the same match-the-hardware-you-have move covered elsewhere on this site where a design reorganises itself around a capability the underlying machine was already offering.

Check the code, do not trust the producer. The industry moved from machine code verification to safe languages, and this paper argues that trade deserves re-examination. In a language-based sandbox the guarantee flows through the compiler: the type checker validates the source and the compiler must faithfully preserve that validity into machine code. The fast implementations use a compiler of over two million lines with hundreds of known miscompilation bugs and quadratic-time algorithms that are awkward inside a security boundary; the careful alternative is 200,000 lines and produces slower code. Verifying the machine code sidesteps the choice, because the artefact being checked is the thing that will actually execute, and the checking is a single linear pass — 300 lines of core logic running at 34 megabytes per second, against 3 for the WebAssembly validator, and every benchmark binary verified in under a third of a second. There is a cost the paper is honest about: the verifier core is small but leans on a disassembler and instruction definitions it did not write, and the author says those dependencies are larger than they would like. Moving a guarantee from the process that produced an artefact to a check on the artefact itself is the same verify-the-output-not-the-pipeline discipline covered elsewhere on this site where a property is established by examining what a tool emitted rather than by trusting how it was emitted.

Needing less turns out to be a security property. The most interesting consequence is one the design did not set out to achieve. WebAssembly requires control-flow integrity — a jump must go to a legitimate target — and enforces it through the trusted compiler plus dynamic checks on indirect calls. Speculative execution can walk straight past those checks, which is how sandbox-breakout Spectre attacks work. This design requires nothing so strong. Jumping anywhere inside your own sandbox is simply legal, speculatively or otherwise; the only enforced property is that jumps stay inside, and that is enforced by the same arithmetic guard as everything else, which speculation cannot bypass because there is no branch to mispredict. So that entire attack class is mitigated by construction. The claim is carefully bounded — the two cross-domain poisoning variants still need a hardware feature that Linux does not enable and that the author could not measure — but the shape of the argument is worth noticing: weakening a requirement removed an attack surface. Finding that a mechanism is safer because it demands a weaker invariant, rather than despite it, is the same fewer-assumptions-fewer-failures observation covered elsewhere on this site where a defence that relies on less turns out to be harder to subvert.

Watching an untrusted load become safe

What happens to one memory access, and what it buys

All figures from the paper, measured on an Apple M1 and a Google Cloud ARM64 instance across 14 SPEC 2017 benchmarks.

Step 1 of 5

Start with an ordinary load through an untrusted pointer

A compiled program contains a load from whatever address a register happens to hold. Nothing constrains that register, so the untrusted program could point it anywhere in the address space, including at a neighbouring sandbox or the runtime. This is the instruction that has to be made safe, along with stores and indirect jumps.

these three instruction categories are the complete list of ways out of a sandbox, since ARM64 is a load-store architecture
some prior systems guard only stores and jumps, which is cheaper but lets sandboxes read each other

Why it works

A 4 gigabyte aligned sandbox means any 32-bit value added to the base lands inside → the bound is enforced by arithmetic with no comparison or branch → so a hostile pointer value cannot escape and there is nothing for speculation to mispredict.
ARM64 already has an addressing mode that adds a zero-extended 32-bit register to a base → the guard folds into the memory instruction itself → costing nothing on the common case, which is where almost all the performance comes from.
The verifier examines finished machine code rather than a source language → the compiler that inserted the guards is outside the trusted base → replacing two million lines of trusted compiler with 300 lines of checking.
Every sandbox lives in one address space and the runtime with it → a domain change is a register update rather than a page table switch → giving an empty system call at 22 nanoseconds and a cross-sandbox call at about 50 cycles.
Only staying inside the sandbox is enforced, not where you jump within it → no control-flow integrity is required and no dynamic check can be speculated past → removing the sandbox-breakout class of Spectre attack by construction.

What it costs (trade-offs and scope)

Every sandbox is exactly 4 gigabytes. That is the size the whole scheme depends on, and it caps what can run: the evaluation uses only single-copy benchmarks with smaller inputs for this reason.
Only one Spectre class is handled by construction. Cross-sandbox and host poisoning need an ARM extension that Linux does not enable and that was not available to measure, so that cost is unknown.
The WebAssembly comparison rests on engines the author modified. Two changes were made to one engine to speed it up, both making it non-conforming, and the comparison covers 7 of 14 benchmarks because WebAssembly lacks features the others need.
The verifier is small but not self-contained. The 300-line core depends on a commercial disassembler, an ELF reader and generated instruction definitions, which the author flags as larger than desired.
It needs its own instrumented libraries. A standard toolchain will not do; the C library, compiler intrinsics and unwinder all have to be rebuilt, and only one such toolchain currently exists.

The evaluation

Fourteen SPEC 2017 benchmarks, on an Apple M1 laptop representing the best available ARM64 performance and a Google Cloud instance representing a virtualised server. Three comparison points, each answering a different question: optimization levels within the scheme itself, to see where the performance actually comes from; the two fastest WebAssembly engines, to compare against what the industry uses; and hardware isolation through Linux processes, a container-style handler and a virtual machine, to compare against what the scheme replaces. Microbenchmarks cover the transitions, which is the whole point of the design.

Property	Hardware isolation	WebAssembly	Verified machine code
Cost while running
Cost of an empty system call
What has to be trusted
Sandboxes per address space
Sandbox-breakout speculation

Geometric mean overhead is 6.4 percent on an Apple M1 and 7.3 percent on a cloud ARM64 instance, against 18.2 and 22.3 percent for the fastest WebAssembly engine and 15.7 percent for a hand-tuned non-conforming variant. Code size grows 12.9 percent in the text segment against 22 percent for WebAssembly. A cross-sandbox call costs about 50 cycles, where published work puts the hardware-protection limit near 400.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The 4 gigabyte sandbox is not a tuning parameter, it is the foundation — the guard works precisely because a 32-bit value cannot escape a 4 gigabyte aligned region — so the size limit is permanent, and it visibly shapes the evaluation, which uses only the smaller-input benchmark variants for that reason. The comparison against WebAssembly deserves care. It covers 7 of 14 benchmarks, because WebAssembly lacks C plus plus exceptions, non-local jumps and some system calls, and two benchmarks needed source changes to compile at all. More delicately, the author improved one competing engine in two ways to make the comparison fairer — removing a barrier that enforces conforming trap behaviour and pinning a base pointer in a register — which makes that engine non-conforming, as the paper states, and means the headline gap is against engines tuned by the person reporting it. On security, the Spectre claim is real but narrow: sandbox breakout is mitigated by construction because nothing depends on control-flow integrity, but cross-sandbox and host poisoning still require an ARM extension that Linux does not enable and that no available hardware allowed the author to measure. And the trusted base, while genuinely much smaller than a compiler, is not 300 lines in practice — that is the core logic, resting on a commercial disassembler, an ELF reader and generated instruction definitions that the author explicitly says are larger than desired.

Background

When an abandoned technique becomes viable again

This paper sits in the site’s security and systems clusters and rhymes with several ideas covered elsewhere on this site. Its sharpest relationship is a disagreement: the hardware-assisted isolation page argues that software sandboxing costs too much and is structurally open to speculative attack, and therefore needs a processor extension, while this work reaches roughly a third of that overhead in pure software and mitigates the sandbox-breakout attack class by needing a weaker guarantee — though both agree that the cross-domain poisoning variants want hardware help. Its argument about where trust should sit, checking a finished artefact rather than trusting the tool that produced it, is the same verify-the-output discipline the compiler consistency and container isolation pages apply to bugs that leave no other trace. Its subject, that the boundary between mutually distrusting code is a design choice with a measurable price rather than a given, rhymes with the flexible OS isolation page, which makes the same point by turning that choice into a build parameter. And its re-examination of an idea the field had written off, on the grounds that the instruction sets and address spaces it was judged on have since changed, is the same the-assumptions-expired argument several architecture pages make when an old trade-off is recomputed against current hardware.

If you remember only three things

1. Isolating untrusted code with hardware is free while running and expensive to leave — thousands of cycles per context switch — which is the wrong shape for platforms running thousands of short-lived programs. Doing it in software puts every sandbox in one address space and makes crossing nearly free, but the classic technique of rewriting machine code to bound every access was written off as costing 20 percent or more and being painful to implement against irregular instruction sets. The industry moved to language-based sandboxing instead, which relocates the cost into a compiler that has to be trusted and is over two million lines with known miscompilation bugs.
2. Three things changed, and the technique gets much better. Make each sandbox exactly 4 gigabytes and align it, and the bound becomes arithmetic rather than comparison: combining the base with any 32-bit value produces an address that cannot leave the region. ARM64 already has an addressing mode doing exactly that inside a load or store, so on common accesses the guard costs nothing at all — moving from a standalone two-cycle guard to this folded form is by far the largest improvement measured. Address spaces are now big enough for 65,000 sandboxes side by side, and a published machine-readable specification of the instruction set makes a verifier practical to build.
3. Full isolation of loads, stores and jumps costs about 7 percent, against 15 to 20 for the fastest WebAssembly engines, with the compiler outside the trusted base because a 300-line verifier checks the finished machine code at 34 megabytes per second. An empty system call is 22 nanoseconds against 129 through Linux, and a cross-sandbox call about 50 cycles where published work puts the hardware limit near 400. And because nothing here depends on control-flow integrity — jumping anywhere inside your own sandbox is legal — the sandbox-breakout class of speculative attack is mitigated by construction, though the cross-domain variants still want hardware support that was not available to measure.

Check yourself

Check your understanding

1. Software fault isolation historically cost 20 percent or more, which is why the industry abandoned it. Predict what makes it cost 7 percent here.
2. WebAssembly is the industry standard for software sandboxing and the paper positions itself against it. Predict the argument about trust, not speed.
3. One class of Spectre attack is mitigated here by construction rather than by an added defence. Predict which, and why the design gets it for free.
4. The paper argues this technique deserves reconsideration now, having been abandoned earlier. Predict what changed, given the technique itself is from the early nineties.

Sources

Zachary Yedidia. “Lightweight Fault Isolation: Practical, Efficient, and Secure Software Sandboxing.” ASPLOS 2024, La Jolla. Stanford University. DOI 10.1145/3620665.3640408.
Full text of the paper was used for this explainer, including the ARM64 overview and addressing modes in Section 2, the basic guard scheme and sandbox layout in Section 3, the zero-instruction guard, stack pointer handling, redundant guard elimination and trampoline-free runtime calls in Section 4, the compiler, verifier and runtime implementation in Section 5, the runtime and code size measurements and the comparisons against WebAssembly engines and hardware isolation in Section 6, and the discussion of speculative attacks and portability in Section 7.