FlexOS: towards flexible OS isolation

Every operating system hard-wires its isolation strategy at design time — monolithic kernels trust all kernel code, microkernels pay for hardware protection everywhere, unikernels drop protection entirely — and revisiting the choice after deployment costs years (Meltdown's KPTI), so FlexOS makes isolation a build-time knob instead, turning a modular library OS into compartments connected by abstract call gates and shared-data annotations that a source-transformation toolchain instantiates as plain calls, Intel MPK domain switches, or per-compartment VMs with RPC — same code, radically different safety — plus a partial-order exploration technique that prunes 80 Redis configurations down to the 5 safest meeting a performance budget.

Why this problem is worth solving

Security engineering inside an OS is a bet placed decades in advance. The

The paper’s own measurements sharpen the cost of rigidity into one comparison: the identical compartmentalization decision — “isolate the scheduler” — costs Redis 43% of its throughput and Nginx 6%. Communication patterns, not principles, decide what isolation costs, and they differ per application. A world where the isolation strategy is a design-time constant is a world where almost every deployment is either less safe than it could afford to be, or slower than it needs to be. FlexOS’s wager is that isolation can be made a late-bound parameter — decided per deployment, changed by rebuild — without paying for the flexibility at runtime.

What people did before, and where it broke

• Monolithic kernels (Linux). Process isolation via address spaces, but the kernel is one trust domain: a bug in any driver owns the machine. Restructuring (breaking processes into isolated pieces, removing user/kernel separation for speed) is each a research project.
• Microkernels (SeL4 + Genode). Hardware isolation around every service, formally verifiable — and in this paper’s SQLite benchmark, 3.1× slower than FlexOS’s three-compartment MPK configuration achieving comparable structure.
• Unikernels / SASOSes (Unikraft and kin). Top performance, zero internal protection — one bad memcpy in the TCP stack owns the database sharing its address space.
• Compartmentalization frameworks (HODOR, ERIM, CubicleOS). Retrofit domains into existing software, but each hard-codes one mechanism and strategy. CubicleOS — the closest relative, also Unikraft-based — runs as a Linux userland process, makes MPK transitions via pkey_mprotect system calls, and lands an order of magnitude slower than FlexOS on SQLite; its trap-and-map sharing automates porting but cannot be turned off or swapped.
• LibrettOS. Can switch between SASOS and microkernel modes — two points, not a space.

How FlexOS works

The idea

Late-bind the isolation decision, like the LibOS late-bound the feature set. The library OS already proved that OS composition can wait until build time; FlexOS extends the same move to safety. The key to making it free is doing the binding by source transformation rather than runtime indirection: a gate between co-located libraries literally disappears into a function call, so the un-isolated configuration measures identical to vanilla Unikraft, and every isolation boundary you do configure compiles to exactly the mechanism-specific minimum. Genericity’s classic tax — the abstraction layer — is paid at compile time, where it costs nothing.

An isolation mechanism is just domains + switches + shared memory. The portability claim rests on a minimal contract: any mechanism that offers protection domains, a switching primitive, and some shared memory can back FlexOS — which covers MPK’s privilege-switching world, EPT’s separate-systems world, and (sketched) CHERI’s capability world, three genuinely different isolation philosophies under one API. The backend differences that remain (per-compartment TCB copies for VM-style backends, stack strategies for MPK) are absorbed by hooks in the core libraries, so adding a mechanism means implementing gates and hooks, not redesigning the OS. When hardware protection breaks — the Meltdown scenario — the response is a rebuild, not a rewrite.

When you cannot rank safety, order what you can and measure the rest. Is 3 compartments + MPK + no hardening safer than 2 compartments + EPT + CFI? There is no defensible scalar answer — so FlexOS refuses to invent one. Partial safety ordering places configurations in a partially ordered set where edges follow only defensible dominance rules (more compartments ≥ fewer; stacked hardening ≥ none; stronger mechanism ≥ weaker), leaving incomparable branches incomparable. Performance, unlike safety, is measurable: the toolchain benchmarks configurations (pruning paths early, since performance decreases monotonically along them), drops those under budget, and returns the poset’s maximal surviving elements — for Redis, 5 configurations out of 80. The human picks from five, not eighty, and every survivor is undominated.

From YAML to hardened image

porting is per-library, one-time, and mechanism-agnostic
the annotation records a security decision, not just an access fact

Why it works

• Unikraft’s micro-library decomposition already draws clean interfaces → compartment boundaries can reuse them at any granularity (merge or split), and the abstract-gate API is just those interfaces made explicit → flexible compartmentalization without a modularization project.
• Source-to-source instantiation happens before the compiler runs → co-located gates become plain calls (zero overhead, measured), isolated gates inline the mechanism’s minimum sequence, and all compiler optimizations apply → the flexibility itself costs nothing at runtime.
• Isolation mechanisms reduce to domains + switch + shared memory → MPK, EPT/VMs, and CHERI fit one API, with mechanism-specific kernel needs absorbed by core-library hooks → a new backend is gates-plus-hooks, and swapping after a hardware break is a rebuild.
• Stack allocation is cheap because bookkeeping happens at compile time → the DSS reuses that bookkeeping for shared variables (shadow = &x + STACK_SIZE in a shared upper stack half) → shared stack data at 2 cycles instead of 100–300+, for kilobytes of memory.
• Safety dominance is partially decidable even when scoring is impossible → a poset ordered by defensible rules, labeled with measurable performance, pruned monotonically → the design space FlexOS opens becomes navigable by non-experts (80 → 5).

What it costs

• Porting is real, and sharing annotations are judgment calls. Days per complex component, with the security quality of the compartmentalization resting on human-marked whitelists; automated shared-data analysis is future work, and the crash-driven loop finds what is accessed, not everything that should be private.
• Unikernel-shaped assumptions. The static, single-app, no-code-loading world makes key tricks sound (PKRU integrity by binary analysis, entry-point checking against known addresses). General-purpose multi-process systems get none of this for free.
• Trust concentrates in the TCB and toolchain. Boot, memory manager, scheduler, interrupt handling, and backend (~3,000 LoC) must be correct under every configuration — compromise there voids all walls; the compiler and hardware are trusted too (Coccinelle is not: transformations are compile-time checked).
• Mechanism limits leak through. MPK offers at most 16 domains, no execute protection, and its CFI is incomplete (a ROP into another compartment only faults on local-data access); EPT pays syscall-class gate latency and duplicates the TCB per compartment; small-payload, chatty boundaries stay expensive on every backend.
• Exploration needs a representative benchmark. Partial safety ordering ranks by measured performance; users who cannot benchmark their workload cannot use it, and the safety ordering is probabilistic dominance, not a guarantee.
• Prototype scope. Two backends complete (CHERI/SGX sketched), four applications ported, and “towards” in the title is honest — this is an architecture and an argument, demonstrated rather than industrialized.

The evaluation

Real hardware (Xeon Silver 4114), four applications (Redis, Nginx, SQLite, iPerf), 160+ auto-generated configurations via the Wayfinder benchmarking platform, and cross-OS comparisons against Linux, vanilla Unikraft, SeL4/Genode, and CubicleOS at structurally equivalent isolation profiles.

Check yourself

Sources

• arXiv preprint 2112.06566 — full text
• ACM DL: FlexOS: Towards Flexible OS Isolation — ASPLOS 2022
• Project FlexOS — source code