To wall off sensitive data (like crypto keys) or a buggy library inside a single program without paying for separate processes, CPUs offer "memory protection keys" — tag each page with a small domain number, and a thread flips its access to a whole domain by writing one register in about 25 cycles, no system call, no cache flush — but Intel and ARM both give you only 16 such domains, hopelessly few when you want one per key or per session or per object, and the software tricks to fake more are brutally slow (evicting a domain forces cross-core TLB shootdowns, over 80% overhead); this paper (Zhejiang University) builds VDom, which VIRTUALIZES the domains to give an unlimited number on top of the 16 hardware keys by spreading virtual domains across multiple page tables and letting a thread cheaply switch page tables (no TLB flush, thanks to address-space IDs) when it needs a domain that isn't currently mapped — delivering unlimited isolated compartments at under 2.65% overhead on real servers, on both Intel and ARM, where a domain switch costs 583 cycles instead of the 30,000-plus of prior systems.
The argument in one glance
-
Problem
You want many isolated compartments inside one process — cheaply
A common security need is to isolate sensitive data or untrusted code WITHIN a single program: keep a cryptographic key readable only by the code that must use it, or wall off a buggy third-party library so a flaw in it cannot corrupt the rest of the process. Doing this by splitting into separate OS processes is expensive — every interaction becomes inter-process communication and a context switch. The appeal of INTRA-process isolation is that the compartments share one address space, so switching between them can be nearly free. Modern applications need this at scale: an OpenSSL server may hold thousands of keys at once, and a browser may link dozens of libraries, many with known vulnerabilities — each ideally in its own compartment.
-
What people did
Use the CPU's hardware memory protection keys
CPUs provide a fast primitive for exactly this: memory protection keys (Intel MPK, ARM Memory Domains). Each memory page is tagged, in its page-table entry, with a small DOMAIN number, and each thread has one permission register (PKRU on Intel, DACR on ARM) holding its access rights — read, write, or no-access — for every domain. To change a thread's access to ALL pages of a domain at once, it simply writes that register: about 25 cycles, with no system call and no TLB flush. So you assign each sensitive compartment its own domain and flip permissions in a handful of cycles as control moves in and out of it.
-
Why it fell short
There are only 16 domains, and faking more is brutally slow
The hardware gives you just 16 domains — the same tiny number on both Intel MPK and ARM Memory Domains. That is nowhere near enough for fine-grained isolation, where an application may want a distinct domain per cryptographic key, per session, per object, or per library — hundreds or thousands of compartments. Existing software attempts to provide more domains all pay dearly. The disable-page approach (libmpk) reuses the 16 keys by EVICTING an old domain — disabling all its pages by editing their page-table entries — which forces cross-core TLB SHOOTDOWNS (every core must invalidate stale entries) and busy-waiting for free keys; in a web server this adds over 80% overhead, and under contention it becomes catastrophic (thousands of percent). The hypervisor approach (EPK) switches among many page tables using virtualization instructions, but pays a standing VM overhead and slows as it scales. So you can have FEW domains fast, or MANY domains slowly — not both.
-
This paper's idea
Virtualize the domains: unlimited virtual domains on 16 hardware keys
This paper (Zhejiang University — Yuan, Hong, Chang, Zhou, Shen, Ren) builds VDom, which VIRTUALIZES memory domains: it presents an UNLIMITED number of virtual domains on top of the 16 hardware keys, fast, on both Intel and ARM. The core trick is to use MULTIPLE page tables. Each page table — a "virtual domain space" — maps a DIFFERENT group of up to 16 virtual domains onto the 16 physical keys, so across many page tables, many virtual domains are simultaneously mapped. A per-thread software "virtual domain register" mirrors the hardware permission register but over unlimited domains. When a thread needs a virtual domain that is not mapped in its current page table, it MIGRATES to a page table where the domain is mapped — and, crucially, switching page tables needs NO TLB flush, because ASIDs (address-space identifiers) tag the TLB entries so nothing goes stale. That flush-free switch (about 583 cycles) is what avoids libmpk's ruinous shootdowns. When no page table holds the domain and none is free, VDom evicts an old domain like a cache, and a carefully-built call gate keeps the permission-register update secure. The result: unlimited isolated compartments at under 2.65% overhead on real servers, versus over 80% for prior software.
Why this problem is worth solving
Compartmentalizing a program so that a bug or exploit in one part cannot
reach another is one of the most effective defenses there is — least
privilege applied inside a single process — and the reason it is not used
everywhere is that it has been too coarse or too slow. Understanding the
fast hardware primitive comes first. A
memory domains are a
way to divide one process’s address space into isolated regions with
per-thread access control, enforced by hardware. On Intel (MPK) and ARM
(Memory Domains), each page is tagged in its page-table entry with a small
DOMAIN number, and each thread carries a single permission register — PKRU
on Intel, DACR on ARM — that holds, for every domain, whether this thread
may read, write, or not access it. The power of the mechanism is the
switch: to change a thread’s permissions on ALL pages of a domain at once,
it writes that one register, which takes about 25 cycles and requires no
system call and no TLB flush. That is orders of magnitude cheaper than the
alternative (calling mprotect, which edits page tables and flushes TLBs),
which is why protection keys are the natural substrate for fast in-process
compartments — you flip a register as control crosses a compartment
boundary
The specific wall, and what it takes to get past it, is the second key
idea. A
the hardware supports only
16 domains, because the permission register and the page-table tag reserve
just a few bits for the domain number. Sixteen is far too few when you want
per-key or per-object isolation, so the domains must be VIRTUALIZED:
present many virtual domains and multiplex them onto the 16 physical keys,
evicting when you run out — exactly like virtual memory multiplexes a huge
virtual address space onto limited physical RAM, or register renaming
multiplexes many logical registers onto few physical ones. The danger is
that multiplexing usually costs a remapping, and here remapping a domain
naively means editing page tables and flushing TLBs across all cores — the
very cost that makes the software approaches slow. So the real problem is
not merely to virtualize the domains, but to virtualize them WITHOUT
paying that remapping cost on the common path. VDom’s answer is to keep
each mapping in a separate page table and switch page tables cheaply,
using address-space identifiers so no flush is ever needed
What people did before, and where it broke
- Raw hardware protection keys (MPK, PKU, ARM domains). Assign each compartment a hardware key and flip the register to switch. Extremely fast (about 25 cycles), but capped at 16 domains — fine for a few compartments, useless for per-key or per-object isolation.
- Disable-page virtualization (libmpk). Reuse the 16 keys by evicting a domain, disabling all its pages via page-table edits. Provides more domains, but each eviction triggers cross-core TLB shootdowns and busy-waiting, adding over 80% overhead in a web server and thousands of percent under contention.
- Hypervisor / extended-page-table switching (EPK). Switch among many page tables with virtualization instructions to get up to hundreds of domains. But it carries a standing VM overhead of 6 to 7% and slows as the number of page tables grows.
- Separate processes. The classic way to isolate: put each compartment in its own process. Strong, but every interaction becomes IPC and a context switch — far too expensive for fine-grained, frequently-crossed boundaries.
- The gap. No approach gave MANY domains AND fast switching AND security AND portability. You could have few-and-fast or many-and-slow. That is the gap VDom closes.
How VDom gives unlimited domains
Spread domains across page tables, switch them flush-free, and evict like a cache
-
The hardware key mechanism and its 16-domain cap
Protection keys tag each page with a small domain number and let a thread flip permissions on a whole domain by writing one register in about 25 cycles, with no syscall or TLB flush. But there are only 16 domains on both Intel and ARM — far too few for per-key or per-object isolation.
-
Spread virtual domains across multiple page tables new
VDom keeps several page tables, each a "virtual domain space" that maps a DIFFERENT group of up to 16 virtual domains onto the 16 physical keys. Across many page tables, many virtual domains are mapped at once, so the effective number of active isolated compartments far exceeds 16.
-
Mirror permissions in a per-thread virtual register new
A per-thread software "virtual domain register" mirrors the hardware permission register but covers unlimited virtual domains (two bits each). Threads call a wrvdr function to set their permission on any virtual domain, without touching the real hardware register directly — which keeps the mechanism secure.
-
Switch page tables cheaply with ASIDs new
When a thread needs a virtual domain not mapped in its current page table, it migrates to a page table where the domain IS mapped by switching the page-table root. Because address-space identifiers (ASIDs) tag TLB entries, this switch needs NO TLB flush — about 583 cycles, versus the tens of thousands libmpk pays on shootdowns.
-
Evict like a cache when domains overflow new
If no page table holds the wanted domain and none has a free key, VDom either evicts an old virtual domain (disabling its pages with only a local TLB flush) or spins up a new page table. A cache-like replacement policy keeps frequently-used and "pinned" domains resident, balancing eviction cost against switch cost.
-
Guard the permission-register update new
A secure call gate wraps the hardware register write: it runs on a protected stack, reads the thread's virtual register from a protected page, performs the register update in a single instruction, and validates that instruction to block control-flow hijacking — so no compartment can switch into another's domain or corrupt the register.
The idea
Virtualize the scarce resource and multiplex it like a cache. The 16-domain wall is the same shape as every resource-scarcity problem an operating system solves: there is not enough of the physical thing, so you present an unlimited LOGICAL abstraction and multiplex it onto the few physical instances, evicting when you run out. Virtual memory does this with RAM, register renaming does it with registers, and VDom does it with protection-key domains — unlimited virtual domains over 16 physical keys, with an eviction policy for when the working set exceeds sixteen. This add-a-layer-of-indirection-to-virtualize-a-fixed-resource discipline is one the site sees elsewhere in a different guise — the microsecond-BFT page bounds an otherwise-unbounded log to a recent window, and caches everywhere keep a hot subset of a larger space — and its payoff is the same: software gets as many logical domains as it wants while the hardware stays small and fast. The novelty is not the idea of virtualizing; it is doing so without the multiplexing cost eating the win.
Route around the expensive operation with a hardware feature. The whole reason prior virtualization was slow is that remapping a domain naively means editing page tables and flushing TLBs across every core — a cross-core shootdown, one of the most expensive things a program can trigger. VDom sidesteps it: instead of re-tagging pages in place, it keeps each mapping in its OWN page table and switches page tables, and because address-space identifiers tag the TLB, switching needs no flush at all. The costly operation is not made cheaper; it is avoided entirely by using an existing hardware feature (ASIDs) that makes the flush unnecessary. This avoid-the-expensive-operation-via-a-hardware-primitive move is the same one the site’s kernel-re-randomization page makes when it relocates code by editing page tables instead of copying bytes, and the same exploit-a- hardware-mechanism move the attention-overlap page makes with SM-scheduling registers and the microsecond-BFT page makes with one-sided RDMA. Find the feature that dissolves the cost, rather than optimizing the cost.
Compartmentalize inside the process to shrink the blast radius. The security purpose underneath the performance work is least privilege at fine grain: give each key, session, or library only the access it needs, so a compromise is contained to one small compartment instead of the whole address space. That is only practical if crossing a boundary is cheap and if the boundaries are plentiful — which is exactly what VDom delivers, and why its careful call gate around the register write matters: the isolation is worthless if a compartment can simply switch itself into another’s domain, so the mechanism that grants permission must itself be untamperable. This shrink-the-blast-radius, make-the-trusted-mechanism- airtight discipline is the same one the site’s systems-security pages share when they minimize what must be trusted and harden the one path that enforces the boundary. Cheap, plentiful compartments turn a security ideal into something a production server can afford.
Turning 16 keys into unlimited compartments
From a hard cap to fine-grained isolation
Virtualizing memory-protection-key domains. Numbers from the paper's Intel and ARM evaluation.
Step 1 of 5
The hard cap
Hardware protection keys switch in about 25 cycles but there are only 16 of them, while fine-grained isolation wants hundreds of compartments. The naive way to reuse keys — disabling an evicted domain's pages — edits page tables and triggers cross-core TLB shootdowns, adding over 80% overhead in a web server.
the 16-domain cap is fixed by the few bits the register and page-table tag reserve for a domain number libmpk's eviction also busy-waits for a free key, compounding the cost under contention
Why it works
- Hardware protection keys switch permissions in about 25 cycles but there are only 16 domains → fine-grained isolation needs hundreds, so the domains must be virtualized → the challenge becomes providing unlimited virtual domains without a slow remapping.
- Naively remapping a domain edits page tables and flushes TLBs across all cores → keeping each mapping in its own page table and switching page tables instead → address-space identifiers make the switch flush-free at about 583 cycles, avoiding the shootdown entirely.
- Many page tables each map a different group of up to 16 virtual domains → many virtual domains are mapped simultaneously → the effective number of active isolated compartments far exceeds the hardware’s sixteen.
- The working set of domains can still exceed capacity → a cache-like eviction policy (with pinned and frequently-used domains kept resident) → overflow is handled gracefully, choosing eviction or page-table switch by which is cheaper.
- Isolation is worthless if a compartment can grant itself another’s access → a secure call gate performs the register update on a protected stack and validates the instruction → no compartment can switch into another’s domain or corrupt the register, so the boundary holds.
What it costs (trade-offs and scope)
- Multiple page tables have their own overhead. Keeping the page tables in sync (allocations, page-table and TLB coordination) grows with the number of page tables — 16 of them add about 39% memory-synchronization overhead — and switching to a non-VDom process is 6 to 7.6% slower. The win is that the common per-request cost stays tiny.
- The eviction-versus-switch choice must be tuned. Frequently-accessed domains favor eviction (avoiding a page-table switch), occasional ones favor switching; a poor policy or a pathological pattern that forces a switch on every access raises cost (704 versus 141 cycles; a string-replace workload sees 16% under eviction versus 7% under switching).
- The trusted base grows. VDom needs a kernel module (about 4290 lines) and a user-space API library, both trusted; a bug in either can break isolation, and security rests on the register-write call gate being airtight, which it defends and validates but which is a delicate path.
- It is an isolation primitive, not a whole defense. Physical, I/O, side-channel, and cold-boot attacks are out of scope, and control-flow hijacking is deferred to a complementary sandbox; address-space integrity depends on the kernel intercepting malicious syscalls that try to rewrite domain bits.
- It requires OS and hardware support. VDom is a modified-Linux, kernel-level deployment on CPUs with protection keys and ASIDs, not a drop-in userspace library, and the ARM results are from a small Cortex-A53 (Raspberry Pi 3).
The evaluation
A Linux kernel module (about 4290 lines) plus a small user API, on an Intel Xeon Gold 6230R (26 cores) and an ARM Cortex-A53 (Raspberry Pi 3), evaluated with microbenchmarks and three applications — Apache httpd with per-key OpenSSL isolation, MySQL with per-thread stacks and MEMORY-engine isolation, and a persistent-memory string workload — against raw MPK, libmpk, and EPK.
Getting more than 16 memory domains
| Property | Hardware keys (MPK / ARM) | Disable-page (libmpk) | VDom (this work) |
|---|---|---|---|
| Domains available | |||
| Overflow handling | |||
| Cost of reusing a key | |||
| Real-app overhead | |||
| Architectures |
Check yourself
Check your understanding
1. Hardware protection keys switch domains in about 25 cycles, so why is providing MORE than 16 domains the hard part rather than the switching itself?
2. The prior system libmpk gets more domains but adds over 80% overhead, while VDom stays under 2.65%. Predict the single design difference that accounts for this.
3. VDom provides unlimited virtual domains but the hardware still has only 16 keys. Predict what general operating-system pattern this design instantiates.
4. VDom lets each thread update its own domain permissions through a wrvdr call, yet wraps that update in a carefully-built call gate. Predict why the gate is essential to security.
Sources
- VDom: Fast and Unlimited Virtual Domains on Multiple Architectures (ACM DL, DOI 10.1145/3575693.3575735) — ASPLOS 2023 (full text read via the ACM PDF).
- ACM Digital Library entry — published version and metadata.
- VDom artifact (source and kernel patches) — implementation and evaluation scripts.