Veil: A Protected Services Framework for Confidential Virtual Machines

A confidential VM keeps the cloud provider out, then hands everything inside to a million-line Linux kernel — Veil builds a privileged monitor inside the VM itself, using a hardware feature that lets one part of a VM lock another part out of its memory.

Why this problem is worth solving

Confidential computing sells a specific promise: you can run sensitive work on someone else’s machine without trusting them. AMD SEV-SNP delivers a real version of that promise — the VM’s memory is encrypted, the reverse map table blocks direct access from outside, and the page tables the hypervisor controls are checked for consistency so it cannot remap its way in. What the promise quietly omits is that a

That is a strange security posture. Enormous effort goes into excluding the cloud provider, and then every secret in the VM is placed at the mercy of whichever driver has a use-after-free this month. The usual fix is a

What people did before, and where it broke

• Trust the kernel. The default. Fine until the first kernel vulnerability, at which point the confidential VM’s guarantees evaporate from the inside.
• Hypervisor-based monitors. Well understood and efficient — a hypervisor-relayed switch to such a monitor costs roughly half what Veil’s does, because it does not need to resume a second virtual CPU. But it requires trusting the cloud provider, which is exactly what a confidential VM refuses to do.
• A separate hardware security layer. Virtualising something like TrustZone is possible, but you are back to trusting the provider and their administrators.
• Compiler-enforced deprivileging. Virtual Ghost instruments the kernel with bounds checks on memory accesses and control-flow integrity on branches. It is general and it works, but the checks run on ordinary code too: system call latency rises 4.9 times on average, whether the computation being run is sensitive or not.
• Manual source instrumentation. Nested Kernel forces monitor calls at sensitive operations such as page table changes, with integrity assured by binary scanning and the x86 write-protect bit. Much faster than compiler approaches, but the write-protect bit only gives integrity. Without confidentiality the monitor cannot hold a secret key, so it cannot shield a program or authenticate a log to a remote party — and it is invoked so often that the reported cost still reaches a 15 to 20% bandwidth reduction in some cases.
• The gap. Every option traded away either the threat model, ordinary-path performance, or the ability to keep a secret. Nobody had a monitor that was internal to the VM, enforced by hardware, and able to protect reads as well as writes.

How Veil builds a privileged layer inside the VM

The idea

Let hardware do the checking, so ordinary code pays nothing. The design principle the authors commit to early is that enforcement must be internal to the VM and must come from hardware rather than inserted software checks. The reason is visible in the numbers they quote about the alternatives: instrumentation taxes every memory access and every branch, so the cost is proportional to how much code runs, not to how much protection is used. Virtual machine privilege levels invert that. Setting a page inaccessible to lower levels costs one instruction at configuration time and nothing thereafter, and the check happens during address translation. The measured consequence is the paper’s cleanest result — with a Veil monitor installed but no protected service in use, SPEC CPU, memcached and NGINX all run within 2% of native. Replacing software instrumentation with a hardware isolation primitive, so that the safety cost falls on rare configuration events rather than on every instruction, is the same trade explored by work pushing past the limits of software fault isolation, covered elsewhere on this site.

Two scarce dimensions multiply into enough. SEV-SNP provides only four privilege levels, and a naive assignment breaks immediately: if all trusted software shares one level, the untrusted kernel can create a malicious enclave that runs there too. Veil’s answer is to stop treating the privilege level as the whole story and pair it with the ordinary protection ring, producing what it calls dual-factor domains. The monitor is level zero and ring zero. Protected services are level one and ring zero — walled off from the monitor’s memory and architecturally unable to create new domains, so they hold least privilege. Enclaves are level two but ring three, and that ring choice is doing real work: since all enclaves share a level, an enclave that could execute supervisor instructions would rewrite the page tables and read its neighbours. Restricting it to user mode is what keeps enclaves apart from each other. Deciding where isolation boundaries fall, and treating that placement as a design parameter rather than something the hardware dictates, is the same stance taken by operating systems built to make their compartment boundaries configurable, covered elsewhere on this site.

A costly switch is fine if you almost never switch. The honest weakness of Veil’s mechanism is that changing domains means exiting the virtual CPU and re-entering with a different saved state, which requires the hypervisor to relay it — 7135 cycles against roughly 1100 for a normal VM exit on non-confidential hardware, most of it the register save and restore that SEV-SNP mandates. The paper meets this head-on with a small piece of analysis: a monitor’s runtime cost is the switch cost multiplied by the switch count, and the two existing families sit at opposite corners. Nested Kernel has a tiny switch cost but is invoked on every page table update, so its count is enormous. Veil has an expensive switch and a count near zero during normal execution, because the delegated privileged operations happen at boot. The same logic explains why Veil is content to add about two seconds to VM startup, most of it spent touching every physical page to set permissions — pushing an unavoidable cost onto initialisation rather than steady state is exactly the calculus that governs how much of a confidential VM’s trusted setup can be trimmed to make startup fast, covered elsewhere on this site.

Following a kernel module load through Veil

This survives the kernel being tricked into disabling its own defences. Even a write gadget that clears a no-execute bit in a page table entry achieves nothing, because the privilege-level permission is checked separately and the kernel cannot change it — an attempt raises a nested page fault and halts the system.

Why it works

• Isolation is a page permission, not a code transformation. Levels are enforced during address translation → ordinary kernel and application code runs unmodified → SPEC CPU, memcached and NGINX show under 2% difference against a native confidential VM.
• The level is immutable once a virtual CPU exists. A CPU cannot raise its own level even running in supervisor mode → the kernel at the lowest level cannot escalate however thoroughly it is compromised → and if it tries to call RMPADJUST on restricted pages, the processor faults and halts rather than silently permitting it.
• Pairing levels with rings makes four levels sufficient. Enclaves share one privilege level but are confined to user mode → they cannot execute supervisor instructions to rewrite page tables → so enclave-to-enclave isolation is provided by address spaces while enclave-versus-kernel isolation is provided by the level.
• Replicating virtual CPUs avoids dedicating them. Each replica shares an identifier but holds its own saved state → every domain can run on every CPU → so the number of services is not bounded by how many CPUs you are willing to sacrifice.
• Both directions of protection come free from the same mechanism. Levels restrict reads as well as writes → the monitor can hold secret keys that the kernel cannot extract → which is what lets Veil shield programs and authenticate logs, the services a write-protection-based monitor structurally cannot offer.

What it costs (trade-offs and scope)

• The domain switch is expensive. 7135 cycles, roughly six and a half times a normal VM exit, dominated by the register save and restore SEV-SNP requires. Veil’s answer is that switches are rare in normal operation, which is true for kernel integrity and logging and much less true for enclaves.
• Enclave system calls are several times slower. Between 3.3 and 7.1 times native, because each one needs two domain switches plus argument copying. Real programs ranged from 4.9% to 63.9% overhead, with the worst case being the workload doing about 36 thousand enclave exits per second. The paper notes this is comparable to SGX and Virtual Ghost rather than better.
• Boot takes about two seconds longer. A 13% increase over an already slow confidential VM boot, more than 70% of it spent executing RMPADJUST across every physical page.
• The trusted computing base grows. VeilMon is about 4100 lines, plus the protected services, plus a system call sanitiser of roughly 1100 lines of Go. Small next to a kernel, but it is new trusted code, and shielded programs must sanitise system call results themselves to resist a malicious kernel returning hostile values.
• The four domains are fixed; the “unlimited” services share them. All enclaves live in one domain and all protected services in another. Isolation within a domain rests on ring separation and address spaces rather than on privilege levels, so a flaw in that reasoning would be a flaw between every enclave at once.
• Log protection depends on the log arriving before the compromise. VeilS-Log stores entries append-only and writes them ahead of the critical event, but the kernel is trusted to report honestly up to the moment it is compromised. Storage is a fixed reservation — the paper’s example sizes it at 1 GB against a typical 1 GB of logs per day, which means the user must collect them daily or lose them.
• Applications need porting. Each of the five shielded programs required around 200 lines of changes plus static linking, which the authors reasonably call minor against codebases of tens of thousands of lines, but it is not transparent.

The evaluation

Check yourself

Sources

The paper is “Veil: A Protected Services Framework for Confidential Virtual Machines” by Adil Ahmad, Botong Ou, Congyu Liu, Xiaokuan Zhang and Pedro Fonseca, published at ASPLOS 2023 Volume 4, pages 378-393, DOI 10.1145/3623278.3624763. The authors are with Arizona State University, Purdue University and George Mason University, and the prototype is open-sourced.

This explainer is written from the full text, read from the author copy hosted at cs.purdue.edu. All figures above come from the paper’s own sections 2, 7, 8 and 9: the 4.9-times system call latency reported for Virtual Ghost, the 15 to 20% bandwidth figure for Nested Kernel, the 7135-cycle domain switch against roughly 1100 cycles for a normal VM exit, the approximately two second and 13% boot increase, the under-2% background overhead on SPEC CPU 2006, memcached and NGINX, the 5.7% and 4.2% module load and unload costs, the 4.9% to 63.9% enclave range with system calls 3.3 to 7.1 times slower, the 1.4% to 18.7% logging overhead against Kaudit’s 0.3% to 8.7%, and the code sizes of roughly 4100 lines for the monitor and under 1200 lines of kernel and hypervisor changes. One transcription note for readers checking against the original: in the discussion of Figure 6 the paper’s text names VeilS-Kci where the figure, the table and the argument all concern VeilS-Log, the audit logging service.