sIOPMP: Scalable and Efficient I/O Protection for TEEs

Secure enclaves protect memory from software but leave a hole where devices are — anything doing direct memory access writes to RAM without the processor's permission checks ever running — and closing that hole with the usual address-translation hardware costs 20 to 38 percent of network throughput; sIOPMP (Shanghai Jiao Tong) argues the mechanism was borrowed from the wrong side of the chip, since device accesses are contiguous ranges rather than scattered pages, and rebuilds the check as a pipelined tree of range comparisons that a hot device passes for free.

Why this problem is worth solving

The security guarantee has a hole in it that most descriptions of these systems skip over.

Which is why the obvious fix keeps getting rejected.

What people did before, and where it broke

• Use the device address translation unit as the protection boundary. A device confined to a virtual address space cannot reach what is not mapped into it. Where it broke. Revoking a mapping requires invalidating a cached translation through an asynchronous command queue, costing between 20 and 38 percent of network throughput in the measurements here. It is page-granular, so sub-page network buffers need copying. And it is too big to trust, since it also does interrupt remapping.
• Batch the invalidations instead of doing them immediately. The default configuration in mainstream kernels defers and groups them, which recovers most of the performance. Where it broke. It trades security for speed by design, leaving a window in which a malicious device can still reach pages that are supposed to be unmapped.
• Pin the mappings so they never need revoking. Later work fixes the device mappings up front, removing the invalidation cost entirely. Where it broke. It gives up dynamic allocation, and it is co-designed with the kernel, so the trusted base grows to include it.
• Split the world in two and give devices a side. One widely deployed architecture partitions all hardware into secure and normal, and normal-side devices cannot reach secure memory. Where it broke. Sixteen memory regions and two possible device roles. That was a reasonable budget for embedded systems and is nowhere near enough for a server with virtual functions.
• Encrypt the memory so a device cannot read it. Ciphertext is useless to an attacker who lacks the key. Where it broke. Without an integrity tree it does not stop a replay attack that rolls a region back to a stale value, and it also blocks legitimate access — the device cannot decrypt either — so data must be copied through an unencrypted bounce buffer, measured here at 23 to 24 percent of network bandwidth.
• Add a dedicated permission table beside the translation unit. What the newest secure architectures do, keeping the trusted base small while still checking device accesses. Where it broke. The table is still page-based and still inside the translation unit, so it inherits the asynchronous invalidation cost, the page granularity, and the extra table walk.
• The gap. Every one of these mechanisms was inherited from how the processor protects itself. Nobody asked whether device accesses have the same shape as processor accesses. They do not — they are contiguous, or a bounded set of contiguous ranges — and paging is the wrong abstraction for that.

How sIOPMP checks a device access

The idea

The isolation mechanism was borrowed from the processor, and device accesses are not shaped like processor accesses. This is the whole argument and everything else follows from it. A processor touches memory in a scattered, unpredictable pattern, which is what paging is for. A device does not — it streams through a contiguous buffer, or through a bounded list of contiguous buffers in scatter-gather mode. Checking an address against a list of permitted ranges is therefore not a compromise for this workload, it is the natural fit, and it drops the two costs that make paging painful here. There is no translation cache, so there is nothing to invalidate and no asynchronous command queue to wait on — reconfiguring a range takes 14 cycles through a memory-mapped register against the effectively unbounded latency of a translation-cache flush. And a range has no inherent granularity, so the sub-page buffers a network stack is full of need no copying. Noticing that a protection granularity inherited from the processor is the wrong shape for the accelerator side is the same observation as the trusted-execution granularity work covered elsewhere on this site.

A device does not care how long its own request takes, so the check can be pipelined. This is the microarchitectural insight, and it is the kind that only becomes visible once you stop treating the device check as a copy of the processor check. On the processor side a permission check sits directly in the path of an instruction that is waiting for its result, so it has to settle within a cycle, which is exactly why comparing a thousand ranges combinationally destroys the clock frequency. On the device side nothing waits — requests pipeline against each other, and their measurements confirm it, showing two saturating streams losing 5.08 against 5.18 bytes per cycle with the pipelined checker, and no measurable loss at all for writes. That frees them to spend cycles, and combining pipelining with a tree-shaped priority reduction takes the design from failing timing at 1024 ranges to holding full clock speed. It also pays off in area for an unobvious reason — left alone, the synthesis tool was burning 17.3 percent of the chip lookup tables as timing buffers for the long combinational chain, and restructuring the logic by hand cut that to 1.21 percent. Finding that a security mechanism can be moved off the critical path once you notice the workload is bandwidth-bound rather than latency-bound is the same move as the confidential inference work covered elsewhere on this site.

Scale by admitting the active set is small, and measure what happens when you guess wrong. The device count cannot be bounded, because virtual functions and plug-in cards make it a software question rather than a hardware one, and the previous design’s mistake was to treat every device as if it needed permanent hardware state. But the number of devices actually moving data at any instant is bounded by cores and bus capacity. So the hardware entries become a cache of the active set, backed by a table in protected memory, with a fault to fill on the first request from a dormant device — 341 cycles, paid once. Promotion and demotion ride on a small content-addressable table that resolves a device identifier to a fast-path slot in one cycle, driven either by an external decision or by a usage bit approximating least-recently-used. What makes this section worth reading is that the paper measures the failure mode rather than asserting it away: with the classification set wrong, so that a genuinely busy device is treated as dormant, it loses 85 percent of its throughput to constant faulting. Treating a bounded active working set as the thing to keep fast while everything else is demoted is the same structure as the mobile hot-launch work covered elsewhere on this site.

Faulting in a dormant device

The table is content-addressable precisely because the two sides have different shapes — device identifiers span a huge range while the slot count is fixed and small, so treating the slot as the address and the identifier as the content lets the lookup complete in one cycle and add nothing to the critical path.

Why it works

• Ranges need no cache, so there is nothing to invalidate. A permitted range is compared directly against the address with no translation step → so revoking permission is a register write rather than a flush through an asynchronous command queue → which is why reconfiguration costs 14 cycles per range and is deterministic, instead of being the dominant overhead.
• Ranges have no inherent granularity. A region is defined by a base and a bound rather than by page boundaries → so a sub-page network buffer is expressible directly → removing the copy that page-based schemes need for exactly the traffic that matters most.
• Nothing waits on an individual device request. Requests from one device pipeline against requests from others → so the check can be split across stages and take extra cycles without costing bandwidth → which is the freedom the processor-side equivalent does not have and the reason nobody had tried it.
• Priority reduction is associative, so it can be a tree. Combining permission results pairwise gives the same answer as walking them in order → so depth grows logarithmically rather than linearly in the number of ranges → and writing that structure by hand rather than leaving it to synthesis cut lookup-table cost by 93 percent.
• The active device set is bounded even though the device count is not. How many devices exist is a software question, but how many can move data at once is limited by cores and bus capacity → so hardware entries can act as a cache of the active set with the rest in protected memory → paid for by one fault, 341 cycles, on a dormant device’s first request.

What it costs (trade-offs and scope)

• The comparison against the page-based mechanism is cross-platform and partly simulated. There is no full implementation of that hardware for the architecture used here, so it was measured on an Intel server while sIOPMP ran on a simulated RISC-V system, and the head-to-head network numbers come from a software model of sIOPMP on the Intel machine with modification and checking costs injected from the hardware measurements. Reasonable given the constraints, and not a like-for-like bake-off.
• The unoptimised baseline is the authors’ own port. No implementation of the region-based unit existed for this architecture either, so they built one from the processor-side equivalent to compare against.
• Misclassification has a cliff, not a slope. A device wrongly treated as dormant loses 85 percent of its throughput at a modest request ratio. The mechanism works because the classification is right; the paper measures this honestly rather than burying it.
• Faulting devices in works for one at a time. The paper states that the mounting mechanism is ideal if only one dormant device is running concurrently. Two genuinely active devices sharing the reserved slot will thrash.
• The limits are bets on what machines look like now. A thousand ranges and 64 fast-path slots are justified by current scatter-gather controllers and core counts, and the paper notes that 128-core processors may require the second number to grow.
• One of the two violation-handling designs needs a bus signal that does not exist. Masking write data uses a strobe already present in common bus protocols, but zeroing response data has no equivalent, so the checker must do it itself using an extra table that costs cycles. The alternative adds a node that generates bus errors and more bus traffic.
• Reconfiguration is cheap per range but scales with range count. Blocking costs 35 cycles and each range 14, so a device with a thousand ranges to reprogram is a five-figure cycle count — still deterministic, unlike a translation flush, but not free.
• This is orthogonal to attesting devices, not a replacement for it. Every off-chip device is assumed potentially malicious. Emerging specifications that let an attested device access enclave memory directly solve a different problem, and the paper says explicitly that they still rely on page-based checking for the isolation half.
• Physical attacks and side channels are out of scope, stated plainly — memory snooping, splicing, cold-boot attacks and in-chip side channels are all excluded.
• No silicon, and a narrow device set. Everything runs on FPGA-based simulation at 60 megahertz, with one network interface and one accelerator, against a processor configuration nominally clocked far higher.

The evaluation

Check yourself

Sources

The paper is “sIOPMP: Scalable and Efficient I/O Protection for TEEs” by Erhu Feng, Dahu Feng, Dong Du, Yubin Xia, Wenbin Zheng, Siqi Zhao and Haibo Chen, published at ASPLOS 2024 Volume 2, pages 1061-1076, DOI 10.1145/3620665.3640378. The first two authors are noted as contributing equally.

This explainer is written from the full text of the published paper. Everything above comes from its own sections 1 through 7: the survey of why current secure architectures decline to trust the device translation unit, covering the cost of invalidating cached translations, the page-granularity problem for sub-page network buffers, the published attack exploiting a shared descriptor ring, and the trusted base argument; the comparison table of existing mechanisms including the deferred configuration attack window, the fixed-mapping approaches that trade flexibility, the sixteen-region two-role limit of the world-splitting architecture, the inability of memory encryption alone to stop replay, and the explicit statement that emerging device-attestation specifications are orthogonal to isolation and still rely on page-based checking; the description of the region-based unit with its source identifiers, memory domains, and static entry priorities; the section 4 design covering the pipelined checker with its block-state monitor, the tree-based priority reduction written at register-transfer level, the extended table in protected memory with fault-in of dormant devices paired to a reserved memory domain, and the content-addressable identifier mapping with explicit and usage-bit-driven promotion; the section 5 implementation covering both violation-handling designs and the read-clear signal that current bus protocols lack, the two consistency hazards and the per-identifier block bitmap that closes them, and the capability-based ownership interface built on an existing RISC-V secure monitor; and the section 6 evaluation covering clock frequency against range count for each checker design, worst-case burst latency, saturated bandwidth, reconfiguration cost, lookup-table usage with and without tree reduction, the network throughput comparison across platforms, the memcached latency measurement, and the misclassification experiment showing 85 percent throughput loss.