BypassD: Enabling fast userspace access to shared SSDs

Modern solid-state drives answer in a few microseconds, so the operating system's own storage code — once a rounding error next to a spinning disk — has become the biggest delay in reading a file, and the usual fix of talking to the drive straight from userspace breaks safe sharing because nothing then checks who may read which blocks; this work from Wisconsin, Athens, and Meta builds BypassD, which borrows the computer's virtual-memory machinery so the memory-protection hardware itself translates file offsets to disk locations and enforces permissions, giving programs direct, kernel-free, yet safely shared access that cuts 4-kilobyte read latency by 42 percent.

Why this problem is worth solving

The trouble starts with a role reversal that fast drives created.

The natural way to remove that software also removes the thing that kept storage safe to share.

What people did before, and where it broke

• Optimize inside the kernel (better scheduling, async, polling). Improve the kernel storage path with smarter I/O scheduling, overlapping requests, and polling instead of interrupts. Where it broke. These help, but the request still enters and leaves the kernel, and modern security defenses make those mode switches more expensive. Dedicating a kernel thread to poll for requests removes the switches but wastes a whole processor core.
• Kernel bypass with no sharing (SPDK). Map the drive into the application and issue requests directly from userspace. Where it broke. It is fast but unsafe to share: with no trusted checker, the program can reach every block on the drive, so the device cannot be shared securely between programs or users. It also forces developers to rebuild the file system and crash-consistency logic themselves.
• Protection on the device (MonetaD and similar). Let userspace go direct, but copy each file’s permission data onto the drive and make the drive check every request. Where it broke. The drive has limited compute and memory. Storing and updating permission data burdens it, competes with real I/O, and causes sudden throughput drops and long tail latencies. A permission-cache miss in one such system raises latency by up to eight times.
• The gap. No system provided low-latency direct access from userspace and safe sharing between programs at the same time, without heavy changes to the drive or the file system, and without modifying applications. Delivering all of that by reusing the computer’s existing address-translation hardware is what BypassD provides.

How BypassD works

The idea

Borrow the machinery of virtual memory to translate and to protect. The heart of BypassD is a reuse. A computer already has hardware that takes a virtual address, looks it up in a page table, translates it to a real location, and checks that the program is allowed to use it — this is how ordinary memory access works. BypassD teaches that same hardware a second job: translate a virtual address that stands for a file offset into a drive block address, and check the file’s read and write permissions along the way. Nothing new and untrusted has to sit on the fast path, because the translate-and-check step is the one the hardware was built to do. Taking the existing address-translation path and pointing it at a fresh target, rather than building a new lookup mechanism from scratch, is the same instinct the direct page-table indexing work covered elsewhere on this site follows, reaching the final translation entry by a direct computation instead of a fresh, purpose-built structure.

Let a trusted hardware checker make sharing safe. The reason direct access could not be shared was that removing the kernel removed the guard. BypassD puts the guard back, but in hardware instead of software: every direct request is checked by the IOMMU using the requesting program’s own page table, so a program can reach only the blocks of files it was granted and cannot forge a raw block address. Because that check runs on every access with almost no cost, many programs can pound the same drive at once, each safely confined to its own files. Giving several tenants direct, low-overhead, yet safely isolated access to one fast drive is the same goal the virtual-SSD work covered elsewhere on this site pursues, handing each tenant its own confined view of a shared drive so they can use it directly without interfering with each other.

Keep the rare, trust-sensitive work on the slow path and make the common work cheap. BypassD does not move everything to userspace. It splits the workload: the infrequent operations that change file structure — creating files, growing them, changing permissions — stay in the mature kernel file system, where safety and crash consistency are already handled. Only the frequent, latency-critical reads and writes take the fast direct path. This split is why applications need no changes and why the design stays simple. Routing the occasional structural operation through a careful, dedicated path while the bulk of accesses run fast and unhindered is the same division the serverless file-system-metadata work covered elsewhere on this site makes, handling metadata through its own managed service so the common data accesses stay quick.

Following a 4-kilobyte read through BypassD

These special entries hold a drive block address and a device identifier, not a memory page address.

Why it works

• Software was the bottleneck, so removing it from the hot path is the win. Because the drive now returns a 4-kilobyte block in about four microseconds while the kernel stack adds nearly as much again → cutting the file layer, block layer, driver, and mode switches out of every read and write → so a 4-kilobyte access drops by 42 percent and lands close to the drive’s raw latency.
• Reusing address translation supplies protection at no new cost. Because the IOMMU and page tables already translate and permission-check addresses in hardware on every memory access → giving them the extra job of turning file offsets into block addresses and checking file permissions adds no new trusted software → so userspace gets direct access that is still safe.
• A trusted hardware checker makes sharing safe. Because every direct request is verified by the IOMMU against the requesting program’s own page table → a program can reach only blocks of files it was granted and cannot forge a raw block address → so many programs share one drive securely, which kernel-bypass tools like SPDK cannot do.
• Splitting rare from frequent keeps the design simple and compatible. Because the infrequent metadata operations stay in the mature kernel file system → only the frequent reads and writes move to userspace → so applications run unmodified and crash consistency remains the kernel’s proven job.
• Host-side checks dodge the drive’s limits. Because the permission checks run on the powerful host IOMMU rather than on the storage device → the drive’s small processor and memory are never loaded with permission bookkeeping → so BypassD avoids the throughput collapses and tail-latency spikes that device-side protection suffers.

What it costs (trade-offs and scope)

• It needs hardware that does not yet exist. BypassD asks for small changes to the IOMMU (a new translation type that maps file offsets to block addresses and checks storage permissions) and to the drive (asking the IOMMU for these translations). No shipping processor has this, so the paper models the translation cost by adding a measured delay rather than running on real modified hardware.
• Opening a large file the first time is slow. Building the file mapping for a file whose entries are not already cached takes real time — negligible for small files, but growing with size, up to about twenty-one milliseconds for a sixteen-gigabyte file. The cost is paid once and amortized over many fast reads, but the current design does not hide it behind other work.
• No benefit when the drive is the bottleneck. Once many threads saturate the drive, the drive’s own service time dominates and the software savings vanish. BypassD helps expose a fast drive’s latency, not to speed up an already-overloaded one, and I/O-light workloads see little gain.
• Fairness now depends on the drive. Because the kernel is off the data path, BypassD cannot run the kernel’s I/O scheduler to balance programs. It relies on the drive’s simple round-robin scheduling across queues, which the paper finds good enough but coarser than a full kernel scheduler.
• It does not share files across virtual machines, and adds a little memory. Isolation between virtual machines is at the block level, so file sharing across them is out of scope. The file-mapping entries also cost memory, about a 0.2 percent overhead, modest but real.
• Writes to the same block must be serialized. To keep concurrent partial writes to one block correct, the userspace library serializes them, a small added constraint on the fast path.

The evaluation

Check yourself

Sources

The paper is “BypassD: Enabling fast userspace access to shared SSDs” by Sujay Yadalam, Chloe Alverti, Vasileios Karakostas, Jayneel Gandhi, and Michael Swift, published at ASPLOS 2024, DOI 10.1145/3617232.3624854. The authors are at the University of Wisconsin-Madison, the National Technical University of Athens and the University of Athens, and Meta.

This explainer is written from the full text of the paper, obtained from the last author’s University of Wisconsin publications page, which matches the ASPLOS 2024 proceedings and is distributed under a Creative Commons Attribution license. Everything above comes from the paper’s own sections: the introduction and motivation with the 4-kilobyte read latency breakdown on an Intel Optane P5800X drive (device time about half, file layer and ext4 about thirty-six percent, block layer about seven percent, driver about three percent, mode switches a few percent); the background on NVMe queue pairs mapped into userspace, the IOMMU, IO virtual addresses, and Shared Virtual Addressing; the BypassD design (its split of kernel-side metadata operations from userspace data operations, the fmap system call, the file table entries that store logical block addresses and a device identifier, the virtual-block-address to logical-block-address translation and permission check in an enhanced IOMMU using the process address-space identifier, and the revocation mechanism); the qualitative security evaluation and threat model; the implementation in Linux with ext4 (about 517 kernel lines, 1303 ext4 lines, 885 driver lines, and 1496 library lines); and the evaluation on an Intel Xeon Gold 5317 with an Optane P5800X drive, including the modeled virtual-block-address translation overhead (PCIe round trip about 345 nanoseconds, IOMMU translation costs, a minimum modeled delay of 550 nanoseconds, and translation as low as 550 nanoseconds), the 42 percent 4-kilobyte read latency reduction and roughly 30 percent average read and write latency improvements over synchronous and asynchronous kernel interfaces, the closeness to SPDK, the 25 percent average improvement over the standard kernel, the device-sharing and scaling results, the memory overhead of about 0.2 percent, the fmap overheads, and the WiredTiger and XRP comparisons (about 20 percent over baseline and about 10 percent over the recent in-kernel approach).