ByteFS: System Support for (CXL-based) Memory-Semantic Solid-State Drives

A new class of SSD can be written a cacheline at a time as well as a block at a time, and file systems that only know how to write blocks turn a 64-byte inode update into a 4-kilobyte one — ByteFS picks the interface per data structure and cuts write traffic by up to 5.1 times.

Why this problem is worth solving

The storage-class memory story of the last decade was supposed to end with non-volatile memory on the memory bus. It did not: Intel shipped Optane in 2019 and shut it down in 2022, and no mature replacement arrived. But the thing people wanted — durable, byte-addressable capacity at a price that scales — is still wanted. The alternative that has emerged is to take a commodity flash SSD and expose its internal DRAM buffer through a base address register, so the BIOS and operating system map it into the host address space and software can touch it with ordinary loads and stores. That drive keeps its conventional block interface as well, so it offers both.

The economics are why anyone cares. The paper’s measurements put non-volatile memory at 300 nanoseconds to read a cacheline and 3.6 dollars per gigabyte; a memory-semantic SSD reads a cacheline in 4.8 microseconds and costs 22 cents per gigabyte, with capacity scaling to terabytes per PCIe slot. That is sixteen times slower and sixteen times cheaper, which makes it a different product rather than a substitute. The awkward part is the software. File systems have spent decades assuming the storage interface is a block, and the consequence shows up in the

What people did before, and where it broke

• Use the block interface, as always. Ext4 and F2FS treat the drive as a sequence of 4-kilobyte blocks. Every superblock, inode, bitmap or directory entry update becomes a full block write. Journaling makes it worse: in Ext4’s ordered mode, 30.7% of all traffic on average comes from journal double writes.
• Use a log-structured file system. F2FS reduces write amplification for data by writing out of place, but that shifts cost into data pointer updates, which reach 26% of write traffic and 16% of read traffic.
• Use a persistent-memory file system. PMFS and NOVA were built for byte-addressable durable memory and do the small updates properly. But they assume near-DRAM latency and no page-granular medium underneath, so they persist small updates efficiently and then fail to exploit spatial locality on reads, and their page-granular copy-on-write for crash consistency generates extra write traffic on a device where writes are expensive.
• Map the flash into a unified virtual address space. FlatFlash integrates the SSD into one virtual memory space with unified address translation. A software approach to the same hardware opportunity, but it does not rethink the file system data structures.
• Let the drive cache pages in its own DRAM. The natural firmware design, and it means the same page sits in both the drive’s DRAM and the host page cache — spending the scarcest resource in the system on data the host already has.
• The gap. One camp optimised for blocks and one for bytes, and the device offers both with different costs. Nobody had asked, structure by structure, which interface each part of a file system actually wants.

How ByteFS decides and where it puts the work

The idea

Ask the data structures, not the device. The most useful part of this paper is section 3, a quantitative study that goes through every core file system structure and asks which interface it wants and why. The answers are not uniform and they are not obvious. The superblock is touched so rarely — 0.23% of writes, 0.02% of reads — that the block interface is fine and simpler. Bitmaps are 5.4% of write traffic but up to 25.2% of reads in Ext4, so they want byte writes and block reads. Directory entries are 8% of reads and 23% of writes, and you want the whole block on a lookup because you need the full entry list. The general pattern that emerges is that metadata writes are small and want bytes while metadata reads want blocks to exploit locality, and file data wants whichever suits the size of the request. Deciding placement and granularity from measured access behaviour rather than from a fixed rule is the same discipline that governs how pages get placed across tiers in a memory system with more than one kind of memory, covered elsewhere on this site.

The mismatch inside the drive is as expensive as the one outside it. PCIe and CXL let the host write a cacheline, but NAND flash still only writes a page. Left alone, that means every byte-granular write from the host triggers a read-modify-write of a whole flash page, replacing one amplification with another one hidden inside the device. ByteFS reorganises the drive’s DRAM into a log-structured region at cacheline granularity, so byte writes append as 64-byte-aligned entries rather than landing in a page cache, and a background cleaner coalesces them before flushing to flash. Finding the lookup structure for that log is a real problem — a single skip list over a terabyte address space would be too slow — so it uses three layers: a partition table dividing the address space into 16-megabyte chunks, a skip list per partition keyed by logical page address, and an ordered chunk list within a page. Recognising that the expensive part of a system is the seam where two components disagree about granularity, and building a translation layer that absorbs the disagreement, is the same diagnosis reached by work reconciling protection granularities across a heterogeneous machine, covered elsewhere on this site.

Do not cache the same page in two scarce places. The third idea is a piece of resource discipline that follows from the first two. A conventional SSD keeps a page cache in its internal DRAM, and the host keeps a page cache too, so a hot page occupies both. ByteFS removes the drive-side page cache entirely: when a flash read is unavoidable, the loaded page is cached only in host DRAM, and the drive’s DRAM is reserved for the write log. This works because the host page cache is much larger and already tracks what is hot, while the drive’s DRAM is the scarce resource whose best use is absorbing and coalescing writes. The cost is that on workloads with very good locality, a page-granular drive cache would absorb more writes and produce less flash traffic — which the paper reports honestly. Deciding which of several memory tiers should hold a given piece of data, on the basis of what each tier is uniquely good for rather than caching everywhere, is the same reasoning behind offloading cold memory out of expensive tiers in datacentres, covered elsewhere on this site.

Following one small write to disk

ByteFS deliberately splits each 128-byte inode so the frequently changing fields sit together. That makes a typical inode update a single 64-byte write, which the byte interface performs atomically. Under a block-only file system this would have been a 4-kilobyte read-modify-write.

Why it works

• Small updates stop being large ones. A 64-byte inode change is written as 64 bytes instead of 4 kilobytes → metadata traffic between host and drive falls by 11.4 times on average against Ext4 and 6.1 times against F2FS → with maxima of 25.3 and 17.2 times.
• Reads keep the benefit blocks were good at. All metadata reads still use the block interface → a directory lookup or an inode miss brings in a whole page and the host caches it → so byte granularity is applied only where it helps, which is why read-heavy workloads are not made worse.
• The log absorbs the flash page granularity. Byte writes append at cacheline granularity and are coalesced in the background → many small writes to one flash page become one flash write → cutting flash traffic by 2.9 and 2.1 times against Ext4 and F2FS, and by 3.2 and 2.2 times against NOVA and PMFS.
• Crash consistency reuses the log instead of double-writing. Transactions commit into the same firmware-level write log → no separate journal double write for metadata → which is why ByteFS also beats NOVA and PMFS, whose page-granular copy-on-write generates extra writes.
• Scarce drive DRAM goes entirely to writes. No page cache inside the drive, only in the host → the 256-megabyte log gets the whole budget → and performance scales with log size because more entries coalesce before a flush.

What it costs (trade-offs and scope)

• Read-heavy workloads gain nothing. Webserver, Webproxy and YCSB-C perform about the same as Ext4 and F2FS, and YCSB-E — uniform range scans with no locality — is also a wash. This is a write and metadata optimisation; the paper is clear about it, but the headline numbers come from write-heavy cases.
• The device is much slower than what it replaces. A cacheline read takes 4.8 microseconds against 300 nanoseconds for non-volatile memory. The case for these drives is cost and capacity, not speed, and any workload that would have been served by persistent memory latency will notice.
• Copy-on-write costs page cache. Keeping a shadow copy of every modified page so the XOR comparison can find changed cachelines consumes 16% of an 8-gigabyte page cache on average. The authors judge this acceptable against the gains; it is still memory taken from caching.
• Sometimes ByteFS moves more data to flash than its rivals. Where locality is good, a page-granular drive cache absorbs more writes than a cacheline log does. The paper reports this and argues it does not matter because cleaning happens in the background — a fair argument, but it is a real reversal on some workloads.
• The hardware is a prototype and an emulator. The real system is an OpenSSD FPGA board with 1 terabyte of flash and a hand-extended PCIe protocol; the sensitivity study, including the CXL configuration with 175-nanosecond cacheline latency, runs on a 2.1-thousand-line emulator built on FEMU. No commercial memory-semantic SSD was used.
• Recovery takes seconds, not milliseconds. After a crash, ByteFS takes 4.2 seconds on average — 0.9 to load the drive DRAM and 2.7 to scan the log and transaction log and flush committed entries.
• It is an Ext4 derivative. 3.9 thousand lines of kernel changes on Linux 5.1, plus 0.4 thousand lines of firmware. Modest, but it inherits Ext4’s structure rather than rethinking it from scratch.

The evaluation

Check yourself

Sources

The paper is “ByteFS: System Support for (CXL-based) Memory-Semantic Solid-State Drives” by Shaobo Li, Yirui Eric Zhou, Hao Ren and Jian Huang, published at ASPLOS 2025 Volume 1, pages 116-132, DOI 10.1145/3669940.3707250. The authors are with the Systems Platform Research Group at the University of Illinois Urbana-Champaign, and Shaobo Li and Yirui Eric Zhou are listed as co-primary authors.

This explainer is written from the full text, read via the arXiv version at arxiv.org/abs/2501.04993. All figures above come from the paper’s own sections 2, 3, 4 and 5: the device characteristics of 4.8 microsecond cacheline reads and 0.22 dollars per gigabyte against 300 nanoseconds and 3.6 dollars for non-volatile memory, the 6.2 times write and 1.7 times read amplification for Ext4 and F2FS, the per-structure traffic shares including 0.23% for the superblock and 30.7% for Ext4 journaling, the 64-byte inode split and bitmap grouping, the 256-megabyte log region and 85% cleaning threshold, the 936-cycle and 14 GB/s AVX2 XOR figures and 16% page cache cost of copy-on-write, the throughput results of 2.5 and 1.48 times on micro-benchmarks and 4.1 times on OLTP, the metadata traffic reductions of 11.4 and 6.1 times on average, the flash traffic reductions of 2.9, 2.1, 3.2 and 2.2 times, the 2.4 times YCSB result, and the 4.2 second recovery time. The prototype uses an OpenSSD FPGA board; sensitivity results including the CXL configuration come from an emulator built on FEMU.