Re-architecting I/O Caches for Emerging Fast Storage Devices

Mohammadamin Ajdari, Pouria Peykani Sani, Amirhossein Moradi, Masoud Khanalizadeh Imani, Amir Hossein Bazkhanei and Hossein Asadi

Enterprise storage has long put a small fast drive in front of a big slow one as a cache, and the design rules for that were worked out when the cache was an SSD and the backend was a spinning disk; this work from Sharif University shows that when the cache becomes an ultra-fast device and the backend is now a fast SSD array, those rules quietly invert — caching every block eagerly and flushing aggressively now makes things slower, because the cache itself, not the backend, has become the bottleneck, and the backend is fast enough to help share the load; the paper measures three real caches, builds a framework to find the right settings, and reaches up to 11 times higher throughput and 30 times lower tail latency.

The argument in one glance

Problem

Cache design rules were tuned for slow-backend systems

An I/O cache puts a small fast device in front of a big slow one. The accepted rules — cache aggressively, promote every miss immediately, flush eagerly, use simple locking — were worked out when the cache was an SSD and the backend was a slow hard-disk array whose only job was to be avoided.
What people did

Reuse those rules with an ultra-fast cache on an SSD array

Keep the same cache architectures and settings, but swap in an ultra-fast cache device (Optane, persistent memory, RAM disk) over a fast all-flash SSD backend, expecting the usual speedups.
Why it fell short

The rules invert, and the cache becomes the bottleneck

On fast devices the bottleneck moves from the backend into the cache software itself. Two architectures that looked identical on hard disks now differ by 5 times; eager promotion and aggressive flushing cause cache contention; and the fast SSD backend is now capable enough that avoiding it entirely is the wrong instinct.
This paper's idea

Re-measure real caches and re-architect for fast devices

The paper empirically analyzes three real I/O caches on fast hardware, builds a framework to systematically test the design choices, and finds a scalable architecture: many threads with fine-grained locks, delayed promotion, and lazy flushing, so the cache stops being the bottleneck and the fast backend shares the load, reaching up to 11 times higher throughput and 30 times lower tail latency.

Why this problem is worth solving

I/O caching is one of the most widely-deployed cost-saving tricks in enterprise storage, and its whole design was shaped by an assumption that no longer holds.

An I/O cache places a small, fast storage device in front of a large, slower one, so that frequently-accessed blocks live on the fast device and most requests are served quickly, while the bulk of the data sits cheaply on the slow backend. This lets an enterprise get much of the performance of an all-fast system at a fraction of the price. For years the canonical setup was a solid-state drive (SSD) cache in front of an array of hard disk drives (HDDs). In that world the backend was thousands of times slower than the cache, so every design decision pointed the same way: cache as much as possible, promote a missed block into the cache immediately so the next access hits, flush dirty blocks out to the backend eagerly to keep room free, and do not worry too much about the cache software’s own overhead — because anything was better than going to a spinning disk. These became the accepted rules of thumb baked into real cache implementations, and they worked well precisely because the backend was the undisputed bottleneck.

The devices have since changed underneath those rules, and the paper’s contribution is to show, with real measurements, that the change is not incremental but qualitative.

Two things have shifted up a level. The cache is now an ultra-fast device — an Optane SSD, a persistent-memory module, or even a RAM disk — delivering millions of I/O operations per second and latencies under a microsecond. And the backend is no longer a slow disk array but a fast all-flash SSD array, itself capable of hundreds of thousands of operations per second. When both layers are fast, the bottleneck stops being the storage medium and moves into the cache software: the threads, locks, and bookkeeping that manage the cache. The paper demonstrates this concretely. Three real I/O caches that all looked bounded to about the same modest throughput on an HDD backend — because the disks hid their differences — suddenly diverge by 5 times when placed on fast hardware, as their internal parallelism is finally exposed. Even the bedrock assumption that a higher hit rate is always better breaks: because the backend SSD array is now fast, a lower hit rate can actually help by letting the system load-balance requests between the cache and the capable backend. The accepted rules were never tested in this regime, so architects were tuning by obsolete intuition.

What people did before, and where it broke

Cache SSDs in front of HDD arrays, by the old rules. Use the well-studied SSD-on-HDD design: promote aggressively, flush eagerly, simple locking, maximize hit rate. Where it broke. Those rules assume the backend is the bottleneck; on a fast SSD backend the bottleneck moves into the cache software, so the rules no longer apply.
Assume caches behave the same on fast devices. Take an architecture validated on slow backends and expect proportional speedups on ultra-fast ones. Where it broke. Two architectures indistinguishable on HDDs differ by 5 times on fast devices, because the slow backend had been masking their real parallelism and overheads.
Study only narrow fast-cache scenarios. Prior work looked at special cases — caching for HPC compute nodes, filesystem-sync buffering, or a backend with similar throughput to the cache. Where it broke. None examined the ultimate performance of general-purpose I/O caches on fast devices over SSD arrays in a storage-area network, leaving the general design question unanswered.
Trust simplistic cache models or simulations. Reason about cache behavior with idealized models rather than real, industry-grade implementations. Where it broke. Real caches have intrinsic complexity — locking, threading, promotion, flushing — whose interactions on fast hardware are exactly what determines performance and cannot be captured by simple models.
The gap. No one had measured how real I/O caches actually behave on emerging fast devices, nor identified which design choices matter and how to set them. Providing that empirical analysis, a framework to explore the choices, and a re-architected design is what this paper does.

How the re-architecting works

From old caching rules to a design measured for fast devices

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

When the fast layer changes, the old rules quietly stop holding. The paper’s central move is empirical humility: rather than assume the decades of SSD-on-HDD caching wisdom carries over, it measures what real caches actually do when the cache is ultra-fast and the backend is a fast SSD array, and discovers the intuitions invert. The clearest evidence is that two cache architectures which appeared equivalent on hard disks — because the slow backend bounded them both to the same modest throughput — pull apart by 5 times once the fast hardware exposes their true internal parallelism. Re-deriving a system’s design from fresh measurements of how software actually behaves on a new class of storage device, instead of porting assumptions tuned for the previous generation, is the same move the heterogeneous-storage key-value work covered elsewhere on this site makes, re-examining data-management choices against the real characteristics of modern fast devices rather than inheriting a design shaped by slower media.

Stop hoarding every block — the fast backend can share the work. The old rule was to keep everything in the cache and never touch the slow backend, but when the backend is a capable SSD array, the smarter policy is to let it help. So the paper delays promotion — a missed block is not pulled into the cache on first sight but only after it is missed again (an nhit policy) — and flushes lazily, bypassing write misses straight to the backend instead of aggressively evicting to make room. Both changes stop the small, fast cache from thrashing under contention and deliberately route some traffic to the fast backend, boosting throughput by over 2 times in medium-locality workloads and cutting read tail latency dramatically. Deliberately deciding what to keep in the fast tier versus what to leave to the capable slower tier, rather than promoting everything eagerly, is the same discipline the storage-tier compaction work covered elsewhere on this site follows, governing movement between fast and slow tiers so the fast tier holds what truly benefits and the slower tier absorbs the rest.

Unlock parallel hardware by shrinking the lock. On slow backends it barely mattered how the cache managed its own metadata, because the disk dominated; on fast devices the cache’s internal synchronization is the whole game. The paper shows that a cache using few threads, or many threads but a single coarse lock over the entire metadata table for every tag search, is capped at a few hundred thousand operations per second, whereas matching threads to CPU cores and using fine-grained locks — one lock per a small number of cache entries — scales to millions. Replacing a single coarse lock over a shared structure with fine-grained locks so that independent operations proceed in parallel, unlocking the throughput fast hardware can sustain, is the same design the write-intensive concurrency-control work covered elsewhere on this site is built on, narrowing the scope of each lock so that many operations touch the shared state at once instead of serializing behind one another.

Following a request through the re-architected cache

Serving I/O on an ultra-fast cache over a fast SSD backend

A trace of the paper's tuned architecture handling reads and writes, following its lookup, promotion, and flushing findings.

Step 1 of 5

A request arrives and the cache is looked up

Many cache threads, matched to the CPU cores, handle incoming requests in parallel. Each looks up the target block using fine-grained locks that cover only a few cache entries, so lookups do not serialize behind one global lock.

Coarse locking here would cap throughput at a few hundred thousand operations.

Why it works

Fast devices move the bottleneck into software. Because the cache device and the backend are both fast now → the storage medium is no longer the limiter and the cache’s threads and locks are → so design effort must shift to the cache software rather than to avoiding the backend.
Fine-grained locking exposes real parallelism. Because a single coarse lock over the whole metadata table serializes every lookup → threads pile up behind it and cap throughput at a few hundred thousand operations → so per-entry locks matched to core count are what reach multi-million-operation scalability.
Delayed promotion avoids cache thrashing. Because promoting every missed block immediately floods and contends the small fast cache → deferring promotion until a second miss keeps only genuinely hot data and offloads the rest to the fast backend → so throughput rises and read latency becomes predictable.
Lazy flushing prevents self-inflicted contention. Because aggressively flushing dirty blocks forces locking during victim selection and saturates the small cache → bypassing write misses to the capable backend instead removes that contention → so both throughput and quality of service improve.
A capable backend changes what a hit is worth. Because the SSD backend can itself serve many operations quickly → a lower hit rate can help by load-balancing between cache and backend → so maximizing hit rate is no longer the single goal it was on slow disks.

What it costs (trade-offs and scope)

The best settings are workload-dependent. The paper’s findings are not one fixed configuration but relationships: delayed promotion helps medium-locality and read-heavy workloads most, larger cache lines only pay off with delayed promotion, and the right nhit value varies. Tuning still requires knowing the workload.
Delaying promotion can cost on some patterns. Not immediately caching a missed block means a genuinely hot block takes an extra miss (served by the backend) before it is cached. This is a good trade when the backend is fast and locality is moderate, but a workload with a slower backend or very high locality could prefer eager promotion.
The gains are relative to a non-optimal baseline. The headline up-to-11-times throughput and up-to-30-times tail-latency improvements are measured against a deliberately non-optimal architecture; the improvement over an already-reasonable configuration is smaller, and the reported ranges (for example, 1.7 to 11 times throughput) show the spread.
It assumes a specific class of system. The study targets enterprise all-flash storage-area networks with an ultra-fast cache over an SSD array. Its conclusions are about that regime; a system with a slow backend, or a very different device hierarchy, would not see the same inversions.
It measures existing caches, not a new engine. The contribution is analysis, a framework, and tuned architectural choices realized in industry-grade open-source caches (OpenCAS, EnhanceIO, DM-Cache), not a clean-slate cache. Findings are constrained by what those implementations can be configured to do.
Results reflect particular hardware and traces. The numbers come from specific fast devices and workloads on one enterprise-grade system; other devices, capacities, or traces would shift the exact figures even if the qualitative findings hold.

The evaluation

Old caching rules versus the re-architected design on fast devices

Property	SSD-on-HDD rules	Naive fast-device port	Re-architected for fast devices
Where the bottleneck sits
Locking of cache metadata
Promotion of missed blocks
Flushing of dirty blocks
Result on fast devices	Not designed for this

How caching design changes when the cache becomes ultra-fast and the backend becomes a fast SSD array, as measured in the paper. Under the old SSD-on-HDD rules the slow backend was the bottleneck, so coarse locking was harmless, promoting every miss and flushing aggressively made sense, and a higher hit rate was always better. Porting those rules naively to fast devices fails, because the bottleneck moves into the cache software: two architectures that looked identical on disks differ by 5 times, a single coarse lock over the metadata table caps throughput at a few hundred thousand operations, eager promotion thrashes the small fast cache, and aggressive flushing causes victim-selection lock contention. The paper measures three industry-grade caches (OpenCAS, EnhanceIO, DM-Cache) on an enterprise all-flash system, groups the design into lookup logic, cache line size and promotion, and flushing, and finds a scalable architecture: many threads matched to CPU cores with fine-grained per-entry locks (reaching multi-million operations per second), delayed promotion via an nhit policy (over 2 times throughput on medium-locality workloads and up to 76 times lower read tail latency), and lazy flushing that bypasses write misses to the capable backend (up to 4.3 times better quality of service). Because the fast SSD backend can itself serve many operations, a lower hit rate can help by load-balancing between cache and backend. Integrated on real workloads, the tuned architecture delivers 1.7 to 11 times higher throughput and 2.6 to 30 times lower 99.99-percentile tail latency than a non-optimal one.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This is a measurement-driven systems paper, and its value is as much in the surprising findings as in the final design. The setup is a familiar cost-saving pattern — put a small fast device in front of a big slow one and cache the hot data — but the paper catches that pattern at a moment of technological change, when the cache has become an ultra-fast device and the backend has become a fast all-flash SSD array. The decades of design wisdom for I/O caches was worked out when the backend was a slow hard-disk array, and every rule pointed the same way: cache everything, promote every miss instantly, flush aggressively, maximize hit rate, and do not worry about the cache software’s own overhead. The paper shows, with real measurements on industry-grade caches rather than models, that in the new regime those rules invert. The bottleneck is no longer the storage medium but the cache software, so a single coarse lock that was harmless on disks now caps throughput at a fraction of what fast devices can sustain, and only fine-grained locking with threads matched to cores reaches multi-million-operation scalability. Because the backend SSD array is itself fast, hoarding every block is counterproductive: delaying promotion until a block is missed again, and lazily bypassing write misses to the backend, lets the two tiers share the load, raising throughput and, strikingly, cutting tail latency by up to 76 times for reads. Even the axiom that a higher hit rate is always better fails, because a lower hit rate can enable load balancing. Read the scope carefully. The results are relationships, not a single magic configuration: the best promotion and cache-line choices depend on the workload’s locality and request sizes, the headline up-to-11-times and up-to-30-times numbers are against a deliberately non-optimal baseline within a wider range, and the whole study is about enterprise all-flash storage-area networks, not every storage system. The durable idea is that caching heuristics are not laws of nature but artifacts of a particular speed gap, and when that gap changes, the heuristics must be re-measured and sometimes reversed.

Background

Why the storage hierarchy keeps forcing software to be re-thought

It helps to see this work within a recurring pattern: each time a new, much faster storage device appears, the software layers above it — caches, filesystems, databases, RAID controllers — have to be re-examined, because they encode assumptions about how slow the storage was. Fast flash and persistent memory have repeatedly turned software overhead that was once negligible into the dominant cost, exposing locks, copies, and eager policies that the slow medium used to hide. The disaggregated-RAID storage work covered elsewhere on this site reflects the same pressure at the array level, rethinking how redundancy and data layout are managed once the underlying flash devices are fast enough that the old controller-centric design becomes the limiter. The common thread is that a storage system’s design is implicitly a bet about where the time goes, and fast devices keep moving the answer: from the platter to the software, from the backend to the cache, from I/O waiting to lock contention. This paper’s specific lesson — that promotion, flushing, and locking policies tuned for slow backends can actively harm performance on fast ones — is one instance of a broader discipline that will keep recurring as devices get faster: profile the real system on the real hardware, find where the new bottleneck actually is, and be willing to reverse long-held rules of thumb rather than assume they still hold.

If you remember only three things

1. An I/O cache puts a small fast storage device in front of a large slow one so hot data is served quickly and cheaply. The accepted design rules — promote every missed block immediately, flush dirty blocks aggressively, use simple coarse-grained locking, and maximize hit rate — were tuned for an era when the cache was an SSD and the backend was a slow hard-disk array that had to be avoided at all costs. The devices have shifted up a level: the cache is now an ultra-fast device (Optane SSD, persistent memory, or RAM disk) and the backend is a fast all-flash SSD array. The paper shows, with real measurements on three industry-grade caches (OpenCAS, EnhanceIO, DM-Cache) rather than simplistic models, that in this regime the old rules quietly invert, because the bottleneck moves from the storage medium into the cache software, and the fast backend is now capable enough to share the load. Two architectures indistinguishable on disks differ by 5 times on fast hardware, and even the rule that a higher hit rate is always better fails.
2. Grouping the design into lookup logic, cache line size and promotion, and flushing, the paper builds a framework of synthetic and real workloads to test each choice and finds a scalable architecture. First, the lookup path must use many threads matched to CPU cores with fine-grained locks (one lock per a few cache entries); few threads or a single coarse lock over the whole metadata table caps throughput at a few hundred thousand operations, while fine-grained locking reaches multi-million operations per second. Second, promotion should be delayed: instead of caching every miss, promote a block only after it is missed again (an nhit policy), which avoids thrashing the small fast cache, offloads reads to the fast backend, and improves both throughput (over 2 times on medium-locality workloads) and read tail latency (up to 76 times lower). Third, flushing should be lazy: bypass write misses straight to the backend rather than aggressively evicting, avoiding victim-selection lock contention and improving quality of service by up to 4.3 times.
3. Integrated on real workloads, the tuned architecture delivers 1.7 to 11 times higher throughput and 2.6 to 30 times lower 99.99-percentile tail latency than a non-optimal one, and the paper releases its evaluation framework. Scope: the findings are workload-dependent relationships rather than one fixed configuration, the headline numbers are against a deliberately non-optimal baseline, the study targets enterprise all-flash storage-area networks with an ultra-fast cache over an SSD array, and it tunes existing open-source caches rather than building a new engine. The durable lesson is that caching heuristics are not laws of nature but artifacts of a particular speed gap between cache and backend; when new fast devices change that gap, the heuristics — promotion, flushing, locking, even the value of a hit — must be re-measured on the real hardware and are sometimes best reversed.

Check yourself

Check your understanding

1. Why do the long-accepted I/O cache design rules break when the cache is an ultra-fast device over an SSD array?
2. What did the paper find about how the cache lookup path should be built?
3. Why does delaying promotion and flushing lazily help on fast devices?
4. What did the re-architected cache achieve, and what are the limits of the result?

Sources

The paper is “Re-architecting I/O Caches for Emerging Fast Storage Devices” by Mohammadamin Ajdari, Pouria Peykani Sani, Amirhossein Moradi, Masoud Khanalizadeh Imani, Amir Hossein Bazkhanei, and Hossein Asadi, published at ASPLOS 2023, DOI 10.1145/3582016.3582041. The authors are at HPDS Research and Sharif University of Technology, Tehran, Iran.

This explainer is based on the full text of the paper, obtained from the authors’ camera-ready copy hosted on the Sharif University Data Storage and Networks group website (dsn.ce.sharif.edu). The problem framing (I/O caching as a small fast device in front of a large slow one, the SSD-on-HDD design rules of eager promotion, aggressive flushing, coarse locking, and hit-rate maximization, and why they were tuned for a slow backend), the empirical findings (that the same architecture behaves completely differently on fast devices, that two caches indistinguishable on HDDs differ by 5 times on fast hardware, that a lower hit rate can help via load balancing, and the measurements of three industry-grade caches OpenCAS, EnhanceIO, and DM-Cache), the design analysis (grouping into lookup logic, cache line size and promotion, and flushing; fine-grained locking with threads matched to cores reaching multi-million operations; delayed promotion via nhit policy giving over 2 times throughput and up to 76 times lower read tail latency; lazy flushing bypassing write misses and improving quality of service by up to 4.3 times; the interaction of cache line size with promotion), and the results (1.7 to 11 times higher throughput and 2.6 to 30 times lower 99.99-percentile tail latency over a non-optimal architecture, and the released evaluation framework) are all drawn directly from the paper. Background explanations of I/O caches, SSDs, storage-area networks, promotion and flushing policies, locking granularity, and tail latency are standard and provided to make the paper’s contribution understandable.