RAID combines several disks into one faster and more reliable virtual disk, and its whole appeal is that adding drives adds speed — datacenters now want to do the same thing with drives that live in other machines, reached over the network, because that lets storage and computing be scaled and replaced independently; the obstacle is a decades-old cost that changes character completely once the drives are remote, since updating part of a protected stripe requires reading the old data and old checksum and writing both back, which locally is extra work for the disks but over a network is extra traffic through the one network card the whole array shares; the arithmetic is brutal, because with a fast card and typical drives the array tops out at 2.6 times a single drive no matter how many drives you add, destroying the property that made RAID worth having; this paper (Peking University) observes that while the extra disk work is unavoidable the extra NETWORK traffic is not, and lets the storage machines exchange data directly with each other instead of routing everything through the client.
The argument in one glance
-
Problem
Putting RAID over the network destroys the property that made it worth having
RAID builds one virtual disk from several real ones, giving more capacity and speed than any single drive plus survival of a drive failure, and its entire appeal is that adding drives adds throughput. Datacenters increasingly want their storage in separate machines reached over the network, so storage and computing can be scaled and replaced independently — and spreading an array across machines is genuinely more robust, since it removes the single point of failure a monolithic storage server represents. But parity RAID carries a decades-old penalty: writing part of a protected group requires reading the old data and old checksum then writing both back, turning a small write into four operations. Locally that is extra work for the disks. Remotely, every one crosses the client's single network card — and the arithmetic is brutal, because a 100 gigabit card against typical drives caps the array at 2.6 times one drive NO MATTER how many drives are added.
-
What people did
Avoid the small writes, or accept the cost
The write penalty is not new and has been attacked for decades. The standard remedy is to avoid triggering it — accumulate small writes in a battery-backed memory cache until enough accumulate to rewrite a whole protected group at once, which needs no reading of old contents and so avoids the penalty entirely. Otherwise you simply pay it, which is what every straightforward combination of RAID with remote storage does today.
-
Why it fell short
Batching moves the cost rather than removing it, and paying it does not scale
Accumulating writes requires non-volatile memory as a cache layer and produces write amplification in the background, so the work is deferred and reshaped rather than eliminated. Paying the cost directly is worse. Measured on a high-end 100 gigabit network card and drives writing at about 19 gigabits each, the doubled write traffic caps a remote array at 50 gigabits, which is 2.6 times a single drive for one parity level and 1.8 times for two — and crucially those ceilings do not move when drives are added, because the limit is the client's network card rather than the drives. An array that cannot scale with its drive count has lost the reason it exists, which is why RAID over remote storage was impractical rather than merely slow.
-
This paper's idea
The extra disk work is unavoidable; the extra network traffic is not
This paper (Peking University — Shu, Zhu, Ma, Huang, Mei, Liu, Jin) makes a distinction the field had not drawn. The additional reads and writes on the DRIVES are inherent to how parity works and cannot be removed. The additional traffic on the NETWORK is an artefact of routing everything through the client, and can be eliminated entirely. Each storage machine has its own network card, its own processor and its own controller, all of which sit idle while the client's single card is saturated — so the storage machines are allowed to exchange data directly with one another, computing partial checksums where the data already is and forwarding only what is needed. The client keeps just the coordination it must own. Three further techniques extract parallelism from the resulting multi-step operation, and reconstruction after a failure is spread according to who has bandwidth to spare. The result scales linearly from 4 to 18 storage machines where the best prior system is architecturally stuck at half the network card.
Why this problem is worth solving
The mechanism that causes the difficulty is old and well understood, so it is
worth stating precisely.
RAID combines several
physical drives into one virtual drive with more capacity and speed than any of
them alone. The widely used parity levels protect against a drive failing by
grouping blocks across the drives into a STRIPE and storing, alongside the data
blocks, a CHECKSUM block computed from them. If any one drive is lost its
contents can be recomputed from everything else in the stripe, which costs only
one drive’s worth of space out of the array. Reading is cheap. Writing is where
the difficulty lies. If a write covers an entire stripe, the checksum can simply
be computed from the new data and written. But if it covers only part of one —
which is the common case, since applications write small blocks — the checksum
must be updated without knowing the rest of the stripe. The standard method
exploits the fact that the checksum operation is reversible: read the OLD data
and the OLD checksum, work out what changed, and write back both the new data
and the new checksum. So a small write becomes two reads and two writes,
four operations for one. This PARTIAL WRITE PENALTY has been attacked for
decades
Moving the drives onto the network changes what that cost means.
DISAGGREGATED
STORAGE places drives in separate machines reached over the datacenter network
rather than inside the machine using them, which lets an operator scale storage
and computing independently, share drives between tenants, and replace failed
hardware from a common pool instead of keeping a spare in every server. Fast
networks have made this practical, and serverless computing depends on it. For
RAID it is doubly attractive, because spreading an array across several machines
means the array survives losing a whole machine rather than only a drive. But it
transforms the write penalty from a disk cost into a NETWORK cost, and those are
not comparable. Extra disk operations are absorbed by drives that have capacity
to spare. Extra network operations all funnel through the client machine’s
single network card, which is shared by everything it does. Measured here, a
high-end card carries about 100 gigabits while a single drive writes at about 19
— so once writes are doubled, the array tops out at roughly 2.6 drives’ worth of
throughput, and adding a ninth or a nineteenth drive changes nothing at
all
What people did before, and where it broke
- RAID inside one machine. The conventional arrangement, with the controller and all its drives in a single storage server. No network cost at all, but the whole array becomes unavailable if that server fails, and capacity must be planned in advance with a dedicated spare drive kept idle in every array.
- Batching small writes into full-stripe writes. The classic remedy for the write penalty. Accumulate writes in a battery-backed memory cache until a whole stripe can be rewritten at once, which needs no reading of old contents. Effective, but it requires that non-volatile cache and produces background write amplification, so the cost is deferred and reshaped rather than removed.
- RAID over remote drives, straightforwardly. Modern operating systems can combine software RAID with remote block devices directly, which works and is simple. But every extra operation crosses the client’s network card, so throughput is capped far below what the drives could deliver.
- Distributed RAID with distributed parity computation. Not a new idea, with research attempts dating to the early nineties, and it does spread the computation. But those designs predate the situation where the network card rather than the drives or the processors is the binding constraint.
- Systems optimizing latency instead. Recent work targets tail latency rather than throughput, and one requires modifications at the device level that commodity drives do not support.
- The gap. Everyone treated the write penalty as a single cost to be reduced or avoided, when over a network it is really two costs with different natures — unavoidable extra work on the drives, and entirely avoidable extra traffic on the network. Nobody had separated them. That is the gap this paper fills.
How the extra network traffic is removed
Let the storage machines talk to each other instead of routing through the client
-
Every extra operation crosses one network card
A partial stripe write needs two reads and two writes, and with the drives remote all four traverse the client machine's single network card. That card carries about 100 gigabits against drives writing about 19 each, so the array caps at roughly 2.6 drives' worth however many drives are added.
-
Separate the two costs new
The extra reads and writes on the drives are inherent to how parity updating works and cannot be removed. The extra traffic on the network is an artefact of routing all data through the client, and is therefore removable — a distinction the earlier work had not drawn.
-
Use the resources sitting idle new
Each storage machine has its own network card, processor and controller, all largely unused while the client saturates its single card. Giving each remote target a controller of its own, connected directly to its peers, means data can move between storage machines without passing through the client at all.
-
Keep at the client only what the client must own new
Because a stripe cannot take two concurrent writes, the client keeps a controller that admits one write per stripe and queues the rest. It also computes checksums itself for whole-stripe writes, since those need no reading of old contents and doing it locally is already optimal.
-
Overlap everything that can overlap new
A single operation now spans several machines and several devices. Checksum reduction is allowed to proceed without waiting for accompanying metadata to arrive, and the network and drive operations within one request are pipelined rather than serialized, extracting parallelism across processors, network cards and drives.
-
Spread reconstruction by who has bandwidth new
Rebuilding a failed drive requires reading every surviving block in each stripe, which can swamp any single machine. Rather than choosing where to combine them at random, a probabilistic model picks machines according to their available bandwidth, avoiding those already busy.
The idea
Two costs that look like one are not the same cost. The write penalty has been treated as a single quantity to be reduced for decades, and the standard remedies attack it as such by arranging not to trigger it. This paper’s move is to notice that once the drives are remote it is really two distinct costs with different natures. The extra reads and writes performed by the DRIVES are inherent to how checksum updating works and cannot be removed by any rearrangement. The extra traffic on the NETWORK, by contrast, exists only because the design routes all of that data through the client, and is therefore an artefact rather than a necessity. Once separated, the second becomes attackable while the first is simply accepted. Distinguishing the part of a cost that is intrinsic from the part that is an accident of your architecture is a discipline the site rewards repeatedly, and it is what turns an apparently fundamental limit into an engineering problem.
Route around the bottleneck using the resources already idle beside it. The binding constraint is one network card on the client, and everything else in the system has bandwidth to spare — each storage machine has its own card, its own processor and its own controller, largely unused while the client saturates. So the storage machines are given controllers of their own and connected directly to each other, letting data move between them without ever passing through the client. The result is that the operation the client requests still costs the drives four operations, but costs the client’s card only one. Discovering that the hardware needed to relieve a bottleneck is already present and idle, and reorganising the data path to use it, is the same exploit-what-is-already-there move covered elsewhere on this site — in the page that presses matrix units built for machine learning into service for cryptographic arithmetic, and in the page that abandons uniform access so that most traffic never touches the shared network at all.
Know exactly where you have no advantage, and say so. The evaluation is notably disciplined about its own boundaries. On ordinary reads all three systems reach the network card’s ceiling, because reads are cheap in every design and there is nothing to improve. On whole-stripe writes the new system matches the baseline exactly, because both handle that case identically and no penalty exists to remove. The improvements appear precisely where the penalty lives and nowhere else. The baseline itself was strengthened before being compared against, by adding a faster arithmetic library and a second parity mode to it. And the authors note their eight-drive test configuration is smaller than more than 80 percent of production arrays, so their measured gains understate what a typical deployment would see — a caveat pointing in the unusual direction. Reporting where a technique does nothing, and improving the competition before measuring against it, is the honest-evaluation discipline covered elsewhere on this site wherever a paper makes its claim easier to trust by narrowing it.
Writing one block into a protected stripe
From four trips across the client's network card to one
A partial stripe write, the operation that decides whether remote RAID scales. Numbers from the paper's testbed.
Step 1 of 5
What the application asked for
A program writes a small block. It lands inside a stripe spread across several remote machines, alongside other data blocks and a checksum block that protects them. Because the write covers only part of the stripe, the checksum cannot be recomputed from the new data alone.
with eight drives and 512 kilobyte chunks a stripe is about 3.5 megabytes, so writes below roughly 1.5 megabytes take this path applications overwhelmingly write blocks far smaller than a stripe, which is why this case dominates
Why it works
- Updating part of a stripe requires the old data and old checksum → with remote drives all four operations cross the client’s single network card → so the array is capped by that card at about 2.6 drives’ worth however many drives exist.
- The extra drive operations are inherent but the extra network traffic is not → letting storage machines exchange differences directly removes the second without touching the first → the drives still do four operations while the client’s card does one.
- Every storage machine has an idle network card, processor and controller → giving each one a controller and peer connections uses capacity that was already paid for → relieving the bottleneck without adding hardware.
- One operation now spans several machines and devices → pipelining its stages and allowing checksum reduction to proceed without waiting for metadata extracts parallelism a sequential protocol forfeits → which is what makes the multi-hop path fast rather than merely cheap.
- Rebuilding a failed drive reads every surviving block in each stripe → choosing where to combine them by available bandwidth rather than at random avoids swamping busy machines → improving reconstruction read bandwidth by 53 percent.
What it costs (trade-offs and scope)
- It only helps where the penalty exists. On ordinary reads all systems reach the network card’s ceiling and there is nothing to gain, and on whole-stripe writes the new design matches the baseline exactly because both handle that case the same way. The benefit is confined to partial writes and degraded reads.
- The drives can become the bottleneck instead. With only eight storage machines the limit shifts from the network to drive bandwidth, and at an even read-write mix the system becomes congested and latency rises. Removing one bottleneck exposes the next.
- It requires controllers on the storage machines. The design places software on every remote target rather than only on the client, so storage servers are no longer passive block devices — a meaningful deployment requirement where storage is operated separately.
- Evaluation is bounded by the testbed. Scaling was measured only to 18 storage machines because the testbed has 19 servers, so the linear trend is demonstrated rather than extrapolated to larger arrays.
- The comparison is against software implementations. The baselines are the operating system’s software RAID and a user-space implementation the authors strengthened themselves; hardware RAID controllers and latency-focused systems are not throughput-comparable and are excluded for stated reasons.
The evaluation
dRAID was built on user-space storage software using the standard fabric protocol over direct memory access, and evaluated on 19 machines each with a 24-core processor, two enterprise flash drives and a 100 gigabit network card. It was compared against the operating system’s software RAID and against the best available user-space implementation, which the authors first strengthened by adding a faster arithmetic library and double-parity support. Raw block-device performance was measured with a standard load generator using random access across I/O sizes, chunk sizes, stripe widths and read-write mixes, in both normal and degraded states, with a key-value store benchmark for end-to-end application results.
Building a RAID array from remote drives
| Property | RAID inside one machine | RAID over remote drives | dRAID |
|---|---|---|---|
| Survives losing a whole machine | |||
| Spare capacity | |||
| Network cost of a partial write | |||
| Scales with drive count | |||
| Reads after a drive fails |
Check yourself
Check your understanding
1. Combining RAID with remote drives caps the array at about 2.6 times a single drive. Predict why adding more drives does not raise that ceiling.
2. The paper distinguishes two costs inside the write penalty. Predict what that distinction is and why it unlocks the solution.
3. The fix relies on the storage machines doing work themselves. Predict why that capacity was available.
4. The evaluation reports that on whole-stripe writes the new system performs the same as the baseline. Predict why including that strengthens the paper.
Sources
- Disaggregated RAID Storage in Modern Datacenters (ACM DL, DOI 10.1145/3582016.3582027) — ASPLOS 2023.
- Author PDF — the full 17-page paper, read in full for this explainer.
- dRAID implementation and artifacts (GitHub) — the released system, testbed setup and experiment scripts.