When thousands of containers share one datacenter SSD, one greedy neighbor can starve the rest, so you need to give each a fair, proportional slice of the device while still letting idle capacity flow to whoever needs it — but the obvious way to measure a container's device usage, counting bytes or IOPS, is badly wrong, because a 4 KB random write can hammer an SSD far harder than a 4 KB sequential read; IOCost (Meta) fixes this by profiling each SSD offline to build a COST MODEL that estimates an IO's true device occupancy, schedules containers by that cost in virtual time, splits the work into a tiny per-IO fast path and a periodic slow planning path so it adds no measurable overhead even at 750,000 IOPS, and corrects the model's imperfection with a feedback loop that watches device latency — cutting SLO violations from dozens to two in a stress test, and now running across all of Meta's millions of machines and upstreamed into Linux.
The argument in one glance
-
Problem
Many containers, one disk, and one greedy neighbor can starve the rest
A datacenter machine runs many containers side by side, and they share a single block-storage device (an SSD). Without isolation, one container hammering the disk — a batch job, a memory leak swapping to storage — starves the latency-sensitive services next to it, blowing their tail latency. So the operating system needs IO CONTROL: give each container a fair, PROPORTIONAL share of the device according to its priority. But it must also be WORK-CONSERVING — if some containers are idle, their unused capacity should flow to whoever can use it, so the expensive device is never left idle while work waits. Fair sharing plus full utilization, on a resource shared by thousands of containers, is the requirement.
-
What people did
Count the IO — cap bytes/IOPS, or fair-queue by sectors
The established tools measure IO and divide it up. blk-throttle sets fixed per-container limits on IOPS or bytes per second. BFQ does proportional fair queueing, scheduling requests in proportion to each container's weight, counted in sectors. IOLatency enforces per-container latency targets. Each takes the intuitive route: quantify how much IO a container is doing — in bytes, IOPS, or sectors — and hand out shares accordingly, the way a CPU scheduler hands out time slices.
-
Why it fell short
Bytes and IOPS are not device occupancy, and the overhead is fatal
The intuitive metrics are the flaw. Bytes and IOPS are poor measures of how much a container actually OCCUPIES a device, because SSD cost is wildly nonlinear: a 4 KB random write can slow the device far more than a 4 KB sequential read, thanks to internal buffering, deferred writes, and garbage collection — so two containers moving equal bytes can impose very unequal load. Fair-by-bytes is therefore unfair in reality. On top of this, the fleet spans dozens of heterogeneous SSD models with different profiles, so fixed limits are brittle and un-tunable; blk-throttle is not work-conserving (idle capacity cannot be shared); and BFQ's per-IO fair-queueing overhead is fatal on modern SSDs — it delivers only about 20% of baseline throughput (an 80% loss) because the device does millions of IOPS and every one pays its scheduling cost. No prior mechanism is at once low-overhead, work-conserving, proportional, fair, and aware of memory-management IO.
-
This paper's idea
Profile the device to learn true IO cost, and schedule by it cheaply
IOCost (Meta, with CMU — Heo, Schatzberg, Tang, Skarlatos, and colleagues) starts from the observation that IO control lacks what CPU scheduling has: a measure of OCCUPANCY. CPU scheduling divides TIME; IO had no comparable unit. So IOCost builds one, by profiling each SSD offline to fit a COST MODEL that estimates how much any IO truly occupies the device from its features (read/write, random/sequential, size) — not its bytes. It then schedules containers in virtual time by that cost, weighted by priority, and keeps per-IO overhead near zero by splitting control into a tiny FAST issue path (per IO) and a periodic SLOW planning path (the heavy reasoning). A feedback loop watches device latency and adjusts a global rate to correct the model's imperfections, and a budget-donation algorithm makes it work-conserving. The result achieves all five properties at once, with no measurable overhead at 750,000 IOPS — deployed across all of Meta's millions of machines and upstreamed into Linux.
Why this problem is worth solving
Storage isolation is the difference between a datacenter that
packs many workloads onto each machine efficiently and one where a
single bad tenant wrecks its neighbors — so getting IO control
right directly affects how densely and safely a fleet can be run.
The reason it is genuinely hard, and not just a matter of dividing
a number, is that unlike CPU there is no natural unit to divide.
The whole problem reduces to
a measure of how
much of a storage device a stream of IO actually consumes — the
storage analogue of CPU time. CPU scheduling is tractable because
it has such a metric: a task’s share is simply the fraction of CPU
TIME it uses, which is directly observable and additive. IO control
has no equivalent. The obvious candidates — bytes transferred or
IO operations per second (IOPS) — are not occupancy, because a
device does not spend equal effort per byte or per operation. A
random write forces an SSD into buffering, read-modify-write, and
garbage collection; a sequential read streams almost for free. So
two containers issuing the same bytes or IOPS can occupy the device
by wildly different amounts, and any scheduler that divides shares
by bytes or IOPS is dividing by the wrong quantity
Two further realities turn a hard problem into a brutal one. The
fleet is heterogeneous — Meta runs dozens of SSD models whose
performance profiles differ enormously — so any fixed configuration
is wrong on most devices, and the metric has to be learned
per-device rather than assumed. And modern SSDs are so fast that
the control mechanism itself becomes the bottleneck, which is the
trap of
the cost of running
the IO-control logic once for every single IO request. On a slow
disk doing hundreds of IOs per second, an elaborate per-request
scheduler is negligible. But a modern SSD can serve MILLIONS of IOPS,
so any work done per IO is multiplied millions of times per second
and can consume a large fraction of the machine. Sophisticated
fair-queueing schedulers like BFQ pay a heavy per-request cost —
sorting, bookkeeping, latency tracking — and on a fast SSD this
collapses throughput to about 20% of what the device can do, an 80%
loss. So an IO controller cannot afford to be clever on the hot
path; whatever intelligence it has must run somewhere cheaper
What people did before, and where it broke
- blk-throttle (fixed IOPS/bytes limits). Cap each container at a configured rate. Simple and cheap, but NOT work-conserving — idle capacity cannot be lent out — and configuring per-device limits across a heterogeneous fleet is brittle and, at scale, intractable.
- BFQ (proportional fair queueing by sectors). Schedule requests in proportion to weight, which is work-conserving and fair — but its per-IO overhead is fatal on fast SSDs (about 20% of baseline throughput), it counts sectors (a poor occupancy proxy), it ignores memory-management IO, and it produces wide latency swings.
- IOLatency (per-container latency targets). Enforce a latency target per container. Good for strict prioritization, but it cannot fairly SHARE among equal-priority containers and needs fragile per-application tuning to balance isolation against utilization.
- Simple IO schedulers (kyber, mq-deadline). Cheap and low-overhead, but they provide no proportional control, no work-conserving fairness, and no memory awareness — they reorder IO, they do not isolate tenants.
- The gap. No mechanism was simultaneously low-overhead, work-conserving, proportional, fair among equals, AND aware of memory-management IO — because none had a cheap, accurate measure of device occupancy to schedule by. IOCost builds that metric and achieves all five at once.
How IOCost works
Learn the cost metric offline, schedule by it cheaply, and correct it with feedback
-
Profile each SSD to learn true IO cost new
Offline, IOCost runs fio saturation benchmarks on a device and fits a linear COST MODEL: an IO's cost is a base cost (chosen by read/write and random/sequential) plus a per-byte rate (by read/write) times its size. This gives an occupancy estimate per IO — for example, a random read costing about 109 microseconds — practical to build even across the roughly 30 device types in Meta's fleet.
-
Charge each IO its cost in virtual time new
Each container (cgroup) has a virtual time advanced, per IO, by the IO's cost divided by the container's hierarchical weight. A global virtual time advances at a rate. An IO is issued immediately if the container's virtual time is within budget, and throttled otherwise until budget accumulates — giving weighted-proportional, work-conserving sharing measured in real device cost.
-
Split control into a fast path and a slow path new
The per-IO ISSUE path is deliberately trivial (microsecond scale): compute the cost, divide by the cached weight, check the budget, issue or throttle. All the sophistication lives in a periodic PLANNING path (millisecond scale): monitor the device, adjust rates, run budget donation. Keeping the hot path this cheap is why IOCost adds no measurable overhead at 750,000 IOPS.
-
Correct the model with a QoS feedback loop new
The linear model is imperfect (it cannot capture caching, reordering, garbage collection), so IOCost closes the loop: the planning path watches device saturation signals (completion latency, in-flight depth) and adjusts the global virtual-time rate — speeding it up when the device is under-used, slowing it when saturated. This corrects up to a 2x model error automatically, so the model need only be approximately right.
-
Reclaim idle capacity via budget donation new
To stay work-conserving, the planning path detects containers using less than their share and lets them DONATE unused budget to busy siblings — reducing donors' weights along the path to the root, while recipients lazily recompute their effective weight on the issue path with no global synchronization. Donors can rescind at any time, and non-donating containers are provably untouched.
-
Charge cost to the true owner, and use debt to avoid inversion new
IO caused by memory management (swapping out a container's pages) is charged to that container, its owner — not to whoever triggered the allocation. If the owner is over budget, blocking its swap would stall an innocent allocator (priority inversion), so IOCost lets the owner incur DEBT (issue now) and repays it by throttling the owner's later IOs, or by briefly blocking it at the next userspace return.
The idea
Invent the occupancy metric the resource never had. The root difficulty is that IO control had no equivalent of CPU time — no directly measurable, additive unit of how much of the device a container consumes — and everyone reached for bytes or IOPS, which are not it, because a device spends unequal effort per byte. IOCost manufactures the missing metric by PROFILING: run saturation benchmarks offline, fit a cost model, and thereafter estimate each IO’s true device occupancy from its features. This is the measure-the-true-cost discipline the site returns to more than any other — the rendering page counting tile-intersections instead of points, the serving page optimizing throughput instead of FLOPs, the FHE page counting rotations instead of SIMD width, the DSA page weighing cache pollution instead of cycles — pushed to its sharpest form: the true cost was not merely mismeasured but UNMEASURED, so IOCost had to construct the yardstick before it could schedule by it. Bytes lie about device load exactly the way FLOPs lie about serving throughput, and the fix is the same — profile reality and account by it.
Put the cheap work on the hot path and the smart work on a slow path. A modern SSD does millions of IOPS, so anything done per IO is multiplied millions of times a second — which is why BFQ’s clever per-request scheduling collapses throughput. IOCost’s structural answer is to split control in two: the per-IO issue path does only a multiply, a divide, and a comparison, while a periodic planning path (running every few milliseconds, not every IO) carries all the expensive reasoning — feedback control, budget donation, saturation monitoring. The hot path stays nearly free; the intelligence lives where it is affordable. This is the keep-the-hot-path-dumb, think-on-the-slow-path discipline the site sees in the persistent-memory database keeping its shared node doing dumb copies while the compute nodes think, and in the off-critical-path amortization the scheduling pages rely on — separate the frequent cheap operation from the occasional expensive decision.
A rough model plus a feedback loop beats a perfect model open-loop. IOCost’s cost model is a simple linear fit that cannot capture an SSD’s caching, reordering, or garbage collection — and it does not need to, because a feedback loop corrects it. The planning path watches the real signal (device latency and saturation) and adjusts the global rate up or down, absorbing up to a 2x error in the model automatically: halve the model’s parameters and the rate climbs to compensate; double them and it drops. So the engineering effort goes not into a perfect model but into a controller that closes the loop on reality. This is the predict-then-correct discipline the site sees when a scheduler forecasts a resource it cannot measure exactly and provisions against the forecast — a cheap approximation made reliable by measuring the outcome and adjusting, rather than an expensive model trusted open-loop.
Why counting bytes fails, and how IOCost accounts correctly
From a fair-by-bytes fiction to a cost-based schedule
Sharing one SSD among containers. Numbers from IOCost's evaluation and deployment.
Step 1 of 5
The shared SSD and the noisy neighbor
Several containers share one SSD. A latency-sensitive service needs its IO to complete fast; a batch job or a memory-leaking process next to it pounds the device. Without isolation the noisy neighbor saturates the SSD and the service's tail latency explodes. The OS must divide the device fairly by priority while never leaving it idle if someone can use it.
IO control is the storage analogue of CPU scheduling, but without a natural unit to divide work-conservation matters because the device is expensive and idling it wastes the fleet
Why it works
- Bytes and IOPS are not device occupancy because SSD cost is nonlinear → IOCost profiles each device offline to estimate an IO’s true cost → it schedules by the real quantity, so fairness is real rather than a byte-counting fiction.
- Modern SSDs do millions of IOPS, so per-IO work is multiplied enormously → IOCost puts only a multiply-divide-compare on the issue path and all reasoning on a periodic planning path → it adds no measurable overhead at 750,000 IOPS where BFQ loses 80%.
- The cost model cannot capture caching and garbage collection → a feedback loop watches device latency and adjusts the global rate → up to 2x model error is corrected automatically, so the model need only be approximately right.
- Idle capacity would otherwise be wasted → a planning-path donation algorithm lends unused budget to busy containers without touching the fast path → the device stays fully utilized (work-conserving) and non-donating containers are provably unaffected.
- Memory-management (swap) IO belongs to the container that owns the memory → IOCost charges it there and lets an over-budget owner incur debt → an innocent allocator is never blocked by another container’s swap, avoiding priority inversion.
What it costs
- The model is approximate, and needs the feedback to survive. The linear cost model does not capture caching, reordering, or garbage collection; the QoS loop corrects up to about a 2x error but requires active monitoring, and a device whose behavior swings more than that stresses the design.
- Consistent devices are preferred over fast-but-bursty ones. Meta found SSDs with steady behavior work better than ones with high but unpredictable peaks, which must be throttled aggressively — harming utilization — so IOCost subtly shapes which hardware is worth buying.
- Configuration is systematic but non-trivial. Each device type needs offline profiling and QoS tuning (via a benchmark suite), and the effort scales with the roughly 30 device types in the fleet; there is no turnkey way to set the parameters in a brand-new environment.
- Donation favors deep hierarchies. The budget-donation algorithm is efficient for deep cgroup trees; very wide hierarchies with many simultaneous donors need more planning-path iterations to settle.
- Model training is manual. Cost models are built offline with fio; there is little online model refinement, so a device or firmware the profiling did not anticipate may be modeled poorly until re-profiled.
The evaluation
Microbenchmarks and real workloads across Meta’s heterogeneous SSD fleet and public clouds (AWS EBS gp3/io2, Google Cloud Persistent Disk), against blk-throttle, BFQ, IOLatency, kyber, and mq-deadline — measuring overhead, proportionality, work-conservation, behavior under memory pressure, and stacked-workload SLO violations — plus two years of production deployment on millions of machines.
IO control mechanisms for a shared SSD
| Property | blk-throttle (fixed limits) | BFQ (fair queueing) | IOCost |
|---|---|---|---|
| Overhead on a fast SSD | |||
| Work-conserving | |||
| Occupancy metric | |||
| Memory-management aware | |||
| ZooKeeper SLO violations (6h) |
Check yourself
Check your understanding
1. Prior IO controllers divide a container's share by bytes, IOPS, or sectors, the way a CPU scheduler divides time. Predict why this seemingly natural choice is the root flaw.
2. IOCost splits control into a trivial per-IO "issue path" and a periodic "planning path" that carries all the sophistication. Predict why this split is what lets it work on million-IOPS SSDs.
3. IOCost's cost model is a simple linear fit that cannot capture SSD caching, reordering, or garbage collection — yet it works well in production. Predict how it copes with an admittedly imperfect model.
4. When a memory-leaking container swaps its own pages to disk, that swap IO must be charged to the leaking container — but if it is over budget, blocking the swap would stall a different, innocent container that was allocating memory. Predict how IOCost resolves this.
Sources
- ACM DL: IOCost: Block IO Control for Containers in Datacenters — ASPLOS 2022 (full text read via the author-hosted PDF; no arXiv preprint).
- Linux kernel iocost documentation — IOCost is upstreamed as the kernel’s io.cost controller, with device-profiling tools open-sourced.