Thesios: Synthesizing Accurate Counterfactual I/O Traces from I/O Samples

Phitchaya Mangpo Phothilimthana, Saurabh Kadekodi, Soroush Ghodrati, Selene Moon and Martin Maas

To decide whether a new disk, cache size, or data-placement policy is worth deploying across a fleet of hundreds of thousands of disks, engineers need to know how it would behave — but the honest way to find out is to actually deploy it and risk production, and even a full recording of today's disks cannot show an untried change; this work from Google builds Thesios, which stitches together the sparse I/O samples already collected across many similar disks into a full-resolution trace of one representative disk, and reshapes it to answer what-if questions, matching real disk behavior within a few percent so teams can test a change on paper first.

The argument in one glance

Problem

Designing storage needs realistic I/O traces, especially for untried changes

To design and tune a data center's storage system, engineers rely on I/O traces — detailed records of the read and write requests disks actually serve. Two kinds matter. Representative traces capture the full I/O experience of a real disk, so you can evaluate today's hardware and policies. Counterfactual traces answer what-if questions — how would a bigger disk, a new cache size, or a different data-placement policy behave before you deploy it? The trouble is that both are hard to get: full tracing is ruinously expensive, and no recording of today's disks can show what a change that has not happened yet would do.
What people did

To answer a what-if, deploy the change and measure, or trace fully

The usual way to evaluate a hypothetical change is to actually roll it out to some fraction of production and watch its effects over time — an A/B experiment. To get representative traces of current behavior, the instinct is to record every operation on the disks, or on a chosen subset, at full resolution. Or, lacking that, to reconstruct behavior by upsampling the sparse samples already kept, or to synthesize a trace from aggregate statistics like average request rates.
Why it fell short

Every option is too costly, too risky, or too lossy

Deploying a change in production is slow, expensive, and risky — it means real code changes affecting live workloads. Full tracing of a fleet of hundreds of thousands of disks would produce petabytes a day and slow foreground work; tracing a subset even creates a feedback loop, since a load balancer sees the traced disks respond slower and routes traffic away, making them unrepresentative. Upsampling sparse traces is lossy and misses bursts, and building a trace from aggregate statistics needs so many modeling assumptions — interarrival times, queuing, caching, locality — that it rarely holds up.
This paper's idea

Stitch the samples you already have into a full disk, then reshape it

Thesios starts from the sparse I/O samples that data centers already collect cheaply, and stitches together the samples from many similar disks into a full-resolution trace of one representative disk — like the Ship of Theseus, a whole rebuilt from many parts. It groups disks so the combined ones truly resemble each other, re-simulates the queuing and ordering a real disk would see, and, to answer a what-if, simply changes which samples it combines — synthesizing the trace of a disk that does not yet exist, without deploying anything.

Why this problem is worth solving

The heart of the matter is that the trace you most want is the one no measurement can give you, so it is worth being precise about what is missing.

An I/O trace records the sequence of read and write requests a storage device serves, with details like each request’s arrival time, size, priority, and latency. A representative trace captures the full experience of a real disk — enough to reproduce its request rate, its utilization, its bursts over hours and days, its access locality, and its tail latency. A counterfactual trace goes further: it captures what a disk would experience under a change that has not been deployed, such as a larger capacity or a policy that places files differently. The obstacle is cost and possibility. Recording every operation on every disk at full resolution is prohibitive — a fleet of roughly 450,000 disks would generate several petabytes of trace data a day — and it slows the very workloads being measured. So operators keep only down-sampled traces, a small fraction of operations, which are too thin to characterize a single disk. Tracing just a subset of disks does not help either, because the extra overhead makes those disks respond slightly slower, so a load balancer routes requests away from them, and they stop being representative — a feedback loop that corrupts the measurement. And crucially, even a perfect full trace of today cannot answer a what-if, because it can only record what did happen, not what would happen under an undeployed change. That gap is exactly what makes synthesizing traces, rather than capturing them, worthwhile.

The reason the alternatives fall short is that each tries to manufacture a full trace the wrong way, ignoring a resource that is already there.

Consider the options for getting a realistic trace without full capture. Upsampling — taking the sparse samples and scaling them up to a full-resolution trace — is fundamentally lossy: a single sampled disk may never have seen a transient burst, so no amount of scaling recreates it, and pushing the sampling rate higher to compensate reintroduces the interference problem. Synthesizing a trace from aggregate statistics, like mean request rates and latencies, requires modeling interarrival times, queuing behavior, caching, system overhead, and spatial and temporal locality all at once, each with its own assumptions and parameters to tune, and small errors compound. And the fallback of simply deploying the change to measure it — an A/B experiment in production — is exactly what one wants to avoid: it demands real code changes, exposes live users to a possibly-bad idea, and takes time to show effects. What all of these miss is that large storage systems already run sampling telemetry, cheaply recording a fraction of requests with rich per-request detail, across every disk in the fleet. That existing data holds, in aggregate, the full behavior of a representative disk — if you knew how to reassemble it. Recognizing and exploiting that already-collected resource, rather than capturing more or modeling from scratch, is what makes the approach both cheap and accurate.

What people did before, and where it broke

Deploy the change in production and measure it (an A/B experiment). Roll the hypothetical policy or hardware out to some disks and watch the effect. Where it broke. It is slow, expensive, and risky — real code changes affecting live workloads — and a bad change can violate service guarantees before you learn it was bad.
Capture full-resolution traces of every disk. Record every operation to get a perfect picture. Where it broke. A fleet of hundreds of thousands of disks would produce several petabytes of trace a day, a prohibitive footprint, and the tracing itself slows the foreground workloads being studied.
Fully trace only a representative subset of disks. Trace fewer disks to cut the cost. Where it broke. Which subset is representative is unclear, and the tracing overhead makes those disks respond slower, so a load balancer routes traffic away from them, and they stop being representative — a self-defeating feedback loop.
Upsample the sparse samples, or model from aggregate statistics. Scale the kept samples up to full resolution, or synthesize from averages. Where it broke. Upsampling is lossy and cannot recreate bursts a single disk never saw, and aggregate models need so many interlocking assumptions — interarrival, queuing, caching, locality — that they rarely match reality.
The gap. Every approach either paid an unacceptable cost, corrupted itself through interference, lost detail, or could not represent an undeployed change at all. What was missing was a way to reassemble the full, representative, and even counterfactual behavior of one disk from the cheap samples already collected across many — which is exactly what data-driven storage design needs.

How Thesios works

Rebuilding one full disk trace from many disks' samples

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Answer the what-if with a synthesized surrogate instead of a risky production experiment. The honest way to learn whether a new disk or policy is worth deploying is to deploy it and watch, but that exposes live users to a possibly-bad change and takes time to reveal its effects. Thesios instead builds a trace of the hypothetical disk from data already in hand and evaluates it offline, so the decision is made on paper before anything reaches production. Substituting a cheap, faithful stand-in for an expensive and risky real run, and trusting it because it preserves the behavior the decision actually depends on, is the same move the noise-resilient quantum-optimization work covered elsewhere on this site makes, running the costly search on a reduced surrogate that shares the essential landscape rather than paying full price on the real system.

Reuse the telemetry the system already produces, for free. The expensive path is to capture more data; Thesios’s insight is that large storage systems already sample a fraction of every disk’s requests with rich detail, and that this cheap, already-collected data holds, in aggregate, the full behavior of a representative disk — if reassembled correctly. It adds no new tracing overhead and no interference, because it reads what the fleet is already recording. Extracting the signal a task needs from information the system already computes about itself, rather than paying to measure anew, is exactly the discipline the adaptive-SLAM-runtime work covered elsewhere on this site follows, reading the algorithm’s own already-computed state instead of collecting an expensive external metric.

Keep what the samples tell you, and recompute only what the synthesis changed. Each sample carries a true disk service time that Thesios preserves unchanged, because that is a physical property of the request. But the order the requests arrive at the disk, and thus their queuing and latency, is specific to the synthesized sequence and must be recomputed — so Thesios re-simulates exactly that, the priority reordering and queuing, and nothing more. Recovering everything the data still lets you keep and re-deriving only the part the change actually disturbs is the same discipline the dynamic-neural-network compiler covered elsewhere on this site applies, holding fixed all it can determine statically and recomputing at runtime only what genuinely depends on the new input.

Following a disk-capacity question through Thesios

Should we deploy larger disks?

Tracing a capacity what-if shows how Thesios evaluates hardware that is not deployed.

Step 1 of 5

Operators want to know how an 18-terabyte disk would behave

Today the fleet runs 12-terabyte disks, and the team is considering 18-terabyte ones, which have about the same bandwidth but hold more data. Deploying them just to see the I/O behavior would be costly and slow, and no trace of the current 12-terabyte disks directly shows it.

The worry is whether the bigger disk would be over-utilized and violate latency guarantees.

Why it works

Stitching samples from many similar disks rebuilds one full disk for free. Full tracing costs petabytes a day and interferes with workloads → but the fleet already samples a fraction of every disk’s requests, and combining those from n similar disks reconstructs the full I/O of one representative disk → so a full-resolution trace comes at no new overhead and no interference.
Grouping by disk and trace characteristics makes the synthesis represent a real disk. Combining arbitrary disks would produce a trace representative of none, since even same-model disks behave very differently → but grouping on capacity, fullness, and hot-cold ratio and clustering on cache rate and throughput keeps the combined disks alike → so the synthesized trace matches a real category, cutting within-category load variance about fivefold.
Re-simulating the order makes the disk-level latency accurate. The stitched sequence is a server-level order, not what any disk actually saw, so its raw latencies are wrong → but a simulator recomputes queuing and priority reordering for the specific synthesized sequence → so disk-level arrival times and read latencies become accurate, matching reality for most requests within a few percent.
Changing which samples are combined synthesizes a disk that does not exist. No trace of today can show an undeployed change → but selecting and re-weighting the samples — more cold data for a bigger disk, filtered files for a new placement policy — directly builds the counterfactual disk → so a bigger, fuller, or differently-filled disk is evaluated without deploying anything.
Replaying on a real-server emulator captures what a model cannot. Some effects, like a larger buffer cache or a prefetcher, resist analytical modeling → but replaying the synthesized trace on an emulator running actual storage-server software and a real disk reproduces them → so Thesios predicted a cache-size doubling’s hit-rate gain to within a tenth of a percentage point of the real change.

What it costs (trade-offs and scope)

Tail latency is imprecise. The simulator’s simple priority-reordering model reproduces most request latencies within a few percent but captures the tail latency of latency-sensitive requests only to about 80 to 84 percent accuracy, so Thesios is not suited to questions that hinge on high-accuracy tail latency without further adjustment.
It cannot capture feedback loops. If a proposed change lowers latency and users respond by sending more requests, Thesios will not see it, because it does not model how users react to the change it is evaluating.
Fine-grained burstiness may be off. It compensates for dropped samples in aggregate, so the overall request count is accurate but the count within a small time window may not be — a limitation the authors note is fixable by re-weighting per interval, not fundamental.
It needs rich existing telemetry and correct labels. The method assumes a sampling service that records per-file samples with detailed fields and a system that correctly labels data hot or cold; where that infrastructure or labeling is absent, the approach does not directly apply.
The grouping and modeling take tuning, and writes are only implicit. Which characteristics to group and cluster on is storage-cluster-dependent and must be chosen, and the simulator does not model writes explicitly — only their effect on read latency — so disk-level traces omit writes.
It gives data-driven confidence, not a guarantee, and is validated on one setting. Thesios improves on today’s high-level, assumption-heavy decisions but is not exact at the 99.9th percentile, and its accuracy is demonstrated on Google’s specific clusters and workloads, so results may differ elsewhere; emulator studies also require careful setup to avoid skew.

The evaluation

Three ways to evaluate a hypothetical storage change

Property	Deploy and measure (A/B)	Full-resolution tracing
What it costs
Answers an undeployed what-if
Representative of a real disk	Yes, it is real	Yes
Latency accuracy	Exact	Exact
Risk to production	High; live users affected	Moderate; slows workloads

Three ways to evaluate a hypothetical storage change, with Thesios validated against real Google production data. Its synthesized traces match actual disks to 95 to 99.5 percent on read/write request counts, 90 to 97 percent on utilization, and up to 99.8 percent on read latency, capture diurnal and weekly fluctuation with over 95 percent confidence, and across four case studies — hypothetical disk capacity and utilization, cold-data placement, low-RPM disks, and buffer-cache size — predicted real outcomes closely, including a cache-doubling hit-rate gain of 3.4 points against an actual 3.5 — because it reassembles cheap existing samples into a full, reshapeable trace instead of tracing fully or deploying for real.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

Thesios is an elegant, pragmatic idea from a production setting: the trace you most need — of a disk under a change that has not happened — cannot be measured, so synthesize it from data you already have. It stitches the cheap, already-collected I/O samples of many similar disks into a full-resolution trace of one representative disk, groups disks carefully so the result represents a real category, re-simulates the queuing and priority reordering a real disk would see, and answers a what-if by simply changing which samples it combines. Validated against Google production, it matches real disks within a few percent on request counts, utilization, and read latency, and four case studies predicted real outcomes closely — most strikingly, a buffer-cache doubling’s hit-rate gain of 3.4 points against an actual 3.5. Read it with its scope. The simulator’s tail latency is only 80-to-84-percent accurate, so tail-sensitive questions need care; it cannot model feedback loops where users react to the change, or fine-grained burstiness; it depends on rich existing sampling telemetry and correct hot-cold labels; writes are captured only through their effect on reads; and its accuracy is shown on one company’s clusters. The durable lesson is that when the measurement you want is impossible or too risky to take, the answer is often to reassemble it from the cheap data you are already collecting — turning telemetry you kept for other reasons into a safe way to test ideas before they touch production.

Background

Why large systems increasingly decide by synthesis, not experiment

It helps to see Thesios as part of a shift in how the largest systems are tuned: away from learning by risky live experiment, toward learning from data already flowing through the system. At the scale of hundreds of thousands of disks, every deployment decision — a new drive, a cache size, a placement policy — carries real cost and real risk, and the old default of trying it in production and watching is a luxury that gets more dangerous as the fleet and the blast radius grow. Yet these systems are also drenched in telemetry, sampled continuously for monitoring and billing, and that exhaust turns out to be rich enough, if reassembled cleverly, to answer questions it was never collected to answer. The deep-learning-on-GPUs retrospective covered elsewhere on this site makes the general version of this point for hardware, arguing that sound design decisions come from measuring what the system actually does rather than from assumptions, so that engineering follows evidence. Thesios extends that evidence-driven stance to a counterfactual: not just what the system does, but what it would do. The broad lesson is that as systems grow too large and too critical to experiment on freely, the decisive capability becomes turning the data they already emit into faithful models of hypotheticals — replacing the risky rollout with a synthesis that can be checked against reality after the fact.

If you remember only three things

1. Designing large storage systems needs realistic I/O traces, and the most valuable ones are counterfactual — how a bigger disk, a new cache size, or a different placement policy would behave — which no measurement can supply, since even a full trace of today records only what happened. Full tracing of a fleet of hundreds of thousands of disks would cost petabytes a day and slow workloads (and tracing a subset creates a load-balancer feedback loop that corrupts representativeness), upsampling sparse traces is lossy, aggregate-statistic models need too many assumptions, and the fallback of deploying a change to measure it is slow, expensive, and risky.
2. Thesios reassembles the sparse I/O samples that data centers already collect cheaply: for a one-in-n sampling rate, it stitches the file-samples from n similar disks into a full-resolution trace of one representative disk (grouping by capacity, fullness, and hot-cold ratio, then clustering on cache rate, read/write ratio, and throughput to cut within-category load variance about fivefold, and re-weighting for samples dropped during bursts). A server simulator then re-simulates priority reordering and queuing to convert the server-level trace into an accurate disk-level trace, preserving each sample's true disk service time while recomputing only the order-dependent latency. To answer a what-if, it changes which samples it combines — more cold data for a larger disk, filtered files for a new placement — and evaluates on the fast simulator or a real-server emulator.
3. Validated against Google production, Thesios matches real disks to 95 to 99.5 percent on read/write request counts, 90 to 97 percent on utilization, and 80 to 99.8 percent on read latency, and captures diurnal and weekly fluctuation with over 95 percent confidence; four case studies (disk capacity and utilization, cold-data segregation, low-RPM disks, and buffer-cache size) predicted real outcomes closely, including a cache-doubling hit-rate gain of 3.4 points versus an actual 3.5. Its limits: the simulator's tail latency is only 80 to 84 percent accurate, it cannot model feedback loops or fine-grained burstiness, it requires rich existing sampling telemetry and correct hot-cold labels, writes are captured only via their effect on read latency, and its accuracy is demonstrated on Google's specific clusters.

Check yourself

Check your understanding

1. Why can no measurement, even a perfect full trace of every disk, answer a counterfactual what-if question?
2. What is the core idea behind how Thesios builds a full-resolution trace?
3. Why does Thesios re-simulate the requests with a server simulator instead of using the sampled latencies directly?
4. Which statement best reflects the limits of Thesios?

Sources

The paper is “Thesios: Synthesizing Accurate Counterfactual I/O Traces from I/O Samples” by Phitchaya Mangpo Phothilimthana, Saurabh Kadekodi, Soroush Ghodrati, Selene Moon, and Martin Maas, published at ASPLOS 2024, DOI 10.1145/3620666.3651337. The authors are at Google, Google DeepMind, and the University of California, San Diego, and the synthesized traces are released open-source.

This explainer is written from the full 17-page text. Everything above comes from the paper’s own sections: the introduction and preliminaries motivating representative and counterfactual I/O traces and arguing the infeasibility of full tracing (petabytes a day for a 450,000-disk fleet), subset tracing (the load-balancer feedback loop), upsampling, and aggregate-statistic modeling; the Thesios methodology of combining down-sampled traces from similar entities, and its concrete instantiation for storage servers and disks using Google’s existing RequestSampler telemetry, file-samples over op-samples for locality, disk and trace characteristics with K-means clustering to identify similar disks (reducing within-category load variance over fivefold), the trace synthesizer combining n-times-w disks with re-weighting for dropped samples, and the trace reorganizer server simulator modeling latency as disk time plus the maximum of queuing and system overhead with per-priority queues and a Gamma-distributed system-overhead fit; the counterfactual synthesis by adjusting hot and cold sample multipliers for capacity, fullness, and utilization and by filtering for placement policies, evaluated via a lightweight simulator or a full server emulator; the validation showing 95 to 99.5 percent accuracy on request counts, 90 to 97 percent on utilization, 80 to 99.8 percent on read latency, and cache-hit rates within five points; and the four case studies on hypothetical disk characteristics, cold-data segregation and disk power-down, low-RPM disks (validated within one percent average disk-time against a real low-RPM disk), and a buffer-cache doubling predicted at 3.4 points versus an actual 3.5, along with the stated limitations on feedback loops, burstiness, and tail latency.