Proactive Runtime Detection of Aging-Related Silent Data Corruptions: A Bottom-Up Approach

Jiacheng Ma, Majd Ganaiem, Madeline Burbage, Theo Gregersen, Rachel McAmis, Freddy Gabbay and 1 other

Chips can quietly start miscomputing as their transistors age, producing silent data corruptions that break the old assumption that hardware either works or fails loudly — and the usual detection tests treat the chip as a black box, so they are huge and run only every few months; this work from Michigan, Washington, and partners goes bottom-up, using the chip's own gate-level layout and validated aging models to pinpoint the exact signal paths that will age into timing failures, then formal verification to synthesize tiny targeted tests that activate just those paths, small enough to run inside a live application every second at 0.8 percent overhead.

The argument in one glance

Problem

Aging chips silently miscompute, breaking the fail-stop assumption

Software has always assumed hardware either works correctly or fails loudly. But transistors age: when one sits idle in a fixed state for a long time, a physical effect called bias temperature instability gradually raises its switching delay, until a signal arrives too late or changes too early and a flip-flop captures the wrong bit. The chip keeps running and silently produces wrong answers — a silent data corruption with no crash, no log, no exception. Meta, Google, and Alibaba have all found such CPUs in their fleets, and most of the corruptions appear only after the chip has been in use for a while, pointing straight at aging.
What people did

Test the chip often with generated test cases

The obvious defense is to run test cases that stress the hardware and flag it when it misbehaves. A body of work generates such tests and schedules them across a fleet. Google's SiliFuzz fuzzes the instruction set to produce test programs; Intel's DCDiag and its open version build tests on top of common libraries. Run them, compare the outputs, and quarantine any chip that fails.
Why it fell short

Black-box tests are huge, so they run far too rarely

These generators treat the chip as a black box, building tests atop an abstract model of it. Lacking the real implementation, they must throw enormous, complex suites at the hardware to be sure every component is stressed — SiliFuzz makes around 500,000 tests, and a full run of Intel's diagnostic takes 45 minutes. Suites that big cannot run often; one cloud operator schedules them just once every three months. But aging is continuous and can surface at any time, so testing that infrequent catches it far too late.
This paper's idea

Go bottom-up from the gate layout to tiny targeted tests

The chip is not actually a black box: its gate-level netlist is known, and transistor aging is well modeled at exactly that level. Vega uses the netlist plus validated aging models to pinpoint the specific signal paths that will age into timing failures, then formal verification to synthesize a short instruction sequence that provokes each one. Because the tests target only the vulnerable paths, the whole suite is a few hundred to a couple thousand cycles — small enough to embed inside a live application and run every second, at under one percent overhead.

Why this problem is worth solving

The failure mode at the heart of this paper is dangerous precisely because it is quiet.

A silent data corruption is a hardware failure that produces a wrong result without any sign that something went wrong — no crash, no error log, no exception, just a bad value handed to the software as if it were correct. That silence is what makes it dangerous. A conventional hardware fault is fail-stop: the component either works or visibly dies, and software has been built for decades on that assumption. A silent corruption violates it. The wrong value flows downstream, contaminating later computation, so a fault in one instruction can surface as a mysterious failure far away in space and time, nearly impossible to trace back. These are not hypothetical: Meta, Google, and Alibaba have all reported finding CPUs in their data centers that consistently miscompute certain operations, and the corruption comes from the processor itself rather than from memory, where error-correcting codes would have caught it.

One of the leading causes of these corruptions is a slow, physical process that a factory test cannot see.

Transistor aging is the gradual physical degradation of a transistor as it is used, and the dominant mechanism in modern chips is bias temperature instability. When a transistor is held in one state for a long time — especially idling — its threshold voltage creeps upward, which means it switches a little more slowly than it used to. Multiply that across a signal path and the path’s total delay grows, until eventually a signal no longer arrives within the narrow timing window a synchronous circuit depends on. Arriving too late is a setup violation, which can at least be patched by slowing the clock; changing too early is a hold violation, which the clock cannot fix and which requires the chip to be physically repaired. The degradation is uneven across a chip, since different paths idle differently, and it is front-loaded in time, with most of a decade’s worth arriving in the first year — so a chip that passed every factory test can quietly cross into misbehavior months into its deployed life.

What people did before, and where it broke

Fuzz the instruction set (SiliFuzz). Google’s tool treats the CPU through its instruction-set architecture and fuzzes it to generate test programs that expose miscomputation. Where it broke. With no view of the implementation, it must generate around 500,000 tests to blanket every component, and a suite that large is far too heavy to run often.
Build tests on common libraries (DCDiag). Intel’s diagnostic and its open-source version construct stress tests atop popular software libraries and run them to shake out faulty chips. Where it broke. A full run takes 45 minutes, so it can only be scheduled during rare maintenance windows, long after an aging fault may have appeared.
Manage and schedule the long test suites. Frameworks decide when to run these expensive suites across a large fleet to amortize their cost. Where it broke. They are still bounded by the underlying runtime of the tests, so one cloud operator ends up running them just once every three months — a cadence that badly lags continuous aging.
Mitigate aging at design time. Circuit-level techniques during design and fabrication slow the aging process itself. Where it broke. They reduce aging but cannot guarantee its elimination, so field chips can still develop corruptions and a runtime detection mechanism is still needed.
The gap. Every detection approach treated the hardware as a black box, so all of them paid the same price: without implementation detail they had to test blindly and exhaustively, which made the suites too large to run frequently — exactly the opposite of what a continuous, accumulating fault like aging demands. What was missing was a way to use the chip’s own known structure to test only where aging would actually strike.

How Vega works

From a gate-level netlist to a test that runs inside an app

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Stop treating the chip as a black box and read its actual wiring. The whole weight of prior detection came from a single self-imposed handicap: pretending not to know how the hardware is built. A black-box generator, unable to see inside, has to assume any component might be the faulty one, so it carpet-bombs the chip with hundreds of thousands of tests and still cannot promise it stressed the right spot. But the chip is not a black box at all — it is synthesized from a gate-level netlist, a precise map of every cell and wire, and transistor aging has been studied for years at exactly that level. Vega throws away the handicap and works from the netlist directly, so instead of guessing where a fault might be it computes where aging will actually land. Refusing to reason about a circuit as an opaque box and instead exploiting its known internal structure to make the analysis precise is the same move the scalable hardware-invariant-learning work covered elsewhere on this site makes, opening up the design to reason about its wiring rather than hammering it blindly from the outside.

Let a formal tool hand you the exact input that breaks a path — or a proof that nothing can. Once Vega knows which paths are vulnerable, it needs a program that provably drives each one to fail, and here it turns to formal verification rather than fuzzing. It models the timing fault logically — the path endpoint latches a wrong value whenever its input just changed — grafts that model into a shadow copy of the affected logic, and poses a single question to the verifier: find an input under which the real output and the shadow output disagree. The tool either returns a cycle-accurate trace that is guaranteed to trigger the fault, or it proves that no legal input ever can, letting Vega drop that path from the suite entirely. Getting back a concrete, guaranteed-to-fail input that pinpoints the fault, rather than a vague sense that one might exist, is the same payoff the compiler-verification work covered elsewhere on this site delivers, where an SMT solver returns the exact operands on which two supposedly equivalent pieces of code diverge.

Bridge the gate level all the way up to a test that rides inside real software. The hard part is not finding a faulty waveform but turning it into something a running application can execute cheaply. Vega closes that distance in two moves: a one-time script per microarchitecture maps the module-level signal trace onto the assembly instructions that reproduce it, and a profile-guided pass finds a place in the target program that is visited routinely but not on every hot path, and quietly embeds the test there. The result runs as part of the application, checking the silicon beneath it every second, for a fraction of a percent of overhead. Carrying an insight from a low hardware level up through the abstraction stack until it becomes actionable in the software a developer actually runs is the same bridging the cross-level debugging work covered elsewhere on this site performs, tying a symptom seen at one level of the system back to its cause at another so the two can be reasoned about together.

Following the reasoning through Vega

Aging a tiny adder into a failure, then testing for it

The paper's worked example is a pipelined two-bit adder synthesized to a handful of gates, which makes each phase concrete.

Step 1 of 5

A small netlist that initially meets timing

The example is a two-bit pipelined adder synthesized into a few flip-flops and XOR and AND cells. As built, every path satisfies its timing constraints — the longest path arrives comfortably before the clock edge, and the shortest holds stable long enough after it. Nothing is wrong yet.

The circuit targets a one-nanosecond cycle, and its longest path lands at 0.9 nanoseconds, inside the required setup window.

Why it works

Opening the netlist replaces blind coverage with precise targeting. A black-box generator must stress every component just in case, producing enormous suites → but the gate-level layout tells Vega exactly which paths can age into failure → so it writes tests only for those, shrinking a job of hundreds of thousands of cases to a few dozen.
Signal probability plus validated models predict where failures will appear. Aging is invisible until a chip already misbehaves → but profiling how long each cell idles in a state, fed through silicon-validated delay models, forecasts which paths will cross their timing limits over ten years → so the tests are aimed at tomorrow’s failures rather than blindly at everything.
Formal verification yields a provable trigger, or a proof a path is safe. Fuzzing can only sample inputs and never conclusively rule a fault out → but a formal tool searches exhaustively for an input that makes a faulty path visibly diverge → so Vega either gets a guaranteed-to-trigger trace or a proof that no input can activate that path, sparing a wasted test.
A shadow replica and a cover property recast a timing fault as a logic question. Formal tools reason about logic, not the physics of a late signal → but modeling the violation as a wrong value injected into a duplicate of the affected cells lets Vega simply ask when the real and shadow outputs differ → so an analog timing problem becomes a discrete query the verifier can answer.
Because the suite is tiny, it can live inside the application and run constantly. A 45-minute diagnostic can only run every few months, long after aging has struck → but a few hundred to a couple thousand cycles slots into a rarely-taken branch of a live program → so the chip is checked every second at about 0.8 percent overhead, catching corruption as it starts.

What it costs (trade-offs and scope)

No physically aged silicon is in the loop. The whole pipeline runs against aging models and simulated failing netlists, not chips that have actually aged. Those models are validated against silicon, but there is no end-to-end demonstration that Vega catches a corruption on a genuinely worn-out part.
It was shown on two units of one small core. The prototype targets the ALU and FPU of the CV32E40P, an open-source in-order RISC-V core, at 28 nanometers — not a full commercial CPU. Working at module scale is deliberate, because whole-CPU formal verification does not scale, but it also bounds what has been demonstrated.
Instruction construction is manual. Turning a signal trace into assembly needs an expert-written script for each microarchitecture and unit — the paper calls it the most labor-intensive step. It is done once per component, but it is not automated and demands deep knowledge of the design.
Worst-case analysis over-flags. The timing analysis assumes pessimistic voltage, temperature, and process, so some paths it marks as aging-prone would never fail in practice. The conservatism guarantees real failures are caught, but it inflates the candidate list with false positives.
Not every vulnerable path yields a test. For some FPU paths the only observable symptom is a status flag already altered by a prior instruction, leaving nothing to compare against; and Vega proves a third to nearly half of candidate paths can never be triggered at all.
Random tests are sometimes competitive. On the FPU, plain random tests detected up to 97.2 percent of injected failures. Vega’s real advantage is the small targeted suite plus formal proofs that certain paths can never fail, not always a higher raw detection rate.
It covers aging-related timing faults only. Corruptions from manufacturing defects or other mechanisms are out of scope, and the authors frame Vega as complementary to broad top-down suites run during maintenance, not a replacement for them.

The evaluation

Approaches to detecting silent data corruptions

Property	DCDiag (library-based)	Random tests
How tests are generated
Knows where aging will strike
Suite size and runtime
Can prove a path can never fail	No	No
Fit for continuous in-app testing

Silent-data-corruption detection approaches, with Vega evaluated on the ALU and FPU of a 28-nanometer RISC-V core against aging models. Vega identifies 11 aging-prone paths in the ALU and over 1,300 in the FPU, generates suites running in a few hundred to a couple thousand cycles at about 0.8 percent application overhead, and formally proves a third to nearly half of candidate paths can never fail; it is framed as complementary to the broad top-down suites, not a replacement.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

Vega is a clever reframing with a clean end-to-end prototype: by using the chip’s own gate-level netlist and validated aging models, it pinpoints the signal paths that will age into timing failures and uses formal verification to synthesize tiny, targeted tests that fit inside a running application at about 0.8 percent overhead — a genuine alternative to 45-minute black-box suites run once a quarter. Read its results with the scope in mind. Nothing here runs on physically aged silicon; the whole loop is driven by aging models and simulated failing netlists, so it demonstrates a workflow, not a field catch. It was shown on the ALU and FPU of one small in-order RISC-V core, deliberately at module scale because whole-CPU formal verification does not scale. Its most labor-intensive step, turning traces into instructions, is manual and per-microarchitecture. The worst-case timing analysis over-flags paths that may never fail, not every vulnerable path can be turned into a test, and on the FPU even random tests sometimes match its detection rate — its distinctive value is the small targeted suite plus the ability to formally prove a path can never fail. The durable idea is the direction of attack: bottom-up from the physics, not top-down from a black box.

Background

Why hardware you cannot trust is a systems problem

It is worth stepping back to see why silent data corruption unsettles the whole computing stack, not just the chip. For decades, software was written against a simple contract: the processor beneath it is either correct or dead. Compilers, operating systems, distributed protocols, and error-handling code all lean on that fail-stop assumption — a machine that miscomputes without warning is simply not in their model of the world. Aging quietly voids the contract, turning a trusted foundation into something that can lie, occasionally and unpredictably, in ways that propagate silently into results a business depends on. That is why detecting it cannot be left to the chip alone; it becomes a systems problem, needing continuous checks woven into the software itself, so a corruption is caught the moment it appears rather than discovered downstream in a corrupted database or a wrong financial figure. Vega is one answer, embedding the check in the application. The same drive to establish trust in hardware by proof rather than assumption animates the secure-speculation verification work covered elsewhere on this site, which reasons formally about a processor’s behavior to rule out a class of flaw instead of hoping it is absent. The broader pattern is that as devices grow more complex and more prone to subtle, silent misbehavior, the burden of guaranteeing correctness increasingly shifts upward, and the systems that endure are the ones that stop trusting the hardware blindly and start checking it.

If you remember only three things

1. Transistor aging gradually slows switching until signals miss their timing windows, so a chip that passed every factory test can begin silently miscomputing — a silent data corruption with no crash or log, breaking the fail-stop assumption software has relied on for decades; because aging accumulates continuously, catching it needs frequent testing, but existing black-box generators build enormous suites (around 500,000 tests, or a 45-minute run) that can only be scheduled every few months.
2. Vega goes bottom-up: it profiles how long each gate idles as a proxy for aging stress, runs aging-aware static timing analysis with silicon-validated models to pinpoint the exact paths that will age into setup or hold violations, models each violation logically in a shadow replica, and asks a formal tool for a provable input trace that triggers it — or a proof that no input can, dropping that path — then lifts the trace to assembly and embeds it in an application with a profile-guided pass.
3. On the ALU and FPU of a 28-nanometer RISC-V core, Vega flags 11 and over 1,300 aging-prone paths, generates suites that run in a few hundred to a couple thousand cycles at about 0.8 percent overhead, and formally proves a third to nearly half of candidate paths can never fail — but everything runs against aging models rather than physically aged silicon, it covers only two units of one small core, instruction construction is manual, the worst-case analysis over-flags, and on the FPU random tests sometimes match its detection rate, so its real edge is the tiny targeted suite plus proofs of non-failure.

Check yourself

Check your understanding

1. Why are aging-related silent data corruptions so hard to handle with existing test suites?
2. What is the core idea behind Vega's bottom-up approach?
3. How does Vega turn a physical timing fault into something a formal verification tool can search for?
4. Which statement best reflects the limits of Vega's evaluation?

Sources

The paper is “Proactive Runtime Detection of Aging-Related Silent Data Corruptions: A Bottom-Up Approach” by Jiacheng Ma, Majd Ganaiem, Madeline Burbage, Theo Gregersen, Rachel McAmis, Freddy Gabbay, and Baris Kasikci, published at ASPLOS 2024, pages 219 to 234, DOI 10.1145/3622781.3674182. The authors are at the University of Michigan, the Technion, the University of Washington, the Hebrew University of Jerusalem, and Google.

This explainer is written from the full 16-page text. Everything above comes from the paper’s own sections: the introduction and background on silent data corruptions reported by Meta, Google, and Alibaba (with 73.5 percent of one operator’s corruptions appearing in already-deployed CPUs), on transistor aging via bias temperature instability with its setup and hold timing violations and its front-loaded, nonuniform progression, and on the reaction-diffusion model, signal probability, and aging-aware static timing analysis; the three-phase Vega workflow of Aging Analysis (signal-probability simulation with free-running counters and a SPICE-derived aging-aware timing library over a ten-year lifetime), Error Lifting (logical models of setup and hold violations, MUX-based failure-model instrumentation, shadow replicas, cover-property trace generation with a formal tool, the initial-value-dependency mitigation, and manual instruction construction), and Test Integration (a software aging library and a profile-guided LLVM-based embedding); the implementation on the ALU and FPU of the CV32E40P RISC-V core at 28 nanometers using SPICE, Cadence Innovus, Yosys, JasperGold, and LLVM; and the evaluation reporting nonuniform aging delay increases, 11 aging-prone paths in the ALU and 1,366 in the FPU filtered to 6 and 41 unique endpoint pairs, test suites of a few hundred to a couple thousand cycles, formal proofs that a third to nearly half of candidate paths cannot fail, detection quality against injected failing netlists including cases where random tests reach up to 97.2 percent on the FPU, and an average profile-guided integration overhead of 0.8 percent.