Hardware Sentinel: Protecting Software Applications from Hardware Silent Data Corruptions

Some processors in a large fleet quietly compute the wrong answer — not crashing, not logging an error, just returning a number that is wrong — and at Meta's measured rate of roughly one per thousand machines that is thousands of corrupting CPUs silently poisoning results; the established defence is to run diagnostic tests on the hardware, which is slow, vendor-specific and misses things, so Hardware Sentinel (Meta with Cornell) inverts the direction and mines six years of crash logs the fleet was already collecting, looking for the tell-tale shape of a hardware fault — several unrelated applications failing on the same CPU core, with no hardware error logged anywhere.

Why this problem is worth solving

The rate is what changed everything about this topic.

What makes it hard is that the fault is conditional in ways a test cannot enumerate.

What people did before, and where it broke

• Assume it does not happen. For decades the working figure was about one defective device per million, drawn largely from studies of cosmic-ray-induced bit flips. Where it broke. Nobody outside a hyperscaler had the fleet size to measure it. When Meta did, the rate came out near one per thousand, and Google and Alibaba independently corroborated the order of magnitude.
• Screen hardware during maintenance windows. Run thorough diagnostics whenever a machine is offline anyway. Where it broke. Coverage is bounded by the maintenance schedule, not by need — about six months for one full sweep. Some architectures and workload families almost never come up, and the paper finds regions and machine families where this approach detected nothing at all that the new method found.
• Co-locate short tests with production work. Run millisecond-scale checks alongside real workloads so machines get tested continuously without being drained. Where it broke. It cannot be used where performance matters most, which is exactly where the paper reports it failing — gaming and edge workloads are too latency-sensitive to tolerate it, so those machines effectively go unchecked.
• Rely on the hardware to report its own faults. Processors expose machine check registers, and boards log environmental and error events. Where it broke. By definition. If the hardware knew, the corruption would not be silent. The paper turns this into a filter: a clean hardware error log is a precondition for suspicion, not a reassurance.
• Treat the resulting crashes as software bugs. Which is what happens by default, since a segmentation fault looks like a segmentation fault. Where it broke. Engineers spend weeks debugging correct code. The paper’s framing is that these failures were never invisible — they were misattributed.
• The gap. Everyone was pushing tests down at the hardware. Nobody had asked whether the failures already streaming out of the application layer, which the fleet logs anyway, contain enough signal to identify the corrupting machine without any hardware-specific knowledge at all.

How Hardware Sentinel finds a corrupting CPU

The idea

The evidence was already being collected; it was being filed under the wrong cause. This is the reframing, and its appeal is that it costs no new instrumentation. A corrupting processor does not stay hidden — the wrong values it produces eventually make something fall over, and that something gets logged like any other crash. What was missing was not data but attribution: a segmentation fault caused by a bad multiply looks exactly like a segmentation fault caused by a bad pointer, so it goes into the software bug pile and an engineer loses a fortnight to it. Mining six years of those logs turns a debugging cost into a detection signal, and does so without any vendor tooling, which is the property that lets someone outside a chip partnership do this work. Reading root causes out of telemetry the system already produces, rather than adding a new probe, is the same approach as the trace-based root-cause analysis work for microservices covered elsewhere on this site.

The signature is concentration, not the crash itself. No individual exception means anything — a fleet this size produces hundreds of thousands daily and nearly all are software. What distinguishes hardware is a statistical shape that software essentially never produces: several unrelated applications, sharing no code path, failing repeatedly on one specific physical core of one machine, while every other core on that machine is fine. A software bug spreads across cores and across the whole fleet; a defective execution unit does not move. The paper sharpens this with a second condition that reads as inverted logic and is exactly right — the machine’s hardware error log must be empty, because a fault the hardware reported is by definition not silent. It also finds that aggregating a core with its hyperthread sibling raises detection, since they share physical hardware. Identifying a fault by the distributional shape of failures rather than by any single observation is the same statistical instinct as the corruption detection work for deep learning covered elsewhere on this site.

Six years of fleet data buys an empirical finding nobody could have guessed. Beyond the framework, the paper reports which kernel exceptions actually correlate with confirmed hardware faults, and the ratios are dramatic: double faults appear 59 times more often among confirmed cases than fleet-wide, stack segment faults 21 times, non-executable memory violations 20 times, invalid opcodes 18 times. The common exceptions everyone sees — lockups, kernel oopses — turn out to carry almost no signal. That is a genuinely new operational fact, and it is only derivable by someone holding both the failure logs and the confirmed defect list for millions of machines over years. Producing a characterisation that is only visible from inside a production fleet, and publishing it so outsiders can use it, is the same kind of contribution as the warehouse-scale memory allocator study covered elsewhere on this site.

Walking one machine through the pipeline

That feedback role matters because the rules are heuristics. If a machine keeps appearing at the top and keeps being filtered out, the filter is wrong, and the control plane gets revisited. The authors are explicit that a static rule set cannot capture the nuances and that the policies need continuous attention.

Why it works

• A defective execution unit cannot move; a software bug does. A bad multiplier is bound to one core of one machine → several unrelated applications crashing repeatedly on that one core, while its neighbours are fine, is a shape software does not produce → concentration becomes the discriminator rather than any individual crash.
• Silence from the hardware is positive evidence, not absence of it. If the processor logged a machine check, the failure was loud and already has a remediation path → so requiring a clean hardware error log narrows the candidates to exactly the silent case the framework is hunting → the filter that would normally weaken a diagnosis strengthens this one.
• Detection is decoupled from machine availability. Testing reaches a machine only when maintenance or a co-location slot allows → mining logs reaches every machine that is running, all the time → which is why the largest gains land on architectures and regions that testing rarely visits, and on latency-sensitive workloads it cannot touch.
• Two applications is the cheapest way to separate hardware from software. A single buggy binary failing across thousands of machines would swamp any per-core statistic → requiring failures from unrelated applications on the same core discards that class immediately → and the remaining single-application cases are held aside as possible corruption-inducing workloads.
• Hyperthread siblings share the silicon, so they should share the evidence. Treating logical cores separately diluted the signal → aggregating a core with its sibling, since they share the physical execution hardware, raised the detection rate → a small modelling choice that follows directly from what the fault actually is.

What it costs (trade-offs and scope)

• The headline improvements are against each baseline separately. The 1.74 times over out-of-production testing and 1.92 times over in-production testing are individual comparisons. Against the two combined — which is what the fleet actually runs — the improvement is 1.41 times. That is still substantial and it is the number that describes the deployment.
• The thresholds are hand-tuned judgement calls. Six reboots in thirty days, three for AI machines, 60% concentration on one core, two distinct applications, a one-week exception window, a configurable top-N. All are described as based on analytical observation, with reasoning given but no derivation, and the authors say plainly that a static rule set is insufficient and the policies require ongoing attention.
• The false-positive story is instructive. An early version flagged machines as having faulty CPUs when the workload crashing them was a fuzzer whose entire purpose is to crash things. The fix was to exclude fuzzer workloads. It is an honest anecdote and it shows how much the method depends on knowing what the workloads are meant to do.
• The validation sample is ten machines. Seven reproduced, one inconclusive, so two did not reproduce. The 70% rate is genuinely high against a typical 15% to 30%, and it rests on ten samples with the CPU generation chosen by the third party rather than at random.
• It does not help everywhere. One architecture in early adoption shows no improvement, because too little work runs on it to generate signal. Four of twenty-seven regions show no significant benefit, because they test frequently and run a machine family whose tests happen to be effective.
• This is detection after the fact, not prevention. The framework finds machines that have already corrupted data — the corruption has propagated into whatever those workloads wrote. It shortens the exposure window and removes the machine; it does not stop the first bad result.
• It presumes an infrastructure almost nobody has. Six years of retained failure data, structured kernel logs across many kernel versions, reboot and repair and hardware-event databases, and the query capacity to correlate across all of them at fleet scale. The vendor-agnostic claim is real and does not make the approach reproducible without a fleet.
• The baselines are the authors’ own prior systems. Which is reasonable, since they are the published state of practice, and does mean the comparison is internal.
• Absolute numbers are withheld. Results are relative detection rates and normalised defect rates, with hundreds of defective CPUs identified over six years. The shape of the finding is clear; its scale relative to the fleet is not stated directly.

The evaluation

Check yourself

Sources

The paper is “Hardware Sentinel: Protecting Software Applications from Hardware Silent Data Corruptions” by Rhea Dutta, Harish Dattatraya Dixit, Rik Van Riel, Gautham Vunnam and Sriram Sankar, published at ASPLOS 2025 Volume 2, pages 482-497, DOI 10.1145/3676641.3716258. The authors are at Meta Platforms.

This explainer is written from the full text, read through a proxy of the ACM Digital Library copy. Everything above comes from the paper’s own sections 2 through 5: the account of how silent data corruption rates moved from an assumed one per million to a measured one per thousand and were corroborated by other hyperscalers; the description of out-of-production and in-production testing with their six-month sweep cycle and their unsuitability for performance-sensitive workloads; the six data sources of the data plane covering structured kernel logs, reboots, system event logs, prior test results, repairs and core concentration metadata; the control plane thresholds of six reboots in thirty days and three for AI machines, a 60% single-core exception concentration, at least two distinct applications, a one-week exception window and a configurable top-N ranking, together with the stated rationale for each; the requirement of a clean hardware error log as a precondition; the hyperthread sibling aggregation that raised detection; the worked example ruled a software anomaly because exceptions were spread evenly across cores, and the fuzzer false-positive case; the results across eight CPU generations manufactured between 2012 and 2023, thirteen workload families and twenty-seven regions, including the architectures and regions where the framework adds nothing; the exception-type correlations at 59.35, 20.77, 20.03, 17.80 and 6.92 times the fleet rate for double fault, stack segment, non-executable memory, invalid opcode and breakpoint exceptions; the improvements of 1.74 times over out-of-production testing, 1.92 times over in-production testing and 1.41 times over the two combined; the 21% reduction in fleet exceptions and the fall from 30% to 9% of kernel exceptions attributable to the top candidate machines; and the independent third-party failure analysis of ten samples with seven conclusive reproductions and one inconclusive.