SEVI: Silent Data Corruption of Vector Instructions in Hyper-Scale Datacenters

Yixuan Mei, Shreya Varshini, Harish Dattatraya Dixit, Sriram Sankar and K. V. Rashmi

A tiny fraction of datacenter CPUs are quietly defective, computing wrong answers for certain operations without any crash or error — "silent data corruption" that spreads bad results through the system; this paper (Carnegie Mellon and Meta — Yixuan Mei, K. V. Rashmi, Harish Dixit, and colleagues) runs the first huge fleet-scale study of these corruptions specifically in vector (SIMD) instructions — 78 trillion test rounds — uncovering undocumented failure patterns and building a cheap in-application check that catches 88 to 100 percent of bad machines at 1.35 percent overhead.

The argument in one glance

Problem

Some datacenter CPUs quietly compute wrong answers

At the scale of a modern datacenter — millions of CPUs running continuously — a small but real fraction of processors turn out to be defective in a particularly insidious way: they compute wrong results for certain operations, under certain conditions, without raising any error. This is Silent Data Corruption (SDC). Unlike a crash or a machine-check exception, an SDC produces no signal at all — the hardware simply returns an incorrect value, which then flows downstream and corrupts everything computed from it, so a single bad core can silently poison data, decisions, and stored results across a system. These faults come from manufacturing defects and hardware degradation that leave particular cores marginal, so they miscompute in ways that ordinary error detection does not catch. This is not hypothetical: Google, Meta, and Alibaba have all publicly reported SDC incidents affecting their production fleets. Because the errors evade detection and propagate, they are among the hardest reliability threats to manage at hyperscale. And there is a specific blind spot: vector instructions — the SIMD operations that process many data elements at once and underpin machine learning, scientific computing, and data processing — are heavily used and have complex datapaths, but how SDCs actually behave in these instructions had not been comprehensively studied.
What people did

Detect bad cores with fleet testing and generic checks

Operators already know SDC is a problem and try to manage it. The main approaches are large-scale fleet testing — periodically running test workloads across many cores to find the ones that miscompute — and adding redundancy or checks to catch corrupted results. But to test and to build detection well, you need to understand how the corruptions actually manifest: which instructions fail, under what conditions, and with what error patterns. Existing understanding of SDC has been broad, but the specific behavior of corruptions in vector instructions, and how to detect them cheaply, was not well characterized.
Why it fell short

Without knowing the patterns, testing and detection stay blunt

The gap was a lack of detailed, instruction- and application-level knowledge of how vector-instruction SDCs behave, which limits both testing and detection. Fleet testing is enormously expensive — it consumes vast amounts of compute to scan cores — and without knowing which vector instructions and conditions are most prone to corruption, that testing is inefficient. Generic detection, meanwhile, either misses the corruptions or is too heavy to run in production: full duplication of every computation, for example, would catch errors but roughly double the cost, which is untenable at datacenter scale. What was missing was, first, a comprehensive characterization of how SDCs actually occur in vector instructions across a real hyperscale fleet — the patterns, the likely underlying causes — and second, a detection mechanism informed by those findings that is cheap enough to leave running in production while still catching the great majority of corrupting machines. Understanding the phenomenon precisely is the prerequisite for both efficient testing and efficient detection.
This paper's idea

Characterize vector-instruction SDCs at scale, then detect them cheaply

SEVI presents the first comprehensive instruction- and application-level analysis of vector-instruction silent data corruptions in hyper-scale datacenters, using a two-stage approach that studies the corruptions both at the level of individual instructions and in the context of real applications. The study is enormous: over 78 trillion test rounds across more than 14 billion CPU seconds of a production fleet. From this, the authors uncover undocumented SDC patterns — regularities in how and when vector instructions corrupt — that give insight into the possible underlying hardware causes and point toward new mitigation strategies. Building on these findings, they propose a low-overhead SDC detection mechanism based on in-application algorithm-based fault tolerance: rather than duplicating whole computations, it adds cheap mathematical checks inside the application (in the spirit of checksums on a computation) that are tailored to catch the corruption patterns the study revealed. This detection achieves an 88 to 100 percent SDC machine detection rate with a time overhead of only 1.35 percent, even for modestly sized inputs — meaning it catches the overwhelming majority of corrupting machines while adding almost nothing to runtime, making it practical to leave on in production.

Why this problem is worth solving

Silent data corruption is uniquely dangerous because it is invisible, and at hyperscale even a tiny per-core rate becomes a constant stream of corrupted results, so characterizing and cheaply catching it is a first-order reliability need.

Most hardware failures announce themselves — a crash, a reset, an error code — and systems are built to handle them. SDC is different and worse precisely because it is silent: a defective core returns a plausible but wrong number, and nothing flags it, so the error slips into the data, gets stored, gets computed on, and propagates until it surfaces as an inexplicable downstream problem, if it surfaces at all. At the scale of millions of CPUs, the probability that some core, somewhere, is silently miscomputing at any moment is not negligible, which is exactly why Google, Meta, and Alibaba have all reported it as a real production issue. The consequences range from corrupted analytics and bad ML training data to incorrect financial or safety-relevant results, and the cost of a corruption that is discovered late — after it has spread — can be enormous. Vector instructions raise the stakes further, because SIMD is so heavily used in the compute-intensive workloads that dominate datacenters, so a corruption pattern specific to vector instructions touches an enormous amount of important computation. Understanding these corruptions precisely, and being able to detect them without doubling the cost of every computation, is therefore not a niche concern but a core requirement for trustworthy hyperscale computing. Both the characterization and the low-overhead detection directly reduce the risk that silent errors quietly undermine results across the fleet.

The key realization is that SDCs are not random noise but follow patterns, and knowing those patterns lets you replace brute-force duplication with cheap targeted checks.

The naive way to catch silent corruption is to do everything twice and compare, but full duplication roughly doubles cost, which is a non-starter at datacenter scale. The alternative is to check results cheaply, but a cheap check only works if you know what you are looking for. SEVI’s bet is that vector-instruction SDCs are not featureless random errors: they have structure — particular instructions, conditions, and error signatures that recur — and a study large enough to see that structure can reveal it. Running tens of trillions of test rounds across a real fleet is what makes the patterns visible, and those undocumented patterns do two things. They hint at the underlying hardware causes, which is valuable for both chip vendors and operators. And, crucially, they tell you how to design a detector that is cheap because it is targeted: instead of duplicating a computation, you add algorithm-based fault-tolerance checks — mathematical invariants the correct computation must satisfy, like a checksum that a matrix operation preserves — chosen to catch precisely the corruption patterns the study found. Because these checks exploit the algorithm’s own structure rather than repeating the work, they cost a tiny fraction of the computation while still flagging the errors. So the deep move is measurement-driven: characterize the phenomenon at massive scale, extract the patterns, and use them to make detection both effective and nearly free. Knowing the enemy is what turns an unaffordable defense into an affordable one.

What people did before, and where it broke

Run large-scale fleet testing. Periodically test cores across the fleet to find those that miscompute. Where it broke. It is enormously expensive in compute, and without knowing which vector instructions and conditions are most prone to SDC, the testing is inefficient.
Duplicate computations and compare. Run everything twice to catch corrupted results. Where it broke. Full duplication roughly doubles cost, which is untenable in production at datacenter scale.
Use generic error checks. Apply broad, workload-agnostic detection. Where it broke. It either misses the specific vector-instruction corruptions or is too heavy to leave running, because it is not informed by how these SDCs actually behave.
Rely on prior, coarse SDC understanding. Lean on broad knowledge that SDC exists. Where it broke. The specific instruction- and application-level behavior of vector-instruction SDCs, and their patterns, had not been comprehensively characterized, so mitigation stayed blunt.
The gap. No comprehensive, fleet-scale characterization of vector-instruction SDCs existed to reveal their patterns, and no low-overhead detector informed by such patterns had been built to catch them cheaply in production.

How SEVI works

From coarse SDC awareness to pattern-driven, low-overhead detection

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Detect silent corruptions in real workloads instead of letting them propagate. SEVI’s purpose is to catch vector-instruction SDCs before their wrong results spread, focusing detection on the computations that actually run in the fleet. Building mechanisms to detect silent data corruptions where they matter — in the running application — rather than hoping generic hardware checks catch them is the same enterprise the work on combating silent data corruptions in deep learning (DR-DNA) covered elsewhere on this site pursues, adding application-level detection for silent errors. SEVI targets the vector instructions that underpin so much datacenter compute.

Learn the phenomenon from a massive real-fleet measurement study. The foundation of SEVI is an enormous characterization — tens of trillions of test rounds across a production fleet — to reveal patterns that only appear at scale. Mining a real, warehouse-scale system with a huge measurement study to uncover behavior and patterns that inform better system design is the same approach the work on characterizing a memory allocator at warehouse scale (warehouse-scale characterization) covered elsewhere on this site takes, letting scale surface structure invisible in small experiments. SEVI’s study surfaces the undocumented SDC patterns that make cheap detection possible.

Make the detector cheap enough to run everywhere. Rather than doubling cost with duplication, SEVI adds targeted algorithm-based checks that catch corruption at a tiny overhead, so detection can be deployed pervasively. Designing an error-detection mechanism to be efficient enough to leave on in production, instead of an expensive check reserved for offline use, is the same goal the work on efficient memory sanitization (GiantSan) covered elsewhere on this site shares, making a heavyweight checking idea affordable at scale. SEVI’s 1.35 percent overhead is what turns SDC detection from a costly audit into an always-on safeguard.

Following a corruption through SEVI

Catching a silently-corrupting core

An illustration, drawn from the paper's abstract, of how SEVI characterizes and detects vector-instruction SDCs. Internal mechanism details are the paper's.

Step 1 of 5

A core silently miscomputes a vector instruction

On some defective core, a SIMD vector instruction returns a wrong result for a particular operation and condition, with no crash or error signal, so the incorrect value would normally propagate downstream unnoticed.

This is the silent corruption SEVI aims to catch.

Why it works

Silent errors propagate, so catching them early is essential. An SDC produces no signal and its wrong value flows downstream → but detecting it in the running application → so the corruption is caught before it spreads and poisons more results.
Massive scale reveals patterns invisible in small tests. A few experiments cannot show how rare corruptions behave → but tens of trillions of test rounds across a real fleet → so undocumented SDC patterns and likely causes become visible.
Knowing the patterns enables targeted, not brute-force, checks. Full duplication doubles cost → but designing checks aimed at the specific discovered corruption patterns → so detection is effective without repeating the whole computation.
Algorithm-based checks exploit the computation’s own structure. A generic check is either weak or heavy → but mathematical invariants the correct result must satisfy → so a small extra computation flags a wrong result cheaply.
Low overhead makes detection always-on. An expensive audit can only run occasionally → but a 1.35 percent overhead → so the detector can run continuously in production and catch most bad machines.

What it costs (trade-offs and scope)

The characterization required an enormous testing effort. The insights come from over 78 trillion test rounds and more than 14 billion CPU seconds on a hyperscale fleet — a scale of measurement only a large operator can perform.
Detection is not perfect. The reported machine detection rate is 88 to 100 percent, so some corrupting machines can still be missed depending on the workload and corruption type.
The checks are algorithm-based and application-specific. Algorithm-based fault tolerance adds invariants tailored to particular computations; it must be integrated into applications and suits computations that admit such checks, rather than being a universal drop-in.
It targets vector-instruction SDCs. The study and detector focus on SIMD vector instructions; SDCs in other instruction classes are outside this specific scope, though the methodology may generalize.
Reported figures depend on inputs and conditions. The 88-to-100-percent detection and 1.35 percent overhead are measured on the paper’s workloads and even for modestly sized inputs; results depend on the application, input size, and corruption patterns.

The comparison

Handling silent data corruption in vector instructions

Property	Fleet testing only	Full duplication	SEVI (pattern-driven ABFT)
Understands SDC patterns
Detection in production
Overhead
Detection rate
Practical to deploy widely

A comparison of ways to handle silent data corruption in vector instructions. Fleet testing is expensive and periodic; full duplication catches errors but roughly doubles cost. SEVI first characterizes vector-instruction SDCs across a hyperscale fleet (78 trillion test rounds) to reveal their patterns, then uses targeted algorithm-based fault-tolerance checks to detect 88 to 100 percent of corrupting machines at only 1.35 percent overhead — cheap enough to run continuously. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s abstract and public metadata, not its full text, which was not available at the time of writing. The high-level claims are the authors’: the paper presents the first comprehensive instruction- and application-level analysis of vector-instruction silent data corruptions (SDCs) in hyper-scale datacenters, using a two-stage approach and performing over 78 trillion test rounds across more than 14 billion CPU seconds. It reports uncovering undocumented SDC patterns that provide insight into possible underlying causes and inspire new mitigation strategies, and proposes a low-overhead SDC detection mechanism leveraging in-application algorithm-based fault tolerance that achieves an 88 to 100 percent SDC machine detection rate with a time overhead of only 1.35 percent even for modestly sized inputs. The paper notes that SDC is a reliability threat reported in production by Google, Meta, and Alibaba. The background above on silent data corruption, vector (SIMD) instructions, fleet-scale testing, and algorithm-based fault tolerance (for example, checksum-style invariants) is standard context provided to make the approach intuitive. The Flow, the step-by-step walkthrough, and the “Why it works” reasoning are illustrations built from the abstract’s description, not detailed accounts from the paper — the specific SDC patterns discovered, the exact two-stage methodology, the algorithm-based checks used, and the full evaluation should be read from the paper. Do not treat specific procedural details here as established beyond what the abstract states.

Background

Why trustworthy computing at hyperscale is increasingly about catching silent hardware errors

It helps to see SEVI within a shift in how the field thinks about reliability. For most of computing’s history, the assumption was that hardware, once it passed manufacturing test, computed correctly, and reliability engineering focused on failures that announce themselves — crashes, resets, detectable memory errors. But as fleets have grown to millions of chips and as devices have been pushed to the edge of their physical margins, a quieter failure mode has become impossible to ignore: individual cores that silently compute wrong answers, some of the time, for some operations. Because these errors leave no trace and propagate through everything computed from them, they undermine the basic assumption that a correct program on working-looking hardware produces correct results. The response emerging across the industry is twofold: measure the phenomenon at scale to understand it, and build detection that is cheap enough to run everywhere rather than reserved for occasional audits. This is a reliability discipline for a world where hardware correctness can no longer be taken for granted, and where the sheer scale that creates the problem also provides the data to study it. This effort to keep large-scale computation trustworthy by detecting and tolerating silent hardware faults rather than assuming they do not happen is the same terrain the work on proactive runtime detection of aging-related silent data corruptions (proactive SDC detection) covered elsewhere on this site occupies, catching silent errors before they corrupt results. SEVI’s particular contribution is to characterize vector-instruction SDCs comprehensively for the first time and to turn that knowledge into an almost-free, always-on detector.

If you remember only three things

1. Silent Data Corruption (SDC) is a serious datacenter reliability threat: a small fraction of CPUs, due to manufacturing defects or degradation, silently compute wrong results for certain operations with no crash or error signal, so the incorrect values propagate downstream and corrupt everything computed from them. Google, Meta, and Alibaba have all reported SDC in production. A specific blind spot was vector (SIMD) instructions — heavily used in ML, scientific computing, and data processing, with complex datapaths — whose SDC behavior had not been comprehensively studied. Existing defenses are blunt: fleet testing is enormously expensive and, without knowing which instructions and conditions corrupt, inefficient; full duplication catches errors but roughly doubles cost; and generic checks either miss the corruptions or are too heavy for production.
2. SEVI presents the first comprehensive instruction- and application-level analysis of vector-instruction SDCs in hyper-scale datacenters, using a two-stage approach across over 78 trillion test rounds and more than 14 billion CPU seconds of a production fleet. This massive scale reveals undocumented SDC patterns — regularities in how and when vector instructions corrupt — that give insight into the underlying hardware causes and inspire new mitigation. Building on the patterns, the authors propose a low-overhead detection mechanism based on in-application algorithm-based fault tolerance: instead of duplicating computations, it adds cheap mathematical invariants (checksum-style checks the correct result must satisfy) targeted at the discovered corruption patterns, so detection exploits the algorithm's own structure rather than repeating the work.
3. The detector achieves an 88 to 100 percent SDC machine detection rate with a time overhead of only 1.35 percent, even for modestly sized inputs — catching the overwhelming majority of corrupting machines while adding almost nothing to runtime, so it can be left running in production. The trade-offs and scope: the characterization required an enormous testing effort only a large operator can perform, detection is not perfect (some machines can be missed), the algorithm-based checks are application-specific and suit computations that admit such invariants, and the work targets vector-instruction SDCs specifically. This explainer is based on the paper's abstract; the discovered patterns, methodology, and full evaluation should be confirmed in the paper.

Check yourself

Check your understanding

1. What makes silent data corruption (SDC) especially dangerous in datacenters?
2. Why is understanding SDC patterns important before building a detector?
3. What are SEVI's two main contributions?
4. What does SEVI's detector achieve, and what are the limits of the approach?

Sources

The paper is “SEVI: Silent Data Corruption of Vector Instructions in Hyper-Scale Datacenters” by Yixuan Mei, Shreya Varshini, Harish Dattatraya Dixit, Sriram Sankar, and K. V. Rashmi (Carnegie Mellon University and Meta), published at ASPLOS 2026, DOI 10.1145/3779212.3790217.

This explainer is written from the paper’s abstract and public metadata; the full text was not available at the time of writing, so the page is marked a draft. As reported there, SEVI presents the first comprehensive instruction- and application-level analysis of vector-instruction silent data corruptions in hyper-scale datacenters, using a two-stage approach across over 78 trillion test rounds and more than 14 billion CPU seconds. It uncovers undocumented SDC patterns that provide insight into underlying causes and inspire new mitigation, and proposes a low-overhead detection mechanism using in-application algorithm-based fault tolerance, achieving an 88 to 100 percent SDC machine detection rate at only 1.35 percent time overhead even for modestly sized inputs. Background on silent data corruption, vector instructions, fleet-scale testing, and algorithm-based fault tolerance is standard context. The specific numbers are the paper’s own; the discovered patterns, methodology, and full evaluation should be confirmed in the paper.