CirFix: automatically repairing defects in hardware design code

Automated program repair can fix software bugs by mutating code and re-running the tests, but it had never worked on HARDWARE designs, because Verilog is nothing like a sequential program — every block runs in parallel each clock cycle, a test's output is a waveform of signal values over time rather than a pass or fail, and you cannot trace which line ran; CirFix (University of Michigan) brings repair to hardware for the first time by rebuilding the two domain-specific pieces — a fault localizer that follows bad values backward through wire and register assignments instead of line coverage, and a fitness function that scores a candidate fix by how many signal bits it gets right at every cycle (turning all-or-nothing into a smooth gradient a genetic search can climb) — then evolving Verilog syntax trees with hardware-aware mutations, producing plausible repairs for 21 of 32 seeded defects and correct ones for 16, matching software repair's success rate.

The argument in one glance

Problem

Hardware bugs are costly and repaired entirely by hand

Debugging and fixing hardware designs written in Verilog or other HDLs is slow, expensive, and manual, and the stakes are high — flaws like Meltdown, Spectre, and the Pentium FDIV bug show how costly an undetected hardware defect can be. In the software world, AUTOMATED PROGRAM REPAIR has become a real tool: systems like GenProg take a buggy program and a test suite, mutate the code, re-run the tests, and evolve toward a version that passes, fixing bugs without a human. The obvious question is why hardware engineers do not have the same thing — why nobody had brought automated repair to the hardware designs where bugs are just as painful.
What people did

Point software repair tools at Verilog

The natural move is to reuse the mature software machinery. Automated program repair has a well-worn recipe: LOCALIZE the fault (find which statements are likely to blame, usually by seeing which lines run more often in failing tests than passing ones), MUTATE the suspect code, and use the TEST SUITE as a fitness signal to steer an evolutionary or search-based process toward a patch that passes. Verilog is text with a grammar and an abstract syntax tree, so in principle you could mutate it the same way and let a testbench judge the results.
Why it fell short

Hardware breaks every assumption software repair rests on

It does not work, because HDLs violate the premises of software repair. CONCURRENCY: a program runs line by line, but a hardware design is inherently PARALLEL — every always-block fires every clock cycle — so ranking statements by how often they execute is meaningless, since everything executes constantly. OBSERVABILITY: you cannot instrument a circuit to trace which statement "ran"; the only things you can observe are the values on wires and registers over time. TEST SIGNAL: a software test passes or fails, but a hardware testbench produces a WAVEFORM — a signal is correct or wrong at every one of many clock cycles — so a bare pass/fail throws away almost all the information about how close a candidate is. Fault localization and the fitness signal, the two things that steer repair, both assume a sequential world that hardware simply is not.
This paper's idea

Rebuild localization and fitness for hardware, then evolve the Verilog

CirFix (University of Michigan — Ahmad, Huang, and Weimer) brings automated repair to hardware for the first time by keeping the domain-general skeleton (evolutionary search over the syntax tree) and rebuilding the two domain-specific pieces. A DATAFLOW FAULT LOCALIZER starts from the wires whose values are wrong and walks backward through assignments and conditionals to a precise set of suspect AST nodes — no execution ranking needed. A WAVEFORM FITNESS FUNCTION scores a candidate by how many output bits it gets right at EVERY clock cycle, turning a flat pass/fail into a continuous [0,1] score with a smooth gradient a genetic algorithm can climb. Then hardware-aware mutations and repair templates evolve the Verilog. The result: plausible repairs for 21 of 32 seeded defects and correct repairs for 16 — a rate comparable to mature software repair.

Why this problem is worth solving

Automated program repair is one of the success stories of software engineering research, and its absence from hardware was a conspicuous gap given how expensive hardware bugs are to find and fix. The reason the software tools could not simply be reused is that they rest on a model of execution hardware does not share. Software repair leans on

the standard software technique for guessing where a bug lives: run the test suite, and for each statement track how often it executes in FAILING tests versus PASSING tests. Statements that run mostly during failures are ranked as more suspicious, producing an ordered list of likely culprits to try mutating first. It works because software executes sequentially — one statement after another — so “which statements ran on the failing input” is a meaningful, discriminating signal. The whole method is built on the idea that different inputs exercise different code paths, and the failing paths point at the bug

. In hardware that premise collapses: a Verilog design is massively parallel, with every always-block evaluating on every clock edge, so essentially every statement “runs” every cycle and execution frequency tells you nothing about where the fault is.

The second broken assumption is even more consequential for the search itself. Software repair uses the test suite as its compass, but a test that only reports pass or fail gives a searing-narrow signal, and hardware makes this worse and better at once through the

the way a hardware testbench specifies correct behavior: not as a single pass/fail verdict, but as the expected VALUE of each output wire and register at every clock cycle — a waveform over time. Running a candidate design through the testbench in a simulator produces its own waveform, and comparing the two is not a binary question but a detailed, bit-by-bit, cycle-by-cycle comparison. This is a curse for naive reuse (there is no simple “the test passed”) but a gift if you exploit it, because it means you can measure PARTIAL correctness: a candidate that gets 80% of the output bits right across the cycles is measurably closer to a fix than one that gets 20% right, even though both “fail”

. A bare pass/fail would waste all of that. So bringing repair to hardware is not a matter of porting code; it requires reinventing how you localize the fault and how you score a candidate, precisely so the rich waveform information becomes the gradient that guides the search. Do that, and an entire proven paradigm crosses the domain boundary.

What people did before, and where it broke

Software automated program repair (GenProg, Angelix). Mature and effective on software (GenProg repairs about 52% of its benchmark, Angelix about 34%), but built on sequential execution, statement-coverage fault localization, and pass/fail tests — none of which hold for parallel, waveform-producing hardware.
Directly reusing spectrum-based localization on HDL. Rank Verilog statements by execution frequency in failing tests. Fails because every always-block executes every cycle, so frequency does not discriminate the faulty statement from the rest.
Pass/fail testbench evaluation. Treat a hardware testbench as a binary oracle. Throws away the per-cycle, per-bit waveform information, leaving the search a flat landscape with no gradient — a candidate one bit from correct looks no better than one that is entirely wrong.
Manual hardware debugging and formal methods. Expert debugging is accurate but slow and expensive; formal verification finds bugs but does not automatically REPAIR the source. Neither gives the mutate-and-test automation software enjoys.
The gap. No system automatically repaired defects at the Verilog source level, because the localization and fitness that drive repair had never been rebuilt for hardware’s parallelism and waveform oracle. CirFix does exactly that, and shows the software paradigm transfers once its domain-specific parts are replaced.

How CirFix works

Localize by dataflow, mutate the AST, score the waveform, evolve, minimize

Parse the Verilog to an AST

PyVerilog converts the design into an abstract syntax tree — the object the repair process will mutate and, after each edit, regenerate back into Verilog to simulate. This is the domain-general skeleton borrowed from software repair.
↓ →
Localize the fault by dataflow, not coverage new

Simulate the buggy design, compare each output wire to its expected waveform, and put every mismatched signal into a "mismatch set." Then iterate to a fixed point: implicate any assignment that writes a mismatched wire, implicate any conditional that reads one, and transitively add the identifiers those depend on. The result is a precise SET of suspect AST nodes — not a ranked list, because parallelism makes all implicated statements equally suspect.
↓ →
Mutate the AST with hardware-aware operators new

Apply genetic operators drawn from the suspect set: REPLACE a suspect node with another of a compatible type, INSERT a statement into an always/initial block, or DELETE a node. Constraining replacements to the same grammar type (fix localization by Verilog BNF) cuts candidates that fail to compile from 35% down to 10%.
↓ →
Try nine hardware repair templates new

Alongside random mutation, apply templates that encode common HDL bug fixes: negate a conditional, change a sensitivity-list trigger (rising edge to falling edge, or to any-change), swap a blocking assignment for a non-blocking one, or increment/decrement a numeric literal. These bake hardware-engineer knowledge into the search so it reaches likely fixes fast.
↓ →
Score each candidate on the waveform new

Regenerate Verilog, simulate it against an instrumented testbench that records every output wire and register at every clock cycle, and compute a continuous fitness in [0,1]: add one for each matching bit, subtract one for each mismatch, and penalize unknown or high-impedance values twice as hard, then normalize. This partial-credit score gives the search a smooth gradient toward a fix instead of a flat pass/fail.
↓ →
Evolve, then minimize new

Run a genetic algorithm — population 5000, tournament selection, crossover, 5% elitism — breeding higher-fitness candidates over generations until one reaches fitness 1.0 or the budget (eight generations / twelve hours) runs out. Finally, delta-debugging strips any extraneous edits, yielding a minimal patch (often, unusually for repair, a MULTI-edit one).

Evaluated on 11 hardware projects (9,770 lines of design plus 2,923 of testbench) from decoders and counters to an i2c controller, SHA3 hash, Reed-Solomon decoder, and SDRAM controller, with 32 defects transplanted by three hardware experts. Simulation via Synopsys VCS; over 90% of repair time is spent simulating candidates.

The idea

Port a whole paradigm by rebuilding only the parts that touch the domain. Automated program repair is a general loop — localize, mutate, score, evolve — wrapped around two pieces that are deeply domain-specific: how you guess where the bug is, and how you tell a candidate is getting warmer. CirFix’s insight is that the general loop transfers to hardware unchanged, but those two pieces must be reinvented, because they encode assumptions (sequential execution, pass/fail tests) that hardware violates. So it keeps the evolutionary search and the AST-mutation machinery and replaces fault localization and fitness with hardware-native versions. This is a discipline the site rewards — identify what is truly general versus what is domain-bound, and re-implement only the latter — and it is why CirFix could bring a decade of software repair progress across a boundary everyone assumed it could not cross.

Turn a pass/fail verdict into a continuous gradient the search can climb. The pivotal design choice is the fitness function. Evolutionary search is only as good as its objective: if every non-solution scores identically (all “fail”), the landscape is flat and the search is a blind random walk. Hardware’s waveform oracle is the escape, because correctness is measured bit-by-bit at every cycle, so CirFix can award PARTIAL credit — a candidate right on 80% of output bits scores far above one right on 20%, and edits that help register as fitness gains. This strong fitness-distance correlation is what lets the algorithm converge incrementally, even assembling MULTI-edit repairs by climbing 0.58 to 0.77 to 1.0 as edits accumulate. It is the reward-shaping lesson in a systems setting: build an objective that is smooth in the direction of the answer, and a dumb search becomes powerful. A binary oracle would have left the same search helpless.

Measure and localize against the domain’s true observable. Software localizes by statement coverage because statements are what execute; hardware’s observables are wire and register values over time, so CirFix localizes by DATAFLOW — start from the signals that are wrong and walk backward through the assignments and conditionals that produced them. And it measures correctness on those same signals across cycles. Because the design is parallel, there is no meaningful order to rank suspects by, so it returns a set rather than a ranked list, a direct consequence of the domain’s structure. This is the match-your-instruments-to-the-real-signal discipline the site sees when a scheduler measures the contention that actually occurs rather than an isolated proxy: localize and score against what the hardware genuinely exposes, not against a software abstraction it does not obey.

Repairing a faulty 4-bit counter

From a wrong waveform to an evolved fix

A 4-bit counter whose overflow output is wrong. Numbers from CirFix's evaluation.

Step 1 of 5

The bug shows up as a wrong waveform, not a failed test

The counter's overflow output holds the wrong value, and the testbench reveals it not as a bare "fail" but as a detailed disagreement: on 17 of 26 clock cycles the overflow signal does not match the expected waveform. CirFix turns that into a fitness score — with mismatches and penalties, this faulty design scores about 0.58 out of 1.0, already telling the search how wrong it is.

a pass/fail oracle would report only "wrong" and give the search nothing to climb
the waveform comparison is per-bit and per-cycle, so partial correctness is measurable

Why it works

Hardware executes in parallel, so statement-coverage localization cannot discriminate the fault → CirFix localizes by dataflow, walking backward from wrong signals through the assignments and conditionals that produced them → a precise suspect set without any execution ranking.
A testbench emits a waveform, not a pass/fail → scoring a candidate by matching output bits at every cycle yields a continuous fitness with strong fitness-distance correlation → the genetic search gets a gradient and can converge incrementally.
Random edits mostly fail to compile → constraining mutations to grammar-compatible types (fix localization by Verilog BNF) → the fraction of non-compiling candidates drops from 35% to 10%, so the search spends its budget on viable programs.
Common HDL bugs follow known patterns → nine repair templates encode the fixes engineers apply by hand (negate conditional, flip clock edge, swap blocking/non-blocking) → the search reaches likely repairs quickly instead of rediscovering them.
The smooth fitness lets partial progress accumulate → so CirFix can build MULTI-edit repairs by climbing through intermediate scores → 7 of 21 minimized fixes needed several coordinated edits, beyond what pass/fail search could assemble.

What it costs

Timing bugs are out of scope. CirFix relies on functional testbenches, so defects that manifest as timing violations or clock-relationship errors — which need waveform-timing analysis a functional testbench does not perform — are not addressed.
It needs an oracle of correct behavior. Fitness requires the expected waveform, obtained from a previous correct design or manual annotation; if neither exists, CirFix cannot be applied. Quality degrades gracefully (correct repairs fall from 16 to 10 when expected behavior is cut to 25%), but the oracle is essential.
A limited operator and template vocabulary. It cannot make edits outside its repertoire — it cannot widen a register from 8 to 12 bits, define a new operator, or fix module-instantiation errors — so defects needing those changes are unreachable.
Plausible is not always correct. Of 21 plausible repairs, 5 overfit to the testbench (pass it but are semantically wrong), the standard automated-repair risk that a patch satisfies the tests without truly fixing the bug.
Slow, and simulation-bound. Average repair takes 2.03 hours with over 90% spent simulating candidates, longer than an expert fix in wall-clock terms; the authors argue machine time is cheaper than developer time, and repairs are validated only in simulation, not for synthesizability.

The evaluation

Eleven hardware projects (9,770 lines of design, 2,923 of testbench) spanning simple circuits (decoder, 4-bit counter, T flip-flop, multiplexer) to complex ones (i2c controller, SHA3 hash, Tate pairing, Reed-Solomon decoder, SDRAM controller), with 32 defects transplanted by three hardware experts, repaired with a 12-hour budget each and compared against an informal brute-force baseline.

Why software repair needed rebuilding for hardware

Property	Software repair assumes	Hardware reality	CirFix's answer
Execution model
Fault localization
Test signal
Mutations
Repair rate

CirFix keeps software repair's general loop (evolve edits to an AST) and rebuilds only the two domain-bound pieces — fault localization and fitness. Repair quality degrades gracefully as the expected-behavior oracle is thinned (16 correct at full behavior, 10 at 25%), and the instrumented testbench even exposed a reset bug the original testbench missed.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

CirFix’s reach is bounded by its ingredients. It repairs functional defects, so timing and clock-relationship bugs — which a functional testbench cannot even observe — are out of scope. It needs an oracle of correct behavior (a prior design or manual annotation); it degrades gracefully as that oracle is thinned but cannot run without one. Its edit vocabulary is finite, so repairs requiring a new operator, a wider register, or a module-instantiation fix are simply unreachable, and among plausible repairs some overfit the testbench (5 of 21) — the perennial automated-repair caveat that passing the tests is not the same as being correct. It is simulation-bound and slow (2.03 hours average, over 90% in simulation) and validates only functional simulation, not synthesizability. And the 32 defects were transplanted by experts rather than harvested from industrial bug trackers, a threat to external validity the authors mitigate with diverse projects and defect types. None of this dulls the contribution — bringing automated program repair to hardware for the first time by identifying that the general repair loop transfers while fault localization and the fitness signal must be rebuilt for a parallel, waveform-producing domain, and achieving a repair rate on par with mature software tools.

Background

Cross a domain by rebuilding the domain-bound parts, and shape a gradient the search can climb

CirFix sits in the site’s verification-and-repair cluster and rhymes with several ideas covered elsewhere on this site. Its central move is to port a whole paradigm across a domain boundary by separating what is general (the evolve-edits-to-an-AST loop) from what is domain-bound (fault localization and the fitness signal) and rebuilding only the latter — the same disciplined transplant the site admires when a systems problem is recast onto a solved model, except here the transplant runs the other way, carrying software repair into hardware. Its most transferable lesson is about search design: an evolutionary or greedy search is only as good as the smoothness of its objective, so turning a flat pass/fail into a continuous, partial-credit waveform fitness is what converts a blind walk into a climb — the reward-shaping insight that echoes the noisy-quantum page building a corroboration signal out of structure and the optimal-inlining page measuring binary size directly rather than trusting a proxy. Indeed CirFix shares the optimal-inlining page’s generate-and-test-against-the-real-tool stance: both refuse a surrogate and let the real toolchain judge each candidate — compile and measure there, simulate and score here. And its dataflow localization and per-cycle scoring are the match-your-instruments- to-the-true-observable discipline the PCIe-collectives page applies when it measures the contention that actually occurs: localize and score against what the hardware genuinely exposes (wires over cycles), not against a software abstraction it does not obey.

If you remember only three things

1. Automated program repair fixes software bugs by mutating code and re-running tests, but it could not be applied to hardware, because Verilog violates its premises: designs execute in PARALLEL (every block every cycle) so statement-coverage fault localization is meaningless, only wires and registers are observable so there is no execution trace, and a testbench produces a WAVEFORM of values over time rather than a pass/fail — so the two things that steer repair, fault localization and the fitness signal, both had to be reinvented.
2. CirFix keeps the domain-general repair loop (evolve edits to the syntax tree) and rebuilds the domain-specific parts: a dataflow fault localizer that walks backward from wrong signals through the assignments and conditionals that produced them (returning a suspect SET, since parallelism makes all equally suspect), and a continuous fitness that scores a candidate by how many output bits match at every clock cycle — turning a flat pass/fail into a smooth gradient the genetic search can climb — plus hardware-aware mutations, nine HDL repair templates, and type-constrained edits that cut non-compiling candidates from 35% to 10%.
3. The result is plausible repairs for 21 of 32 expert-seeded defects and correct repairs for 16 (comparable to GenProg's ~52% and Angelix's ~34% on software), including 7 multi-edit repairs the smooth fitness made reachable, at an average 2.03 hours dominated by simulation — with honest limits that it needs an oracle of correct behavior, cannot fix timing bugs or edits outside its vocabulary, and can overfit the testbench. The durable lessons: cross a domain by rebuilding only its domain-bound pieces, shape an objective that is smooth toward the answer, and measure against the domain's true observable.

Check yourself

Check your understanding

1. Software automated program repair localizes faults by ranking statements on how often they run in failing versus passing tests. Predict why this technique is useless on a Verilog hardware design.
2. CirFix scores each candidate repair with a continuous fitness in [0,1] based on matching output bits at every cycle, rather than a pass/fail verdict. Predict why this continuous score is essential to making the evolutionary search work.
3. CirFix keeps the general repair loop of software tools (evolve edits to an AST) but rebuilds fault localization and the fitness function. Predict why this split — general skeleton, rebuilt domain parts — is the right way to bring repair to a new domain.
4. Among CirFix's 21 plausible repairs, 5 pass the testbench but are semantically wrong (overfit), and repairs are validated only in simulation. Predict what this reveals about the guarantees automated repair can and cannot offer.

Sources

ACM DL: CirFix: Automatically Repairing Defects in Hardware Design Code — ASPLOS 2022 (full text read via the author-hosted University of Michigan PDF; no arXiv preprint).
Artifact (Zenodo, DOI 10.5281/zenodo.5846419) — open-source CirFix implementation and defect benchmark.