Debugging in the brave new world of reconfigurable hardware

Jiacheng Ma, Gefei Zuo, Kevin Loughlin, Haoyang Zhang, Andrew Quinn and Baris Kasikci

Chip bugs used to be catastrophic — once silicon is manufactured a mistake is ruinously expensive and sometimes permanent, which is why traditional hardware development spends enormous effort proving designs correct before anything is built; reconfigurable chips changed that economics completely, because their logic can be rewritten after deployment, so developers began working the way programmers do, testing quickly and patching later, and deliberately doing far less verification; the trouble is that the tools never followed, since everything built for hardware assumes you are trying to prove a design correct beforehand rather than work out why the thing in front of you is misbehaving, leaving developers to stare at recordings of every wire in the circuit; this paper (Michigan) collects and reproduces 68 real bugs from open-source designs and shows that they sort into three families that mirror the classic families of SOFTWARE bugs almost exactly, which licenses importing software debugging techniques — then builds five such tools, including something as basic and as absent as a print statement.

The argument in one glance

Problem

Hardware development became agile, and its debugging tools did not

A bug found after a chip is manufactured is ruinously expensive and sometimes impossible to fix, which is why traditional hardware development pours resources into simulation and formal verification before any silicon exists. RECONFIGURABLE chips overturn that. Their logic can be rewritten at any time, even in production, so a bug is a patch rather than a catastrophe — and developers have responded rationally by working the way programmers do, testing quickly and fixing later while deliberately doing far less verification. One major company has publicly adopted exactly this methodology. But relaxed verification means more bugs escape into production, and here the analogy breaks, because the tooling never followed. Nearly every hardware tool that exists is built to PROVE a design correct in advance, not to explain why the thing in front of you is misbehaving.
What people did

Record the circuit and look at it

What developers actually do is observe. They capture a WAVEFORM — a recording of the value of every wire in the circuit at every clock cycle — and read through it looking for the moment things went wrong. Or, since capturing everything is usually impossible, they choose a handful of signals they suspect, re-synthesize the design to record those, run it again, and repeat. Both are entirely general and require no assumptions about the bug.
Why it fell short

That is looking at everything, which is nearly the same as looking at nothing

A waveform contains every wire at every cycle with no indication of which matters, so extracting a cause is a manual search through an enormous undifferentiated recording. The alternative, choosing signals and re-synthesizing, replaces the search with a guessing game where each guess costs a full synthesis run. Existing automated fault localization covers only specific protocols and algorithms; checkpointing and tracing help but still demand substantial manual work; and software techniques such as data-race detectors cannot simply be pointed at a circuit, because the programming model is not the same. Unsurprisingly, most developers in a recent survey said they need better debugging tools — and the field is immature enough that among the 50 most popular open-source projects, 56 percent have no public bug tracker and 88 percent ship no test cases to reproduce a bug.
This paper's idea

Study the bugs, notice they mirror software bugs, and import the tools

This paper (Michigan, with Santa Cruz — Ma, Zuo, Loughlin, Zhang, Quinn, Kasikci) argues you cannot build the right tools without first knowing what you are debugging, so it does the study nobody had done. The authors collect 68 real bugs from open-source designs — mining commit histories and talking to developers, since bug trackers largely do not exist — reproduce them reliably, and classify them by root cause into three families. The finding that unlocks everything is that those three families correspond almost exactly to the three classic families of SOFTWARE bugs: data mis-access mirrors memory bugs, communication mirrors concurrency bugs, and semantic mirrors semantic. That correspondence is what licenses importing decades of software debugging technique into a domain that had none of it. They then build five tools on that basis, beginning with the most elementary thing hardware lacked — a print statement — and show that 18 of 20 instrumented designs still meet their original timing.

Why this problem is worth solving

The shift that created this problem is real and recent, so it is worth being precise about what changed.

A FIELD PROGRAMMABLE GATE ARRAY is a chip whose logic is not fixed at manufacture but configured afterwards, and can be reconfigured any number of times. Developers describe a circuit in a HARDWARE DESCRIPTION LANGUAGE, which specifies what happens on every clock cycle, and that description is SYNTHESIZED into a configuration the chip adopts. The consequence is economic rather than technical. In conventional chip design, a bug found after fabrication is ruinously expensive and occasionally unfixable, which justifies enormous investment in simulation and formal verification before anything is built — correctness has to be established in advance because there is no second chance. Reconfigurable hardware removes that finality, since the logic can be rewritten even in production, so a bug becomes a patch. Developers have reacted by adopting a software-like cycle, doing far less verification in exchange for faster iteration, and at least one major company has publicly said it works this way. The reasoning is sound, and it means more bugs now escape into production by design — which is fine only if you can find them afterwards

. That last condition is the one nobody had provided for.

Finding a bug afterwards in hardware is a different activity from finding one in software.

Debugging happens in two settings. SIMULATION runs the design in software, which avoids the long synthesis step and so iterates quickly, but executes orders of magnitude slower than real hardware and therefore cannot reach the complex workloads where many bugs appear. ON-FPGA testing runs at full speed but gives up nearly all visibility. What visibility exists takes the form of a WAVEFORM, a recording of every wire in the circuit at every clock cycle, which is comprehensive and almost unusable — it shows everything with no indication of what matters. Because on-chip storage is tiny, recording everything for long is impossible, so developers instead pick a few signals they suspect, re-synthesize the design to capture those, and run again, with every guess costing a full synthesis. Software techniques do not simply transfer, because a circuit has no call stack, no heap, and no notion of one instruction following another — everything happens everywhere, every cycle. And crucially, any debugging logic added becomes real circuitry that consumes area and can break the design’s timing, so an instrument that works in simulation may not fit on the chip

What people did before, and where it broke

Simulation-based testing and formal verification. The mainstay of traditional hardware development, and very effective at proving properties before fabrication. But they are built to establish correctness in advance, not to explain an observed failure — and the whole premise of the new development style is doing less of them.
Reading waveforms. Fully general and assumption-free, but a recording of every wire at every cycle offers no guidance about which parts matter, so finding a cause is an unaided manual search.
Iterative signal selection. Choosing suspect signals, re-synthesizing to record them, and running again. This narrows the recording to something manageable, but converts debugging into a guessing game in which each guess costs a full synthesis run.
Protocol-specific fault localization. Automated tools do exist, but only for particular protocols and algorithms, so they cannot help with an arbitrary design.
Checkpointing and tracing infrastructure. Genuinely useful for capturing what happened, but they supply raw material rather than a diagnosis, and turning that material into a root cause remains substantial manual work.
Software fault-localization tools. Decades of mature technique exists for software, but data-race detectors and uninitialized-memory detectors cannot be pointed at a circuit, because the programming model is fundamentally different.
The gap. Development methodology moved to a software-like cycle while the instruments stayed in the verification era, and nobody had established what hardware bugs actually look like — which is the prerequisite for building anything better. That is the gap this paper fills.

How a study of real bugs turns into a toolkit

Collect the bugs, classify them, notice the software correspondence, then build

Debugging is manual because nothing localizes causes

Reconfigurability let developers relax verification and patch later, so more bugs reach production, but hardware tooling proves designs correct in advance rather than diagnosing observed failures. Developers are left reading recordings of every wire, or guessing signals and re-synthesizing.
↓ →
Collect real bugs, the hard way new

Since 56 percent of the most popular open-source projects have no public bug tracker and 88 percent ship no reproducing test cases, bugs are recovered by mining commit histories and issue discussions, reading the patches, and in some cases asking developers directly — across a hypervisor, CPUs, a network card, a WiFi controller and more.
↓ →
Make each one reproduce reliably new

Every collected bug is rebuilt into a testbed where it can be triggered on demand, which is what makes it possible to evaluate whether a proposed tool actually helps, and is released so others can use it.
↓ →
Classify by root cause, and find the correspondence new

The 68 bugs sort into three families — data mis-access, communication, and semantic — which turn out to correspond almost exactly to the three classic families of software bugs, namely memory, concurrency and semantic. That correspondence is the licence to import software debugging technique.
↓ →
Build the missing instruments new

Five tools follow from the symptoms observed. The foundation is a print statement for hardware, which compiles into recording logic that works identically in simulation and on the chip. On top of it sit monitors for state machines, for the provenance of a value, and for event statistics, plus a checker that localizes where data is being lost.
↓ →
Make sure the instruments still fit new

Because debugging logic becomes real circuitry, the tools are evaluated on whether an instrumented design still meets its timing and how much area it costs — 18 of 20 designs kept their original target frequency, with resource cost growing at worst linearly in the recording buffer size.

Bugs were drawn from open-source designs including an FPGA hypervisor, two RISC-V processors, a graphics processor, a network card, a WiFi controller, a Bitcoin miner and several hardware libraries, and the testbed and tools are released openly.

The idea

Study the bugs before building the tools. The paper’s structure is its argument. Everyone agreed better hardware debugging tools were needed, but nobody had established what hardware bugs actually are — what causes them, how they announce themselves, how they get fixed — so any tool would have been built on intuition. So the first contribution is empirical: 68 real bugs collected, reproduced, and classified by root cause. That the collection was genuinely hard is itself a result, since most popular open-source hardware projects have no bug tracker and ship nothing that reproduces a failure, forcing the authors to mine commit histories and consult developers. Establishing what the problem actually consists of before designing a solution is the same characterize-before-you-build discipline covered elsewhere on this site, where an exhaustive measurement campaign on real devices reveals structure that the existing generic configuration had ignored entirely.

The correspondence with software bugs is what makes the toolkit possible. The central finding is a mapping. Hardware bugs sort into data mis-access, communication, and semantic families, and these line up almost exactly with the classic software families of memory, concurrency, and semantic bugs — a hardware buffer overflow really is a buffer overflow, a producer-consumer mismatch really is the bounded-buffer problem, using a signal before it is valid really is using uninitialized memory. That correspondence is not a decorative analogy; it is the justification for importing decades of software debugging technique into a domain that had almost none, and it explains why the first tool built is a print statement. Recognising that an unfamiliar domain’s problems are a known domain’s problems in disguise is the map-onto-a-solved-problem move covered elsewhere on this site, where an intractable-looking placement search turns out to be a knapsack, and where two unrelated workloads turn out to reduce to the same set operations.

Many hardware bugs are indistinguishable from deliberate technique. The subtlest thread running through the study is that the bug patterns are also legitimate idioms. Truncating a value to fewer bits is a bug when accidental and a standard technique when intended. Dropping data is a fault in one place and correct behaviour in another. Some developers even rely on an index wrapping around past the end of a buffer. This is why the tools localize rather than decide, and it shows up honestly in the results — the data-loss checker reports one false positive by flagging an intentional drop the test case never exercised, and misses one real bug precisely because the loss occurs at a register where dropping is sometimes deliberate. The same ambiguity produces both errors. Accepting that a tool cannot separate intent from accident, and building it to narrow the search for a human rather than to render a verdict, is a discipline the site rewards wherever the honest answer is to assist judgement instead of replacing it.

Diagnosing a hang in an image accelerator

From a frozen application to the responsible logic

The paper's own case study: an image-conversion accelerator that hangs. Numbers from the evaluation.

Step 1 of 5

The symptom

An accelerator converts images to grayscale. Software on the processor loads an image, hands it to the accelerator, and waits for the result written back to memory. Sometimes the task simply never completes. All the software can report is that it hung — there is no stack to inspect and no error.

the underlying fault is a buffer overflow, but nothing about the symptom indicates that
recording every wire in the circuit would capture the answer somewhere inside an enormous undifferentiated trace

Why it works

Reconfigurability lets bugs be patched after deployment → developers rationally relax verification in exchange for speed → so more bugs reach production and the need shifts from proving correctness beforehand to localizing causes afterwards.
Nobody knew what hardware bugs consist of → so collecting, reproducing and classifying 68 real ones comes first → giving tool design an empirical basis rather than intuition.
The three hardware bug families correspond to the three software families → mature software debugging techniques become applicable → which is why the foundational tool is a print statement, absent from hardware but universal in software.
Bugs surface in simulation and on the chip, and reproducing them in the slower setting is often impossible → one recording mechanism serving both means a tool can be applied wherever the bug appears → without maintaining two implementations.
Debugging logic becomes real circuitry that consumes area and affects timing → the tools must be evaluated on whether instrumented designs still build → and 18 of 20 keeping their target frequency is what makes them usable rather than merely interesting.

What it costs (trade-offs and scope)

Instrumentation is real circuitry and sometimes does not fit. Two of 20 evaluated designs failed to keep their original target frequency once instrumented. Observing a circuit at full speed costs area and timing in a way that has no equivalent in a simulator.
The tools localize rather than decide. Because bug patterns coincide with legitimate idioms, the data-loss checker produced one false positive and missed one genuine bug, both from the same inability to distinguish intentional drops from accidental ones.
The study is bounded by what could be recovered. Bugs came from open-source projects via commit archaeology, since most such projects have no bug tracker, so the sample reflects bugs that were found, fixed, and discussed publicly — and commercial designs, where practices differ, are absent.
The testbed is a subset. Of the bugs studied, 20 were reproduced for evaluation, chosen because the authors were familiar with those platforms, with the rest reproducible only with further effort.
It targets hardware description languages specifically. The work addresses designs written directly in circuit-describing languages, which the paper notes still dominate because higher-level synthesis costs roughly six times the performance and five times the resources.

The evaluation

The study collected 68 bugs from open-source hardware — an FPGA hypervisor, two RISC-V processors, a graphics processor, a network card, a WiFi controller, a Bitcoin miner, an SD-card driver, bus endpoint implementations, a transform implementation and several libraries — plus a floating-point adder contributed directly by a developer. Bugs were recovered from commit histories and issue discussions because bug trackers largely do not exist, then classified by root cause into three classes and thirteen subclasses. Twenty were rebuilt as a reliably reproducible testbed, on which the five tools were evaluated for diagnostic usefulness, generated analysis code, area cost and timing impact across two commercial FPGA platforms.

Finding the cause of a hardware bug

Property	Simulation and formal verification	Waveforms and signal selection	These tools
What it is for
Works on the real chip
Manual effort required
Fits the agile cycle
Distinguishes bug from idiom

The 68 bugs classify into three families — 28 data mis-access, 20 communication and 20 semantic — mirroring the classic software families of memory, concurrency and semantic bugs. The largest single subclasses are bit truncation with 12 instances, signal asynchrony with 10, and erroneous expressions with 10. Across the reproduced testbed the tools generate dozens to thousands of lines of analysis logic per bug, with the data-loss checker alone emitting between 522 and about 19 thousand lines.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

Read the scope of the study before generalising from it. The bugs were recovered by mining commit histories and issue discussions, because most popular open-source hardware projects have no bug tracker at all and 88 percent ship no reproducing test cases — so the sample necessarily consists of bugs that were found, fixed and discussed in public, and commercial designs with different practices are absent entirely. Of the 68 studied, 20 were rebuilt into the reproducible testbed, chosen for platform familiarity rather than representativeness. The tools themselves carry an honest limitation the paper does not hide: because hardware bug patterns coincide with legitimate design idioms, truncation and data dropping being deliberate techniques as often as faults, the data-loss checker both reported a false positive and missed a real bug, in each case because it could not tell intent from accident. Instrumentation is also real circuitry rather than a simulator convenience, and two of the 20 designs failed to keep their original target frequency once instrumented, with area growing at worst linearly in the recording buffer. Finally the work targets designs written directly in circuit-describing languages, which the paper notes still dominate because higher-level synthesis costs roughly six times the performance. Within that scope the lessons travel well: when a field’s working methods change, check whether its instruments changed too; establish what your problem actually consists of before designing a solution for it; and when a tool cannot separate intent from accident, build it to narrow a human’s search rather than to deliver a verdict.

Background

The methodology moved, the instruments did not, and the bugs turned out to be familiar

This paper sits in the site’s hardware-design and correctness cluster and rhymes with several ideas covered elsewhere on this site. Its diagnosis is that a field’s working practice changed — reconfigurability made bugs patchable, so verification was rationally relaxed — while its instruments stayed built for the previous regime, which is the same the-ground-moved recognition the persistency-race page makes when a compiler freedom that was harmless becomes harmful once memory survives power loss. Its response, building the missing instrument rather than arguing about what the answer should be, is the same move the open-hardware graphics page makes when it supplies a complete stack because the question could not otherwise be asked. Its foundational empirical step, collecting and classifying real bugs before designing anything, is the characterize-first discipline the flash-memory page shows when exhaustive measurement of real chips reveals that a vendor-supplied configuration ignored the structure actually present. Most importantly, its central finding that hardware bug families mirror software bug families almost exactly is the map-onto-a-solved-problem move that recurs across the site — where a placement search turns out to be a knapsack, or two unrelated workloads turn out to reduce to the same set operations — and here the payoff is decades of borrowed technique, starting with a print statement. And its candour that the same ambiguity between deliberate idiom and accidental fault produces both its false positive and its one miss is the report-where-it-breaks habit the site rewards, since a tool that admits it narrows rather than decides is more useful than one that overclaims.

If you remember only three things

1. A bug found after a chip is manufactured is ruinously expensive and sometimes unfixable, which is why traditional hardware development invests enormously in verification before anything is built. RECONFIGURABLE chips overturn that economics, since their logic can be rewritten even in production, so developers have rationally adopted a software-like cycle of quick testing and later patching with far less verification. The consequence is that more bugs now reach production by design — but the tooling never followed, because nearly every hardware tool exists to prove a design correct in advance rather than to explain an observed failure. Debugging is therefore manual inspection of recordings of every wire, or guessing signals and re-synthesizing at a full build per guess.
2. The paper argues you cannot build the right tools without knowing what you are debugging, so it collects 68 real bugs from open-source designs — mining commit histories and consulting developers, since 56 percent of the most popular projects have no bug tracker and 88 percent ship nothing that reproduces a failure — makes each reliably reproducible, and classifies them by root cause. The unlocking finding is that the three resulting families correspond almost exactly to the three classic families of SOFTWARE bugs: data mis-access mirrors memory bugs, communication mirrors concurrency bugs, semantic mirrors semantic. A hardware buffer overflow really is a buffer overflow, and using a signal before it is valid really is using uninitialized memory.
3. That correspondence licenses importing decades of software technique, and the five tools built on it begin with the most elementary thing hardware lacked — a print statement that compiles into recording logic working identically in simulation and on the chip — with monitors for state machines, value provenance and event statistics layered on top, plus a data-loss localizer. Because debugging logic is real circuitry rather than a simulator convenience, the decisive practical result is that 18 of 20 instrumented designs still met their original timing. The transferable lessons: when a field's methods change, check whether its instruments did; establish what the problem consists of before solving it; and when a tool cannot distinguish intent from accident, build it to narrow a search rather than deliver a verdict.

Check yourself

Check your understanding

1. Reconfigurable chips let bugs be patched after deployment, which sounds like unambiguously good news. Predict the problem it created.
2. The paper spends its first half classifying 68 bugs rather than building tools. Predict why that ordering matters.
3. The data-loss checker reports one false positive and misses one real bug. Predict what single underlying difficulty causes both.
4. The evaluation emphasises that 18 of 20 instrumented designs kept their original target frequency. Predict why that metric is the decisive one.

Sources

Debugging in the brave new world of reconfigurable hardware (ACM DL, DOI 10.1145/3503222.3507701) — ASPLOS 2022.
Author PDF — the full 16-page paper, read in full for this explainer.
Reproducible hardware bug testbed (GitHub) — the released collection of reproducible bugs, with the evaluation artifact alongside it.