Stepwise Debugging for Hardware Accelerators

Domain experts now build custom hardware accelerators from high-level programs, but when one misbehaves they have to debug the compiled circuit by scrolling through a waveform of every wire on every clock cycle — like a debugger that can only step through assembly, with no breakpoints and no link back to their source; this paper (Cornell) builds Cider, a debugger that runs the accelerator at an intermediate level that still remembers the program's control flow, so it can offer real breakpoints, single-stepping, and watchpoints tied to source-level constructs, while faithfully simulating the hardware so it catches the timing and numeric bugs that only appear once code becomes a circuit.

The argument in one glance

Problem

You can write an accelerator in a high-level language, but you must debug it as a raw circuit

High-level programming models let a domain expert describe a specialized hardware accelerator quickly, in something close to ordinary code, and a compiler turns that description into a circuit. But when the accelerator misbehaves, the debugging tools drop the programmer all the way down to the hardware. The mainstream option is register-transfer-level simulation, which records the value of every wire and register on every clock cycle and shows it as a waveform. Scrolling through that waveform is nothing like using a normal software debugger: there is no single-stepping at any granularity coarser than one clock tick, there are no breakpoints, all program state is flattened into anonymous wire signals, and nothing relates the circuit's state back to the high-level source the programmer actually wrote. It is like debugging with a tool that can only step through assembly instructions, never source-level statements. The very control-flow structure that made the source readable has been dissolved into hardware.
What people did

Debug the program in its original software semantics instead

The natural alternative is to debug the program before it becomes hardware, using the source language's own semantics: write an interpreter for it, or compile it to a normal software executable and use a standard debugger like GDB. Now you get breakpoints, single-stepping, and source-level state, all the familiar comforts of software debugging.
Why it fell short

Software-level debugging cannot see the bugs the hardware compiler introduces

Debugging in software semantics finds functional, algorithmic bugs, but it is blind to the bugs that only arise once the program becomes a circuit — and those are common. Turning a high-level description into hardware forces the compiler to introduce concerns that have no counterpart in the source: cycle-level timing, explicit fine-grained parallelism, custom numerical formats like fixed-point, and physical sharing of resources. Each of these can be miscompiled, and bugs pervade even mature commercial high-level-synthesis tools. Some efforts try to map waveform data back to high-level state, but each is a point solution welded to one specific C-based synthesis tool; every new compiler has to build the capability from scratch. So the programmer is stuck between a debugger that sees the hardware but not the program, and one that sees the program but not the hardware.
This paper's idea

Debug at an intermediate level that still remembers the control flow

The reason waveform debugging loses the program is that register-transfer-level code has already dissolved all control flow into state machines and multiplexer logic. Cider debugs one level up, at Calyx, an intermediate language for accelerator compilers that deliberately preserves the imperative control flow — sequences, loops, parallel blocks — from the high-level source, alongside the hardware structure. Cider is an interpreter for Calyx that executes it directly, without lowering to a circuit first, so it can offer software-like breakpoints, single-stepping, watchpoints, and state inspection keyed to the program's own logical steps, and map them back to source lines. Because it still faithfully simulates the hardware semantics, it catches the timing, overflow, and memory bugs that software-level debugging misses, and by skipping the inactive parts of the design it runs competitively with real hardware simulators.

Why this problem is worth solving

The gap is not that accelerators are hard to write, but that when one is wrong there is no good way to ask why.

The standard way to debug a hardware design is register-transfer-level simulation, which computes and records the value of every signal in the circuit on every clock cycle, then presents it as a waveform the designer scrolls, zooms, and pans through in linear time. This is exhaustive and faithful to the hardware, but it reflects the circuit’s semantics, not the program’s, so it is poorly suited to bugs that originate in the high-level language. It makes bugs hard to localize in time, because the only notion of a time step is a clock cycle, not a logical step of the program; and hard to localize in space, because at any given cycle every signal is potentially relevant and nothing marks which ones matter. There are no breakpoints, no logical single-stepping, and no mapping from a wire back to the source variable it came from. The experience is like a software debugger that can only step through assembly, never source statements. For a domain expert who wrote their accelerator in a high-level language, this is a debugging tool that has forgotten the very thing they wrote, and reconstructing the program’s logic from the waveform is exactly the work the high-level language was supposed to save.

The other option avoids the hardware entirely, which is precisely why it fails to find hardware bugs.

Instead of simulating the circuit, a designer can run the program in the source language’s own semantics — through an interpreter or by compiling to a normal executable and using a software debugger — which restores breakpoints, stepping, and source-level state. But this only exercises what the program means as software, and hardware compilation introduces a whole layer of concerns that software execution never sees: it must assign cycle-level timing, expose fine-grained parallelism, choose custom numerical representations such as fixed-point with a fixed number of integer and fraction bits, and share physical resources among logically separate operations. Any of these can be introduced incorrectly, and in practice compilers are frequently unfaithful to the source semantics — bugs pervade even popular commercial high-level-synthesis tools. So a functional bug and a hardware-compilation bug look completely different, and only the hardware-aware view can catch the latter. Prior attempts to bridge this by mapping waveform data back to high-level state exist, but they are point solutions tied to individual C-based synthesis flows, not reusable infrastructure. Closing this gap generally, for the many languages that compile to hardware, is what Cider is after.

What people did before, and where it broke

Register-transfer-level waveform simulation. Simulate the compiled circuit and record every signal on every clock cycle for the designer to scroll through. Where it broke. No breakpoints, no logical stepping coarser than a clock cycle, all state flattened into wires, and no link back to the high-level source, so bugs are hard to localize in time or space.
Debug in the source language’s software semantics. Interpret the program, or compile it to software and use a standard debugger, before making hardware. Where it broke. Finds only functional bugs; it is blind to the bugs introduced by compiling to hardware — timing, parallelism, custom number formats, resource sharing — which are common even in mature tools.
Raise post-compilation debugging to rule-based hardware languages (for example Cuttlesim). Improve register-transfer-level debugging by lifting it to the abstraction of a hardware design language. Where it broke. These tools are built for hardware designers describing arbitrary circuits, not for the high-level, domain-specific accelerator languages a domain expert uses, so the source-level gap remains.
Map waveform data back to high-level state. Reconstruct high-level program state from the circuit’s signals to relate bugs to the source. Where it broke. Each such effort is a point solution specific to one C-based synthesis tool; every new compiler must rebuild the capability from scratch, so there is no general infrastructure.
The gap. No reusable debugger gave software-like, source-level abstractions — breakpoints, stepping, watchpoints tied to logical program steps — while still faithfully simulating hardware so it catches hardware-compilation bugs, across the many languages that compile to accelerators. Building that on a control-preserving intermediate language is what Cider contributes.

How Cider works

Interpreting a control-preserving intermediate language to debug accelerators

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Debug where the control flow still exists, not after it has been dissolved into wires. Cider’s central move is to pick the right level of abstraction to debug at. Register-transfer-level code is too low: by the time a compiler has emitted it, the program’s loops and branches have become state machines, multiplexer signaling, and unstructured hardware, and no debugger can recover the logical steps a programmer thinks in. Calyx sits one level up and deliberately keeps the imperative control operators — sequence, while, parallel — from the source, so the structure of the computation is still explicit. By interpreting Calyx directly and attaching source-position metadata, Cider can offer breakpoints and stepping keyed to logical program steps and then relate them back to source lines. Preserving a faithful correspondence between a program’s source-level structure and the lower form it is compiled into, so that tools can map one back to the other, is the same guarantee the work on compilation consistency modulo debug information covered elsewhere on this site pursues, keeping the thread from source to compiled code intact so debugging stays meaningful.

Faithfully simulate the hardware, so the hardware’s bugs are actually caught. A debugger that only reasoned about control would miss the point, because the bugs that matter here are the ones the hardware introduces. So underneath the control simulator, Cider runs a structural simulator that executes each group exactly like classic hardware simulation — an inner loop settling combinational assignments until they stop changing, an outer loop advancing sequential register state — and it enforces Calyx’s semantics with a fail-stop policy, turning silent undefined behavior into named, timely errors: an integer overflow reported against the specific adder, an out-of-bounds memory index flagged rather than silently wrapped, a data race between parallel arms detected instead of resolved arbitrarily. Executing a hardware design cycle-accurately, at the level of registers and settling wires, is the same core task the work on parallel register-transfer-level simulation covered elsewhere on this site is built around, and Cider reuses that classic machinery as its engine while wrapping it in control.

Aim the whole thing at finding real bugs, cheaply enough to use while iterating. The payoff is a debugger that surfaces concrete, previously-unreported bugs in real accelerator generators — a bounded loop miscompiled into an unbounded one, a counter register too narrow to hold its range, a fixed-point constant translated off by a shift, a parallel write that races — and localizes each back to a source construct. And because it exploits the control information to simulate only the parts of the design that are active, rather than every wire every cycle, it runs fast enough to be practical, finishing simulation before a compiled simulator has even finished compiling. Hunting down and pinpointing defects in hardware design code is the same objective the work on fast symbolic repair of hardware designs covered elsewhere on this site takes up, and Cider contributes the interactive, source-level front half of that story: seeing clearly where and when the bug happens.

Following a bug through Cider

Localizing a fixed-point miscompilation in a counter accelerator

A trace, following the paper's running example, of how Cider finds a bug that register-transfer-level simulators silently pass over.

Step 1 of 5

A high-level for-loop misbehaves as hardware

A small program adds four times a step value to its input using a bounded for-loop. Compiled to a circuit and run in a register-transfer-level simulator, it either loops forever or produces a wrong result, with no indication why — the waveform shows only wires changing over cycles.

The source has a single bounded loop, so an infinite loop already signals a compilation bug, but the waveform cannot say where.

Why it works

The intermediate language kept what register-transfer code threw away. Once a design is lowered to wires, its loops and branches are gone, encoded as state machines → but Calyx retains explicit imperative control operators alongside the hardware structure → so Cider has real logical steps to break on, step over, and map to source, which no waveform can offer.
Groups give a natural, larger-than-cycle unit of time and space. A clock cycle is the only step a register-transfer simulator knows, and every signal is equally relevant → but Calyx groups are self-contained units of work with a done signal → so Cider can single-step logical chunks and inspect only the cells active at a computationally relevant moment.
Faithful structural simulation preserves the hardware’s real behavior. Debugging in software semantics would hide timing and numeric bugs → but Cider’s engine settles combinational assignments and advances registers exactly like classic hardware simulation → so the bugs that only appear as hardware still appear under Cider.
Fail-stop on undefined behavior turns silent corruption into a named error. Verilog silently wraps overflow and skips non-power-of-two bounds checks → but Cider implements Calyx’s semantics and stops on overflow, out-of-bounds access, and races, naming the offending cell → so subtle bugs surface immediately and localized, instead of propagating into a wrong final value.
Exploiting control lets it skip the inactive design and stay fast. A lowered circuit must be simulated in full every cycle → but Cider knows from the control program which groups are active and simulates only those → so high-level simulation runs about 20 times faster than simulating the equivalent lowered design, keeping the debugger practical.

What it costs (trade-offs and scope)

It is tied to the Calyx ecosystem. Cider debugs languages that compile to Calyx; it is not a general debugger for arbitrary hardware description languages or existing register-transfer-level codebases. Its reach grows only as more frontends and flows (including a Calyx dialect in a mainstream compiler framework) target Calyx.
Source-level debugging depends on frontends adding metadata. End users get native, source-mapped debugging only once a compiler is modified to attach position metadata to the intermediate code; for frontends whose input has no natural source positions, such as a systolic-array generator, that metadata must be synthesized by hand.
The simulator is deliberately basic and not the fastest. Cider’s structural engine uses a straightforward form of classic simulation and is largely unoptimized; it beats an interpreter-based Verilog simulator but is 2.9 times slower than a compiled one, and on the largest design studied a compiled simulator’s up-front compilation pays off and pulls ahead by 3.4 times.
It defines an informal semantics with concrete choices. The only prior Calyx implementation was a compiler that resolved undefined behavior arbitrarily for efficiency; Cider had to distill an informal semantics and make concrete decisions about ambiguities, especially the timing of parallel blocks, so its behavior reflects those particular choices rather than a formally settled standard.
It debugs down to Calyx, not through the final lowering. Cider catches bugs at and above the Calyx level and helps localize compiler bugs, but the final Calyx-to-register-transfer lowering is a step it does not itself execute, so faults introduced purely in that last translation are outside its direct simulation.

The evaluation

Debugging a high-level accelerator: three levels of visibility

Property	RTL waveform simulation	Software-semantics debugging
Debugging abstractions
Link to high-level source
Catches hardware-compilation bugs
Speed vs interpreter simulator	Baseline (Icarus Verilog)
Speed vs compiled simulator	Baseline (Verilator)	n/a

From the paper's evaluation. Cider provides software-like, source-mapped debugging while faithfully simulating hardware, catching bugs both other approaches miss. It runs 4.2 times faster than the interpreter-based Icarus Verilog and 2.9 times slower than the compiled Verilator; because it exploits control information, high-level simulation is about 20 times faster than simulating the lowered design (124 times on the largest control program), and it finishes simulating before Verilator finishes compiling (about 53 times faster than that compile phase), making the end-to-end debug loop fast.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The contribution is a genuinely useful reframing: debug accelerators at a control-preserving intermediate language rather than at raw hardware, and you can have software-like breakpoints, stepping, and watchpoints that map to source, without giving up faithful hardware simulation. Cider interprets Calyx directly, keys its debugging abstractions to the program’s groups, and enforces the language’s semantics with a fail-stop policy that names overflows, out-of-bounds accesses, and data races the usual simulators pass over silently. It finds previously-unreported bugs in real accelerator generators and, by simulating only the active design, runs competitively with open-source hardware simulators. Read that within scope. It works only for languages that compile to Calyx, and source-level mapping needs frontends to add position metadata; its simulator is basic and unoptimized, beating an interpreter-based simulator but trailing a compiled one, especially at scale; and it rests on an informal Calyx semantics with concrete choices, notably about parallel timing, and does not itself execute the final lowering to hardware. Within those bounds it is a clean demonstration that the right intermediate abstraction can make hardware debugging feel like software debugging.

Background

Why high-level hardware design created this gap

It helps to see where this problem comes from. For most of computing’s history, building a custom chip meant writing at the register-transfer level by hand, and the tooling — simulators, waveform viewers — grew up to serve that world. But the demand for specialized accelerators, and the arrival of high-level programming models and high-level synthesis, changed who writes hardware: now a domain expert can describe an accelerator in something close to ordinary code and have a compiler generate the circuit. The productivity of that shift is real, but it opened a gap, because the debugging tools never moved up with the languages. The person now writing accelerators thinks in loops and arrays and functions, yet when something breaks they are handed a waveform of anonymous wires, an artifact from the era when the wires were the program. The broader project of raising hardware design to a higher level of abstraction — letting people express what they want computed and having a compiler handle the circuit — is exactly the terrain the work on hierarchical dataflow compilation for high-level synthesis covered elsewhere on this site occupies, generating good hardware from high-level intent. Cider fills in the debugging half of that same shift, so the tools finally speak the language the designer wrote in.

If you remember only three things

1. High-level programming models let domain experts generate custom hardware accelerators quickly, but debugging is stuck at the low level: register-transfer waveform simulation records every signal on every clock cycle with no breakpoints, no logical single-stepping, all state flattened into wires, and no link to the source, while debugging the program in its software semantics finds only functional bugs and is blind to the timing, parallelism, and numeric-format bugs the hardware compiler introduces, which are common even in mature tools.
2. Cider debugs one level up, at Calyx — an intermediate language that preserves the imperative control flow (sequence, while, parallel blocks over named groups) that register-transfer code dissolves into state machines — by interpreting Calyx directly rather than lowering to a circuit; a structural simulator runs each group like classic hardware simulation (settling combinational assignments, then advancing registers), a control simulator walks the logical program with a program counter, and a fail-stop policy turns silent undefined behavior (overflow, out-of-bounds access, parallel data races) into named, localized errors, with debugger commands keyed to groups and mapped back to source lines via frontend metadata.
3. Cider found previously-unreported bugs in real accelerator generators (a racing parallel memory write in the Dahlia compiler, a parallel scheduling bug in a softmax kernel) and localized them to source constructs, and by exploiting control information to simulate only the active design it runs 4.2 times faster than the interpreter-based Icarus Verilog and about 20 times faster than simulating the lowered design, finishing before the compiled Verilator finishes compiling; its scope is the Calyx ecosystem, source mapping requires frontend metadata, its simulator is basic (2.9 times slower than compiled Verilator, more at scale), and it rests on an informal Calyx semantics with concrete choices about parallel timing.

Check yourself

Check your understanding

1. Why is debugging a high-level accelerator with register-transfer-level waveform simulation so painful?
2. Why does debugging the program in its software semantics not solve the problem?
3. What is Cider's key idea, and how does its interpreter work?
4. What did Cider achieve, and what are its limits?

Sources

The paper is “Stepwise Debugging for Hardware Accelerators” by Griffin Berlstein, Rachit Nigam, Christophe Gyurgyik, and Adrian Sampson (Cornell University), published at ASPLOS 2023 (Volume 2), DOI 10.1145/3575693.3575717.

This explainer is written from the full text of the authors’ open-access version. Everything above comes from the paper’s own sections: the introduction and background on the two mainstream debugging options and their failures — register-transfer-level waveform simulation, which records every signal on every clock cycle with no breakpoints, no logical stepping, flattened state, and no source mapping, and software-semantics debugging, which finds functional bugs but is blind to the timing, parallelism, custom-numerical-format, and resource-sharing bugs that hardware compilation introduces (common even in mature commercial tools); the description of Calyx as an intermediate language that combines hardware dataflow with explicit imperative control operators (sequence, while, if, parallel, invoke) over named groups, preserving control information that register-transfer code loses; the design of the Cider interpreter, including its fail-stop policy for undefined behavior and the informal Calyx semantics it required, the structural simulator (an inner stabilization loop settling simultaneous group assignments and an outer loop advancing sequential state until a group’s done signal), the control simulator (interpreting the control program, with parallel blocks run as copied “parallel universe” environments merged with race detection), the tree-structured environment model with a program counter over the control schedule, and the detection of overflow, out-of-bounds memory access, parallel data races, and intra-group driver conflicts that register-transfer simulators handle silently; the debugger commands (break, step, step-over, watch, continue, print with format codes, where) keyed to groups, and the flexible source-level mapping via frontend position metadata demonstrated on Dahlia and a systolic-array generator; the case studies finding previously-unreported bugs, including a parallel memory-write race in the Dahlia compiler (reported upstream) and a parallel scheduling bug in a softmax kernel; and the evaluation reporting Cider at 4.2 times faster than Icarus Verilog and 2.9 times slower than Verilator, high-level simulation about 20 times (up to 124 times) faster than simulating the lowered design, Cider finishing simulation before Verilator finishes compiling (about 53 times faster than that compile phase), and the LeNet scalability study (3,598 lines of Calyx across 17 components) where Cider is 8.3 times faster than Icarus Verilog while Verilator is 3.4 times faster than Cider.