Simulator Independent Coverage for RTL Hardware Languages

Kevin Laeufer, Vighnesh Iyer, David Biancolin, Jonathan Bachrach, Borivoje Nikolic and Koushik Sen

Measuring how thoroughly a chip's tests exercise its design normally means using whatever coverage metrics the simulator vendor happened to build in, so results cannot be merged across tools and cannot even be mapped back to modern hardware languages like Chisel; this work from UC Berkeley moves coverage out of the simulator and into the compiler — every metric becomes a compiler pass that lowers to one tiny cover primitive, so adding a metric works on every backend and adding a backend unlocks every metric, results merge trivially across software simulators, FPGA emulation, and formal tools, and coverage maps cleanly back to the original source.

The argument in one glance

Problem

Coverage tells you how well your tests exercise a chip

Before a chip is manufactured, it is verified by simulation: engineers write tests that drive the design and check its behavior. But how do you know the tests are thorough? Coverage metrics answer that — line coverage checks every statement ran, toggle coverage that every wire took both values, finite-state-machine coverage that every state and transition was reached. Engineers craft stimuli to hit more and more of these coverpoints, gaining confidence that the test suite has exercised the design and not left dangerous corners untested.
What people did

Use the coverage features built into your simulator

Traditionally coverage is a feature of the simulator itself. You run your design in a simulator that supports line or toggle coverage, turn on its coverage flag, and it reports which parts of the design were exercised. The simulator authors decide which metrics exist and how they are computed, and each simulator ships its own implementation baked into its engine.
Why it fell short

Monolithic coverage cannot be shared, extended, or mapped back

Locking coverage inside each simulator causes three problems. Most open-source and research simulators, and FPGA emulators, offer no automated coverage at all. The ones that do use incompatible custom formats, so results from a fast software run, an FPGA emulation, and a formal tool cannot be merged into one picture. And modern hardware languages like Chisel compile down to Verilog before simulation, so coverage on the generated Verilog does not map back to the source — the compiler may rewrite a branch into a conditional assignment, so full Verilog coverage can still miss lines of the original design.
This paper's idea

Move coverage from the simulator into the compiler

The framework lifts coverage out of the simulator entirely. Each metric becomes a pass in the hardware compiler that rewrites the design in terms of a single tiny primitive — a cover statement that counts how often a signal is true at a clock edge — plus metadata mapping those counts back to source lines or states. A simulator only has to support that one primitive to gain every metric, and a report generator turns the counts into a readable report. Because every backend reports the same simple counts, results merge trivially across software simulators, FPGA emulators, and formal tools.

Why this problem is worth solving

The thing being measured is a stand-in for a question no test suite can ever fully answer.

A coverage metric is a way to measure how thoroughly a set of tests exercises a design, standing in for the impossible ideal of testing every possible behavior. The simplest kinds are structural. Line coverage asks whether every statement the designer wrote actually ran at least once — if a line never executes, either it is dead code or the tests missed it. Toggle coverage asks whether every wire took both a zero and a one value; a wire stuck at a single value is a wire the tests never fully worked. Finite-state-machine coverage goes further, checking that every state and every transition of a control machine was reached. None of these prove the design correct — they measure the test suite, not the chip — but they are the practical yardstick verification engineers use to decide whether they have written enough tests, and to find the corners of a design that no stimulus has yet touched.

The reason mapping coverage back to the source is hard is the way modern hardware is written.

A hardware construction language is a way of describing circuits by writing a program in an ordinary language like Scala that generates the hardware, rather than writing the hardware out directly. Chisel is the leading example: a designer writes Scala code using circuit primitives as library objects, and running that program produces the design. This makes hardware highly reusable and parameterizable — a single generator can stamp out a whole family of chips from a configuration — but it introduces a gap. The Chisel design is not simulated directly; a compiler first lowers it, through an intermediate representation called FIRRTL, into plain structural Verilog that ordinary tools understand. Along the way the compiler rewrites constructs, so a branch in the source may become a conditional assignment in the Verilog. That is exactly why coverage measured on the generated Verilog no longer lines up with the Chisel the designer actually wrote, and why getting source-level coverage needs the compiler’s help.

What people did before, and where it broke

Turn on the simulator’s built-in coverage. Run the design in a simulator that supports line or toggle coverage and let it report what was exercised. Where it broke. You get only the metrics that simulator’s authors chose to implement, and most open-source simulators, research prototypes, and FPGA emulators offer no automated coverage at all.
Use one tool’s custom coverage format. Each simulator that does support coverage reports it in its own way. Where it broke. The formats are incompatible, so results from a fast software run, an FPGA emulation, and a formal tool cannot be merged into a single picture of what the whole verification effort covered.
Measure coverage on the generated Verilog. Since hardware construction languages compile to structural Verilog, just use the Verilog simulator’s coverage on the output. Where it broke. The compiler rewrites the code on the way down — a branch becomes a conditional assignment — so full coverage of the Verilog does not imply full coverage of the source, and design patterns like state machines vanish entirely.
Hand-annotate functional coverpoints. Have engineers manually mark the scenarios they want to see covered. Where it broke. It is labor-intensive and specific to each design, it does not scale to structural metrics across a whole SoC, and it is still tied to whatever simulator will collect it.
The gap. Every path left coverage welded to a single tool: unavailable in most simulators, unmergeable across the ones that had it, and unable to speak the language the designer actually wrote in. What was missing was a way to define a coverage metric once, independently of any backend, collect it identically from a software simulator, an FPGA emulator, or a formal tool, and map the results all the way back to the original source.

How simulator-independent coverage works

Lifting coverage out of the simulator and into the compiler

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Define each metric once against a single primitive, so backends and metrics stop multiplying. The waste in the old world is a many-to-many explosion: every coverage metric has to be reimplemented inside every simulator that wants it, and most simply never bother. The framework breaks that by inserting one thin layer of indirection — a single cover primitive that just counts how often a signal is true at a clock edge. Every metric is written as a compiler pass that lowers to that primitive, and every backend needs to understand only that one primitive; then any metric works on any backend. Collapsing a matrix of tools-times-features into a single shared interface, so that supporting the interface once gives a new tool all the features and adding a feature once gives it to all the tools, is the same move the FPGA-middlebox work covered elsewhere on this site makes, where implementing one small primitive in a new simulator unlocks the entire framework rather than requiring bespoke integration each time.

Instrument the compiler’s own representation so the coverage still points at the source. The reason coverage on generated Verilog is misleading is that the compiler has already rewritten the design, dissolving the branches and state machines the metric cares about. The framework sidesteps this by doing its work inside the compiler, on the intermediate representation, before that lowering happens — so it can see a branch as a branch, insert a cover point right there, and record exactly which lines of the designer’s Scala that point represents. When the counts come back, the report generator walks that metadata to annotate the original source. Reaching across the layers of a compiled system to tie a measurement taken at a low level back to the high-level code a human wrote is the same bridging the cross-level debugging work covered elsewhere on this site performs, connecting what is observed after compilation to the source it came from so a developer can reason about the two together.

Once coverage is a compiler artifact, it can serve tools far beyond simulation. Freeing coverage from the simulator does more than make it portable; it changes what coverage can be used for. Because the cover primitive is emitted as an ordinary assertion, the very same instrumented design a simulator counts can be handed to a formal model checker, which will search for inputs that reach each cover point or prove that none exist — turning a coverage pass into a tool for exploring a design and even for finding bugs in the coverage analysis itself. Building a verification capability as a compiler pass so that a solver can take the same artifact and reason about it exhaustively is the same strategy the compiler-verification work covered elsewhere on this site follows, expressing the check as a compiler-level construct that an automated reasoner can then attack to find a concrete input or a proof.

Following the reasoning through the framework

Measuring line coverage of a two-way branch

Following a tiny Chisel branch through the flow shows how the coverage survives compilation and comes back mapped to the source.

Step 1 of 5

A branch whose two arms we want to check

The design has a Chisel when-otherwise: when the input is set, the output becomes one, otherwise it becomes two. Line coverage should tell us whether the tests ever took each arm — whether both the one-line and the two-line ran at least once.

If only one arm ever fires, a whole behavior of the design is going untested, and structural coverage is meant to surface exactly that.

Why it works

A single cover primitive collapses a many-to-many problem into a many-plus-many one. Baking every metric into every simulator means each of N metrics must be reimplemented for each of M tools → but if each metric compiles down to one shared cover primitive, a metric is written once and a backend supports it once → so adding a metric lights it up on every backend and adding a backend unlocks every metric, taking a fifth tool only about five hours to support.
Instrumenting the compiler’s IR before it lowers preserves the link to the source. Measuring coverage on the generated Verilog loses the mapping, because the compiler has already rewritten branches into other constructs → but inserting cover points in the intermediate representation, while the source structure is still visible, lets the pass record exactly which lines each point stands for → so the counts can be annotated back onto the Chisel the designer actually wrote.
Because every backend reports the same counts, merging across them is free. Custom per-simulator formats make it impossible to combine a software run with an FPGA emulation or a formal result → but a uniform map from cover-point name to count is trivially additive → so coverage from a fast software suite can be merged with, or subtracted from, an expensive FPGA run, cutting the counters that FPGA needs by nearly half.
Emitting the primitive as a standard assertion lets formal tools reuse it unchanged. A coverage counter and a formal cover goal look like different things → but a cover statement is just an assertion that a signal can be true, which a formal tool already knows how to chase → so the same instrumented design a simulator counts, a model checker can use to find inputs that reach a point or prove none exist.
Living in the compiler lets new metrics be added cheaply and even audited. A metric locked inside a simulator is expensive to write and impossible to inspect from outside → but as a compiler pass a new metric like ready-valid coverage took only a few hours to build on the shared machinery → and a formal tool run on the output even exposed a bug in the framework’s own state-machine analysis by proving some reported transitions unreachable.

What it costs (trade-offs and scope)

It is plumbing, not a new capability. The metrics themselves — line, toggle, state-machine — are standard; the contribution is portability, mergeability, and source mapping, demonstrated on one ecosystem, Chisel and FIRRTL. It makes coverage better connected, not fundamentally more powerful.
The “no slowdown” claim is narrow. It holds against Verilator’s built-in coverage on isolated raw simulation, where Verilator internally does something similar. The generic approach does slow event-driven commercial simulators, numbers the authors cannot report; they argue only that Verilator with their coverage still beats commercial tools with native coverage.
FPGA coverage costs real area. Wide, accurate counters inflate resources — logic use rises 2.8 times at 32-bit counters — while the cheap version with tiny counters only tells you covered-or-never-covered, not accurate counts, and the widest counter configuration on the out-of-order core did not even fit on the FPGA.
The state-machine analysis is conservative and can miss. It over-approximates transitions, so it never misses one but can over-report; and a machine whose next-state logic crosses a module boundary is invisible to the module-scoped pass, as happened on a real RocketChip design.
It depends on compiler access and the cover model. The approach needs a hardware compiler with an instrumentable IR; generalization to other intermediate representations and languages is argued but only shown on FIRRTL, and the report generators are bare-bones ASCII rather than polished tooling.
Value-bin coverage blows up. Counting how often a signal lands in each of many mutually exclusive bins with separate cover statements is exponential; doing it efficiently needs a not-yet-implemented cover-values primitive backed by a counter array or block RAM.
Merging leans on identical instrumentation. Filtering FPGA points already caught in software works only because the exact same instrumentation runs on both, and the 42-percent counter saving is only as good as the initial software test suite that pre-covers them.

The evaluation

Where coverage lives, and what that costs

Property	Simulator built-in	On generated Verilog	Manual coverpoints
Where coverage is defined
Metrics available
Merges across software, FPGA, formal
Maps back to the source language	Not applicable
Cost to add a metric or a backend		Not applicable	Per design, by hand

Approaches to RTL coverage, with this work prototyped for Chisel and FIRRTL across five backends (Treadle, Verilator, ESSENT, the FPGA-accelerated FireSim, and the SymbiYosys formal tool). It collects line coverage while booting Linux on FireSim at 65 MHz, adds no measurable overhead over Verilator's native coverage, merges software and FPGA results to cut FPGA counters by 42 percent, and let the authors add a fifth backend in about five hours.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This is a clean, well-executed piece of verification infrastructure: by defining each coverage metric as a compiler pass that lowers to one small cover primitive, it makes coverage portable across five very different backends, mergeable across software, FPGA, and formal tools, and mappable back to the original Chisel — and it does so with little or no simulation overhead. Read it for what it is. The metrics themselves are standard; the value is the plumbing, shown on one compiler ecosystem. The zero-overhead result is against one open-source simulator’s built-in coverage on isolated raw runs, and the generic approach does slow event-driven commercial simulators. On the FPGA, accurate wide counters cost real area, so the cheap mode only distinguishes covered from never-covered, and the largest configuration did not fit. The state-machine analysis is conservative and can miss machines that cross module boundaries, value-bin coverage still needs an unbuilt primitive, and the whole approach presumes a hardware compiler with an instrumentable IR. The durable idea is the architecture: pull coverage out of the simulator, express it once in the compiler, and let a single primitive carry it to every tool.

Background

Why verifying hardware is so much harder than testing software

It helps to see why hardware teams pour so much effort into measuring coverage in the first place. A software bug can usually be patched after release; a bug baked into a fabricated chip cannot, and a re-spin costs months and millions, so a design has to be as close to correct as possible before it is ever manufactured. But a modern chip has astronomically many possible states, far too many to test exhaustively, so verification is a game of confidence rather than certainty: engineers write ever more tests and use coverage to measure how much of the design those tests have actually touched, hunting for the untested corners where a lurking bug could hide. That is why coverage is not a nicety but a core instrument of pre-silicon verification, and why being able to collect it everywhere — fast software runs, FPGA emulation booting a real operating system, formal tools proving parts unreachable — and combine it into one picture is genuinely valuable. The same seriousness about establishing hardware correctness before trusting it drives the silent-data-corruption work covered elsewhere on this site, which assumes even a fabricated, tested chip can quietly go wrong and builds checking around it rather than presuming it is sound. The broader pattern is that as designs outgrow any hope of exhaustive testing, the discipline shifts to measuring and reasoning about how much confidence the testing has actually earned.

If you remember only three things

1. Coverage metrics — line, toggle, finite-state-machine — measure how thoroughly a chip's tests exercise its design, but they are traditionally baked into each simulator, so most open-source and research simulators offer none, results from different tools cannot be merged, and coverage on the Verilog that hardware construction languages like Chisel compile to cannot be mapped back to the source because the compiler rewrites branches into other constructs.
2. This work moves coverage out of the simulator and into the compiler: each metric becomes a FIRRTL compiler pass that lowers to one small cover primitive (a counter that samples a signal at a clock edge) plus metadata mapping counts back to source lines or states, so a backend needs only to support that one primitive to gain every metric, a new metric works on every backend, results merge trivially across backends because they all report the same counts, and the same instrumentation drives software simulators, an FPGA emulator, and a formal tool.
3. Prototyped for Chisel across five backends, it collects line coverage while booting Linux on the FPGA-accelerated FireSim at 65 MHz, adds no measurable overhead over Verilator's native coverage, merges software and FPGA runs to cut FPGA counters by 42 percent, and let a fifth backend be added in about five hours — though it is infrastructure rather than a new capability, FPGA counters cost real area, the state-machine analysis is conservative and can miss cross-module machines, value-bin coverage needs an unbuilt primitive, and it presumes a hardware compiler with an instrumentable IR.

Check yourself

Check your understanding

1. Why is coverage hard to get for languages like Chisel that compile down to Verilog?
2. What is the core idea of this work?
3. Beyond portability, what does putting coverage in the compiler enable?
4. Which statement best reflects the limits of this work?

Sources

The paper is “Simulator Independent Coverage for RTL Hardware Languages” by Kevin Laeufer, Vighnesh Iyer, David Biancolin, Jonathan Bachrach, Borivoje Nikolic, and Koushik Sen, published at ASPLOS 2023, pages 605 to 614, DOI 10.1145/3582016.3582019. The authors are at the University of California, Berkeley, with two affiliated with SiFive and JITX.

This explainer is written from the full 10-page text. Everything above comes from the paper’s own sections: the introduction and background on coverage metrics (line, toggle, functional, and finite-state-machine) and on hardware construction languages, their FIRRTL intermediate representation, and the lowering to structural Verilog that breaks source-level coverage (with the when-to-conditional-assignment example); the simulator-independent interface built around a single cover primitive that samples a signal at a clock edge and reports a saturating count, and its implementation across five backends — the FIRRTL interpreter Treadle (about 200 lines, under a work week), Verilator via SystemVerilog assertions, the FPGA-accelerated FireSim via saturating counters on a per-clock-domain scan chain, the SymbiYosys formal tool for trace generation, and the ESSENT simulator (about five hours and sixty lines); the instrumentation passes and report generators for branch and line coverage, toggle coverage with a global alias analysis to avoid redundant instrumentation, conservative over-approximating finite-state-machine coverage using ChiselEnum annotations, and a ready-valid coverage metric built in about three hours; and the evaluation showing overhead on par with or below Verilator’s native coverage, FireSim FPGA resource and maximum-frequency scaling with counter width on Rocket and BOOM SoCs, booting Linux at 65 and 40 MHz with millisecond scan-out of thousands of cover counts, a 42-percent reduction in FPGA counters by merging with software coverage, the use of coverage as fuzzing feedback via AFL, and formal trace generation on RISC-V Mini that revealed unreachable cache write-path code and a bug in the framework’s own state-machine analysis, plus the limitation motivating a future cover-values primitive.