Manticore: Hardware-Accelerated RTL Simulation with Static Bulk-Synchronous Parallelism

Mahyar Emami, Sahand Kashani, Keisuke Kamahori, Mohammad Sepehr Pourghannad, Ritik Raj and James R. Larus

Designing a chip means simulating it, and the fastest simulators run three orders of magnitude slower than the hardware they model — which is strange, because a circuit is nothing but fine-grain parallelism, thousands of tiny independent computations per clock edge. This paper (EPFL) shows why ordinary processors cannot use that parallelism, with a model where adding a second core makes simulation slower, then builds a 225-core machine whose synchronization is entirely scheduled at compile time and beats the fastest open-source simulator on 8 of 9 benchmarks while running at a tenth of the clock speed.

The argument in one glance

Problem

You cannot build the chip until you have simulated it, and simulation is glacial

Every accelerator goes through many rounds of design, debugging and software development before anyone commits it to silicon, and detailed simulation sits at the centre of all of it. The fastest software simulators run a design at somewhere between 1 and 1000 kilohertz, which is more than a thousand times slower than the circuit itself would run. That turnaround time directly limits how much a designer can explore and how thoroughly anything gets tested, and it is why long simulations get measured in days.
What people did

Use the parallelism that is obviously there

A hardware description is not an accidentally parallel program — it is explicitly a set of independent components that compute simultaneously and only synchronize at clock edges. Written as a graph, a circuit is a mass of independent paths that an evaluator could walk at the same time. Since multicore processors are the standard answer to making software faster, and this workload advertises its own concurrency, running the simulation across many cores looks like the obvious move.
Why it fell short

The tasks are far too small for a general-purpose machine

Designers want circuits to clock fast, which limits how much logic can sit between two clock edges, so a realistic design decomposes into an enormous number of very tiny tasks. Everything must then synchronize twice per simulated cycle. The paper models this with only the synchronization cost counted and finds that for small circuits performance DROPS when you go from one processor to two, and that the fastest open-source simulator manages at best a four-times speedup and is often slower multithreaded than single-threaded.
This paper's idea

Build a machine where synchronizing costs nothing

This paper (EPFL — Emami, Kashani, Kamahori, Pourghannad, Raj and Larus) observes that hardware descriptions almost never contain long divergent code paths, so a compiler can predict exactly what every core will be doing on every cycle. That makes it possible to replace the runtime barrier with a schedule computed ahead of time, filling gaps with delays so everyone arrives together. Nothing waits, nothing checks, nothing arbitrates — and cores that need none of that machinery become small enough to fit hundreds on a chip.

Why this problem is worth solving

The workload has an unusual shape, and it is worth being precise about what is being simulated and how.

A hardware description is a NETLIST — a graph whose nodes are circuit elements (gates, registers, memories) and whose edges are the wires between them. Split every register into its current value and its next value and the graph becomes acyclic, which turns simulating one clock cycle into a well-defined task: evaluate the graph until every next register value has been computed from the current ones, then swap them and repeat. This work targets FULL-CYCLE, CYCLE-ACCURATE simulation, which means values are only observed at clock edges and the order of evaluation is fixed ahead of time rather than driven by which signals happen to change. That sounds wasteful, since unchanged parts of the circuit get evaluated anyway, but it is generally faster in practice because monitoring signals to decide what to skip costs more than the skipping saves

The obstacle is a mismatch between how small the tasks are and how expensive coordination is.

Simulating one clock edge needs two synchronization points: one when all the computation has finished and one when all the resulting values have been exchanged. That is the minimum any correct parallel scheme requires. The trouble is what sits between them. Because designers push clock frequencies up, very little logic fits between two clock edges, so a real design becomes a huge number of tiny tasks — and the paper measures the consequence with a deliberately simplified model that counts only the synchronization and ignores data movement entirely, so its results are an upper bound on any system. For small circuits, where a barrier arrives every few hundred instructions, going from one processor to two makes simulation SLOWER. Real measurements agree: the fastest open-source simulator reaches at most a four-times speedup, and on smaller designs runs slower with threads than without

What people did before, and where it broke

Compile the design to optimised C++ and run it on one core. This is what the fastest open-source simulator does, and it is genuinely good — highly optimised, a decade of development behind it.
But it leaves a thousandfold gap to real hardware. Between 1 and 1000 kilohertz against a circuit that would run at gigahertz, which is what makes long simulations take days.
Run the same simulation across many cores. The concurrency is explicit in the source language, so this should be the easy case.
But the tasks are too small to be worth coordinating. Two barriers per simulated clock edge, with only hundreds of instructions between them, so the measured ceiling is about four times and often below one.
Prototype the design on an FPGA instead. Runs at real speed, which is why it is used for operating system bring-up and software development.
But you lose visibility and turnaround. Compiling to an FPGA is slow and seeing inside the design is much harder than in a simulator, which is precisely what you need while debugging.
Use a GPU. Recent work gets up to forty times the throughput of a software simulator.
But by running many simulations at once. With a single stimulus, which is the common case while debugging, it runs an order of magnitude slower.
The gap. The parallelism in the workload is real and abundant, and every existing machine either cannot exploit it at that granularity or exploits a different kind of parallelism entirely.

How you make synchronization free

Removing every runtime decision so that fine-grain parallelism becomes affordable

Compile the design to fast code and run it

Turn the hardware description into optimised C++ with a fixed evaluation order, and execute it. This is the established approach and the baseline throughout, and its single-core performance is the bar everything else has to clear.
↓ →
Measure the ceiling before designing anything new

Build a synthetic model of a parallel simulator that counts only the synchronization and ignores data movement entirely, giving an upper bound on what any system could achieve. It shows three regimes, and reveals that for small designs adding a second processor actively hurts.
↓ →
Replace the runtime barrier with a compile-time schedule new

The classical bulk-synchronous model alternates computation and communication phases separated by a barrier. Because hardware descriptions rarely contain long divergent paths, a compiler can instead compute exactly how long each core will take and insert delay instructions so all cores arrive at the next phase together.
↓ →
Strip the cores of anything the compiler can do instead new

No interlocks, no scoreboards, no branch prediction — branches become predication and every path executes. Instructions live in fixed-latency on-chip memory so the front end never stalls, and each core touches only its own registers and scratchpad. What remains is small enough that hundreds fit on one chip.
↓ →
Give the compiler the network as well new

The interconnect is a unidirectional torus whose switches have no buffers at all: a message arriving for a busy link is simply dropped. Correctness depends entirely on the compiler having scheduled communication so that collision never happens, which it can do because execution is fully deterministic.
↓ →
Split the circuit graph per output, then merge back to fit new

Carve the design into one independent task per register it writes, which maximises parallelism but duplicates work. Then merge tasks back until they fit the available cores, starting from the smallest and merging along communication edges, so that no core becomes a straggler and traffic falls at the same time.

Prototyped as 225 cores at 475 megahertz on a datacenter FPGA, with an 18,000-line compiler, evaluated against the fastest open-source RTL simulator on desktop and server processors.

The idea

The parallelism was always there; the machine could not pay for it. What makes this paper convincing is that it measures the obstacle before proposing anything. The synthetic model is deliberately unfair to the conclusion — it counts only barrier cost and assumes data movement is free, so whatever it shows is a ceiling no real system can exceed. And it shows that for a small design, where a barrier arrives every few hundred instructions, a second processor makes things worse rather than better; only when a simulated cycle reaches hundreds of thousands of instructions does parallelism pay, and by then each cycle is so expensive that the absolute rate is poor anyway. Real measurements of the best available simulator land exactly where the model predicts, topping out around four times and frequently going backwards with threads. So the diagnosis is not that software simulators are badly written; it is that the work-to-synchronization ratio of this workload falls in a region general-purpose processors cannot serve. Establishing that a bottleneck is structural rather than incidental, by bounding what any implementation could achieve before building an alternative, is the same prove-the-ceiling-first discipline covered elsewhere on this site where a paper demonstrates the limit is in the machine model rather than in the effort spent.

Hand everything to the compiler, including the parts hardware normally owns. The design is an unusually pure version of a bargain that has been struck before: if the compiler can predict execution exactly, the hardware need not check anything. Here that is taken further than usual. There are no interlocks or scoreboards, because the compiler inserts delays to respect dependencies. There are no branches, because control flow becomes predication and every path runs. Off-chip memory has the same predictable latency as on-chip memory from the compiler’s point of view, because an access stalls every core and the entire network until it completes. And most strikingly, the network switches have no buffers whatsoever — a message that arrives for a busy link is dropped and lost, and the only thing preventing data loss is that the compiler has scheduled communication so a collision cannot occur. This is only defensible because the workload permits it: hardware descriptions rarely contain long divergent paths, so a conservative static schedule is not much worse than a perfect dynamic one. The payoff is that the cores become tiny, and a core using under 0.021 percent of the chip is what makes hundreds of them possible. Removing runtime machinery on the strength of a property of the workload, so that what remains is small enough to replicate, is the same trade-flexibility-for-density argument covered elsewhere on this site where a specialised datapath wins mostly by omitting what its workload never needs.

Winning by a wide margin just to draw level. The paper is unusually direct about the size of the hole it starts in. The prototype runs at 475 megahertz against a desktop processor at 4.6 to 4.9 gigahertz, executes one instruction per cycle where the desktop manages one to two and a half, and has a narrower datapath with a simpler instruction set. Put together, the machine must extract somewhere between ten and twenty-five times from parallelism before it has matched a single x86 thread — and the authors say plainly that a large fraction of the observed gain goes into making up that deficit rather than into pure advantage. The results show the point sharply: on eight of nine benchmarks it wins, but on the ninth, which has real sequential dependencies in a table lookup that cannot be parallelised, it has both the highest software rate and the lowest hardware rate, with parallelism improving it by only about 17 percent. Amdahl applies here as everywhere. Reporting a handicap in the same breath as a headline, so that a reader can see how much of a result is recovery rather than gain, is the same account-for-the-baseline-honestly practice covered elsewhere on this site where the interesting number is what remains after the comparison is made fair.

Following one clock edge through 225 cores

What it takes to simulate a single simulated cycle without a barrier

Figures from the 225-core prototype at 475 megahertz on a datacenter FPGA, measured against the fastest open-source simulator.

Step 1 of 5

Turn the circuit into a graph with no cycles

Every register is split into its current value and its next value, which makes the netlist acyclic. Simulating one clock edge means evaluating that graph until every next value has been computed from current ones, then swapping. The graph makes the available parallelism explicit, since independent paths can be walked simultaneously.

this is full-cycle simulation, where the evaluation order is fixed ahead of time rather than driven by which signals changed
that wastes work on unchanged logic but avoids the larger cost of monitoring signals to decide what to skip

Why it works

Hardware descriptions rarely contain long divergent code paths → a compiler can predict exactly how long every core will take → so the runtime barrier can be replaced by inserted delays that cost nothing.
With execution fully deterministic, message arrival times are known in advance → the network needs no buffers, arbitration or flow control at all → and switches that simply drop contending messages are safe because contention never happens.
Interlocks, scoreboards and branch prediction all exist to handle uncertainty the compiler has removed → each core shrinks to under 0.021 percent of the chip → which is what makes 225 of them fit, with room for far more.
Splitting the circuit graph per output maximises parallelism but duplicates work → merging back from the smallest task along communication edges balances load and cuts traffic together → avoiding stragglers, which would otherwise set the pace for everyone.
Every simulated cycle costs the same number of instructions regardless of design state → the compiler can count cycles exactly and report the critical path → making scaling measurable without running anything.

What it costs (trade-offs and scope)

It is a prototype, not a tool, and says so. Single-clock designs only, most of SystemVerilog unsupported, and the authors state directly that this does not replace a simulator with a decade of development behind it.
No waveform debugging. The authors call this an essential part of a designer’s toolkit; a hardware design for collecting waveforms exists but is unevaluated.
Accurate timing simulation is incompatible. Not merely missing but structurally at odds with the approach, and the paper says retrofitting it would be hard.
The whole design must fit on chip. The prototype holds about 900,000 instructions across 225 cores with 14.4 megabytes of memory, and benchmarks were sized so their state fits the scratchpads.
Compilation is slower than a conventional compiler. Most of it is graph partitioning, which the authors argue is inherent to parallel simulation rather than to their machine, but it is real today.

The evaluation

Three separate questions. Whether general-purpose processors can serve this workload at all, answered by a synthetic model that counts only synchronization and therefore bounds any possible system, cross-checked against measurements of the fastest open-source simulator on desktop and server machines. Whether the prototype beats that simulator, answered on nine designs including a machine learning accelerator. And whether it scales, answered by having the compiler report the critical path length, which is exact because every simulated cycle costs the same regardless of state. A cost comparison using cloud instance prices rounds it out.

Property	Optimised software simulation	FPGA prototype of the design	Statically scheduled simulator
Simulation rate
Benefit from more cores
Turnaround to first result
Visibility into the design
Where it fits

The prototype is faster than the software simulator on 8 of 9 benchmarks, by up to 16 times against its single-threaded performance and 11 times against its multithreaded performance, despite running at 475 megahertz against desktop processors near 4.9 gigahertz and executing one instruction per cycle against one to two and a half. The single loss is a benchmark whose sequential table lookup resists parallelisation, where parallelism buys about 17 percent.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The authors are notably candid that this is a research prototype rather than something anyone should adopt. Only single-clock designs work, most of SystemVerilog is unsupported, and advanced language features needed to write serious testbenches are missing — they state outright that it is not a replacement for the simulator it beats, which has had more than a decade of active development. Two absences matter more than the feature list. Waveform debugging, which the paper itself calls essential to a hardware designer, does not exist yet; a hardware design for it is sketched and left unevaluated. And accurate timing simulation, as opposed to cycle-accurate, is described as incompatible with the approach rather than merely unimplemented. There are capacity limits too: the whole design must fit on chip, which for the prototype means roughly 900,000 instructions and 14.4 megabytes, and the benchmarks were deliberately sized so their state fits the scratchpads. On the numbers themselves, read the speedups alongside the handicap the paper reports in the same section — a ten to twenty-five times deficit in clock speed, instructions per cycle and datapath width, meaning a large fraction of the parallel gain is spent recovering serial ground rather than adding to it. The scaling curves are also compiler predictions rather than measurements, which the authors justify because instruction counts are exact without off-chip access, and the single-core baseline they are relative to mostly cannot be run at all because programs do not fit one core.

Background

When the workload is parallel but the machine cannot afford it

This paper sits in the site’s accelerators and compilers clusters and rhymes with several ideas covered elsewhere on this site. Its central finding — that a workload full of obvious parallelism goes unexploited because coordination costs more than the work between coordination points — is the same granularity-mismatch diagnosis the memory disaggregation and graph processing pages reach, where adding workers makes things worse until something changes about what a worker costs to synchronize. Its method of proving the ceiling first, with a model deliberately generous to the alternative, is the same bound-it-before-you-build-it move the performance analysis pages argue for. Its central bargain, moving everything a processor normally decides at runtime into a compiler that can only do it because the workload is unusually predictable, is the same static-scheduling trade the vectorizing compiler generation and network card synthesis pages make, and the network card page reaches almost the same conclusion from the opposite direction, generating a pipeline whose correctness rests on compile-time analysis of a language restricted enough to permit it. And its subject, that verifying hardware is itself a computing problem worth building hardware for, sits beside the FPGA compilation and high-level synthesis pages that attack the other half of the same design loop.

If you remember only three things

1. Chip design depends on cycle-accurate simulation, and the fastest software simulators run between 1 and 1000 kilohertz — over a thousand times slower than the circuit being modelled. The parallelism appears to be free for the taking, since a hardware description is explicitly a collection of independent components synchronizing only at clock edges. But designers push clock frequencies up, which limits the logic between edges, so a real design becomes an enormous number of tiny tasks that must all synchronize twice per simulated cycle. A model counting only synchronization cost, and therefore bounding any possible system, shows that for small designs a second processor makes things SLOWER.
2. The fix is to notice that hardware descriptions almost never contain long divergent code paths, so a compiler can predict exactly what every core does on every cycle and replace the runtime barrier with inserted delays. That permits stripping the cores of everything that exists to handle uncertainty: no interlocks, no scoreboards, no branches at all since control flow becomes predication. It extends to the network, whose switches have no buffers and simply drop a message arriving for a busy link — safe only because the compiler has scheduled communication so contention cannot occur. A core then occupies under 0.021 percent of the chip, which is what makes hundreds of them fit.
3. The prototype runs 225 cores at 475 megahertz and beats the fastest open-source simulator on 8 of 9 benchmarks, by up to 16 times against its single-threaded performance, with compiler-predicted scaling still improving at 200 to 300 cores. The paper is honest about what that number contains: at a tenth the clock speed with a fraction of the instructions per cycle, the machine needs 10 to 25 times from parallelism just to match one x86 thread, so much of the gain is recovery rather than advantage — and on the one benchmark with genuine sequential dependencies, it loses.

Check yourself

Check your understanding

1. A hardware description is explicitly parallel, and RTL simulation is full of independent work. Predict why running it across many cores of a general-purpose processor barely helps.
2. The design replaces the runtime barrier with a compile-time schedule. Predict what property of the workload makes that possible, and how far it is taken.
3. The prototype beats the fastest software simulator on 8 of 9 benchmarks. Predict what the paper says about how much of that is genuine advantage.
4. The compiler splits the circuit graph into one task per output register and then merges tasks back together. Predict why merging cannot use a standard graph partitioning tool.

Sources

Mahyar Emami, Sahand Kashani, Keisuke Kamahori, Mohammad Sepehr Pourghannad, Ritik Raj and James R. Larus. “Manticore: Hardware-Accelerated RTL Simulation with Static Bulk-Synchronous Parallelism.” ASPLOS 2023, Volume 4. EPFL, with contributions from interns at the University of Tokyo, Sharif University and IIT Roorkee. DOI 10.1145/3623278.3624750. Preprint at arXiv 2301.09413.
Full text of the paper was used for this explainer, including the simulation background in Section 2, the static bulk-synchronous execution model in Section 3, the instruction set and architecture in Section 4, the pipeline and network implementation in Section 5, the parallelisation and scheduling passes in Section 6, the limits-of-parallelism model, benchmark comparisons, scalability analysis and cloud cost estimates in Section 7, and the authors’ stated limitations in Section 8.
The hardware design, Verilog frontend and backend compiler were released publicly under a permissive licence.