Parendi: Thousand-Way Parallel RTL Simulation

Chip designers test their designs by simulating them cycle by cycle in software, and as chips grow the single-threaded simulators cannot keep up, while multi-core versions stall at a few dozen threads because synchronizing threads on an ordinary server costs thousands of cycles — Parendi compiles a chip design into a bulk-synchronous program for the Graphcore IPU, a 1,472-core chip with a hardware barrier and a fast on-chip network, partitions the design's smallest units of work across up to 5,888 cores with a duplication-aware merging algorithm, and simulates large designs up to 4 times faster than the best x64 servers at about half the cost.

Why this problem is worth solving

A team designing a 100-core system on chip cannot tape it out without simulating it. The RTL description is compiled into a program that evaluates every register and every gate for one clock cycle, then the next, for millions or billions of cycles. Verilator, the fastest open-source tool, simulates the largest design in this paper, a 15 by 15 mesh of RISC-V cores with about 20 million gates, at 430 simulated cycles per second on one thread of a modern Intel server. Booting an operating system takes billions of cycles. At that rate, one billion cycles is 27 days.

Multithreading helps, but only so far. The paper’s Figure 1 plots single-thread performance against transistor count and shows the gap widening every year; the dashed line, which estimates the cores needed to simulate a current chip at the 2006 rate, already reaches into the thousands. Yet on the servers the authors tested, Verilator’s best speedup on that 20-million-gate design is 21.6 times on 28 threads, giving 9.2 kHz, and adding threads beyond a socket or chiplet boundary makes it slower, not faster. The question the paper asks is whether RTL simulation can use thousands of cores, and what kind of machine that requires.

What people did before, and where it broke

• Verilator on x64. Compiles RTL to C++ and, since version 4, runs it on multiple threads with fine-grained point-to-point synchronization between tasks. Where it broke. Synchronization on x64 is expensive. In the paper’s microbenchmark, a barrier across 56 threads costs a few thousand cycles, and a thread needs hundreds of thousands of instructions of work per simulated cycle to hide it. Small designs see no speedup; large ones stop scaling at chiplet (8 threads on AMD) or socket (28 threads on Intel) boundaries.
• RepCut on x64. A bulk synchronous simulator that partitions the design with a hypergraph and replicates logic to cut communication, reaching superlinear speedups up to 32 cores. Where it broke. Same ceiling: no gains past 32 threads on a 48-core machine.
• GPU simulators. RTLFlow runs many copies of one design in lockstep, and GCS accelerates gate-level simulation. Where it broke. RTL is irregular, so single-instruction multiple-data execution wastes most of the GPU, and an H100 has only about 50 MiB of on-chip memory, far too little to hold a large design’s state next to the cores.
• Custom hardware: Manticore, Nexus, ASH. Purpose-built cores for bulk-synchronous RTL simulation, prototyped on FPGAs or in simulation. Where it broke. FPGAs have too little SRAM to hold large designs, and ASH exists only as a simulated architecture.
• The gap. Nobody had run RTL simulation on thousands of real cores. Doing so needs a machine where synchronization is cheap and on-chip memory is large, and a compiler that decides which of tens of thousands of tiny tasks go on which core.

Where Parendi sits

The idea

The execution model: bulk synchronous parallel. A digital circuit at RTL is a set of registers and the combinational logic between them. In cycle-accurate simulation, each register has a current value read at the start of a cycle and a next value written at the end.

Why the IPU and not x64. The paper measures each cost on both machines. Synchronization: with independent random-number generators as the workload, adding cores costs the IPU almost nothing once each core has a few thousand instructions of work, because its barrier is in hardware and takes a few hundred cycles. On x64, even 47,104 tiny tasks per thread lose 25 percent to synchronization, and 736 per thread lose more than 75 percent. Communication: within one IPU, latency depends only on how many bytes each core sends, not on how many cores are talking, since the experiment uses 13 percent of the 7.7 TiB per second exchange. Between IPUs, latency grows with total volume, because the 107 GiB per second link runs at 82 percent utilization, so off-chip traffic must be minimized. Computation: an IPU core is slow, 84 times slower than an x64 core on one small design, so it must win through parallelism.

Fibers and the straggler. A

Merging is submodular, which makes partitioning hard. Two fibers often share intermediate logic. Put them on the same core and the shared part runs once, so the merged cost is less than the sum, and any communication between them disappears. The cost of a process is therefore a

IPU-specific compilation. Three optimizations matter. Differential exchange: a process that reads a register file or cache bank would need a full copy each cycle, so static analysis counts how many entries can change per cycle and only those updates are sent. Aggressive block splitting: Verilator’s splitting pass is pushed to split every clocked block for maximum parallelism, accepting code growth. Aggressive inlining: the generated program has no function calls at all, which would bloat an x64 instruction cache but is fine on an IPU, where each tile has 200 KiB of code memory with no cache and the chip totals about 300 MiB of instruction space.

Tracing one simulated cycle on two tiles

f1 = {a1,a2,a3,b1,c1} -> write1 | f2 = {a3,a4,b2} -> write2 | f3 = {a5,b3,b4} -> write3

Why it works

• A hardware barrier makes fine-grained parallelism affordable. The IPU hides synchronization behind a few thousand instructions per tile; x64 needs hundreds of thousands. Because RTL naturally splits into tens of thousands of tiny fibers, the IPU can use all of them while x64 must lump them into a few dozen coarse threads.
• Private on-chip memory removes the cache problem. Each item in an RTL simulation is touched once per simulated cycle, which may be millions of machine cycles later, so caches perform badly. The IPU’s 900 MiB of on-chip SRAM, split into 624 KiB per tile, holds the design’s state and code next to the cores with no cache at all.
• Cheap on-chip communication means more tiles never hurt within an IPU. Since intra-IPU latency depends on bytes per tile rather than tile count, performance on one IPU rises monotonically with tiles. x64 loses that monotonicity at chiplet and socket boundaries.
• Duplication-aware merging keeps the straggler from dominating. Merging fibers that share logic runs the shared part once and eliminates their communication, and the heuristic never merges past the current worst process, so t_comp stays at its floor while t_comm falls.
• Off-chip traffic is cut before it exists. Because cross-IPU latency grows with total volume, partitioning fibers across IPUs with a min-cut hypergraph partitioner before merging (rather than after) gives the most freedom to find small cuts. Going from 1 to 4 IPUs still yields a 60 percent gain on one large design.
• Bigger designs scale better. Fiber imbalance limits strong scaling of small designs but helps weak scaling: doubling a system on chip doubles the fibers, but only a few are stragglers, so the spare tiles absorb the new work and the rate stays flat over a long range where Verilator’s rate keeps falling.

What it costs

• A slow core needs a lot of parallelism. One IPU tile is 37 to 84 times slower than an x64 core on the small designs. Parendi does not match Verilator’s single thread on a small RISC-V core or a small pipelined Rocket core even using every tile, and it loses on the two-core mesh designs (0.62 and 0.67 of Verilator’s speed). The wins begin at a few hundred thousand gates.
• BSP forbids overlap and dynamic balancing. The IPU’s exchange is statically scheduled and must start at a barrier, so computation cannot overlap communication and load cannot be rebalanced at run time.
• Memory limits are hard. A design whose code and state exceed about 4 times 900 MiB will not compile, and a single Verilog array larger than roughly 400 KiB must be split by hand. The paper only explored up to 4 IPUs; 16 or 64 are future work.
• Language coverage is partial. One top-level clock plus gated or divided clocks; Verilog test drivers only, since a C++ testbench interacting every cycle would be impractically slow on the IPU; no combinational loops; DPI support is experimental. A compiler bug prevented evaluating the BOOM out-of-order core.
• Off-chip scaling is modest. Four IPUs over one gives 60 percent on the large designs, not 4 times, and for some designs fewer IPUs are faster.
• Compile time is minutes, not seconds. Up to 40 minutes and 55 GiB for the largest design, though that is 12 times faster and 18 times leaner than Verilator’s 8 hours and 1,043 GiB for the same design.

The evaluation

The benchmarks are an option-pricing Monte Carlo engine (mc), the VTA machine-learning accelerator, and two families of Chipyard systems on chip: srN, an N by N mesh network on chip with N squared minus 3 small Rocket RISC-V cores, for N from 2 to 15, and lrN with large Rocket cores including floating point and virtual memory, for N from 2 to 10. Sizes range from about 200 thousand gates (sr2) to about 20 million (sr15, lr10), excluding SRAMs. Parendi runs on a Graphcore M2000 with four MK2 IPUs at 1.35 GHz (5,888 tiles, 2020 hardware); Verilator v5.006 at -O3 runs on a dual-socket AMD EPYC 9554 (64 cores per socket, chiplets of 8) and a dual-socket Intel Xeon 6348 (28 cores per socket), with 2 to 32 threads. All benchmarks use Verilog drivers without DPI calls on both sides.

Three further results complete the picture. Cost. On cloud prices at the time (IPU-POD4 at 2.13 dollars per hour; a 16-core Azure Dv4 at 0.77), a one-billion-cycle run of sr15 takes 9 hours and 19.20 dollars on the IPU versus 57 hours and 43.78 dollars on x64, and the IPU stays cheaper unless Verilator achieved 3.2 times superlinear scaling. For nightly regressions of many one-million-cycle tests, the IPU is cheaper under both scheduling strategies tried. Measured power is 185 W for the four IPUs; the authors estimate the x64 runs at 80 to 118 W, so at 3.4 to 4.9 times the speed the IPU uses about half the energy. Versus research simulators. Against RepCut on an older Rocket SoC, Verilator wins small designs, RepCut medium ones, and Parendi the largest; RepCut’s code for the 32-core SoC crashes the compiler. Against Manticore’s published numbers, Manticore’s huge register file wins a small bitcoin design, and Parendi’s 1,472 tiles win vta and mc. Partitioning choices. On a single IPU, Parendi’s bottom-up merge and RepCut’s hypergraph approach each win some designs. Across four IPUs, partitioning fibers before merging beats partitioning processes after, and ignoring the multi-IPU structure is far worse.

Check yourself

Sources

The paper is “Parendi: Thousand-Way Parallel RTL Simulation” by Mahyar Emami, Thomas Bourgeat and James R. Larus, EPFL, published at ASPLOS 2025, Volume 2, DOI 10.1145/3676641.3716010, under a Creative Commons Attribution licence. This explainer was written from the full text of the arXiv preprint (2403.04714, version 2, March 2025). Numbers are taken from Tables 1 to 3 and Figures 4 to 17. Parendi is open source at github.com/epfl-vlsc/parendi.