Ratte: Fuzzing for Miscompilations in Multi-Level Compilers Using Composable Semantics

MLIR is a popular compiler toolkit built from mix-and-match language fragments called dialects, but it has no precise written-down meaning for those dialects and only one implementation, which makes it very hard to catch the worst kind of compiler bug — a miscompilation, where the compiler silently produces wrong code; this work from Imperial College builds Ratte, which writes each dialect's meaning as a composable reference interpreter alongside a matching test generator, so the generator only ever produces well-defined programs and the interpreter gives a trusted answer to compare against, letting Ratte become the first MLIR fuzzer to find miscompilations — six of them, in the production compiler.

The argument in one glance

Problem

MLIR has no precise meaning to check compilers against

MLIR is built from mix-and-match language fragments called dialects, but their meaning exists only as English documentation, and the meaning of dialects combined is even less specified. Without a precise reference, the worst compiler bug — a miscompilation that silently produces wrong code — is nearly impossible to catch.
What people did

Differential-test the compiler on generated programs

Generate lots of MLIR programs and compare compiler outputs, either across two implementations or across optimization levels, flagging any disagreement as a bug.
Why it fell short

No second implementation, and invalid programs prove nothing

MLIR is one unifying framework, so there is no independent implementation to compare against, and cross-optimization-level testing misses bugs in the lowering passes every path runs through. Worse, most randomly generated programs have undefined behavior, so any output difference could be blamed on the program, not the compiler.
This paper's idea

Co-develop composable semantics and fuzzers (Ratte)

Ratte writes each dialect's meaning as a composable reference interpreter, developed alongside a matching program generator. The interpreter runs in-step with generation so every produced program is well-defined, and it also serves as the trusted oracle for differential testing. Dialect interpreters and generators compose, so combinations are cheap to test. Ratte is the first MLIR fuzzer to find miscompilations — six, in the production compiler.

Why this problem is worth solving

MLIR sits under a growing amount of important software, which makes its correctness matter, and also makes it unusually hard to test.

MLIR, Multi-Level Intermediate Representation, is a compiler framework designed to cut the cost of building new compilers. Its central idea is the dialect: a modular, reusable fragment of an intermediate representation that captures one aspect of a language — control flow, memory access, floating-point arithmetic — or one domain, such as machine learning or hardware design. A compiler author assembles a compiler by combining existing dialects rather than building everything from scratch, and lowers a program step by step from high-level dialects down to a low-level, executable one like the LLVM dialect. Underneath, MLIR is a single-static-assignment representation extended with regions, which are nested pieces of program that encode scoping directly in the IR; an operation has a name, operands, results, attributes, and possibly regions. Dialects are composable, meaning a single program can freely mix operations from several of them, and real MLIR compilers almost always do. That composability is MLIR’s strength, but it is also why specifying its meaning is hard: not only does each dialect need a precise semantics, but dialects interact when composed, and those interactions are even less documented.

The gap between MLIR’s importance and its lack of a precise, checkable meaning is exactly where the most dangerous compiler bugs hide.

A miscompilation is the worst kind of compiler bug: the compiler silently generates wrong code that still runs and returns a plausible but incorrect answer, with no crash and no error message. Finding one requires knowing what the program should have computed and having a valid program to run. The standard technique is differential testing: compile the same program in two ways and compare the outputs, so any disagreement reveals that at least one path is wrong. On MLIR this runs into three walls. First, there is no formal reference semantics to compare an output against — only English documentation, which leaves edge cases ambiguous. Second, MLIR is a single unifying framework, so there is rarely a second independent implementation to cross-check, and comparing across optimization levels alone will not catch a bug in a lowering pass, because every compilation path goes through the same lowering. Third, effective differential testing needs many valid test programs that are free of undefined behavior — because a program with undefined behavior can be compiled any way at all, so a difference proves nothing — and generating such programs is hard for one dialect and combinatorially harder across dialect combinations.

What people did before, and where it broke

Rely on the English documentation as the spec. Treat the prose description of each dialect as the semantics developers and testers work from. Where it broke. Prose leaves edge cases ambiguous and says even less about how dialects behave when composed, so there is no precise oracle to declare an output wrong.
Differential-test across optimization levels. Compile a program at different optimization settings and compare, flagging disagreements. Where it broke. Every compilation path runs the same lowering passes, so bugs in lowering produce the same wrong answer at every level and slip through undetected.
Fuzz MLIR for crashes with existing generators. Use a tool like MLIRSmith to generate syntactically valid MLIR and look for compiler crashes. Where it broke. These generators produce an unpredictable mix of valid and undefined-behavior programs that often do not compile fully, so they can find crashes but cannot detect miscompilations, where you must know the correct answer.
Build a custom generator per dialect combination. Treat each combination of dialects as its own language and write a bespoke, hand-tuned program generator for it. Where it broke. Fuzzers are tightly coupled to one language, so each combination repeats the same generation and analysis work — prohibitively costly for MLIR’s large and growing family of dialects.
The gap. No approach gave a precise, composable reference meaning for dialects together with a way to cheaply generate valid, well-defined programs across their combinations. Providing both, in a mutually reinforcing loop, is what Ratte does.

How Ratte works

From English docs to a composable interpreter that generates its own tests

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Grow the test generator and the reference meaning together. Ratte’s foundation is a co-developed pair: for each dialect it writes both a reference interpreter that defines what the operations mean and a program generator that produces test programs, and it runs the interpreter in-step with generation so that the value of every variable is known at every point and the generator never emits a program with undefined behavior. This creates a harmonious cycle — the fuzzer exercises and validates the semantics by generating programs, while the semantics let the fuzzer generate high-quality, well-defined programs whose correct answers it knows. Building a generator that respects a language’s validity constraints so that every produced test is meaningful, rather than generating blindly and discarding the vast majority, is the same move the deep-learning compiler fuzzing work covered elsewhere on this site makes, weaving the rules a valid program must satisfy into the generator itself so that testing effort lands on programs the compiler is actually obliged to handle.

Build each dialect’s meaning as a snap-together brick. Because MLIR is an open, ever-growing family of dialects, Ratte cannot specify all of it at once, so it makes both the semantics and the generators modular and composable: each dialect’s interpreter and generator are defined independently, and combining dialects is as cheap as combining their pieces. Achieving this requires solving the expression problem — adding both new operations and new interpretations without rewriting existing code — for which Ratte uses effect systems, and it builds the generators with property-based testing libraries so they compose the same way. Designing a system as independent, reusable components that compose into whatever combination a problem needs, instead of a monolith rebuilt for each case, is the same discipline the composable sparse-compilation work covered elsewhere on this site follows, expressing a capability as modular abstractions that snap together so new combinations come almost for free rather than as fresh from-scratch engineering.

Catch silent wrong code by holding it against a trusted answer. A miscompilation produces no crash, so the only way to detect it is to compare the compiler’s output against an independent source of truth for what the program should compute. Ratte’s reference interpreter is exactly that oracle: it computes the correct result for each well-defined program it generates, so when the production MLIR stack compiles and runs the same program and disagrees, the difference is an unambiguous bug — and crucially this catches lowering-pass bugs that comparing across optimization levels cannot, because the interpreter is a genuinely separate authority rather than another run of the same passes. Using differential comparison against an independent reference to surface bugs that leave no visible symptom is the same design the compiler-driven differential-testing work covered elsewhere on this site is built on, cross-checking outputs that should agree so that a silent divergence becomes a signal instead of an answer nobody thought to doubt.

Following a test program through Ratte

Generating a well-defined MLIR program and catching a miscompilation

A trace of Ratte building one test program and differential-testing it, following the paper's composable-semantics design.

Step 1 of 5

Pick a set of dialects to test together

The user chooses a dialect combination — for example arith, func, and vector, lowered to the LLVM dialect. Ratte assembles the matching generators and reference interpreters by composing the per-dialect pieces.

Composability means the combination needs no bespoke generator.

Why it works

A well-defined program makes disagreement conclusive. Because Ratte generates only programs free of undefined behavior → any difference between the compiler’s output and the reference must be the compiler’s fault, not the program’s → so a mismatch is an unambiguous bug rather than noise.
An independent interpreter catches lowering bugs. Because the reference semantics is a separate authority, not another run of the compiler → it disagrees even when every optimization level makes the same lowering mistake → so bugs invisible to cross-optimization-level testing are exposed.
Running semantics in-step avoids wasted generation. Because the interpreter evaluates each partial program and guides the next choice → Ratte rarely has to generate a program and throw it away for undefined behavior → so it efficiently produces the valid, well-defined tests that miscompilation-finding requires.
Composable pieces make combinations cheap. Because each dialect’s interpreter and generator are modular and snap together → testing a new dialect combination reuses existing pieces instead of a bespoke generator → so coverage scales with MLIR’s growing family of dialects.
The cycle strengthens both sides. Because the fuzzer exercises the semantics while the semantics enable high-quality tests → errors in either the spec or the compiler surface as differences → so the reference semantics and the test generator improve together.

What it costs (trade-offs and scope)

Someone must write the reference semantics. Ratte’s power comes from an executable interpreter for each dialect, which is real specification effort. It pays off as reusable, validated semantics, but a dialect with no interpreter cannot be tested this way.
Generation is slower than blind fuzzing. Because Ratte interprets the program as it builds it, its throughput is lower — on the order of a thousand programs in roughly three minutes versus about one minute for a crash-focused generator like MLIRSmith. The trade buys well-defined programs a faster fuzzer cannot produce.
It is limited to loop-free programs. Like the YARPGen approach it generalizes, Ratte generates loop-free programs so it can track variable values exactly during generation. It still exercises some looping by generating higher-level operations that lower into loops, but direct loop generation is out of scope.
Coverage is per-dialect, not all of MLIR. MLIR is open-ended and users can add new dialects, so no fuzzer can cover the whole framework. Ratte was demonstrated on several key dialects; extending to others means writing their composable interpreters and generators.
The bug count reflects one testing campaign. The eight bugs found (six miscompilations, of which two were lowering-pass bugs, plus two wrongly-rejected valid programs) come from roughly a week of overnight runs on a laptop against a specific MLIR version; more compute or newer versions would shift the numbers.
The oracle is the semantics, so the semantics must be right. A difference could in principle mean the reference interpreter is wrong rather than the compiler. The harmonious cycle mitigates this by continuously cross-checking, and indeed the work also uncovered and fixed ambiguities in the MLIR specification itself.

The evaluation

Three ways to test an MLIR compiler for bugs

Property	Cross-optimization-level testing	Crash fuzzing (MLIRSmith)
Reference for correct output
Programs are well-defined	Not guaranteed
Finds miscompilations
Scales to dialect combinations	Not applicable	One fixed generator
Generation throughput	Not applicable

Three ways to test an MLIR compiler, as compared in the paper. Comparing a compiler against itself at different optimization levels needs no reference semantics but cannot catch bugs in the lowering passes every path runs through, and gives no guarantee the test programs are well-defined. Crash-focused fuzzing like MLIRSmith generates syntactically valid MLIR quickly but produces an unpredictable mix of valid and undefined-behavior programs that often do not fully compile, so it can find crashes but not miscompilations, where the correct answer must be known. Ratte co-develops, for each dialect, a composable reference interpreter and a matching program generator; it runs the interpreter in-step with generation so every program is free of undefined behavior by construction, uses the interpreter as an independent oracle for differential testing, and composes the per-dialect pieces so any dialect combination is cheap to test. It generates more slowly (about a thousand programs in three minutes versus roughly one minute for MLIRSmith) because it interprets as it builds, but that is what lets it become the first MLIR fuzzer to find miscompilations — eight previously-unknown bugs in the production compiler, six of them miscompilations (two caused by lowering passes) and two valid programs wrongly rejected — while also surfacing and fixing ambiguities in the MLIR specification itself.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

Ratte is a careful piece of compiler-testing research that turns a specification gap into a testing method. The problem it attacks is real and underappreciated: MLIR has become foundational infrastructure for many compilers, especially in machine learning and hardware, yet the meaning of its dialects lives only in English prose, and the meaning of dialects composed together is barely documented at all, so the most dangerous kind of compiler bug — a miscompilation that silently returns a wrong answer with no crash — had no reliable way of being caught. The insight is to co-develop, for each dialect, an executable reference interpreter and a matching program generator, and to run the interpreter in-step with generation so that every program produced is well-defined by construction and its correct output is already known. That yields a harmonious cycle: the generator stress-tests the semantics, and the semantics let the generator produce exactly the kind of valid, well-defined programs that miscompilation-finding needs, with the interpreter doubling as the independent oracle for differential testing. Because both the semantics and the generators are built as composable pieces — using effect systems to solve the expression problem and property-based testing to combine generators — new dialect combinations are cheap to test, which matters for an open, ever-growing framework. Read the results with their scope. Ratte needs someone to write each dialect’s semantics; it generates more slowly than blind fuzzers because it interprets as it goes; it is limited to loop-free program generation (testing loops only indirectly through operations that lower into them); and its eight bugs, six of them miscompilations, come from about a week of overnight runs on a specific MLIR version. The oracle is the reference semantics, so a difference could in principle indict the interpreter rather than the compiler — but the cycle that continuously cross-checks the two is precisely what also let the authors find and fix ambiguities in the official MLIR spec. The durable idea is that for a composable IR, the meaning and the tests should be built together and composably, so that specifying a dialect and testing it become two halves of the same act.

Background

Why precise, composable meaning is the hard part of modern IRs

It helps to see Ratte within a shift in how compilers are built: away from monolithic, hand-written intermediate representations and toward modular, reusable IR frameworks assembled from fragments. MLIR is the leading example, and its dialects make it enormously productive, but they relocate the hard problem from writing a compiler to pinning down what the pieces mean, individually and in combination. The MLIR-based hardware IR work covered elsewhere on this site reflects the same trend from the design side, using MLIR’s dialect machinery to raise the level at which accelerators are described so that reusable abstractions replace bespoke, from-scratch tooling. The common thread is that composability is a double-edged gift: it multiplies what you can build cheaply, but it also multiplies the interactions whose meaning must be specified before anyone can say whether a compilation is correct. Ratte’s contribution is to take that seriously and make the specification executable, composable, and self-testing, so the growth of the dialect ecosystem is matched by a way to keep it trustworthy. As more languages and hardware flows are built on MLIR and frameworks like it, the pattern Ratte exemplifies — treating the semantics of composable IR fragments as a first-class artifact that is co-developed with the tests that validate it — is likely to be how these systems are kept correct as they scale.

If you remember only three things

1. MLIR is a fast-growing compiler framework built from modular, composable language fragments called dialects, and it underlies many production compilers. But its dialects have no formal semantics — only English documentation — and the meaning of dialects composed together is even less specified, which makes it very hard to catch miscompilations, the worst kind of compiler bug, where the compiler silently generates wrong code that runs and returns an incorrect answer with no crash. Differential testing, the standard way to find compiler bugs, is doubly hard on MLIR: there is no second independent implementation to cross-check against, comparing across optimization levels cannot catch bugs in the lowering passes every path runs through, and effective testing needs many valid, undefined-behavior-free programs whose correct output is known, which are hard to generate per dialect and combinatorially harder across dialect combinations.
2. Ratte co-develops, for each dialect, a composable reference interpreter (the meaning) and a matching modular program generator (the fuzzer). It runs the interpreter in-step with generation, building each program one operation at a time and incrementally evaluating the partial program to track every variable's type and value, so the generator only makes choices that keep the program free of undefined behavior — avoiding the waste of generating and rejecting. The finished, well-defined program is given an oracle (printing selected values) and differential-tested by compiling through the real MLIR stack and comparing against the reference interpreter, which is an independent authority and so catches lowering-pass bugs that cross-optimization-level testing misses. The semantics and generators are made composable using effect systems (to solve the expression problem) and property-based testing, so any dialect combination is cheap to test. Fuzzer and semantics reinforce each other in a harmonious cycle.
3. Ratte is the first MLIR fuzzer able to find miscompilations. It found eight previously-unknown bugs in the production MLIR compiler — six miscompilations (two of them caused by lowering passes) and two valid programs wrongly rejected — including a case where minus 1 times minus 1 on a 1-bit integer wrongly produced minus 1 instead of 1. It also surfaced and fixed ambiguities in the MLIR specification, and yields a reusable, validated reference semantics for several key dialects. Scope: it needs each dialect's semantics written, generates more slowly than crash-only fuzzers because it interprets as it builds, is limited to loop-free programs (testing loops only via operations that lower into them), and covers selected dialects rather than all of open-ended MLIR. The durable lesson is that for a composable IR, meaning and tests should be built together and composably, so specifying a dialect and testing it become two halves of the same act.

Check yourself

Check your understanding

1. Why is it especially hard to catch miscompilations in MLIR?
2. What is the core idea of Ratte?
3. Why can Ratte catch lowering-pass bugs that comparing across optimization levels cannot?
4. What did Ratte achieve, and what are its main limits?

Sources

The paper is “Ratte: Fuzzing for Miscompilations in Multi-Level Compilers Using Composable Semantics” by Pingshi Yu, Nicolas Wu, and Alastair F. Donaldson, published at ASPLOS 2025, DOI 10.1145/3676641.3716270. All three authors are at Imperial College London.

This explainer is based on the full text of the paper, obtained from the authors’ open-access copy (the CC-BY version) hosted on Alastair Donaldson’s Imperial College homepage. The problem framing (MLIR as a composable framework of dialects lacking formal semantics, the difficulty of catching miscompilations, and why differential testing struggles on MLIR because there is no second implementation, cross-optimization-level testing misses lowering bugs, and valid undefined-behavior-free programs are hard to generate across dialect combinations), the Ratte design (co-developed composable reference interpreters and modular generators, effect systems to address the expression problem for composable semantics, property-based testing for composable generators, incremental generation with in-step semantic evaluation to keep programs undefined-behavior-free, the appended oracle and differential testing against the reference interpreter, and the harmonious cycle in which fuzzer and semantics validate each other), and the results (eight previously-unknown bugs found in the production MLIR compiler, six of them miscompilations with two caused by lowering passes and two being wrongly-rejected valid programs, the minus-1-times-minus-1 example on a 1-bit integer, the comparison with MLIRSmith on compileability and undefined-behavior-freeness and generation throughput, the limitation to loop-free programs, and the fixes contributed to the MLIR specification) are all drawn directly from the paper. Background explanations of MLIR, single static assignment, dialects, miscompilations, differential testing, and undefined behavior are standard and provided to make the paper’s contribution understandable.