Compilation Consistency Modulo Debug Information

Production binaries ship without debug information, so when one crashes you rebuild with the same flags plus the debug flag and point a debugger at the core dump — which only works if adding that flag does not change the machine code, a property compilers are supposed to have and nobody had ever systematically tested; this paper (Waterloo) names it, then builds the first tool to hunt violations by mutating programs until they demand absurdly complicated debug information, and finds 23 real bugs in GCC and Clang with 21 confirmed or fixed.

The argument in one glance

Problem

The binary that crashed is not the binary you can debug

Software ships without debug information, for two good reasons: it makes a binary far easier to reverse engineer, and it is expensive — building LLVM with full debug information takes 56 percent longer and produces binaries thirty times larger, which is why industrial teams and anything running on a constrained device switch it off. Then a released binary crashes in the field, the operating system writes a core dump, and somebody has to work out why. The information needed to make sense of that dump is exactly the information that was deliberately left out.
What people did

Rebuild the same program with the debug flag turned on

The standard recovery is straightforward. Take the same source, the same compiler, the same optimization flags, add the flag that emits debug information, and point a debugger at the new binary together with the old core dump. Since a core dump records registers, stacks and memory that are specific to particular machine instructions, this works on one condition: the new binary must contain the same instructions as the one that crashed. Everyone assumes it does, because why would asking for extra metadata change the code.
Why it fell short

Asking for the metadata can change the code

Compilers record symbol and statement information while parsing, then carry it through every optimization pass in the middle end, which must keep it faithful while transforming the program. A pass that mishandles its presence behaves differently, and the output differs. Then the debugger misreads the dump, or shows instructions as unreadable, or the bug you were chasing simply stops reproducing. In the worst case it is not a debugging annoyance at all — enabling debug information once caused GCC to miscompile the Linux kernel outright, through bad data dependencies in instruction scheduling.
This paper's idea

Name the property, then generate programs that stress it

This paper (University of Waterloo — Wang, Tian, Dong, Xu and Sun) names the property Compilation Consistency Modulo Debug Information, and observes that despite a GCC flag and an LLVM script existing to check it, no research had ever tried to break it on purpose. Their tool Dfusor takes a randomly generated program and mutates it three ways, all aimed at forcing the compiler to emit more and stranger debug information rather than at making the program itself unusual. Compile each variant twice with identical optimization flags and different debug flags, and compare the machine code. Any difference is a bug.

Why this problem is worth solving

The property only matters because of a specific workflow, and that workflow only exists because debug information is expensive.

DEBUG INFORMATION is metadata a compiler can emit alongside the machine code, describing how the source relates to the binary — the names, types, scopes and positions of variables, functions and instructions — stored in a format such as DWARF and read by a debugger. It is optional, controlled by a flag with four verbosity levels from none to maximum. Released software almost always omits it, for two reasons. It hands an attacker a decompilation aid. And it is startlingly expensive: measured on a full build of LLVM 14, the maximum level takes 56 percent longer, and an INCREMENTAL build after touching a single source file goes from 4 minutes 54 seconds to 34 minutes 47, a 610 percent increase. The binaries grow from 3.18 gigabytes to 99.42, about thirty times. Sony PlayStation developers disable it for build time alone, and on an embedded device it is simply not affordable

The reason the flag can change the code at all is that it is not a separate output channel.

Emitting correct debug information is not something the back end can do alone, because by the time code is generated the program has been transformed beyond recognition. So the front end records information about symbols and statements while parsing, and every OPTIMIZATION PASS in the middle end is then obliged to do two jobs at once: transform the program, and keep the debug information faithful to the source through that transformation. Passes therefore see extra material in the intermediate representation that would not be there otherwise. If a pass fails to distinguish that material from real instructions, it makes different decisions. One of the bugs found here is exactly this — a loop optimization checking whether two instructions are adjacent counted an intervening debug marker as an instruction, so it declined an optimization it would otherwise have applied. Of the eleven bugs whose fixes the authors traced, every single root cause was in the middle end or the back end

What people did before, and where it broke

Trust that a metadata flag is only metadata. It sounds like it should be true, and for most programs it is.
But nobody had checked it adversarially. GCC ships a flag that compares compilations with and without debug information, and LLVM has a script that runs the same check over its test suite, so both projects clearly care — yet no research had ever generated programs designed to break the property.
Test compilers for miscompilation. A mature line of work generates random programs, compiles them at different optimization levels, and compares what they do when run.
But that is a different property. It asks whether the binary means the same thing as the source. It never asks whether two binaries from the same source contain the same instructions.
Test whether the debug information itself is correct. Other work checks that a debugger stops at the right places and prints the right values for optimized code.
But that assumes the code is fixed and asks about the metadata. This problem is the mirror image: it assumes the metadata is whatever it is and asks about the code.
Build with debug information and then strip it afterwards. This genuinely works for users, since both binaries then contain identical instructions by construction.
But it makes every build pay the debug cost. That is the 610 percent incremental build penalty on every developer, and it hides the compiler bugs rather than fixing them, which does nothing for the people maintaining GCC and LLVM.
The gap. A property that two large compiler communities actively maintain, whose violation can miscompile the Linux kernel, had no test generator at all.

How you generate a program that breaks a debug flag

Mutating for complicated metadata rather than for complicated behaviour

Generate a random seed program

Start from an existing random C program generator, or from a mutation-based one seeded with the compiler regression tests. This part is off the shelf and produces programs with interesting control flow and arithmetic, which is what compiler fuzzing has always needed.
↓ →
Lower the code into many more small pieces new

Randomly pick statements and expressions and break them apart — a loop becomes conditional jumps and labels, a compound expression becomes a chain of assignments to fresh variables. This resembles what a compiler does internally, except that it is deliberately non-deterministic and its goal is to multiply the number of named things the compiler must describe.
↓ →
Lie about where every line of code lives new

Positions are a large fraction of debug information, so falsify them. The language has a directive that overrides the file and line of the code following it, and unlike the alternatives it needs no real file to exist, produces a single source file, and accepts any random string as a filename. Scatter these through the program, plus random whitespace and line breaks.
↓ →
Order the optimizer around at statement granularity new

Command-line flags only turn whole optimization levels on and off. Compiler ATTRIBUTES are finer: randomly attach them to types, functions, variables and labels to force inlining, forbid it, change alignment or packing, or declare a code region hot or cold — pushing the optimizer down paths a plain program would never reach.
↓ →
Compile twice and compare the instructions new

Build each variant with the same optimization flags and two different debug settings, then diff the machine code. This is the entire oracle. It needs no reference compiler, no execution, and no reasoning about what the program is supposed to compute.
↓ →
Shrink the failing program and report it new

A variant that triggers a difference is far too large to be a bug report, so run it through automatic program reducers until every remaining token is necessary. The only manual step in the whole pipeline is analysing the reduced program and filing it upstream.

Ten variants per seed, run continuously against nightly builds of GCC and Clang across optimization levels from none to maximum, with and without link-time optimization.

The idea

There was a third property, sitting between two well-studied ones. Compiler testing has spent years on whether a binary computes what the source says, and a smaller but real effort on whether the debug information correctly describes the optimized code. This paper points out that neither of those asks the question the crash-debugging workflow actually depends on, which is whether two compilations of the same source with the same optimization flags produce the same instructions. The property has a name in neither the literature nor the documentation, but both compiler communities were already maintaining it by hand — GCC ships a dedicated comparison flag, LLVM has a validation script and a tracking issue for making its passes invariant to debug information. What was missing was anyone trying to break it on purpose. Naming a property that a system was already informally relying on, and discovering that nobody had ever tested it because it had never been written down, is the same unstated-invariant-becomes-a-target move covered elsewhere on this site where a compilation guarantee people assumed held turns out to be violated routinely once somebody constructs the comparison.

The oracle costs nothing, and that changes what programs are usable. The hardest part of compiler testing is normally the oracle. To claim a miscompilation you must run the program, which means the program has to be free of undefined behaviour, and generating interesting C that avoids all of it is a research problem in itself. Here the oracle is a diff of two binaries. No execution, no reference implementation, no semantic reasoning — and therefore no requirement that the test program be well defined. It only has to COMPILE. The authors make this concrete in the sharpest way available: they take the GCC bug that miscompiled the Linux kernel, which was originally found as a miscompilation, and show it is also a violation of this property, then reproduce it with a program containing a signed integer overflow. That program is worthless for miscompilation testing and perfectly serviceable here. Rebuilding a hard testing problem around a cheaper oracle so that a much larger space of inputs becomes admissible is the same weaken-the-validity-requirement trade covered elsewhere on this site where a generator gets dramatically more productive once the property being checked stops demanding well-formed executions.

Mutate for metadata, not for behaviour. Existing compiler fuzzers mutate programs to diversify what they compute and which optimizations fire. Dfusor mutates for a different target: making the compiler emit more debug information, and stranger debug information, on the theory that the code paths handling that material are where the bugs live. Each of the three transformations attacks a different part of the metadata. Lowering multiplies the variables and control-flow structures that must be described. Position obfuscation falsifies the file and line of everything, using a directive that is far more common in real compilation than it looks, since preprocessors and parser generators emit it constantly. Attributes push the optimizer into corners. The result is roughly three times as many debug entries per program and 36 percent more distinct kinds of entry, worth up to 6.82 percent additional branch coverage inside the compiler, and 14 of the 23 bugs were reachable only through these transformations rather than through the seed generators alone. Aiming a mutation strategy at the compiler’s own internal bookkeeping rather than at the program’s semantics is the same target-the-machinery-not-the-workload instinct covered elsewhere on this site where a test generator is redirected at the part of a system nobody was exercising.

Following one program from seed to filed bug

How a randomly generated C program becomes a GCC bug report

Measurements from the paper: nightly GCC and Clang builds tested over ten months on a desktop with a six-core processor and 64 gigabytes of memory.

Step 1 of 5

Start with a random program that compiles

An existing generator produces a seed. Notably, it does not need to be free of undefined behaviour, because nothing here will ever run it. That single relaxation makes a far larger space of programs usable than miscompilation testing allows, and it is what lets this technique reach bugs that other approaches structurally cannot.

three different generators are used as sources, and they alone account for 9 of the 23 bugs, three each
the remaining 14 bugs require the transformations described next, so most of the value is in the mutation rather than the generation

Why it works

The property is checked by comparing two outputs of the same compiler → no reference implementation and no execution are needed → so the oracle is a diff and costs essentially nothing to evaluate.
Nothing is ever run, so the test program need not be well defined → programs containing undefined behaviour become usable inputs → opening a space of tests that miscompilation fuzzing must exclude by construction.
Debug information is carried through every optimization pass rather than generated at the end → passes see extra material they must not mistake for real instructions → so mutating for metadata volume reaches the optimizer logic where the bugs actually are.
Compiler attributes control inlining, alignment and hotness per declaration → the optimizer can be steered far more precisely than command-line flags allow → reaching code paths that no plain program would trigger.
A position-overriding directive needs no real file and accepts any string as a name → source positions can be falsified arbitrarily inside one file → giving maximal metadata disruption while keeping the failing program easy to reduce.

What it costs (trade-offs and scope)

The bug count is modest and the timeline is long. 23 bugs over ten months, most of which went into building the tool. This is a first result establishing that the problem exists, not a high-throughput bug finder.
It only says the binaries differ, not which one is wrong. A difference is evidence of a mishandled pass, but diagnosing it still requires reading fixing commits and talking to compiler developers, and the authors could only trace root causes for 11 of the 18 fixed bugs.
One transformation contributes little on its own. Position obfuscation barely moves coverage and does not change the number of distinct metadata entry types at all, since it only rewrites line records.
Seeds must be small single-file programs. Real open-source code cannot be used, because existing reducers cannot shrink large multi-file programs, so the test inputs are all synthetic.
A user-level workaround already exists. Building with debug information and stripping it afterwards guarantees identical machine code. It is impractical because of the build cost, and it hides the bugs rather than fixing them, but it does mean this is a compiler-maintainer problem more than an end-user one.

The evaluation

Three things are measured. Whether the transformations actually complicate debug information, by counting entries and distinct entry kinds in the emitted DWARF across 100 seeds and their variants. Whether they reach more compiler code, by measuring line, branch and function coverage of GCC and Clang while compiling. And whether they find real bugs, by running continuously against nightly builds and reporting upstream. The coverage and metadata results are statistically significant, and the bug results were taken seriously by both projects — GCC confirmed all nine reports and fixed seven of them quickly.

Property	Testing for miscompilation	Testing debug info correctness	Testing this property
The question being asked
What the oracle requires
Constraint on test programs
Where the bugs turn out to live
Research attention before this

Across 100 seeds, the transformations produce 214 percent more debug information entries and 36 percent more distinct entry types, with individual variants reaching over 22 times the entries of their seed. Branch coverage of the compiler rises by 6.00 percent for Clang and 6.82 percent for GCC. Of the 23 bugs found, 14 required the transformations rather than the seed generators alone, and 4 required all three at once.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This is a first result, and its scale reflects that: 23 bugs over ten months, with at least half that time spent building the tool rather than running it. The technique also stops at detection. A difference between two binaries proves a pass mishandled debug information somewhere, but not where — the authors had to read fixing commits and compiler developer feedback to characterise root causes, and could only do so for 11 of the 18 fixed bugs, because seven of the Clang fixes had no identifiable commit. The three transformations are unequal. Code lowering carries the result, contributing the most bugs and the most coverage. Attribute insertion more than doubles the coverage gain in GCC but barely moves it in Clang. Position obfuscation adds least of all and, by construction, does not change the number of distinct metadata entry types at all, since it only rewrites line records. Test inputs are entirely synthetic, because real open-source programs are too large and too multi-file for existing reducers to shrink. And it is worth keeping the practical stakes in proportion: a user can already sidestep the whole problem by building with debug information and stripping it afterwards, which guarantees identical machine code. That costs the full 610 percent incremental build penalty and does nothing to help GCC and LLVM actually implement the property, which is the audience this work is aimed at.

Background

When the cheap oracle is the whole contribution

This paper sits in the site’s compilers and correctness clusters and rhymes with several ideas covered elsewhere on this site. Its closest sibling is the unstable code page, which uses the same shape of test — compile one source twice with different flags, diff the results, and treat any difference as evidence — except that there the varying flag is the optimization level and the culprit is undefined behaviour, while here the varying flag is debug information and the culprit is a pass that cannot tell metadata from instructions. Its central insight, that relaxing what a test program must satisfy is worth more than making the generator cleverer, rhymes with the deep learning compiler testing page, where the hard part is producing inputs that are simultaneously valid and interesting, and with the mutation-based compiler fuzzing page, where the question is what a mutation operator should be aiming at. Its subject — that the tooling around a system can be quietly broken while the system itself is fine, and that debugging support is a first-class property rather than a convenience — is the same the-tools-are-part-of-the-system argument the reconfigurable hardware debugging page makes. And its method of testing an implementation against a property nobody had written down is the same contract-shaped testing the speculation contract page applies to processors.

If you remember only three things

1. Released binaries omit debug information, because it makes reverse engineering easier and because it is expensive — a full LLVM build takes 56 percent longer with it, an incremental build after one file changes goes from under 5 minutes to almost 35, and the binaries grow about thirtyfold. So when a released binary crashes, the recovery is to rebuild with identical flags plus the debug flag and read the core dump. That silently assumes the two builds contain the same instructions. Compilers carry debug information through every optimization pass, and a pass that mistakes it for real code makes different decisions, which hides the bug, confuses the debugger, or once miscompiled the Linux kernel.
2. The property was already being maintained by hand — GCC ships a comparison flag, LLVM has a validation script and a tracking issue — but nobody had ever tried to break it on purpose. Dfusor does, by mutating random programs for a target other than behaviour: lowering statements and expressions into many small pieces to multiply the entities that must be described, falsifying source positions with a directive that overrides file and line and needs no real file, and attaching compiler attributes to force or forbid inlining and set alignment and hotness per declaration. The result is roughly three times the debug entries and 36 percent more distinct entry kinds, worth up to 6.82 percent more branch coverage inside the compiler.
3. The oracle is a diff of two binaries, which means nothing is executed, which means the test program does not have to be well defined — only compilable. That is a much weaker requirement than miscompilation testing imposes, and the authors demonstrate what it buys by reproducing the Linux kernel miscompilation as a violation of this property, using a program containing a signed integer overflow that would be useless for miscompilation testing. Over ten months the tool found 23 bugs, 9 in GCC and 14 in Clang, with 21 confirmed or fixed, and every root cause traced was in the middle end or back end.

Check yourself

Check your understanding

1. A flag that asks the compiler for extra metadata seems like it should not affect the generated instructions. Predict why it can.
2. Miscompilation testing needs random programs free of undefined behaviour, which is itself a hard research problem. Predict what this technique requires of its test programs, and why that matters.
3. Existing compiler fuzzers mutate programs to diversify behaviour and trigger different optimizations. Predict what Dfusor mutations aim at instead, and how they achieve it.
4. A developer could sidestep this entire problem by always building with debug information and then stripping it from the release binary, which guarantees both binaries contain identical machine code. Predict why the paper does not treat that as a solution.

Sources

Theodore Luo Wang, Yongqiang Tian, Yiwen Dong, Zhenyang Xu and Chengnian Sun. “Compilation Consistency Modulo Debug Information.” ASPLOS 2023, Vancouver. University of Waterloo. DOI 10.1145/3575693.3575740. The tool is named Dfusor.
Full text of the paper was used for this explainer, including the background and formal problem statement in Section 2, the three program transformations in Section 3, the bug results, root cause analysis, coverage measurements and debug information metrics in Section 4, and the discussion of the strip workaround and the Linux kernel miscompilation reproduction in Section 5.