Go was designed to make concurrency safe by having goroutines talk through channels instead of sharing memory — but the order in which a select statement picks among ready channels is non-deterministic, so a program must be correct under EVERY possible order, and the nastiest bugs hide in the orders your tests never happened to run; this paper (Penn State) builds GFuzz, a fuzzer that deliberately REORDERS concurrent messages — rewriting each select so a chosen case goes first, with a timer fallback so it never invents a fake deadlock — steers the search with a channel-state feedback signal (new operation pairs, buffer fullness, close events) rather than generic branch coverage, and adds a runtime sanitizer that tracks how channel references flow between goroutines to catch the partial deadlocks Go's own detector misses; run on Docker, Kubernetes, gRPC and four other large systems it found 184 previously unknown bugs, 124 confirmed and 67 fixed, versus 25 for the best prior tool.
The argument in one glance
-
Problem
Go made concurrency "safer" with channels — but channel bugs are still everywhere
Go is a language built for concurrency. Its designers pushed message passing over shared memory: instead of goroutines (lightweight threads) reading and writing shared variables behind locks, they should communicate by sending messages through channels, and wait on several channels at once with the select statement. The slogan is "share memory by communicating," and the belief is that message passing is less error-prone. Go runs critical infrastructure — Docker, Kubernetes, gRPC — on this model. And yet an empirical study found that channel-related concurrency bugs are still common in Go, and in some circumstances MORE likely than bugs from misusing traditional primitives like mutexes. They come in two flavors: BLOCKING bugs, where goroutines get stuck forever (a send nobody will receive, a select or a range that never proceeds), and NON-BLOCKING bugs, where everything finishes but wrongly (a panic from sending on a closed channel).
-
What people did
Point existing bug detectors at Go
There are several ways to hunt concurrency bugs. Shared-memory race detectors (the standard tools for C, C++, and Java) monitor shared variables and locks. Model checkers can systematically explore every possible ordering of events. Static analyzers for Go — Staticcheck, the vet tool, and the state-of-the-art GCatch — reason about channel operations without running the program. And ordinary dynamic detectors watch a program as it runs and report the bugs they see. Each of these has been used to look for concurrency problems.
-
Why it fell short
None of them force the untried order where the bug lives
Grade them against Go's message-passing bugs. Shared-memory race detectors do not understand channels at all — they watch the wrong primitive. Model checking can, in theory, examine every message order, but a real Go system has an astronomically large order space, so exhaustive checking does not scale. Static detectors for Go cover only limited patterns, cannot compute precise alias or loop-iteration information, report false positives, and (GCatch) detect only blocking bugs, missing the non-blocking ones. Worst of all, ordinary dynamic detectors only report bugs that surface in the order that HAPPENED to run — they have no way to ALTER the execution to expose a bug hiding in an order the test never exercised. And Go's own built-in deadlock detector fires only when ALL goroutines are blocked, so it misses partial deadlocks where just a few goroutines are stuck. The bug is in the order you never ran, and nothing here goes looking for it.
-
This paper's idea
Fuzz by reordering the messages — decide who goes first
This paper (Penn State — Liu, Xia, Liang, Song, Hu) builds GFuzz, a dynamic detector that actively MUTATES the processing order of concurrent messages to drive a program into orders it would not normally take. The logic: because select is non-deterministic by design, a correct Go program must work under every order, so systematically trying different orders — deciding "who goes first" — surfaces the orders programmers forgot to handle. GFuzz has three parts. (1) It identifies which messages are concurrent (those waited on by the same select) and ENFORCES a chosen order by rewriting each select into a switch that prioritizes one case, with a timer fallback so it never invents a deadlock that could not really happen. (2) It searches the huge order space like a coverage-guided fuzzer, but with a feedback signal made of CHANNEL STATES — new operation pairs, fuller buffers, new close events — rather than branch coverage, favoring orders that show new channel behavior. (3) It adds a runtime sanitizer that tracks how channel references propagate among goroutines and flags a goroutine that can never be unblocked, catching partial deadlocks the Go runtime misses. On seven large systems it found 184 previously unknown bugs (124 confirmed, 67 fixed), versus 25 for the best prior tool.
Why this problem is worth solving
Go’s whole concurrency story rests on the claim that message passing is
safer than shared memory, and it is the substrate of some of the most
important server software running today — so if channel bugs are common
and hard to catch, that matters for real reliability. The mechanism to
understand first is the trio at the center of Go concurrency. A
a GOROUTINE is a lightweight
thread, cheap enough to spawn thousands of, scheduled by the Go runtime
in user space. A CHANNEL is a typed conduit for passing messages between
goroutines: one goroutine sends a value into it, another receives; a
channel can be unbuffered (a send blocks until a receiver is ready) or
buffered (a send blocks only when the buffer is full), and it can be
closed. A SELECT statement lets a goroutine wait on several channel
operations at once and proceed with whichever becomes ready first, with
an optional default clause and no guaranteed choice among simultaneously
ready cases. Together they replace locks and shared variables with
explicit communication — but they add their own failure modes, because
whether an operation blocks depends on the buffer, the closed state, and
which other goroutines still hold the channel
The specific failures fall into two kinds, and knowing the difference
explains both what GFuzz must trigger and how it must detect. A
a BLOCKING bug leaves one or
more goroutines stuck forever at a channel operation that will never
complete — a goroutine sending into an unbuffered channel that no one
will ever receive from, a select waiting on channels that will never be
ready, or a range loop reading a channel that is never closed. The
program hangs (fully or partially) rather than crashing. A NON-BLOCKING
bug is the opposite: every goroutine finishes, but something goes wrong
along the way — most often a panic, such as sending a value on a channel
that has already been closed, or closing an already-closed channel,
which Go turns into a runtime error. The two need different detection:
the Go runtime already catches non-blocking channel panics, but nothing
in the runtime reliably catches a goroutine that is merely stuck and
can never make progress
What people did before, and where it broke
- Shared-memory race detectors (for C/C++/Java). Monitor shared variables and locks for data races. They watch the wrong primitive entirely — they neither understand channels nor detect any message-passing bug.
- Model checking. Systematically explore all possible message orders to find liveness and safety violations. Correct in principle, but a real Go program has a huge number of possible orders, so exhaustive exploration does not scale to million-line systems.
- Static Go detectors (Staticcheck, vet, GCatch). Analyze channel operations without running the program. They cover limited buggy patterns, cannot compute precise alias information or loop-iteration counts, report false positives, and GCatch — the best of them — detects only blocking bugs, missing non-blocking ones.
- Ordinary dynamic detectors. Watch a running program and report the bugs that appear. But they only see the order that happened to execute; without the ability to alter the execution, they miss every bug that requires a different order.
- Go’s built-in deadlock detector. Fires only when ALL goroutines are blocked, so it cannot report a partial deadlock where a few goroutines are stuck while the program otherwise runs — which is what most real blocking bugs look like.
How GFuzz finds order-dependent bugs
Reorder concurrent messages, steer with channel-state feedback, and detect the stuck goroutines
-
Start from the non-determinism
A select picks non-deterministically among ready channel operations, so a correct program must work under EVERY order. The bug hides in an order the test suite never happened to run. GFuzz treats "which message goes first" as the thing to fuzz.
-
Identify which messages are concurrent new
GFuzz takes a simple, safe rule: the channel operations waited on by the SAME select statement are concurrent — they have no happens-before relation, so any of them could legitimately go first. Reordering only these preserves the program's semantics and cannot fabricate an impossible schedule.
-
Enforce a chosen order without inventing deadlocks new
GFuzz rewrites each select into a switch that prioritizes one chosen case, implementing that case as a small select with a TIMER. If the preferred message arrives before the timeout it goes first; if it genuinely cannot arrive, the timer expires and execution falls back to the original select — so forcing an order never creates a false deadlock that real runs could not produce.
-
Mutate orders and keep the interesting ones new
Starting from orders observed on the given inputs, GFuzz randomly changes an exercised choice to another valid one, generating new orders. It keeps and prioritizes orders that reveal NEW channel behavior, measured by a channel-state feedback signal rather than branch coverage.
-
Score orders by channel state, not branches new
The feedback tracks new consecutive channel-operation pairs, new channels created or closed, channels left open, and new maximum buffer fullness — the states in which channel bugs tend to occur. A weighted score decides how many mutations each order earns, spending effort on the orders closest to novel channel behavior.
-
Detect the bugs the runtime cannot new
The Go runtime already catches non-blocking panics. For blocking bugs, a runtime sanitizer reifies the goroutine-to-channel reference graph and, every second and at program exit, flags any blocked goroutine that no live goroutine could ever unblock — catching partial deadlocks the built-in all-goroutines-blocked detector misses.
The idea
Do not wait for the bug — perturb the schedule to force it out. The pivotal move is to stop being a passive observer. An ordinary dynamic detector reports whatever bug the current run happens to expose; GFuzz instead treats the message order as an input to fuzz and deliberately drives the program into orders it would not otherwise take, so order-dependent bugs that were always latent finally trigger. This actively-perturb-to-expose discipline is the same one the site’s security-fuzzing pages use — the page that fuzzes a black-box CPU against a speculation contract generates adversarial cases to make hidden leakage manifest, and the mutation-based program-repair and compiler-testing pages search a space of mutations rather than waiting for a failure to walk in. The insight shared across all of them is that the interesting behavior lives at the edges of the input (or schedule) space, and you reach the edges only by generating them on purpose.
Pick a feedback signal that matches the bugs you want, not the generic one. A fuzzer is only as good as the signal that tells it which inputs are worth pursuing, and standard fuzzers use branch coverage. GFuzz’s observation is that channel-related bugs occur when a channel is in a particular STATE — sending into a full buffer, closing an already-closed channel, leaving a channel open with a goroutine still waiting — so it measures novelty in channel states (new operation pairs, new buffer fullness, new close events) instead of code coverage, and favors orders that reach new channel states. This choose-the-signal-that-reflects-the- real-thing discipline recurs across the site whenever a system refuses a convenient proxy for the quantity that actually matters — the pages that schedule on measured device occupancy rather than a queue-length proxy, or that rank work by a task-specific measure rather than a generic count. The right coordinate to search in is the one that lights up when the bug is near.
Respect the real structure so you never chase a phantom. Two design choices keep the search sound. First, GFuzz only reorders messages that are genuinely concurrent — those under the same select, which by construction have no happens-before relation — so a reordering can never violate a real dependency and produce a schedule that could not occur. Second, the timer fallback means that when a preferred case truly cannot proceed, execution reverts to the original select instead of hanging, so enforcement never manufactures a false deadlock. Using a structural property to constrain a combinatorial search — rather than exploring blindly and filtering impossible results afterward — is the same exploit-the-structure move the site’s concurrency and side-channel pages make when a monotonicity or ordering property lets them prune or bound a space that is hopeless to enumerate. It is also what lets GFuzz sidestep the model-checker’s scalability wall: fuzz a structurally-safe subspace guided by feedback, rather than enumerate everything.
Forcing the order that breaks the program
From a latent order to a confirmed bug
Following GFuzz on a Docker blocking bug and the fuzzing loop. Numbers from the paper's evaluation on seven Go systems.
Step 1 of 5
The bug that hides in an order
A Docker parent goroutine runs a select over three cases: a one-second timeout, a value channel, and an error channel. A child goroutine will send on the value or error channel. If the timeout fires FIRST, the parent returns and drops its references to both channels — so the child blocks forever on its send into an unbuffered channel nobody can receive from. Normal testing never happens to make the timeout win.
both channels are unbuffered, so the child's send blocks until a receiver appears — and none ever will the bug needs exactly the "timeout goes first" order, which offline runs essentially never produce
Why it works
- Go’s select order is non-deterministic, so a correct program must work under every order, but tests exercise only a few → deliberately mutating the order drives the program into unexercised schedules → the bugs that hide in orders nobody tested finally trigger.
- Messages under the same select have no happens-before relation → reordering only those is always a schedule the program could really reach → forced orders expose real bugs without fabricating impossible ones.
- A preferred case is enforced with a timer that falls back to the original select → a case that genuinely cannot proceed never hangs the program artificially → enforcement never reports a false deadlock.
- Channel bugs occur in particular channel states → a feedback signal built from channel states (operation pairs, buffer fullness, close events) favors orders reaching new states → the search heads toward channel behavior where bugs live, far faster than blind mutation.
- A blocked goroutine is a real bug only if nothing can ever unblock it → a sanitizer that reifies the goroutine-channel reference graph can decide exactly that → partial deadlocks are caught that the runtime’s all-goroutines-blocked detector cannot see.
What it costs (trade-offs and scope)
- It only reorders messages. GFuzz mutates the processing order of concurrent messages, not program inputs, shared-memory access orders, or the interleaving of other synchronization primitives — so bugs that require one of those to surface are outside its reach.
- “Concurrent” means same-select only. It treats messages waited on by one select as concurrent, a deliberately conservative rule; genuine concurrency across different selects or goroutines is not exploited, so some reorderable bugs are missed.
- Coverage is bounded by the input tests. GFuzz reuses existing unit tests and does not generate program inputs, so it can only find bugs in code the tests actually exercise; static tools sometimes catch bugs in code no test reaches.
- Order enforcement is best-effort and adds overhead. The timer fallback means the exact chosen order is not always achieved, the window must be tuned (500ms worked best on gRPC), and the extra waits plus information collection cost about 3.0x runtime; the sanitizer adds under 20% for two apps but up to 75.2% for Go-Ethereum.
- Slowdown can cause a few false positives. If GFuzz slows a run so much that the test framework kills it (unit tests are killed after 30 seconds), it may not finish tracking channel references and can report a false alarm — the source of all 12 false positives.
The evaluation
Implemented on Go 1.16 and run on seven popular systems (Kubernetes, Docker, Prometheus, etcd, Go-Ethereum, TiDB, gRPC — 272K GitHub stars, 6887K lines, 8199 unit tests) on an Intel i7-8700K with 32GB RAM, using five parallel workers and reusing each project’s unit tests as seeds.
Finding channel-related Go bugs
| Property | Shared-memory race detectors | Static detector (GCatch) | GFuzz |
|---|---|---|---|
| Understands channels | |||
| Triggers new orders | |||
| Finds non-blocking bugs | |||
| Partial-deadlock detection | |||
| New bugs in three hours |
Check yourself
Check your understanding
1. Ordinary dynamic detectors run a Go program and report the concurrency bugs they observe, yet GFuzz argues that is fundamentally insufficient. Predict why, and what GFuzz does differently.
2. GFuzz is a coverage-guided fuzzer, but instead of branch coverage its feedback signal is built from channel states — new operation pairs, buffer fullness, close events. Predict why that signal choice matters.
3. When GFuzz forces a particular case of a select to "go first," it uses a timer that falls back to the original select if the preferred message does not arrive in time. Predict why that fallback is essential.
4. Go's built-in deadlock detector fires only when ALL goroutines are blocked, so GFuzz builds its own sanitizer. Predict how that sanitizer detects a blocking bug the runtime would miss.
Sources
- Who Goes First? Detecting Go Concurrency Bugs via Message Reordering (author PDF) — ASPLOS 2022 (full text read).
- ACM DL: Who Goes First? (DOI 10.1145/3503222.3507753) — published version.
- GFuzz source and artifact — code and experimental data (also archived at Zenodo 10.5281/zenodo.5893373).