Dynamic Partial Deadlock Detection and Recovery via Garbage Collection

In Go programs, a background thread can get stuck forever waiting on a message that will never come — a bug called a partial deadlock that silently leaks memory until a long-running service crashes; this work from Aarhus, Washington University, and Uber makes a clever observation that the garbage collector already knows the answer, because a thread whose communication channel is no longer reachable in memory can never be woken, and it builds GOLF, a small extension to Go's collector that detects these stuck threads for free and shuts them down to reclaim their memory, catching 252 real leaks in a day on an Uber service.

The argument in one glance

Problem

Stuck Go threads silently leak memory until a service crashes

In Go, lightweight threads (goroutines) that wait on a message or lock can become permanently blocked while the rest of the program runs on — a partial deadlock, or goroutine leak. These stuck threads hold their memory, so a long-running service slowly bloats and eventually crashes with out-of-memory. The garbage collector cannot reclaim them because it does not know they are stuck forever.
What people did

Detect leaks with test tools or production profilers

Use a test-suite tool that checks for lingering goroutines after each test, or a production profiler that heuristically flags suspicious blocked goroutines, or static analysis of the source code.
Why it fell short

None is both production-ready and sound

The test-suite tool cannot run in long-lived production services. The production profiler reports both false alarms and missed leaks, so it is unsound and cannot be trusted to fix anything automatically. Static analyses struggle at the scale of real Go codebases. Nothing both runs cheaply in production and is sound enough to safely reclaim the leaked memory.
This paper's idea

Let the garbage collector prove the deadlock (GOLF)

The key insight is that memory reachability, which the garbage collector already computes, soundly over-approximates whether a blocked goroutine can ever wake up: if the channel or lock it waits on is unreachable from any live goroutine, nothing can ever unblock it, so it is definitely deadlocked. GOLF extends the Go collector to piggyback this detection on its marking phase, then forcefully shuts down the deadlocked goroutines to reclaim their memory — at negligible overhead, catching 252 real leaks in a day at Uber.

Why this problem is worth solving

Go’s concurrency is easy to use, which is exactly why a certain class of bug is everywhere in Go code.

Go is built around lightweight concurrent threads called goroutines, managed by the Go runtime, that communicate either through shared memory guarded by locks or, more idiomatically, by passing messages over channels — first-class queues that goroutines send to and receive from. A goroutine blocks when it has to wait: to receive a message that has not arrived, to send one that no one is ready to receive, or to acquire a lock someone else holds. If every goroutine in the program ends up blocked at once, the runtime detects a global deadlock and reports a fatal error. That case is rare and loud. The dangerous case is quiet: a partial deadlock, also called a goroutine leak, where some but not all goroutines are permanently blocked, unable to make progress in any possible future execution, while the rest of the program carries on normally. Because the program does not halt, nothing announces the problem. And partial deadlocks are not exotic — they arise frequently in real Go programs whenever an unexpected code path or thread schedule leaves a goroutine waiting for something that will never happen.

The reason these leaks are so damaging is that the language’s own safety net cannot see them.

A blocked goroutine is not free. It holds its stack and everything reachable from it, so a leaked goroutine is a leaked chunk of memory. In a short program this is harmless, but in a long-running service — exactly what Go is used to build — leaked goroutines accumulate silently over days and weeks, quietly raising memory use and CPU, until the service runs out of memory and crashes. The authors show a real production service at Uber whose count of blocked goroutines climbed into the tens of thousands, hidden by routine weekday redeployments but spiking over weekends and holidays when no restart cleared it. The cruel part is that Go’s garbage collector, which automatically reclaims memory no longer in use, is powerless here: it has no idea that a blocked goroutine will never wake, so it conservatively keeps that goroutine and its memory alive forever. Existing tools each fall short. One checks for lingering goroutines after a test run, but cannot be used in a live production service. Another runs in production but relies on heuristics, so it both raises false alarms and misses real leaks. Static analyses that inspect the source code struggle at the scale of large Go codebases. What is missing is a detector cheap enough to run in production and sound enough that its verdict can be trusted to automatically fix the leak.

What people did before, and where it broke

Check for lingering goroutines after tests. Run the test suite and flag goroutines still alive at the end. Where it broke. This only works in a test harness that ends; it cannot be applied to a long-running production service where the leaks actually accumulate.
Heuristically profile production goroutines. Watch blocked goroutines in production and flag suspicious ones. Where it broke. The heuristics are unsound, producing both false alarms and missed leaks, so the results cannot be trusted to safely act on.
Analyze the source code statically. Reason about the program’s concurrency without running it. Where it broke. Static analysis of concurrency has had mixed success and struggles to scale to large real-world Go codebases.
The gap. No approach was simultaneously lightweight enough for production and sound enough that its deadlock verdicts could be trusted to automatically reclaim the leaked memory. Providing exactly that, by leaning on the garbage collector, is what GOLF does.

How GOLF works

From invisible goroutine leaks to garbage-collector-proven deadlocks

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Teach the collector to reclaim goroutines that are dead but look alive. A leaked goroutine is dead in every meaningful sense — it will never make progress — yet the garbage collector keeps it and its memory alive because it cannot tell. GOLF closes that gap, giving the runtime the ability to recognize these permanently-stuck goroutines and reclaim the memory they were needlessly holding. Teaching the memory system to identify data that is effectively dead and stop wasting capacity keeping it around is the same instinct the warehouse-scale memory-traffic work covered elsewhere on this site is built on, recognizing when data no longer matters so that resources are not spent preserving or moving what nothing will ever use again.

Only declare a goroutine dead when it is provably unable to wake. The heart of GOLF’s soundness is that memory reachability over-approximates liveness: a blocked goroutine can only be woken by another goroutine acting on the same channel or lock, so if that object is unreachable from every live goroutine, no such wake-up can ever occur, and the goroutine is provably deadlocked — never a false alarm. Acting only when a conservative check proves the action is safe, so that an automatic optimization can never produce a wrong outcome, is the same discipline the copy-on-write pinning work covered elsewhere on this site follows, taking the drastic step only in the cases it can prove are safe and otherwise leaving well enough alone, so correctness is never at risk.

Read a hidden property off the memory graph. Nowhere does a Go program record which goroutines are deadlocked; GOLF infers it, deducing an unstated fact — permanent blocking — purely from the shape of the reachability graph the collector traces, turning a structure kept for one purpose into the answer to a different question. Recovering information that was never explicitly written down by tracing how memory and pointers connect is the same technique the binary-lifting work covered elsewhere on this site uses, reconstructing structure that is absent from the raw form by following how the pointers are actually used, so a property no one recorded can still be read out.

Following a leaked goroutine through GOLF

Detecting and reclaiming a goroutine stuck on a send that never completes

A trace of GOLF handling one partial deadlock, following the paper's reachability-based detection.

Step 1 of 5

A goroutine waits to hand off a result

A goroutine is launched to send a result over a channel to a worker, but due to an unexpected path the worker exits without ever receiving, and drops its only reference to that channel.

The send has no partner and never will.

Why it works

Partial deadlocks are common and silently costly. Because some goroutines can block forever while the program keeps running → they leak memory and raise CPU with no error → so a long-running service degrades and crashes with no obvious cause.
Reachability soundly proves permanent blocking. Because a blocked goroutine can only be woken by another goroutine acting on the same object → if that object is unreachable from any live goroutine, no one can ever wake it → so unreachability is a sound proof of deadlock, never a false alarm.
The garbage collector already computes reachability. Because the collector’s marking phase traces exactly which memory is reachable from live goroutines → the information detection needs is produced anyway → so piggybacking on it makes detection nearly free.
Soundness makes automatic recovery safe. Because GOLF flags a goroutine as deadlocked only when it is provably unable to wake → shutting it down cannot kill a goroutine that might still make progress → so the runtime can safely reclaim its resources automatically.
Low overhead makes it usable in production. Because the analysis reuses work the collector does regardless and adds little → its runtime overhead is negligible → so it can run continuously on real services, unlike test-only or unsound tools.

What it costs (trade-offs and scope)

Reachability over-approximates, so some leaks are missed. Reachability soundly proves a goroutine is deadlocked when the object is unreachable, but a goroutine can be deadlocked while its channel is still reachable from some live-but-idle goroutine. GOLF is sound (no false positives) but not complete — it detects the reachability-provable subset, which is why it caught 94 percent of microbenchmark leaks but only about half in a large industrial test suite.
It focuses on concurrency-operation blocking, not system calls. The analysis targets goroutines blocked on channels, locks, and similar concurrency objects. Blocking caused by system calls, such as network or file input and output, is outside its scope.
It requires modifying the language runtime. GOLF is an extension to the Go garbage collector, which demands careful attention to many details of the Go runtime. Its benefits come from being built into the runtime, not from a library a program can simply import.
Forcefully shutting down goroutines has semantics to respect. Automatic recovery terminates deadlocked goroutines. Because they are provably unable to make progress, this is safe, but forcibly ending a goroutine is an intervention whose interaction with program cleanup and resource release must be handled correctly.
The results are early but real. The paper presents preliminary results; the detection rates vary by workload, and the industrial numbers reflect specific Uber codebases and services. The production deployment is strong evidence, but broader validation across ecosystems is future work.
It is specific to garbage-collected, message-passing runtimes. The core idea depends on having a garbage collector whose reachability corresponds to liveness. Languages without such a runtime would need a different mechanism.

The evaluation

Three ways to find partial deadlocks in Go

Property	Goleak (test suites)	Production profiler
Runs in long-lived production		Yes
Sound (no false alarms)	Not applicable
Automatically reclaims leaked memory
Runtime overhead	Test-time
Mechanism	Lingering goroutines after tests	Heuristic profiling

Three ways to find partial deadlocks (goroutine leaks) in Go, as compared in the paper. Goleak checks for goroutines still alive after a test run — effective but usable only in test suites, not long-lived production services. A production profiler can watch live services but relies on heuristics that both raise false alarms and miss real leaks, so it is unsound and cannot be trusted to fix anything. GOLF instead extends the Go garbage collector: its key insight is that memory reachability soundly over-approximates liveness, so a goroutine blocked on a channel or lock unreachable from any live goroutine is provably deadlocked. It computes this during the collector's marking phase, so overhead is negligible, and it forcefully shuts down deadlocked goroutines to reclaim their memory. In experiments it detected 94 percent of partial deadlocks across 73 microbenchmarks, reported 180 of 357 known leaks across 3,111 Uber packages (about 50 percent), and, deployed on a real Uber micro-service, revealed 3 programming errors causing 252 partial deadlocks over 24 hours — with negligible overhead, demonstrating production usability. Being sound but not complete, it never falsely accuses a goroutine but does miss leaks whose objects remain reachable.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

GOLF is an elegant piece of runtime engineering built on a genuinely clever observation. Partial deadlocks — goroutines stuck forever while the program runs on — are a pervasive and costly bug in Go, silently leaking memory until long-running services crash, and Go’s own garbage collector was blind to them. The insight that turns this around is that reachability, which the collector already computes, soundly over-approximates liveness: a blocked goroutine can only be woken by another goroutine acting on the same channel or lock, so if that object is unreachable from any live goroutine, the block is permanent and provable. By computing this during the collector’s existing marking phase, GOLF gets detection almost for free, and because the verdict is sound — a goroutine is only ever declared deadlocked when it truly cannot wake — the tool can safely go beyond detection to automatic recovery, shutting the goroutine down and reclaiming its memory. The evidence is compelling and refreshingly real: 94 percent of microbenchmark leaks detected, about half of known leaks across thousands of Uber packages, and, deployed on an actual Uber service, 252 partial deadlocks from three programming errors caught in a single day, all at negligible overhead. Read the scope precisely. GOLF is sound but not complete: it detects only the leaks that reachability can prove, so a deadlock whose channel is still reachable from an idle goroutine is missed — hence the roughly 50 percent rate on the industrial suite versus 94 percent on microbenchmarks. It targets concurrency-operation blocking, not system calls; it requires modifying the language runtime; and its automatic shutdown of goroutines, though safe because they are provably stuck, is a real intervention. The durable idea is that a mechanism a system already runs for one purpose — here, reachability tracing for memory management — can, with the right insight, soundly answer an entirely different and hard question, and do so cheaply enough to run in production.

Background

Why repurposing an existing analysis is such a powerful move

It helps to see GOLF within a recurring and productive pattern: some of the best systems results come not from building a new mechanism but from noticing that a mechanism already running, for a completely different reason, secretly contains the answer to a hard problem — and can be tapped almost for free. Here the mechanism is the garbage collector’s reachability tracing, run to manage memory, and the hard problem is proving that a concurrent thread is permanently deadlocked; the bridge is the observation that reachability soundly bounds liveness. The value is compounding: because the work is already being done, the new capability costs almost nothing, which is exactly what makes it viable in production where a heavyweight analysis would not be. The speculative-execution-testing work covered elsewhere on this site reflects a kindred instinct on the defensive side, systematically and automatically surfacing latent, hard-to-see defects that human reasoning routinely misses, so that problems are caught by machinery rather than by luck. The common thread is that latent bugs in large concurrent or speculative systems are too subtle and too numerous to find by inspection, and the winning strategy is to derive their presence soundly from information the system already exposes. As software grows more concurrent and longer-running, this way of thinking — reuse the reachability, the traces, the checks you already compute, and read new guarantees out of them — is likely to yield more of these quietly powerful tools.

If you remember only three things

1. Go programs use lightweight threads called goroutines that communicate by passing messages over channels or by sharing memory under locks, and a goroutine blocks when it waits on such an operation. If all goroutines block, the runtime reports a global deadlock (rare); the common and insidious case is a partial deadlock, or goroutine leak, where some goroutines are permanently blocked forever while the program runs on. These stuck goroutines hold their stacks and reachable memory, so in a long-running service they accumulate silently, raise memory and CPU, and eventually crash it with out-of-memory. The garbage collector cannot help because it does not know a blocked goroutine will never wake, and prior tools are either test-only, unsound in production, or poor at scale.
2. GOLF (Goroutine Leak Fixer) rests on the insight that memory reachability, which the garbage collector already computes, soundly over-approximates the liveness of concurrency operations: a goroutine blocked on a channel or lock can only be woken by another goroutine acting on that same object, so if the object is unreachable from any live goroutine, the goroutine is provably deadlocked. GOLF extends the Go garbage collector to compute this check during its existing marking phase, making detection nearly free, and — because the verdict is sound — it goes beyond detection to automatic recovery, forcefully shutting down the deadlocked goroutines so the collector reclaims their memory. Overhead is negligible, so it runs in production.
3. GOLF detected 94 percent of partial deadlocks across 73 microbenchmarks, reported 180 of 357 known leaks across 3,111 Uber packages (about 50 percent), and, deployed on a real Uber micro-service, revealed 3 programming errors causing 252 partial deadlocks over 24 hours, all at negligible overhead. Scope: it is sound but not complete, so it misses leaks whose objects remain reachable (why the industrial rate is lower than on microbenchmarks); it targets concurrency-operation blocking, not system calls; it requires modifying the language runtime; and its automatic goroutine shutdown, though safe because provably stuck, is a real intervention. The durable lesson is that a mechanism a system already runs for one purpose — reachability tracing for memory management — can, with the right insight, soundly answer an entirely different hard question, cheaply enough for production.

Check yourself

Check your understanding

1. What is a partial deadlock in Go, and why is it dangerous?
2. What is the key insight that lets GOLF detect these deadlocks soundly?
3. Why does GOLF implement its detection inside the garbage collector, and what does that enable?
4. GOLF detected 94 percent of leaks in microbenchmarks but only about 50 percent in a large industrial test suite. What explains this, and is it a soundness problem?

Sources

The paper is “Dynamic Partial Deadlock Detection and Recovery via Garbage Collection” by Georgian-Vlad Saioc, I-Ting Angelina Lee, Anders Moller, and Milind Chabbi, published at ASPLOS 2025, DOI 10.1145/3676641.3715990. The authors are at Aarhus University, Washington University in St. Louis, and Uber.

This explainer is based on the full text of the paper, obtained as the open-access (CC-BY) author copy hosted on an author’s university website. The problem framing (Go’s goroutines and channels, the distinction between global deadlocks and partial deadlocks or goroutine leaks, the memory-leak and out-of-memory consequences in long-running services, the illustrative Uber production service with tens of thousands of blocked goroutines, and the garbage collector’s inability to reclaim deadlocked goroutines), the limitations of prior tools (Goleak for test suites, LeakProf for production but unsound, and static approaches at scale), the key insight (that memory reachability soundly over-approximates the liveness of concurrency operations, so a goroutine blocked on an object unreachable from any live goroutine is provably deadlocked), the GOLF design (extending the Go garbage collector to compute reachable liveness during marking, and forcefully shutting down deadlocked goroutines to recover memory), and the evaluation results (94 percent detection across 73 microbenchmarks with 121 relevant goroutine-creating instructions, 180 of 357 known leaks reported across 3,111 Uber packages, 3 programming errors causing 252 partial deadlocks detected over 24 hours on a real Uber service, and negligible runtime overhead) are all drawn directly from the paper. Background explanations of goroutines, channels, locks, garbage collection, and reachability are standard and provided to make the paper’s contribution understandable.