Triton-Sanitizer: A Fast and Device-Agnostic Memory Sanitizer for Triton with Rich Diagnostic Context

Hao Wu, Qidong Zhao, Songqing Chen, Yang Chen, Yueming Hao, Tony C. W. Liu and 3 others

Out-of-bounds memory accesses are among the most common bugs in GPU code, and today's GPU sanitizers catch them by instrumenting every memory instruction in the binary — slow, tied to one vendor's hardware, and offering almost no clue about what went wrong; this paper (George Mason and Meta — Hao Wu, Qidong Zhao, Songqing Chen, Yang Chen, Yueming Hao, Tony C. W. Liu, Sijia Chen, Adnan Aziz, Keren Zhou) presents Triton-Sanitizer, the first device-agnostic memory sanitizer for Triton, which exploits Triton's tile-oriented semantics to build symbolic expressions for the addresses and guard masks and checks them with an SMT solver, falling back to eager simulation only for data-dependent indirect accesses — so it detects errors precisely with no false positives and no per-access instrumentation, and reports which tensor and which address computation is to blame, finding 24 unknown bugs across 7 repositories while running 1.07 to 14.66 times faster than compute-sanitizer.

The argument in one glance

Problem

GPU memory bugs are everywhere, and the tools that find them are slow and unhelpful

Memory access errors — reading or writing outside the bounds of an array — are among the most pervasive bugs in GPU programming, and they are especially common in the hand-written kernels that power deep learning. Finding them matters because an out-of-bounds access can silently corrupt results or crash, and GPU code is hard to debug. The standard tools are GPU memory sanitizers, such as NVIDIA's compute-sanitizer, which detect these errors while the kernel runs. But the way they work is costly: they instrument every memory instruction in the low-level intermediate representation or binary, adding a check around each access. That imposes high runtime overhead, slowing the program under test, and because they operate on the vendor's low-level binary, they are tied to a specific device. Worst of all, when they do catch an error, they hand back minimal context — an address that was out of bounds — which is far from enough to actually find and fix the bug in the source.
What people did

Instrument every memory instruction in the binary

Existing GPU sanitizers take the general, device-level approach: they operate on the compiled kernel and wrap every memory instruction with a bounds check at run time. This catches out-of-bounds accesses wherever they occur, and it works for any kernel because it does not need to understand the source, only the memory instructions in the binary.
Why it fell short

Per-access instrumentation is expensive, device-specific, and diagnostically poor

Instrumenting every memory instruction has three drawbacks. It is slow: adding a check to every access imposes high overhead, so sanitized runs are much slower than normal, which discourages routine use. It is device-specific: because it works on the vendor's low-level IR or binary, it is bound to that hardware and does not port across GPU vendors. And it is diagnostically poor: at the binary level the tool has lost the program's structure, so it can report that some address was out of bounds but not which tensor the access was meant for, what code path led there, or which address computation produced the bad value — leaving the developer to reconstruct all of that by hand. What is missing is a sanitizer that is fast, works across devices, and explains its findings well enough to fix them — which requires reasoning at a higher level than individual memory instructions.
This paper's idea

Reason symbolically about tile addresses, and explain the bug

Triton-Sanitizer is the first device-agnostic memory sanitizer designed for Triton, a domain-specific language for writing portable, efficient GPU kernels for deep learning. Its key insight is to exploit Triton's tile-oriented semantics: Triton kernels compute addresses and guard masks over tiles of tensors in a structured way, so instead of instrumenting every access, Triton-Sanitizer constructs symbolic expressions for the memory addresses and the masks that guard them, and verifies with an SMT solver whether any access could fall out of bounds. For the cases symbolic reasoning cannot resolve — indirect accesses whose addresses depend on runtime data (like gather or scatter) — it selectively falls back to eager simulation, actually evaluating those accesses. This hybrid analysis detects memory errors precisely, with no false positives, and without the cost of per-access instrumentation. Beyond detection, it generates rich diagnostic reports: it attributes each violation to the tensors nearest the violated address, tracks the complete call path, and exposes the symbolic operations responsible for the incorrect address. Evaluated on seven widely used open-source Triton kernel repositories, Triton-Sanitizer found 24 previously unknown memory access errors (8 already fixed and upstreamed by the authors) and ran 1.07 to 14.66 times faster than compute-sanitizer, averaging 1.62 times.

Why this problem is worth solving

Hand-written GPU kernels are the performance-critical core of modern deep learning, and their memory bugs are both common and costly.

A GPU kernel runs the same code across thousands of parallel threads, each computing memory addresses into large tensors from thread indices, tile offsets, and strides. This address arithmetic is intricate and easy to get subtly wrong — an off-by-one in a boundary condition, a missing guard on the last tile, a stride miscalculation — and when it is wrong, a thread reads or writes outside the intended array. Such out-of-bounds accesses are among the most pervasive bugs in GPU programming, and they are pernicious: because the GPU executes so many threads with little checking, a bad access may silently corrupt data, produce wrong results that are hard to trace back, or crash the program non-deterministically. Debugging is hard because GPU execution is massively parallel and opaque, and the error surfaces far from its cause. As deep-learning practitioners increasingly write custom kernels to squeeze out performance, the volume of such hand-written code — and thus of these bugs — has grown. A sanitizer that reliably finds these errors is therefore valuable, but only if it is cheap enough to run routinely and informative enough to actually fix the bug it finds, which is exactly where the existing tools fall short.

The reason the existing approach is limited is that it works at the wrong level of abstraction.

General GPU sanitizers like compute-sanitizer operate on the compiled kernel: they insert a bounds check around every memory instruction in the low-level intermediate representation or binary and run them at execution time. This universality is also the source of its problems. First, cost: checking every single access, across thousands of threads and every instruction, adds large overhead, so a sanitized run can be many times slower than an ordinary one, which makes developers reluctant to run it often. Second, portability: because the instrumentation targets a particular vendor’s low-level IR or binary format, the tool is tied to that device and cannot be reused across GPU vendors. Third, and most limiting for actually fixing bugs, diagnostics: by the time code is a binary, the high-level structure is gone — the tool sees a stream of loads and stores over raw addresses, not tensors, tile computations, or call structure. So when it detects a violation it can say little more than which numeric address was out of bounds, leaving the developer to reverse-engineer which tensor was involved, how execution reached that point, and which arithmetic produced the wrong address. The information a developer most needs to fix the bug is precisely what the binary level has discarded. Recovering it, and avoiding the per-access cost, requires analyzing the kernel where its structure is still visible — at the level of the domain-specific language — which is the opening Triton-Sanitizer exploits.

What people did before, and where it broke

Compute-sanitizer and similar binary-level sanitizers. Instrument every memory instruction in the vendor’s low-level IR or binary with a runtime bounds check. Where it broke. High per-access overhead makes sanitized runs slow, the approach is bound to a specific device, and it reports only a bad address with minimal context.
Manual bounds checking and code review. Have developers reason carefully about address arithmetic and add their own guards. Where it broke. The address math is intricate and parallel, so subtle boundary and stride bugs slip through, and review does not scale to large kernel codebases.
Generic dynamic memory tools ported to GPUs. Reuse CPU-style memory-error detectors on GPU code. Where it broke. They are not designed for the massively parallel, tile-structured nature of GPU kernels, and they inherit the same instrumentation cost and poor, structure-free diagnostics.
Testing with many inputs. Run kernels on varied inputs hoping to trigger and observe errors. Where it broke. Out-of-bounds accesses may not surface on the inputs tried, may corrupt silently, and testing gives no explanation of the cause when they do appear.
The gap. No sanitizer reasoned about GPU memory safety at the level of a domain-specific language’s tile semantics — symbolically checking addresses and masks, falling back to simulation only where needed, and reporting the responsible tensor, call path, and address computation — to be fast, device-agnostic, and diagnostically rich at once.

How Triton-Sanitizer works

Checking a Triton kernel's memory safety by symbolic analysis of tile addresses

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Analyze the kernel where its structure is still visible, not in the binary. The pivotal choice in Triton-Sanitizer is to work at the level of Triton, the domain-specific language, rather than the compiled binary. Triton’s tile-oriented semantics mean the address arithmetic and the masks that guard accesses are expressed in structured, analyzable form, so the sanitizer can reason about them directly instead of watching individual loads and stores go by. This both avoids the per-access cost of binary instrumentation and preserves the high-level information — tensors, tile operations, call structure — that a binary throws away. Reasoning about a program at a high-level intermediate representation that retains its structure, so analysis and optimization can be precise and portable, is the same principle the work on an intermediate representation for optimized tensor computations on GPUs covered elsewhere on this site is built on, meeting the computation where its structure is intact. Triton-Sanitizer uses that level for memory-safety checking rather than performance, which is what makes it both device-agnostic and diagnostically rich.

Prove safety symbolically, and only simulate what you must. The core mechanism is a hybrid analysis: construct symbolic expressions for addresses and masks and let an SMT solver prove whether any access can go out of bounds, catching real errors with no false positives and without running a check on every access. Where symbolic reasoning cannot resolve an access — indirect, data-dependent addresses — Triton-Sanitizer selectively falls back to eager simulation, paying that cost only where it is unavoidable. Using an SMT solver to verify a safety property precisely over all cases, rather than testing or instrumenting, is the same solver-backed verification discipline the work on scaling symbolic execution of hardware designs with query caching covered elsewhere on this site is built on, replacing per-instance checks with a proof. Triton-Sanitizer’s twist is the selective fallback, keeping the analysis complete for the indirect accesses a pure symbolic approach would miss.

Turn a detected error into an explanation a developer can act on. Detection is only half the job; the other half is telling the developer what went wrong well enough to fix it, which binary sanitizers cannot. Triton-Sanitizer attributes each violation to the tensors nearest the bad address, tracks the full call path, and exposes the symbolic operations that computed the wrong address, so the report points at the tensor, the path, and the arithmetic to blame. Providing rich, structure-aware diagnostics that guide a developer to the root cause of a GPU memory problem, rather than a bare address, is the same usability goal the work on guiding memory optimization for GPU-accelerated applications covered elsewhere on this site pursues, connecting a low-level symptom back to the source-level cause. Triton-Sanitizer applies it to correctness bugs, which is why it could not just detect but help fix two dozen real errors in widely used kernels.

Following a kernel through Triton-Sanitizer

Catching an out-of-bounds access in a Triton kernel

An illustrative trace, based on the paper's abstract, of how Triton-Sanitizer analyzes a kernel and reports a bug. Internal details beyond the abstract were not available for this draft.

Step 1 of 5

Take a Triton kernel to check

Triton-Sanitizer is given a Triton kernel that operates on tiles of tensors, with loads and stores that compute addresses and apply masks. Somewhere in it, a boundary condition is subtly wrong, so some threads can access out of bounds.

Working at the Triton level keeps the tensors, tiles, and masks visible.

Why it works

Tile semantics make addresses symbolically expressible. At the binary level addresses are just numbers with no structure → but Triton’s tile-oriented operations let the tool derive symbolic address and mask expressions → so it can reason about all possible accesses analytically.
An SMT proof replaces per-access checks. Instrumenting every access is slow and only observes the run at hand → but proving symbolically that no masked access exceeds bounds covers all cases at once → so detection is both precise and free of per-access runtime cost.
Selective simulation covers what symbols cannot. Pure symbolic analysis is stuck on data-dependent indirect addresses → but eagerly simulating just those accesses resolves them → so the hybrid stays complete without simulating everything.
No false positives make it trustworthy to run. A sanitizer that cries wolf gets ignored → but the symbolic-plus-simulation analysis reports only genuine violations → so developers can act on every finding, which is how it upstreamed real fixes.
Structure-level analysis yields fixable diagnostics. A bare out-of-bounds address gives the developer little → but reasoning over tensors, call paths, and address computations names the tensor, path, and arithmetic at fault → so the report guides the fix rather than just flagging the crash.

What it costs (trade-offs and scope)

This is a draft built from the paper’s abstract. The full paper was behind a publisher access wall for this draft (no arXiv version, no open-access or author copy located, and the ACM Digital Library returned a bot-verification challenge), so this account is assembled from the paper’s complete abstract plus standard background on GPU sanitizers and symbolic analysis. The exact symbolic construction, the SMT encoding, the eager-simulation fallback, the diagnostic generation, and the evaluation setup are not reproduced here and should be read from the paper.
It is specific to Triton. Triton-Sanitizer’s power comes from Triton’s tile-oriented semantics; the approach applies to Triton kernels, not to arbitrary GPU code written in other languages, so it complements rather than replaces general binary-level sanitizers for non-Triton code.
Indirect accesses fall back to simulation. Data-dependent addresses cannot be resolved symbolically and require eager simulation, which is more expensive than the symbolic path and depends on the runtime data exercised; kernels dominated by indirect accesses would see less of the symbolic speedup.
The speedup range is wide and workload-dependent. The reported 1.07 to 14.66 times speedup over compute-sanitizer, averaging 1.62 times, varies a great deal across kernels; the low end is only marginally faster, and the specific benchmarks and baseline behind these numbers are not detailed in this draft.
Detection is scoped to memory access errors. Triton-Sanitizer targets out-of-bounds memory accesses; other classes of GPU bug (races, numerical errors, synchronization issues) are outside its stated scope, and its guarantees are with respect to the addresses and masks it analyzes.

The evaluation

Finding memory access errors in GPU kernels

Property	compute-sanitizer (binary)	Manual review / testing	Triton-Sanitizer
How it detects errors
Runtime overhead
Device-agnostic
Diagnostic context
Reported outcome (per abstract)

A qualitative comparison based on the paper's abstract; the paper's main body, including the symbolic construction, SMT encoding, and evaluation setup, was not available for this draft. The 24-bug and 1.07-to-14.66-times-speedup figures are the abstract's own reported results.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This is an honest draft assembled from the paper’s complete abstract plus standard background on GPU memory sanitizers, Triton, and symbolic analysis; the full text was not accessible (no arXiv preprint, no open-access or author copy found, and the ACM Digital Library version returned a bot-verification challenge), so the internal design is not reproduced. What is solid and comes directly from the abstract: memory access errors are pervasive in GPU programming, and existing sanitizers like compute-sanitizer instrument every memory instruction in low-level IRs or binaries, imposing high overhead and giving minimal diagnostic context; Triton-Sanitizer is the first device-agnostic memory sanitizer designed for Triton; it leverages Triton’s tile-oriented semantics to construct symbolic expressions for memory addresses and masks, verifies them with an SMT solver, and selectively falls back to eager simulation for indirect accesses; this hybrid analysis enables precise detection without false positives while avoiding per-access instrumentation; it generates rich diagnostic reports attributing violations to the tensors nearest the violated addresses, tracking the complete call path, and exposing the symbolic operations responsible for incorrect addresses; and, evaluated on seven widely used open-source Triton kernel repositories, it uncovered 24 previously unknown memory access errors (8 already fixed and upstreamed by the authors) and achieved speedups of 1.07 to 14.66 times over compute-sanitizer, averaging 1.62 times. The explanations of why binary instrumentation is slow, device-bound, and uninformative, and why tile semantics enable symbolic reasoning, are standard background and the natural reading of the abstract, not claims about Triton-Sanitizer’s specific internals. Read the exact symbolic construction, SMT encoding, fallback, diagnostics, and evaluation in the paper itself; treat this page as a faithful account at the level of the abstract, not of the implementation.

Background

Why domain-specific languages enable better tools

It helps to see Triton-Sanitizer as an example of a broad and powerful idea: when computation is written in a domain-specific language that captures its structure, tools can do things that are impossible on general-purpose code. A general GPU sanitizer must work on the compiled binary because it cannot assume anything about the program’s shape, and at that level it can only watch raw memory instructions — losing the tensors, tiles, and intent that would make analysis cheap and diagnostics meaningful. A DSL like Triton, by contrast, expresses kernels in terms of tiles, masks, and structured address arithmetic, and that structure is exactly what a sanitizer needs to reason symbolically about which accesses are safe and to explain violations in terms the programmer understands. This is a recurring payoff of DSLs across systems: because the language encodes the domain’s abstractions, compilers can optimize better, and analyzers can verify and diagnose better, than they ever could on lowered code. As deep learning has driven the rise of kernel DSLs for portable, high-performance GPU code, it has also created the opportunity to build correctness and performance tooling that meets those kernels at their own level. That broader movement toward reasoning about GPU tensor programs at a structured, high-level representation rather than the binary is the same shift the work on an intermediate representation for optimized tensor computations on GPUs covered elsewhere on this site rides. Triton-Sanitizer shows that the same structure that helps compilers go fast can also help tools catch and explain bugs.

If you remember only three things

1. Memory access errors (out-of-bounds reads and writes) are among the most pervasive bugs in GPU programming, especially in the hand-written kernels behind deep learning. Existing GPU sanitizers like compute-sanitizer detect them by instrumenting every memory instruction in the low-level IR or binary, which is slow (high per-access overhead), device-specific (tied to the vendor binary), and diagnostically poor (it reports a bare out-of-bounds address, not which tensor or which address computation was at fault).
2. Triton-Sanitizer is the first device-agnostic memory sanitizer for Triton, a domain-specific language for portable deep-learning GPU kernels. It exploits Triton's tile-oriented semantics to construct symbolic expressions for memory addresses and their guard masks and verifies with an SMT solver whether any access can go out of bounds, falling back to eager simulation only for data-dependent indirect accesses. This hybrid analysis detects errors precisely with no false positives and no per-access instrumentation, and it produces rich diagnostics that attribute each violation to the nearest tensor, track the full call path, and expose the symbolic operations that produced the bad address.
3. On seven widely used open-source Triton kernel repositories, Triton-Sanitizer uncovered 24 previously unknown memory access errors (8 already fixed and upstreamed by the authors) and ran 1.07 to 14.66 times faster than compute-sanitizer, averaging 1.62 times. This explainer is a draft from the abstract, so the symbolic construction, SMT encoding, fallback, and evaluation should be confirmed against the paper. The approach is specific to Triton, falls back to costlier simulation for indirect accesses, has a wide workload-dependent speedup range, and targets memory access errors specifically.

Check yourself

Check your understanding

1. Why are existing GPU memory sanitizers like compute-sanitizer limited?
2. What is Triton-Sanitizer's core mechanism?
3. What makes Triton-Sanitizer's diagnostics more useful than a binary sanitizer's?
4. What did Triton-Sanitizer achieve, and what are its main limits?

Sources

The paper is “Triton-Sanitizer: A Fast and Device-Agnostic Memory Sanitizer for Triton with Rich Diagnostic Context” by Hao Wu, Qidong Zhao, Songqing Chen, Yang Chen, Yueming Hao, Tony C. W. Liu, Sijia Chen, Adnan Aziz, and Keren Zhou (George Mason University and Meta), published at ASPLOS 2026 (Volume 2), DOI 10.1145/3779212.3790241. It is licensed CC BY.

This explainer is a draft. The paper’s main text was not accessible at the time of writing (there is no arXiv preprint, no open-access or author copy was located, and the ACM Digital Library PDF returned a bot-verification challenge), so it is written from the paper’s complete abstract together with standard background on GPU memory sanitizers, the Triton domain-specific language, and symbolic analysis with SMT solvers. The abstract supplies the problem and approach: memory access errors remain one of the most pervasive bugs in GPU programming, and existing GPU sanitizers such as compute-sanitizer detect them by instrumenting every memory instruction in low-level IRs or binaries, which imposes high overhead and provides minimal diagnostic context; Triton-Sanitizer is the first device-agnostic memory sanitizer designed for Triton, a domain-specific language for developing portable, efficient GPU kernels for deep learning workloads; it leverages Triton’s tile-oriented semantics to construct symbolic expressions for memory addresses and masks, verifies them with an SMT solver, and selectively falls back to eager simulation for indirect accesses; this hybrid analysis enables precise detection of memory access errors without false positives while avoiding the cost of per-access instrumentation; it generates rich diagnostic reports that attribute violations to the tensors nearest the violated addresses, track the complete call path, and expose the symbolic operations responsible for incorrect addresses; and, evaluated on seven widely used open-source repositories of Triton kernels, it uncovered 24 previously unknown memory access errors, of which 8 have already been fixed and upstreamed by the authors, achieving speedups ranging from 1.07 to 14.66 times over compute-sanitizer, with an average improvement of 1.62 times. The explanations of why binary-level instrumentation is slow, device-bound, and uninformative, and why tile semantics enable symbolic reasoning, are standard background and the natural reading of the abstract, not claims about Triton-Sanitizer’s specific internal design. The paper’s exact symbolic construction, SMT encoding, eager-simulation fallback, diagnostic generation, and evaluation setup are not reproduced here and should be read from the paper. No numbers beyond the abstract’s have been introduced.