DrGPUM: Guiding Memory Optimization for GPU-Accelerated Applications

GPU memory is the resource that decides whether a job runs at all, yet no profiler could tell you which data object was holding it and why — this paper (UC Merced) names ten recurring patterns of memory waste, from allocating an array long before anything touches it to allocating an array almost none of whose elements are ever read, and builds DrGPUM, a profiler that finds all ten in unmodified optimized binaries and points at the source line; guided by its reports, a graduate student with no prior knowledge of the programs cut peak memory by up to 83 percent and sped some of them up by up to 2.48 times, each fix under nine lines of code.

Why this problem is worth solving

The reason no existing profiler could answer the question is that they were all built around the wrong noun.

The second thing worth naming is what the tool is not.

What people did before, and where it broke

• Automatic memory-saving systems. Recomputation drops intermediate tensors and regenerates them on demand; swapping offloads to CPU memory and prefetches back; compression shrinks the data with a lossless or error-bounded scheme. All three are mature and all three genuinely save memory.

• The gap. Each targets a specific class of workload, usually deep learning, and each is a mechanism rather than a diagnosis. They save memory on the objects you point them at. They cannot tell you that an array in your program is never read.

• CUDA Unified Memory. The most general answer available. It lets a program oversubscribe GPU memory transparently, with the runtime migrating pages on demand, so no code change is needed at all.

• The gap. Generality is paid for at run time. Page migration on faults can make a program several times slower than the hardware allows, so it rescues a job that would otherwise not run rather than making a job efficient.

• Value-aware GPU profilers. A line of work that instruments memory instructions to look at the values flowing through them, finding redundant writes, duplicated arrays, and numbers stored far wider than their range.

• The gap. Value-aware and capacity-aware waste are almost disjoint. Run against the same programs, that line of tools detected exactly one of the ten patterns here, and only indirectly, because a value-agnostic mistake like allocating early leaves no trace in the values at all.

• Memory-error checkers. The vendor sanitizer detects leaks, out-of-bounds accesses, and misaligned accesses, and it is the standard tool people already reach for when a GPU program misbehaves around memory.

• The gap. An error checker looks for incorrectness. Nine of the ten patterns here are perfectly correct code that merely wastes memory, so the checker reports nothing. Of the ten, it found only leaks.

How DrGPUM turns a running program into a list of named mistakes

The idea

Waste has a small number of recurring shapes, and each one is a predicate over a timeline. The paper first fixes a mental model — a timeline of every GPU operation, with each data object drawn underneath it and a mark wherever an operation touched that object. Once you have that picture, most memory waste becomes a statement about gaps. An object is allocated early if operations happened between its allocation and its first touch. It is deallocated late if operations happened between its last touch and its free. It is temporarily idle if operations happened between two consecutive touches, which is the deep learning case where an activation computed in the forward pass sits untouched until the backward pass. It is unused if nothing ever touched it, and leaked if it was never freed. It suffers a dead write if two writes hit it with no read in between. Seven patterns, all of them the same shape of question asked at different points on the line, all of them checkable in one walk. That move — turning an informal complaint into a predicate a machine can evaluate — is the same discipline that turns a vague sense of slowness into a named value pattern in the value-centric GPU profiler covered elsewhere on this site, whose authors overlap with this one.

Some waste is inside an object, not around it. The other three patterns need a finer instrument, because they are about which elements of an array were used rather than whether the array was used. Overallocation means only a small fraction of the elements are ever touched, which the paper finds in a molecular docking application where a buffer is sized to a constant maximum while the actual input determines a far smaller need, leaving well under one percent of it touched. Non-uniform access frequency means some slices of the object are read far more often than others, which is a placement opportunity rather than a capacity one — put the hot slices in shared memory. Structured access means each invocation of a kernel touches a different, non-overlapping slice, which means the whole array never needed to exist at once. This finer view is what the automatic capacity-stretching systems cannot supply, because a page-migration scheme like the unified-memory prefetcher covered elsewhere on this site can only move what the program allocated, never question whether it should have been allocated.

A suggestion is worthless unless it lands on a line of code. The last third of the design is about making the report actionable. Every allocation is recorded together with the call path that created it, unwound at run time, so the report can say where the object came from rather than merely that it exists. Offline, debug information is read out of the binary and its shared libraries to turn those addresses into file names and line numbers. Objects are then ranked by whether they participate in the program’s largest memory peaks, so the report opens on the ones that are actually binding rather than on a list of hundreds. The evaluation is the proof that this mattered — the fixes were written by a graduate student who knew CUDA but knew nothing about any of the programs, and each took about an hour and a half. That dependence on debug information being both present and correct is a quiet assumption, and the papers on incomplete debug information covered elsewhere on this site show how often that assumption is violated in optimised builds.

Walking the timeline for one object

The call path is what later lets the report point at a source line rather than at a hex address. The map is the only structure that connects a raw address seen inside a kernel back to the object it belongs to.

Why it works

• The patterns are predicates, not heuristics. Each definition is a precise statement about the timestamped trace → the detector can decide it exactly rather than guessing → the paper can claim no false positives, every reported inefficiency is a real one.
• The object map turns addresses back into nouns. Intercepting allocation and deallocation gives a live address-range table → any access seen anywhere can be attributed to the object it belongs to → the profile is organised around data rather than around code, which is what a memory question needs.
• Topological order makes concurrency analysable. Building a dependency graph from within-stream succession and cross-stream data flow → a single global ordering exists even when calls on different streams overlap in real time → the same before-and-after predicates work unchanged on multi-stream programs.
• The heavy counting happens where the data already is. Copying the object map to the GPU and setting hit flags there → only a few flags cross the bus per kernel instead of a stream of addresses → the object-level analysis of one deep learning framework drops from about an hour and a half to twelve seconds.
• Two collection modes, chosen per kernel launch. Updating access maps on the GPU with atomics is far faster but the maps must fit in device memory → the tool sizes the maps against free capacity before each launch and falls back to CPU-side updating when they would not fit → speed where possible without ever exhausting the memory it is trying to study.

What it costs (trade-offs and scope)

• The tool diagnoses, it does not repair. Every fix in the evaluation was written by a human reading the report. The suggestions are mechanical enough to follow, but the tool has no mechanism for applying them, and safety is the user’s judgement.
• Findings are input-dependent. This is a dynamic profiler, so an object that is unused for the input you profiled may be essential for another. The paper recommends representative inputs and warns that postmortem reasoning is required before acting.
• Instrumentation is not free. Object-level analysis costs a median of about 1.3 to 1.5 times native execution time, with a geometric mean above two. Intra-object analysis, which instruments every load and store, costs a median of roughly 3.5 to 4 times even with kernel sampling turned on.
• Custom allocators need bespoke support. Deep learning frameworks preallocate a large pool and hand out tensors from it, which the vendor interception layer cannot see. The authors wrote a dedicated hook for one framework; supporting another is future work.
• One vendor. The implementation rests on that vendor’s sanitizer interface, so it runs on their GPUs only. The methodology carries over to any device with a comparable binary instrumentation engine, but the code does not.

The evaluation

Check yourself

Sources

DrGPUM: Guiding Memory Optimization for GPU-Accelerated Applications. Mao Lin, Keren Zhou, Pengfei Su. ASPLOS 2023, Volume 3, pages 164 to 176. DOI 10.1145/3582016.3582044. Full text read from the ACM Digital Library PDF.