T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives

When a large language model is split across GPUs, every layer ends with a mandatory data exchange that nothing else can hide — T3 starts that exchange the instant the first slice of the answer is written, using the memory controller rather than the compute units.

Why this problem is worth solving

Language models grew faster than any single accelerator did, so splitting them across devices stopped being an optimisation and became a requirement.

That addition is a

What people did before, and where it broke

• Just let it be serial. The baseline runs the matrix multiply to completion, stores the output, then launches the collective. Simple, portable, and it wastes the entire duration of the collective.
• Overlap it the way data parallelism does. In data parallelism the gradient exchange overlaps with the backward pass of other layers, because nothing in the current iteration needs the result. Tensor parallelism has no such independent work — the consumer of the exchange is the immediate next kernel.
• Split the producer into chunks and interleave. Decompose the matrix multiply into several smaller launches and slip a communication step between them. This works, but each extra launch carries setup and synchronisation cost, and the chunk size has to be retuned per shape.
• Write a fused compute-plus-collective kernel. CocoNet and Google’s decomposition both go this route. It gives fine-grained overlap, but it means touching the matrix kernels themselves — and BLAS libraries hold hundreds of them, each tuned by an expensive autotuning process for a particular input size and GPU generation. Multiply that by every collective type and topology-specific implementation and the combinatorics become unmanageable.
• Give the collective its own compute units. Run the collective as a concurrent kernel on a slice of the GPU. The paper measures this directly: an ideal overlap would give 1.67x geomean speedup, but once the two kernels actually share compute units it falls to 1.18x, and the matrix multiplies slow down by roughly 21% on average.
• Push the reduction into the network switches. In-network aggregation does the summing inside the fabric. It reduces traffic, but it cannot remove the serialised step from the critical path, and it depends on having the right kind of switch in the topology.
• The gap. Every existing approach forces a choice between rewriting the compute kernels and paying a contention tax on both halves. Nobody had made the overlap happen without the compute side knowing it was happening.

How T3 fuses the matrix multiply with the exchange

The idea

The overlap opportunity was already inside the kernel. A large matrix multiply does not run all at once. Its output is tiled, and there are far more tiles than the GPU has room for, so the kernel proceeds in waves. Tensor parallelism cuts along the summed dimension, which shrinks the arithmetic each tile does but changes neither the output size nor the number of tiles nor the number of waves. So the shape of the computation already offers a natural pipeline: while wave three is computing, wave two’s output is finished and could be travelling. The producer does not need to be split — it is already split, and nobody was using the seams. This is the same reframing that lets a collective be decomposed into pieces small enough to interleave with the computation that depends on it, covered elsewhere on this site, except that T3 finds the pieces in the hardware’s own scheduling rather than creating them in software.

The trick is to change what a store means, not what the kernel does. T3’s central move is to configure the producer’s output address space before launch. Part of the output array is mapped so that writing to it lands in a neighbour GPU’s memory using the peer-to-peer store support GPUs already have; another part is mapped so that completing it triggers a DMA. The matrix kernel issues the same stores it always did. It does not know it is communicating. That is what makes the approach survive contact with a BLAS library holding hundreds of tuned kernels — none of them need editing. Treating the collective as a programmable object that the library configures rather than a monolithic call is a theme that shows up whenever collectives get first-class language support, covered elsewhere on this site, and here the programmable object is an address map plus a table of DMA descriptors.

Doing the work in a place that has no compute to steal. The contention measurements are what make the rest of the design inevitable. If a collective must borrow compute units, overlap gives back most of what it promised. So T3 puts every piece of the collective somewhere that has no compute units to borrow: the tracking happens at the memory controller, the transfer uses DMA engines that already exist, and the summation happens in adders near the DRAM banks. What remains is contention for memory bandwidth alone, and that is handled by an arbitration policy rather than by dedicating resources. The idea that a fixed round-robin between two traffic classes is worse than a dynamic policy watching queue occupancy also drives the memory-side scheduling in accelerators built for mixture-of-experts overlap, covered elsewhere on this site, and it reappears here because bursty collective traffic has exactly the shape that fills queues and stalls a latency-sensitive reader.

Following one tile through the pipeline

The output is chunked four ways for four devices. Chunk one is remote_mapped straight to GPU-3. Chunks two and three are dma_mapped, meaning they land in local memory but their completion triggers a DMA. Chunk four is purely local. The tracker and the DMA descriptor table are programmed from this map.

Why it works

• The pipeline is free because the waves already existed. Slicing in the summed dimension shrinks arithmetic per tile → but leaves tile count and wave count untouched → so wave n’s output can travel while wave n+1 computes, with no kernel splitting at all.
• Transparency comes from indirection at the address level. The kernel’s stores are unchanged → but the pages they target are mapped to remote memory or to DMA-triggering regions → so fusion is achieved by configuration rather than by code, and hundreds of tuned BLAS kernels keep working untouched.
• Nothing borrows a compute unit. Tracking sits at the memory controller, transfer uses existing DMA engines, reduction uses near-memory adders → so the matrix multiply keeps all 80 compute units → which is exactly the resource whose loss dropped the ideal 1.67x overlap speedup to 1.18x in the contention study.
• Near-memory reduction removes traffic, not just latency. Partial copies are never read into cores and the final reduction step vanishes → reduce-scatter reads fall by 2.4x geomean and matrix-multiply reads by 1.56x → giving a 22% average drop in total DRAM traffic alongside the speedup.
• Dynamic arbitration beats both round-robin and strict priority. Round-robin lets bursty collective traffic fill the DRAM queues → strict priority cannot evict traffic already queued, and dedicating the compute stream wastes bandwidth → so gating communication on a queue-occupancy threshold keeps room for the compute stream while still using idle cycles, which is what lifts T3 to a 30% geomean from T3’s own 13%.

What it costs (trade-offs and scope)

• It is a hardware proposal, not a library. T3 needs a tracker in the memory controller, an arbitration policy change, and near-memory op-and-store support in DRAM. The tracker is small — 19 KB — and the near-memory units are something memory vendors have already proposed, but none of this runs on a GPU you can buy today.
• The tracking assumes a tiled kernel where each wavefront writes a complete tile. This holds for all the matrix kernels evaluated, and the paper argues other implementations can be handled, but the clean mapping between a wavefront and a memory region is what makes a 19 KB table sufficient.
• Bypassing the last-level cache for output writes sometimes hurts. Near-memory update requires the writes to skip the cache, and for some cache-sensitive shapes that costs more than the traffic saving returns — the paper names FC-1 in T-NLG at sixteen-way slicing as a case where the overlapped runtime actually rises.
• The arbitration threshold is a tuned constant. The right DRAM queue occupancy threshold depends on how memory-intensive the producer is, so it is a parameter that has to be set rather than a self-adjusting mechanism.
• Everything is simulated. The evaluation extends Accel-Sim to model a multi-GPU system, models only a single GPU’s activities on the argument that all GPUs behave identically, and injects the inter-GPU traffic rather than simulating the DMA engine. The communication model is validated to 6% geomean error against real hardware over 6-192 MB transfers, but this is not a measurement on a built chip.

The evaluation

Check yourself

Sources

The paper is “T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives” by Suchita Pati, Shaizeen Aga, Mahzabeen Islam, Nuwan Jayasena and Matthew D. Sinclair, published at ASPLOS 2024 Volume 2, pages 1146-1164, DOI 10.1145/3620665.3640410. The authors are with the University of Wisconsin-Madison and Advanced Micro Devices.

This explainer is written from the full text, read via the arXiv preprint at arxiv.org/abs/2401.16677. All quantitative claims here — the 30% geomean sublayer speedup, the 22% traffic reduction, the 1.67x-to-1.18x collapse under compute sharing, the 19 KB tracker, the 6% simulation validation error and the end-to-end 10% and 12% geomean figures — come from the paper’s own sections 3, 5 and 6. One point worth flagging for readers comparing sources: the abstract quotes end-to-end benefits less prominently than the sublayer numbers, and section 6.4’s summary of large-model results states “up to 12% and 14%” for training and inference while section 6.3’s detailed breakdown gives maxima of 12% and 15% for the smaller models; both are reported here as the paper states them rather than reconciled.