Training a neural net across GPUs leans on collective operations like AllReduce, and when that communication lags the GPUs sit idle — but the standard library NCCL picks its communication paths assuming each link has a fixed bandwidth, which is true on NVLink DGX boxes yet badly wrong on the PCIe-based clusters that are actually about 90% of real machines, where concurrent transfers sharing PCIe switches, CPU memory, and NUMA hops throttle each other by up to 65% with no theoretical conflict; TCCL (Seoul National University) profiles the real CONCURRENT bandwidth of transfers running together, then searches for the best ring ordering of GPUs and the best placement of bounce buffers across NUMA nodes using that measured contention, and swaps the discovered path into NCCL — running collectives up to 2.07x faster than NCCL and MSCCL.
The argument in one glance
-
Problem
When the GPUs finish computing, they wait on each other to communicate
Training a large neural network splits the work across many GPUs, and after each step they must exchange and combine results with COLLECTIVE operations — AllReduce sums everyone's gradients, AllGather assembles a full tensor from pieces, ReduceScatter does both at once. These collectives sit on the critical path: when communication cannot keep pace with computation, the GPUs go idle waiting for data, and training throughput drops. So how fast the collectives run directly limits how fast the whole cluster trains, which is why the communication library underneath matters enormously.
-
What people did
Use NCCL, and trust its path choices
Everyone uses NCCL, NVIDIA's collective communication library. NCCL inspects the interconnect topology and picks communication paths — which GPUs form the ring, which links carry which transfers — using a backtracking search over the topology, guided by each link's known bandwidth. It works beautifully on the systems it was tuned for: DGX servers where GPUs talk over NVLink and NVSwitch, high-bandwidth dedicated interconnects that bypass the CPU entirely. Point NCCL at the topology, let it choose the paths, and communication is fast.
-
Why it fell short
On PCIe clusters, links are not independent — concurrent transfers collide
The trouble is that most machines are not DGX boxes. PCIe-dependent systems — where GPUs communicate over shared PCIe switches, host bridges, CPU memory, and across NUMA nodes — cover over 90% of the machines in real datacenter traces, and NCCL was not built for them. On PCIe, a link's bandwidth is NOT a fixed number: it depends on what other transfers are running at the same time, because they share switches and memory controllers. The paper measures a GPU reading CPU memory at 13.09 GB/s and writing cross-NUMA to another GPU at 10.86 GB/s — but run simultaneously they collapse to 4.27 GB/s, a 65% loss, despite "no theoretical conflict." NCCL, choosing paths from isolated per-link bandwidths, cannot see these collisions because they simply do not happen on NVLink, so it routinely picks rings whose transfers throttle each other.
-
This paper's idea
Profile transfers running together, then search for a collision-free ring
TCCL (Seoul National University) rebuilds path selection around measured CONTENTION instead of isolated bandwidth. A PROFILER runs whole SETS of transfers concurrently and measures how long they take together, so it captures the real bandwidth each gets under contention — not the fictional isolated number. A PATHFINDER then searches for the best ring: the ordering of GPUs and the placement of bounce buffers across NUMA nodes that maximizes the slowest hop's measured bandwidth, using a priority-queue search made provably optimal by a monotonicity property (adding a transfer can only lower bandwidth). A modified NCCL RUNTIME executes the discovered path while keeping NCCL's GPU kernels and API. The search is a one-time cost at install (up to 8 hours), and it makes collectives up to 2.07x faster than NCCL and MSCCL.
Why this problem is worth solving
Collective communication is the quiet tax on every distributed
training run, and on the hardware most people actually own it is
higher than it needs to be. The workhorse operation is
AllReduce, which sums
a tensor (typically the gradients) across all GPUs and gives every
GPU the total, is the most common collective in data-parallel
training. The standard way to run it efficiently is the RING
algorithm: arrange the GPUs in a logical ring, and pass data
around it so each GPU receives a chunk from its predecessor, adds
its own contribution, and forwards it to its successor. After
enough steps around the ring, every GPU holds the full sum. The
ring is bandwidth-optimal in principle because every link is used
at once, but it has a catch — the whole ring runs at the speed of
its SLOWEST hop, since every GPU must wait for the chunk coming
from behind it. So the entire performance question reduces to one
thing: which ordering of GPUs, and which physical links between
them, makes the slowest hop as fast as possible
NCCL chooses it by reading each link’s bandwidth and searching
for a good ring, which is exactly right on NVLink but wrong on
PCIe, because on PCIe the numbers it reads are not stable. The
core reality it misses is
the fact that on a
PCIe-based system, the links are not independent pipes. Transfers
share PCIe switches, the CPU’s host bridges, the memory
controllers of NUMA nodes, and the inter-socket link between
them — so two transfers that each run fast alone can throttle each
other badly when run together, because they compete for a switch
or a memory controller they both traverse. The paper documents
collisions that isolated profiling cannot predict: a GPU-to-CPU
read and a cross-NUMA GPU-to-GPU write, each fine alone, lose 65%
of their combined bandwidth when concurrent; a direct cross-NUMA
GPU-to-GPU transfer that manages only 0.42 GB/s where a same-NUMA
one gets 7.45; four GPUs bouncing through one NUMA node’s memory
that speed up 40% simply by moving one buffer to a different NUMA
node. None of these appear on NVLink, where communication bypasses
the CPU, so a library tuned there is blind to them
What people did before, and where it broke
- NCCL (topology search on isolated bandwidths). Picks rings and paths with a backtracking search guided by each link’s standalone bandwidth. Excellent on NVLink/NVSwitch, but on PCIe it cannot see that concurrent transfers contend for shared switches and memory controllers, so it chooses rings whose hops collide and lose up to 65% of their bandwidth.
- MSCCL (schedule synthesis via optimization). Synthesizes clever collective SCHEDULES (using integer programming) to improve on NCCL’s algorithms. But it inherits NCCL’s choices of transfer TYPE (which link, direct versus bounce-buffered, which NUMA node), so it optimizes the schedule on top of the same contention-blind path model — TCCL beats it by up to 2.07x.
- Isolated micro-benchmarks (OSU, nvbandwidth). Measure point-to-point bandwidth one transfer at a time. Useful for characterizing a link in isolation, but by construction they cannot reveal how transfers interact, which is the entire problem on PCIe — so they give the very numbers that mislead NCCL.
- DGX-focused collective research. A large literature optimizes communication for NVLink/NVSwitch topologies. Valuable for those systems, but PCIe-dependent machines are over 90% of real deployments and behave differently, so the results do not transfer to where most training actually happens.
- The gap. No prior system chose collective paths from the bandwidth transfers actually achieve WHEN RUN TOGETHER on a PCIe cluster, and searched the ring ordering and bounce-buffer placement against that measured contention. TCCL does exactly this, and keeps NCCL’s kernels and API so it is a drop-in.
How TCCL works
Profile concurrent transfers, search the ring against contention, then run it in NCCL
-
Profile whole SETS of transfers at once new
Instead of timing one transfer in isolation (as OSU and nvbandwidth do), TCCL keeps a process pool that launches an entire SET of transfers simultaneously and measures when they all finish — yielding B(T), the real bandwidth each transfer achieves while contending with the others. This is the only way to see the collisions that decide PCIe performance.
-
Lean on the monotonicity property new
A proven fact anchors the whole search: if one transfer set is contained in another, the smaller set's bandwidth is at least the larger's — adding a concurrent transfer can only degrade bandwidth, never improve it. This monotonicity is what lets a greedy priority-queue search be provably optimal rather than a blind combinatorial hunt.
-
Search intra-node rings (Dijkstra-like) new
For the GPUs within one node, a priority-queue search expands partial rings ordered by their measured bandwidth, at each step appending a candidate transfer (a direct GPU-to-GPU hop, or a GPU-to-CPU-memory-to-GPU bounce through a chosen NUMA node). Because of monotonicity, the first COMPLETE ring popped from the queue is guaranteed the best possible.
-
Exploit NUMA symmetry to shrink the search new
The candidate space is enormous — about 25 million transfer sets for a 4-node, 4-GPU AMD cluster. But sets that map to each other by permuting NUMA nodes achieve identical bandwidth, so TCCL caches that equivalence and reuses results; 99.2% of profiling queries become cache hits, making the otherwise-intractable search feasible.
-
Merge nodes into a cluster ring (dynamic programming) new
To join per-node rings into one cluster-wide ring, a dynamic program computes D[n][h][t] — the best bandwidth of a chain from node 1 to node n with given head and tail GPUs — by combining the best sub-chain with each inter-node transfer type. It assumes the inter-node network itself is not the bottleneck, so a merged chain's bandwidth is the minimum of its parts.
-
Substitute the path into NCCL new
A modified NCCL runtime installs the discovered ring ordering and per-transfer types (direct or bounce-buffered, and which NUMA node hosts each bounce buffer), while leaving NCCL's communication kernels and external API untouched — so existing training code runs unchanged and just goes faster. The pathfinding runs once at system setup (one to eight hours) and is amortized over every subsequent run.
The idea
Measure the cost that actually occurs, not the one that is convenient to measure. NCCL reads each link’s standalone bandwidth and reasons as if links were independent pipes. On PCIe they are not: a transfer’s real bandwidth depends on everything else sharing its switches and memory controllers at that moment, and the gap between the isolated number and the concurrent number is enormous — up to 65%. TCCL’s foundational move is to profile transfers RUNNING TOGETHER, so its cost model reflects reality under contention. This is the measure-the-true-cost discipline the site keeps returning to, and here it is almost the same diagnosis the GPU-microservice scheduling page makes about bandwidth GPUs cannot isolate: the resource that breaks your plan is the one you did not measure under realistic conditions. Optimizing against an isolated proxy is how NCCL ends up choosing rings that collide.
Find the structural property that turns a combinatorial hunt into a provably optimal search. The space of ring orderings and bounce-buffer placements is astronomically large — tens of millions of transfer sets — so brute force is hopeless and a greedy heuristic would normally risk missing the optimum. TCCL escapes this with a monotonicity property it can prove: adding a concurrent transfer to a set can only lower the bandwidth, never raise it. That single fact makes a Dijkstra-style priority-queue search exact — once a complete ring is popped, every remaining partial path already has lower bandwidth and can only get worse, so nothing unexplored can beat it. It is the same exploit-a-structural-property-to-search-well move the site sees when a locality property makes a curve-based placement right by construction or a subset relation makes a bound tight: the right property collapses a search you could not afford into one you can trust.
Do the expensive reasoning once, and make it a drop-in. The profiling and pathfinding take up to eight hours, which sounds prohibitive until you notice it runs ONCE, at system installation, and every training run thereafter inherits the better paths. The cost is off the critical path and amortized to nothing, the same solve-once-offline economics the heterogeneous-serving page uses for its MILP and the noisy-quantum page uses for its post-processing. And because TCCL only substitutes NCCL’s path and transfer-type choices while keeping its kernels and API, adopting it changes no user code — the heavy one-time search buys a permanent, transparent speedup on hardware the owner already has.
Why the isolated map lies
Two links that are fast alone, and slow together
An AllReduce ring on a PCIe cluster. Numbers from TCCL's measurements on AMD-V100.
Step 1 of 5
The ring runs at its slowest hop
AllReduce passes data around a ring of GPUs, so every GPU waits on the chunk coming from behind it, and the whole collective runs at the speed of the slowest hop. Choosing the ring means choosing which GPUs are adjacent and which physical links (direct GPU-to-GPU, or bounced through CPU memory on some NUMA node) carry each hop — all to make that slowest hop as fast as possible.
the ring is bandwidth-optimal in principle because all links work at once but one throttled hop drags the entire collective down to its speed
Why it works
- On PCIe a link’s bandwidth depends on what shares its switches and memory controllers → profiling transfers CONCURRENTLY captures the real contended bandwidth → the cost model reflects what actually happens, unlike NCCL’s isolated-link numbers.
- Adding a concurrent transfer can only lower a set’s bandwidth, never raise it (monotonicity) → a priority-queue search that pops the highest-bandwidth partial ring first is provably optimal → the tens-of-millions-set space is searched exactly, not heuristically.
- Sets equivalent under a permutation of NUMA nodes achieve equal bandwidth → TCCL caches those equivalences → 99.2% of profiling queries are cache hits, turning an intractable search into a feasible one.
- Contention concentrates when many transfers pile onto one NUMA node’s memory → distributing bounce buffers across NUMA nodes spreads the load → a four-GPU set gains 40% from moving a single buffer, and whole collectives speed up 1.5 to 2x.
- The profiling and search are expensive but one-time → they run once at installation and only NCCL’s path choices are swapped in → every later training run inherits the speedup with zero code change.
What it costs
- Ring algorithms only. TCCL optimizes ring-based collectives (AllReduce, AllGather, ReduceScatter); tree- and recursive-doubling-based collectives are outside its current scope, though the authors argue the methodology could extend.
- A primitive mismatch can backfire. The Reduce ring does not use every link (the root sends nothing to its successor), but TCCL’s model assumes full utilization — so in one case (AMD-3090 Reduce on a single node) TCCL is actually 17% SLOWER than NCCL.
- End-to-end training gains are modest. Communication is only about 15% of a data-parallel step, so a 2x collective speedup becomes roughly 1.09x on the whole run; tensor parallelism adds pipeline bubbles and memory overhead that further dilute the benefit, so realizing TCCL’s value needs careful choice of parallelism strategy.
- The inter-node search assumes no network congestion. The dynamic program that merges per-node rings takes a merged chain’s bandwidth to be the minimum of its parts, which assumes the inter-node network is not itself a bottleneck — an assumption that can fail under heavy cross-node traffic.
- Hardware and setup constraints. It targets NVIDIA GPUs with Mellanox InfiniBand and requires disabling virtualization features (SR-IOV, IOMMU) for peer-to-peer transfers, and the one-time pathfinding still costs up to eight hours per system configuration.
The evaluation
Three PCIe-dependent clusters — AMD-V100 (4 nodes x 4 GPUs), AMD-3090 (PCIe Gen4, GPUs without direct peer-to-peer), and Intel-V100 — across AllReduce, Broadcast, Reduce, AllGather, and ReduceScatter, against NCCL and MSCCL, with end-to-end deep-learning training as the ultimate test.
Three ways to route a collective on a PCIe GPU cluster
| Property | NCCL (isolated-link paths) | MSCCL (schedule synthesis) | TCCL (contention-profiled discovery) |
|---|---|---|---|
| How link bandwidth is modeled | |||
| Accounts for concurrent contention? | |||
| What it optimizes | |||
| Search guarantee | |||
| Collective speedup (PCIe clusters) |
Check yourself
Check your understanding
1. NCCL works beautifully on NVLink DGX systems but chooses poor communication paths on PCIe clusters. Predict the root cause of this mismatch.
2. TCCL's profiler measures the bandwidth of whole SETS of transfers running concurrently, rather than one transfer at a time like OSU or nvbandwidth. Predict why this concurrent profiling is essential rather than merely nice.
3. The space of possible rings and bounce-buffer placements is tens of millions of transfer sets, yet TCCL finds the provably OPTIMAL ring with a priority-queue search. Predict what makes an exact search possible at that scale.
4. TCCL delivers up to 2.07x on the collectives themselves but only about 1.09x on end-to-end data-parallel training. Predict what this gap reveals, and why it is not a failure.
Sources
- ACM DL: TCCL: Discovering Better Communication Paths for PCIe GPU Clusters — ASPLOS 2024 (full text read via the ACM PDF; no arXiv preprint).
- OpenReview: TCCL — paper page and reviews.
- Artifact (Zenodo, DOI 10.5281/zenodo.10799424) — open-source TCCL implementation.