ARC: Warp-level Adaptive Atomic Reduction in GPUs to Accelerate Differentiable Rendering

Sankeerth Durvasula, Adrian Zhao, Fan Chen, Ruofan Liang, Pawan Kumar Sanjaya, Yushi Guan and 2 others

Training a 3D scene model from photographs — the technique behind 3D Gaussian Splatting — spends nearly half its time in one step where millions of GPU threads all try to add their gradients into the same few parameters at once, and those atomic additions pile up at a small pool of specialised units in the shared L2 cache until the whole chip stalls; this paper from the University of Toronto proposes ARC, which notices that almost every warp has all its threads hitting the same location, sums their contributions inside the core using registers, and sends just one atomic update outward — while steering the lightly loaded warps straight to the L2 units so both sets of hardware stay busy.

The argument in one glance

Problem

One step of training a 3D scene eats nearly half the time, drowning in atomic adds

Rasterization-based differentiable rendering — the family that includes 3D Gaussian Splatting — learns a 3D scene from 2D photographs by gradient descent. Profiling on an NVIDIA RTX 4090 shows the gradient-computation step of the backward pass takes 44 percent of training time on average, and up to 66 percent on large photorealistic scenes. Each thread handles one pixel, and every pixel adds its gradient into the parameters of every primitive that coloured it. Because many pixels touch the same primitive, those additions have to be atomic — and there are millions of them per image.
What people did

Let the GPU handle the atomics the normal way, at the L2 cache

A GPU already has dedicated atomic units — the ROP units in the L2 memory subpartitions — and the standard atomicAdd instruction ships every atomic there to be applied safely. This is correct and asks nothing special of the programmer.
Why it fell short

A handful of atomic units cannot absorb millions of requests

There are only a few ROP units, shared across every streaming multiprocessor on the chip. When millions of atomic requests pour in, they queue at the load-store units and back up into the cores, which then stall waiting to issue. On the RTX 4090 the effect is severe: 128 multiprocessors feed just 176 atomic units. Prior fixes buffer atomics in each core local memory, but the sheer request count overwhelms the load-store units before any buffering can take hold.
This paper's idea

Sum a warp of identical updates in registers first, and use both sets of units at once

Two measurements change the game. Over 99 percent of warps have all their threads updating the very same memory location, and only some threads in a warp are active at any moment. So ARC sums a warp contributions inside the core using registers and sends just one atomic update outward, collapsing dozens of requests into one. And it routes warps with few active threads straight to the L2 units, so the cores and the L2 atomic units both stay busy rather than one sitting idle.

Why this problem is worth solving

The workload is not a niche — it is the training loop behind a fast-growing way of representing three-dimensional scenes.

Differentiable rendering treats the act of making a 2D image from a 3D scene as a mathematical function you can differentiate, so that the scene itself — the positions, colours, and shapes of its primitives — can be learned directly from photographs by gradient descent, exactly the way a neural network’s weights are learned. 3D Gaussian Splatting, the state-of-the-art member of this family, represents a scene as a cloud of fuzzy ellipsoids and renders them with a fast rasterizer; because rasterization is far quicker than the ray-marching used by earlier neural radiance fields, it renders images at high speed. But a separate model has to be trained from scratch for every individual scene, so how long training takes is the everyday cost of using the technique at all, and that cost is what this paper attacks.

The reason one step dominates is a traffic jam at one specific kind of hardware.

An atomic operation is a read-modify-write that the hardware guarantees happens indivisibly, so that when many threads add into the same memory location none of their updates is silently lost. On a GPU these are carried out not in the cores but by a small pool of dedicated units, the ROP units, that live in the L2 cache all the cores share. During gradient computation every pixel-thread must atomically add its contribution into the parameters of the primitives it saw, which produces a flood of atomic requests aimed at that shared pool. When the request rate exceeds what those few units can process, the requests queue at each core’s load-store unit, and the stall propagates back into the core until it cannot issue new instructions — so the bottleneck is not arithmetic but contention at a scarce, shared resource.

What people did before, and where it broke

Send every atomic to the L2 units with atomicAdd. The default path, correct and simple, letting the ROP units serialise the updates. Where it broke. Millions of requests meet a handful of units, so the cores spend their time stalled waiting for the queue to drain; this is exactly the baseline the paper improves on.
Buffer and aggregate atomics in each core’s L1 cache (the PHI approach). Combine commutative updates at the L1 before they travel to the L2, to cut interconnect and ROP traffic. Where it broke. The requests overwhelm the load-store units before they ever reach the L1 buffering, and every atomic also incurs an L1 address lookup; measured here it gives almost nothing, about 1.01 to 1.03 times.
Reserve part of each core’s fast SRAM to aggregate atomics (the LAB approach). Dynamically partition the L1/shared memory so atomic updates accumulate locally in each multiprocessor. Where it broke. Differentiable rendering already uses shared memory, leaving little to partition; and even an idealised version handed free extra SRAM reaches only about 1.4 times, because it aggregates per-core rather than per-warp and still suffers eviction and capacity limits.
Call an existing warp-reduction library (CCCL or CUB). These libraries already sum values across the 32 threads of a warp with tuned code. Where it broke. They assume every thread in the warp is active, but the gradient kernel’s condition checks leave only a subset active, so the routines either misbehave or miss most of the reduction opportunity; ARC beats CCCL by 1.58 times.
Accelerate NeRFs instead, the other rendering family. A great deal of prior hardware targets neural radiance fields. Where it broke. NeRFs are bottlenecked by computation, not atomics, and store their parameters in hash tables and trees whose accesses have almost no intra-warp locality, so their accelerators do not transfer to raster-based rendering.
The gap. Everyone either threw all the atomics at the few L2 units or buffered them per-core, and nobody exploited the fact that in this workload nearly every warp’s threads hit the identical location — which means a warp’s entire contribution can be collapsed to one atomic before it ever leaves the core, and the cores’ own idle capacity can share the load.

How ARC drains the atomic pileup

From millions of atomics at the L2 to one reduced update per warp

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Collapse the many identical updates into one before they reach the contended unit. The whole jam exists because a warp of threads, all wanting to add into the same parameter, each fires a separate atomic at a shared pool that can only serve a few at a time. But if those threads are updating the same location — and the measurement says over 99 percent of warps are — then their additions can be summed first, right there in the core, using registers that are private and fast, and only the single total needs to make the trip to the L2 units. Dozens of requests become one; the scarce resource sees a fraction of the traffic. The move is to do the combining upstream, close to where the data is produced, so that the bottleneck downstream only ever handles already-merged results. That is the same principle the in-network gradient-aggregation training work covered elsewhere on this site relies on, where partial results are summed as they pass through the network fabric so the endpoint that would otherwise be swamped receives one reduced value instead of many.

Do not move all the work to the cores — balance it against how loaded each unit is. The tempting overcorrection is to reduce every warp in the core and spare the L2 units entirely, but that just moves the traffic jam: warps with only a couple of active threads gain nothing from an in-core reduction, and if every warp funnels through the added reduction unit that unit becomes the new bottleneck while the perfectly good ROP units sit idle. ARC instead treats the two kinds of atomic hardware as a pair to be kept jointly busy. A warp whose active-thread count exceeds a balancing threshold is reduced in the core; a lighter warp is sent straight to the L2 units; and a greedy scheduler watches for stalls so that whenever the ROP units are free, work flows to them directly. Splitting a stream of reduction work across two sets of units so that neither starves nor saturates is the same balancing act the edge-colouring sparse-matrix work covered elsewhere on this site performs, scheduling the accumulations so the parallel hardware is used evenly rather than piling onto one contended path.

The win is domain-specific, and it comes from having measured the domain first. None of this would help a general atomics-heavy program, because the entire leverage is the 99-percent intra-warp locality, and that number is a property of rasterization — neighbouring pixels fall on the same primitive, so neighbouring threads update the same parameter. A graph algorithm scattering updates across a huge vertex array has essentially none of this locality, under a tenth of a percent, and ARC would do nothing for it. What makes the paper work is that its authors profiled a real, important, and previously uncharacterised workload, found that atomics — not arithmetic, not memory bandwidth — were the dominant cost, and then built a mechanism that exploits a structural regularity that only this class of workload has. Reaching a large speedup by first characterising exactly where a specific emerging workload spends its time and then designing narrowly for that is the same method the Gaussian-splatting architectural-support work covered elsewhere on this site follows, earning its efficiency by specialising for the rendering pipeline rather than by making the whole GPU generically faster.

Following one warp of gradient updates through ARC

What ARC does with a single warp during gradient computation

A warp is 32 threads, each a pixel. In this workload they almost all want to add a gradient into the same primitive parameter. This traces one such warp through the software version, ARC-SW.

Step 1 of 5

The warp reaches the atomic update

Each of the 32 threads has computed a gradient contribution for the primitive covering its pixel. In the baseline, all 32 would now issue separate atomicAdd requests to the L2 units for the same parameter.

This single line in the kernel, repeated across every warp and every primitive in every training image, is the source of the atomic flood that stalls the streaming multiprocessors.

Why it works

Intra-warp locality turns many requests into one. Over 99 percent of warps have all active threads updating the same location → so a warp’s whole contribution can be summed in registers and sent as a single atomic → which drops the request count at the scarce ROP units far enough to unblock the stalled cores.
Adaptive balancing keeps both kinds of unit busy. Reducing every warp in the core would idle the L2 units and overload the added reduction hardware → so ARC sends light warps straight to the L2 and heavy ones through the core reduction → which uses the ROP units and the cores together and raises the total atomic throughput past either alone.
Registers avoid the costs that per-core buffering pays. Buffering atomics in L1 or shared SRAM brings eviction, capacity, and lookup overheads that blunt prior schemes → but ARC only aggregates within a warp, directly in registers, adding no SRAM → so it sidesteps the very overheads that held LAB and PHI to modest gains.
Reducing per sub-core multiplies the parallelism. A GPU core is split into sub-cores, and ARC places a reduction unit in each of the four → so four warps can be reduced at once inside one core rather than serialising through a single per-core buffer → which is a large part of why ARC beats even an idealised per-core aggregator.
Watching stalls is a cheap, accurate load signal. Knowing whether the L2 units are saturated normally needs extra tracking hardware → but a busy ROP pool already shows up as stalls backing into the core’s load-store units → so a greedy scheduler reads that existing signal to route work, adapting to contention with almost no added logic.

What it costs (trade-offs and scope)

The headline step-speedup is not the end-to-end speedup. ARC accelerates gradient computation by about 2.6 times on real hardware, but that step is only 44 to 66 percent of training, so the whole-training speedup is 1.41 times on average (up to 2.4 times) — Amdahl’s law caps the payoff, and the larger number should not be read as the training being 2.6 times faster.
The hardware design is evaluated only in simulation. ARC-HW’s numbers — 2.06 times on average, up to 8.59 times, and its energy savings — all come from the GPGPU-Sim simulator, not silicon; only the software version, ARC-SW, was measured on real GPUs, so the hardware case rests on a model.
The balancing threshold is a tuned, per-workload knob. Its best value depends on the dataset, the GPU, and the reduction method, and a poor choice can cause slowdowns in compute-bound workloads where the extra instructions are not repaid; the automatic tuning helps but adds a profiling step and assumes the workload is stable across iterations.
The fastest reduction method does not always apply. Butterfly reduction needs all lanes active and so requires the programmer to remove control divergence and zero-pad inactive threads; where that is infeasible, as in the Pulsar sphere workload, only the slower serial reduction is available.
The benefit is confined to high-locality workloads. The whole mechanism leans on the 99-percent intra-warp locality of rasterized rendering; NeRFs and graph workloads have almost none (pagerank measures under 0.1 percent), so ARC does nothing for them, and the claim of generality to other atomics-heavy code is asserted rather than demonstrated.
ARC-SW asks work of the programmer and the GPU. It requires rewriting the gradient kernel to call ARC in place of atomicAdd, tuning the threshold, and a GPU that supports the needed warp primitives — which some do not, for example Intel GPUs.
ARC-HW asks for changes to the chip and its rules. It adds a new ISA instruction, a floating-point unit per sub-core (a small area cost, about 0.047 percent of the RTX 4090’s transistors), and updates to the coherence protocol; correctness depends on the atomic updates being commutative, relaxed, and strongly atomic within the kernel, which holds for gradient computation but not for arbitrary code.
The comparisons are partly idealised and hand-tuned. The prior schemes LAB and PHI are evaluated in the same simulator, LAB-ideal is given free extra SRAM it would not have in practice, and LAB’s partition size and ARC’s threshold are both set to their empirically best values per workload.
Where the gains land is uneven and hardware-shaped. Speedups are largest on big photorealistic scenes with many primitives and on the RTX 4090 (whose low ratio of atomic units to cores makes the bottleneck worse); on the RTX 3060 and on scenes where gradient computation is not the bottleneck, such as some NvDiffRec and Pulsar cases, the end-to-end benefit shrinks.

The evaluation

Three ways to absorb the flood of gradient atomics

Property	Baseline: atomicAdd to the L2 units	Per-core buffering (LAB-ideal, PHI)
Gradient-computation speedup
End-to-end training speedup	1 times	Modest, since the step gain is modest
Where the combining happens	Nowhere — every request goes to the L2
Energy for gradient computation	The reference	Not the focus of those works
Measured on real silicon or simulator	Both

Comparison on 3D Gaussian Splatting, NvDiffRec, and Pulsar workloads across NVIDIA RTX 4090 and RTX 3060 GPUs; software numbers are measured on the real GPUs and hardware numbers modelled in GPGPU-Sim, all normalised to the atomicAdd baseline.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

Read the speedups at the right scope. The much-quoted 2.6-times figure is for the gradient-computation step alone; because that step is 44 to 66 percent of training, the end-to-end speedup is 1.41 times on average, and the training is not 2.6 times faster. The biggest per-step and energy numbers — 2.06 times, 3.9 times energy — come from the hardware design ARC-HW, which is evaluated only in the GPGPU-Sim simulator; only the software ARC-SW ran on real GPUs, and it too needs programmer effort, per-workload tuning of the balancing threshold, and warp primitives that not every GPU has. The whole approach depends on the 99-percent intra-warp locality of rasterized rendering, so it helps this workload family and not general atomics-heavy code like graph analytics, where locality is under a tenth of a percent; the paper’s claim of broader generality is asserted, not shown. The comparisons flatter ARC somewhat — the prior schemes are simulated, LAB-ideal is handed free SRAM, and thresholds and partition sizes are set to their best per-workload values — and the fastest reduction mode cannot be used where control divergence is unavoidable, as in Pulsar. What is solid is the diagnosis: atomics, not arithmetic, dominate this step, and collapsing a warp’s identical updates into one while balancing them against the L2 units genuinely removes most of the stalls, cutting mean stalls per instruction from 38.3 to 10.3 cycles.

Background

Why this workload, and why now

It is worth situating why a paper about atomic-unit contention arrives now and targets this particular application. Learned 3D representations have moved very fast: neural radiance fields showed that a scene could be a trained model rather than an explicit mesh, and 3D Gaussian Splatting then made both training and rendering far faster by swapping ray-marching for the GPU’s high-speed rasterization pipeline. That shift is what created this bottleneck. Ray-marching NeRFs are limited by raw computation and store parameters in scattered structures, so their costs lie elsewhere; rasterized splatting concentrates the backward pass into dense, overlapping per-pixel gradient updates that land on the same handful of primitives, which is precisely the pattern that turns into an atomic pileup. So the bottleneck is a direct consequence of the technique that made the field practical, and it will matter more as scenes grow, because gradient-computation time rises faster with scene complexity than the forward pass does. This is the same pressure that the real-time neural-rendering pruning work covered elsewhere on this site responds to, attacking the cost of these learned-scene models so they can run within a real budget — one line of work trims how much there is to compute, while ARC speeds up the specific hardware step that the raster-based training loop keeps hitting. Both are signs that learned 3D scenes have become important enough that their inner loops now justify dedicated systems and architecture work.

If you remember only three things

1. In rasterization-based differentiable rendering, such as 3D Gaussian Splatting, the gradient-computation step of training takes 44 to 66 percent of the time, and the cost is not arithmetic but contention: millions of pixel-threads atomically add their gradients into a few shared parameters, and those atomics overwhelm the small pool of atomic units in the L2 cache, stalling the cores.
2. ARC exploits two measured facts — over 99 percent of warps have all their threads updating the same location, and only some threads in a warp are active at a time — to sum a warp contributions inside the core using registers and send one atomic outward instead of dozens, while a balancing threshold and a stall-watching scheduler route lightly loaded warps straight to the L2 units so the cores and the atomic units stay busy together.
3. A software version (ARC-SW) runs on stock GPUs and gives a 2.44-times gradient-step and 1.41-times end-to-end speedup on an RTX 4090, and a hardware version (ARC-HW), evaluated in simulation, adds a per-sub-core reduction unit for about 0.047 percent area to do better still — but the gains hinge on the workload high intra-warp locality, so they do not extend to atomics-heavy code like graph analytics, and the headline step-speedup is bounded end to end by how much of training that step is.

Check yourself

Check your understanding

1. What is the actual bottleneck ARC targets in differentiable-rendering training?
2. What key measurement makes warp-level reduction in registers pay off, and why?
3. Why does ARC not simply reduce every warp in the core and bypass the L2 atomic units entirely?
4. Which qualification most affects how the results should be read?

Sources

The paper is “ARC: Warp-level Adaptive Atomic Reduction in GPUs to Accelerate Differentiable Rendering” by Sankeerth Durvasula, Adrian Zhao, Fan Chen, Ruofan Liang, Pawan Kumar Sanjaya, Yushi Guan, Christina Giannoula, and Nandita Vijaykumar, published at ASPLOS 2025, pages 64-83, DOI 10.1145/3669940.3707238. The authors are at the University of Toronto and the Vector Institute.

This explainer is written from the full 20-page text. Everything above comes from the paper’s own sections: the background on atomic processing in GPUs, on differentiable rendering for 3D scene reconstruction, and on rasterization with 3D Gaussian Splatting; the motivation section profiling training on the RTX 4090, breaking down the gradient-computation bottleneck and the two observations of intra-warp locality (over 99 percent of warps) and partial warp participation; the approach section with its two design challenges, the warp-level reduction and dynamic core-versus-L2 scheduling ideas, and the ARC-HW and ARC-SW implementations including the atomred instruction, the address-coalescing thread-mask trick, the reduction unit with its added floating-point unit, the serialized and butterfly reductions, and the balancing threshold with its automatic tuning; the area-overhead estimate of about 0.047 percent from Yosys; and the evaluation on 3D Gaussian Splatting, NvDiffRec, and Pulsar across the RTX 4090 and 3060, covering gradient and end-to-end speedups, shader-stall reductions, comparisons against LAB, LAB-ideal, PHI, and the CCCL library, the ARC-HW-versus-ARC-SW gap, the balancing-threshold sensitivity, the stalls-per-instruction drop from 38.3 to 10.3 cycles, and the energy analysis, together with the related-work discussion of why NeRF accelerators and prior atomic-buffering schemes do not capture this locality.