Letting the network switch add up gradients accelerates distributed training, but every prior system had to throw away the transport layer to do it — NetReduce hides aggregation BENEATH an untouched RoCE stack by splicing connections at the switch (payloads mutate, volumes and checksums stay valid), keeping RDMA hardware speed and ~850 lines of host code while training 12–69% faster than flat or hierarchical all-reduce.
The argument in one glance
-
Problem
Training is a network problem wearing a GPU costume
Data-parallel training synchronizes every iteration: each worker computes a gradient, all gradients are summed (all-reduce), and every worker gets the sum back. For communication-heavy models this dominates — AlexNet spends 77.7% of each iteration in communication on a 100 Gbps cluster. In-network aggregation (INA) attacks the root cause: let the SWITCH add gradients as packets fly through it, so N workers' traffic becomes one stream. SwitchML and ATP showed real wins this way.
-
What people did
Build a custom network stack to talk to the switch
A switch that merges N packets into one, and rewrites their contents, violates the basic contract of TCP and RDMA: an end-to-end IMMUTABLE byte stream where every byte sent arrives unchanged. So every prior INA system concluded the transport layer had to go, and rebuilt packetization, flow control, reliability, and congestion control from scratch in a custom software stack (DPDK in SwitchML, parallel DMA in ATP).
-
Why it fell short
The custom stack costs more than the switch saves
Modern datacenters moved transport into RoCE NIC hardware — near-zero CPU, microsecond latency, line rate. Custom INA stacks cannot ride on it: measured, ATP's software stack manages 50.3 Gbps and 751 µs RTT while burning 128% of a CPU core, versus RoCE's 84 Gbps at 5.7 µs. About 41% of ATP's code re-invents standard transport functions; integrating it with one training framework took four months and 12K changed lines. And in shared clusters, a custom protocol needs its own QoS isolation or it starves against DCTCP traffic.
-
This paper's idea
Aggregate beneath the transport layer, invisibly
NetReduce keeps ordinary RoCE connections between workers arranged in a ring — and puts a "man-in-the-middle" FPGA at the switch that SPLICES them: it holds each worker's gradient packets, sums the payloads, then sends the result onward over each connection with volumes, sequence numbers, and recomputed checksums all valid. The NICs never notice. Hosts keep hardware transport (93% of 100 Gbps, 6.3 µs RTT, one CPU core) with ~850 lines of host code, and training runs 12.3–57.9% faster than hierarchical all-reduce.
Why this problem is worth solving
A data-parallel training job is a loop with a built-in traffic jam.
Every iteration, each GPU computes a gradient — often hundreds of
megabytes — and all workers must exchange and sum them before
anyone can take the next step. This synchronization is the
the collective operation that takes each
worker’s vector, computes the element-wise sum across all workers,
and delivers the complete sum back to every worker — the heartbeat
of data-parallel training
arrange P workers in a ring; each gradient
chunk travels the ring twice — once accumulating each worker’s
contribution hop by hop, once distributing the finished sum — so
bandwidth use is balanced but latency grows with P hops
In-network aggregation is the radical fix: the switch already sees
every worker’s packets, so let it do the addition. N flows in, one
flow out — traffic shrinks by roughly the worker count, the incast
disappears, and transmission time drops. SwitchML reported up to
5.5× for single jobs; ATP 38–68% in shared clusters. The physics is
sound. What this paper fixes is the price prior systems paid in
software: they couldn’t touch the datacenter’s fastest transport,
RDMA over Converged Ethernet — a transport protocol
implemented entirely in NIC hardware. The NIC packetizes messages,
tracks sequence numbers, retransmits losses, and manages congestion
by itself, delivering ~100 Gbps at microsecond latency with almost
no CPU involvement
What people did before, and where it broke
The conflict is precise. A transport connection promises an end-to-end immutable byte stream: no loss, in order, contents untouched. Its endpoints keep that promise with sequence numbers, acknowledgments, sliding windows, and checksums. In-network aggregation breaks the promise twice — it CONSUMES packets (N gradient packets become one result packet, so the receiver’s sequence accounting no longer matches what the sender transmitted) and it MUTATES payloads (a sum replaces each contribution, so integrity checks fail). Faced with that, SwitchML, ATP, PANAMA, iSwitch, and Flare all made the same call: replace the transport layer with an INA-specific stack. Three costs followed:
- Performance. The custom stacks live in software. Measured head-to-head on the paper’s testbed, ATP’s parallel-DMA stack and SwitchML’s DPDK stack reach roughly half of NetReduce’s throughput with hundreds of microseconds of RTT and more than a full CPU core busy, while a plain RoCE NIC moves 84 Gbps at 5.7 µs nearly free. The switch saved traffic; the host gave the savings back.
- Development. Dissecting ATP’s code: 40.81% of it re-implements packetization, flow control, reliability, and congestion control — functions every mature transport already has. Wiring ATP into the BytePS training framework took over four months and 12K lines of changes. ML system developers must learn a bespoke primitive instead of the sockets and RDMA verbs they already use.
- Operation. A custom protocol in a shared cluster needs its own virtual network and QoS rules, because it doesn’t share bandwidth fairly with the incumbents — ATP’s ECN-based congestion control gets starved when competing with DCTCP flows. Deployment also crosses team boundaries: ML developers need cluster operators to bless a new protocol.
There is a hardware exception worth naming:
NVIDIA/Mellanox’s commercial in-switch aggregation,
built into InfiniBand switch ASICs with NIC support — proof the
idea works in silicon, but tied to one vendor’s InfiniBand stack
rather than commodity Ethernet
How NetReduce works
NetReduce's architecture
-
Ring setup
Workers form a ring exactly as in Ring AllReduce: each opens one ordinary RoCE Reliable Connection to its successor. The application-facing abstraction (NCCL rings, RDMA verbs) is completely standard — which is why frameworks need no changes. On the switch side, an aggregator array is allocated per ring on the FPGA.
-
Send with a stowaway header new
A host flow-control module chunks each tensor into fixed-length messages and prepends a small INA header (RingID, Rank, MsgID, MsgLen) to the DATA of each message — not to the packets. The RoCE NIC packetizes the message as usual; only the first packet happens to carry the header.
-
Header recovery at the switch (CLT) new
The FPGA cannot ask the NIC to tag every packet, so it remembers instead: the first packet of a message fills a Connection Lookup Table entry mapping (SrcIP, DstIP, DstQP) → (RingID, Rank, MsgID, first PSN). Every later packet of that message looks up the table and recovers its aggregation context from its connection identity plus sequence number.
-
Aggregate and splice new
Each packet lands in an aggregator slot addressed by its message ID and relative sequence number; a per-rank bitmap tracks arrivals into a packet buffer. When all ranks' packets are present, the FPGA sums the payloads, rebuilds one result packet PER CONNECTION using each original header, recomputes the RoCE integrity checksum (ICRC), and sends them on. Each worker's NIC receives a perfectly ordinary packet from its predecessor.
-
Reliability without new protocol new
RoCE Reliable Connection mode handles loss end-to-end (NACK + go-back-N retransmission), untouched. The switch stays correct under retransmission because its aggregation is IDEMPOTENT — packets are buffered and re-summed, never accumulated in place — and a message whose first packet was lost is detected by a sequence-range check and dropped until retransmitted.
-
Multi-rack and multi-GPU new
Across racks, switches translate connections through a Connection Translation Table along a spine-leaf aggregation hierarchy (virtual switch-to-switch connections carry partial sums). Within multi-GPU machines, Parallel NetReduce runs reduce-scatter over NVLink, then n parallel NetReduce rings (one per GPU position) across machines, then all-gather — balancing load that Tencent AllReduce piles onto one master GPU per machine.
The idea
Splice, don’t terminate. Prior systems saw two irreconcilable data flows: workers-to-switch (aggregation) and switch-to-workers (multicast), neither a valid end-to-end connection. NetReduce’s central observation is that VIEWED TOGETHER they can be paired into flows that ARE valid connections: splice worker W0’s outgoing gradient flow with the result flow heading to W1, and the composite W0 → W1 flow carries exactly as many bytes as W0 sent, in order, with a correct checksum — because the switch recomputes it. The payload was swapped mid-flight, but a transport layer doesn’t check MEANING, only volume, order, and integrity. Satisfy those three invariants and the NICs’ hardware state machines — sequence tracking, acknowledgments, sliding windows, congestion control — all run unmodified. The ring topology makes the pairing natural: every worker already has exactly one successor to receive its spliced connection.
Ride the message boundary. The switch needs per-packet INA metadata (which ring, which rank, which message) but hardware NICs offer no way to tag packets. NetReduce exploits the one thing it DOES control: message contents. Prepend the INA header to the message data, and after NIC packetization the first packet carries it. A connection outlives its packets, so the switch caches the header per connection (the CLT) and derives every subsequent packet’s aggregation address from just its connection ID and sequence number — the two fields any transport, RoCE or TCP, already exposes. This is also why the design claims transport-generality: swap RoCE for TCP and only the two extracted fields change.
Make the switch logic idempotent so reliability stays free. SwitchML and ATP accumulate values into switch registers, so a retransmitted packet threatens double-counting, and they carry special detection logic. NetReduce buffers each rank’s packet and sums only when the bitmap is complete — replaying any packet just overwrites its buffer slot and re-derives the same sum. That single property lets the untouched RoCE go-back-N retransmission machinery provide all reliability: whatever arrives twice is harmless, whatever is lost gets retransmitted by the NIC. Aggregators are recycled with the same care — a result could need retransmitting, so an aggregator is freed only by a gradient packet one full window ahead, which is why the array holds 2N messages for a window of N.
Parallelize the last bandwidth gap. With NVLink inside a machine (~150 GB/s) dwarfing the 100 Gbps NIC, Parallel NetReduce splits each gradient into n pieces, reduce-scatters them across the machine’s n GPUs over NVLink, runs n NetReduce rings across machines simultaneously (each ring carrying 1/n of the data), and all-gathers the results locally. Unlike a physical ring, all machines transmit CONCURRENTLY and their packets meet at the switch — so communication time is essentially independent of the GPU count P, where flat rings degrade with P hops of accumulated latency.
One gradient’s journey
A gradient message through a 3-worker NetReduce ring
Rank = a worker's position on the ring. MsgID = the message's index in the gradient stream. PSN = RoCE's per-packet sequence number. CLT = Connection Lookup Table on the FPGA. N = sliding-window size in messages (2 in the prototype; each message is 170 packets of 1 KB payload). ICRC = RoCE's end-to-end integrity checksum.
Step 1 of 7
Setup: a ring of ordinary RDMA connections
Workers W0, W1, W2 each open one RoCE Reliable Connection to their successor — W0 to W1, W1 to W2, W2 to W0 — through the switch, exactly as NCCL's Ring AllReduce would. The switch routes training traffic through the attached FPGA, where an aggregator array is reserved for this ring.
application sees: standard NCCL ring + RDMA verbs no custom protocol anywhere on the host
Why it works
- A transport endpoint verifies volume, ordering, and integrity — not semantics → a switch that preserves byte counts, reuses original headers, and recomputes the ICRC is invisible to NIC hardware → all of RoCE’s machinery (packetization, go-back-N, congestion control, DCQCN fairness with neighboring traffic) is inherited instead of re-implemented.
- The INA header travels in message data rather than packet headers → no NIC modification is needed → the CLT converts connection ID + sequence number (fields every transport exposes) back into aggregation context, so the same trick would work over TCP.
- Buffered aggregation is idempotent → retransmission needs no special handling at the switch → the reliability protocol is literally RoCE’s own, and the switch logic stays simple enough to fit in ~10% of a mid-size FPGA’s LUTs.
- All workers transmit simultaneously and their packets meet at the switch → communication latency stops scaling with worker count P → where Flat Ring’s latency term grows with P hops and Tencent AllReduce’s with P/n, NetReduce’s stays flat (confirmed in simulation to 1024 GPUs).
- Aggregation runs where the traffic already converges → N workers’ gradient volume becomes 1 stream → freed bandwidth and removed incast compound with RoCE’s efficiency rather than trading off against it.
What it costs
- Switch ports. The FPGA middlebox connects to the commodity switch through six 100 Gbps ports — half the testbed switch’s ports. The paper’s defense: Tofino-based alternatives pay comparably or worse (ATP recirculates packets, costing 50% of internal bandwidth; SwitchML RDMA chains four pipelines in loopback, saturating 75% of ports).
- Extra hops. Traffic detours through the FPGA and back — two extra switch hops. Measured RTT is still 6.3 µs, far below the 751 µs of ATP’s software stack.
- Aggregator memory bounds the window. In-flight traffic must not exceed the aggregator array, so a message-level sliding window (sized by the bandwidth-delay product; N=2 in the prototype) throttles senders — a second flow-control loop layered on RoCE’s own.
- Synchronous, uniform sending. Workers must send the same tensors in the same order, chunked to the same MsgLen, for same-index packets to meet at the same aggregator — the standard INA assumption, inherited unchanged.
- New hardware to rack. An FPGA board per switch: +2.40% capital cost and +0.68% power on the testbed — small, but a real deployment step that pure-software overlap techniques don’t need.
- Compute-bound models gain little. ResNet50, at 25.8% communication share, improves only 6.9–15.1% — the acceleration is proportional to how network-bound the job is.
The evaluation
Two testbeds: six single-GPU machines (RTX 2080, 100 GbE) to validate the protocol against SwitchML, ATP, and Ring AllReduce, and four 8×V100 machines with NVLink (100 GbE) for the parallel variant against Flat Ring and Tencent AllReduce — plus a flow-level simulation extending the comparison to 1024 GPUs. Workloads span CNNs (AlexNet, VGG16, ResNet50 on ImageNet) and Transformers (BERT and GPT-2 pretraining, GLUE and SQuAD fine-tuning). Training quality is checked, not assumed: NetReduce’s loss curves track Ring AllReduce’s within 0.2% (one ResNet50 excursion to 1.5%), so the speedup isn’t bought with accuracy.
Ring AllReduce vs custom-stack INA vs NetReduce
| Property | Ring AllReduce (NCCL) | SwitchML / ATP (custom-stack INA) | NetReduce |
|---|---|---|---|
| Host transport path | |||
| Code to build and integrate | |||
| Training throughput, single-GPU machines | |||
| Multi-GPU machines (4×8 V100, NVLink) | |||
| Switch-side cost | None |
Check yourself
Check your understanding
1. An INA switch merges the gradient packets of 4 workers into one result packet and rewrites its payload with the sum. Predict what an unmodified receiver-side transport layer does when this traffic arrives.
2. The switch needs each packet's aggregation context (ring, rank, message ID), but RoCE NICs packetize messages in hardware with no way to tag individual packets. How does NetReduce get the metadata to the switch?
3. SwitchML and ATP accumulate gradient values into switch registers as packets arrive. NetReduce instead BUFFERS each rank's packet and sums only when its bitmap shows all ranks present. Why is this seemingly wasteful choice load-bearing?
4. On 4 machines × 8 GPUs, Parallel NetReduce beats Flat Ring by up to 68.8% and Tencent's hierarchical AllReduce by up to 57.9%. What are the two structural reasons?