MSCCL++: Rethinking GPU Communication Abstractions for AI Inference

Changho Hwang, Peng Cheng, Roshan Dathathri, Abhinav Jangda, Saeed Maleki, Madan Musuvathi and 9 others

When a large AI model runs across many GPUs, they spend a big slice of their time just shuffling data between each other, and to make that fast the experts throw out the standard communication libraries and hand-write brittle custom code for every workload and every new chip; this work from Microsoft argues the fix is three layers of abstraction — a thin primitive layer that exposes the real hardware instead of hiding it, a small language for writing custom communication algorithms, and a drop-in library on top — which together beat the standard libraries by around 1.7 times and now run in Microsoft Azure and AMD's own library.

The argument in one glance

Problem

To make multi-GPU communication fast, experts abandon the standard libraries

AI runs across many GPUs that constantly exchange data, and communication is a major bottleneck — up to 10 to 40 percent of LLM inference. Vendor libraries like NCCL provide standard collective operations, but to reach peak performance practitioners hand-write custom communication code instead, because a general library cannot tune to each workload or exploit brand-new hardware in time.
What people did

Hand-write custom communication code from scratch

Bypass the standard library and write bespoke communication kernels tuned to the exact workload, message sizes, and hardware, as even NVIDIA's own inference stack does for its AllReduce.
Why it fell short

Custom code is error-prone, brittle, and non-portable

Hand-writing communication is enormously costly: new hardware is immature and poorly documented, and developers must manage subtle synchronization and data-consistency issues across GPUs, CPUs, and network cards. The result is fast but error-prone and non-portable, and it must be redone for each workload and each new chip.
This paper's idea

Three layered abstractions for performance, portability, and productivity (MSCCL++)

MSCCL++ is a design methodology with three layers. A Primitive API exposes the fundamental hardware transfer modes (one-sided, asynchronous) while hiding synchronization and staying portable across vendors. A DSL lets developers write custom communication algorithms with a global view of all GPUs, with an executor handling synchronization and optimization. A Collective API packages tuned algorithms behind the same standard interface as NCCL. It beats the standard libraries by a 1.7 times geomean (up to 5.4 times), runs in Microsoft Azure, and was adopted by AMD's RCCL.

Why this problem is worth solving

As AI models spread across ever more GPUs, the time spent moving data between them has become a first-order cost.

A large AI model no longer fits or runs on a single GPU, so inference and training are spread across many GPUs that must constantly exchange partial results. This communication is not a minor overhead: in many large-language-model inference tasks, the communication kernels account for roughly 10 to 40 percent of the total time. The GPUs coordinate through collective operations — standardized patterns in which all the GPUs cooperate to move and combine data. The central one is AllReduce, which sums each GPU’s partial result and gives every GPU the total; it can be decomposed into ReduceScatter, which sums the inputs and distributes pieces of the result across the GPUs, and AllGather, which collects a distributed buffer onto every GPU. Because these operations are so common and so performance-critical, GPU vendors ship optimized libraries to implement them — NVIDIA’s NCCL, AMD’s RCCL, and Microsoft’s MSCCL — which choose an algorithm (a ring, a tree, and so on) based on the data size. Making these collectives fast directly speeds up AI workloads, which is why so much effort goes into optimizing them.

The striking thing is that, to go truly fast, the experts throw the standard libraries away.

To reach peak performance, practitioners routinely write custom communication code from scratch rather than use the vendor library — even NVIDIA’s own inference stack implements a custom AllReduce instead of relying on NCCL. There are three reasons. First, peak performance requires a careful balance between latency and bandwidth tuned to a specific workload’s message sizes and communication patterns, and a general-purpose library built to serve every workload cannot always optimize for any one of them; the custom AllReduce beats the library on small messages and falls back to it on large ones. Second, collectives often involve computation, such as the reduction in AllReduce, whose performance depends on exploiting parallelism in ways that mesh with the communication, favoring workload-specific customization. Third, chips and interconnects are evolving so fast that practitioners rush to exploit new hardware features long before a general library adapts. But writing custom communication code is enormously expensive. New hardware capabilities are often immature and poorly documented, forcing developers into low-level details with little guidance, and the interactions among GPUs, CPUs, and network cards raise subtle synchronization and data-consistency problems that must be handled exactly right to avoid both bugs and bottlenecks. The standard libraries’ own low-level primitives do not help — NCCL’s send-and-receive is synchronous and hides one-sided, asynchronous hardware capabilities.

What people did before, and where it broke

Use a general-purpose vendor library (NCCL, RCCL, MSCCL). Call the standard optimized collective library for all communication. Where it broke. A one-size-fits-all library cannot tune latency versus bandwidth to each workload, cannot overlap communication with a workload’s computation as well as a custom kernel, and lags new hardware.
Hand-write custom communication code. Write bespoke kernels tuned to the workload and hardware, as expert teams do. Where it broke. It is error-prone and hugely expensive, forces developers through poorly documented low-level details and subtle cross-device synchronization, and is not portable across workloads or new chips.
Use the standard libraries’ low-level primitives. Build on the send-and-receive style primitives the libraries expose. Where it broke. They are vendor-specific and mask hardware features — NCCL’s synchronous send-and-receive prevents one-sided asynchronous communication, and other abstractions block interconnect-specific optimizations.
The gap. No approach delivered performance, portability, and productivity together — letting experts exploit new hardware fully while letting ordinary users adopt the results effortlessly. Providing that through layered abstractions is what MSCCL++ does.

How MSCCL++ works

From a single general library to three layered communication abstractions

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Expose the real hardware instead of hiding it. The base layer, the Primitive API, deliberately exposes the fundamental building blocks of the hardware interconnects — the different data-transfer modes such as memory-mapped, port-mapped, and switch-mapped input and output, with one-sided and asynchronous semantics — where prior primitives masked them, so experts can exploit exactly the capabilities they otherwise hand-code, while the layer still hides the tricky synchronization and stays portable across vendors. Giving software direct access to the fast hardware mechanisms rather than trapping it behind a restrictive general abstraction that hides them is the same move the direct-storage-access work covered elsewhere on this site makes, exposing the underlying fast path so the performance-critical code can use the hardware as it really is instead of through a lowest-common-denominator interface.

Let a language capture the algorithm and a compiler handle the hard parts. The middle layer is a domain-specific language in which a developer writes a communication algorithm with a global view of all the GPUs, and an executor takes over the error-prone concerns — inserting the needed synchronization, fusing instructions, and removing redundant memory accesses before lowering to the primitives — so a workload-specific algorithm that used to demand painstaking manual synchronization becomes something a developer can write quickly and correctly. Providing a higher-level language in which the algorithm is stated and a compiler automatically discharges the low-level optimization and correctness burden is the same design the graph-network compiler work covered elsewhere on this site follows, decoupling what a developer expresses from the tedious kernel-level tuning the tool generates on their behalf.

Package the results behind the interface everyone already uses. The top layer is a library of algorithms, built in the DSL and tuned for different scenarios, that exposes the very same collective API as the standard library, so an ordinary application gets the specialized, faster implementations with no code changes and no expertise required. Delivering an advanced capability through a simple, familiar standard interface so that non-experts adopt it with essentially no effort is the same strategy the shared-prediction-service work covered elsewhere on this site uses, wrapping a sophisticated mechanism behind a minimal common interface so the whole ecosystem can benefit without having to understand the internals.

Following an AllReduce through MSCCL++

Speeding up the AllReduce at the heart of LLM inference

A trace of MSCCL++ handling a collective, following the paper's three-layer design.

Step 1 of 5

Several GPUs must sum and share partial results

During LLM inference, several GPUs each hold a partial result that must be summed and shared with all of them — an AllReduce — and this communication is a large chunk of the inference time.

The collective sits on the critical path of every decode step.

Why it works

General libraries cannot be optimal for every workload. Because peak communication performance needs latency-versus-bandwidth choices tuned to a workload’s exact message sizes and patterns → a one-size library built for all workloads cannot match a tailored implementation → so experts abandon it and write custom code.
Exposing the hardware, not hiding it, enables the key optimizations. Because prior primitives mask one-sided and asynchronous transfer modes behind restrictive abstractions → developers cannot exploit compute-communication overlap or interconnect-specific tricks → so a primitive layer that exposes the real transfer modes unlocks the performance experts hand-code.
A language turns expert tricks into ordinary work. Because writing correct custom communication means managing synchronization and consistency across GPUs, CPUs, and network cards → a DSL that gives a global view and auto-inserts synchronization and fuses instructions removes that burden → so workload-specific algorithms become feasible to write quickly.
A standard interface spreads the benefit to everyone. Because most users are not communication experts → packaging the tuned algorithms behind the same API the standard library uses → so ordinary applications get the speedups with no code changes.
The right abstractions survive new hardware. Because the primitives are organized by data-transfer mode rather than by vendor → new hardware features slot into the existing layers → so supporting them takes weeks, not a rewrite, keeping the stack ahead of fast-evolving hardware.

What it costs (trade-offs and scope)

Peak performance still needs expertise at the lower layers. The layering lets ordinary users adopt tuned algorithms effortlessly, but squeezing the last performance out still requires an expert to write in the DSL or the Primitive API; the top layer is only as good as the algorithms someone put in the library.
Primitives are organized by transfer mode, not fully universal. The building blocks are specific to the data-transfer modes hardware supports and are portable across vendors that support the corresponding mode; hardware with a fundamentally different transfer model would still need new primitives, even if the layering makes that far cheaper.
The abstractions must expose hardware without leaking too much. Exposing one-sided, asynchronous transfer modes gives power but shifts some responsibility upward; the design puts clear consistency and synchronization semantics in the primitive implementation, but reasoning about asynchronous communication remains inherently more involved than synchronous send-and-receive.
The speedups vary by workload and are sometimes modest. Collective speedups are large on average (a 1.7 times geomean, up to 5.4 times), but the end-to-end inference gains are smaller (a 1.2 times geomean, up to 1.38 times), since communication is only part of inference; the benefit depends on how communication-bound a workload is.
It is a substantial system to adopt and maintain. MSCCL++ is a full stack — primitives, a DSL and its executor, and an algorithm library — not a small patch; its value is proven by production use, but adopting it is a real engineering commitment.
Evaluation is on specific hardware and baselines. The reported gains are measured on NVIDIA A100 and H100 and AMD MI300x GPUs against NCCL, RCCL, and MSCCL; other hardware, interconnects, and workloads would shift the numbers.

The evaluation

Three ways to do multi-GPU communication for AI

Property	Standard library (NCCL)	Hand-written custom code
Peak performance for a specific workload
Exposes one-sided, asynchronous hardware		Yes, by hand
Engineering effort	Low, drop-in	Very high, error-prone
Portable across vendors and new hardware	Per vendor
Outcome	Baseline	Fast but brittle

Three ways to do multi-GPU communication for AI, as framed by the paper. A general-purpose vendor library (NCCL, and the NCCL-based RCCL and MSCCL) is easy to adopt but a one-size-fits-all design that cannot tune to each workload or fully exploit new hardware. Hand-writing custom communication code reaches peak performance but is error-prone, non-portable, and hugely expensive, drowning in cross-device synchronization. MSCCL++ offers three layers: a Primitive API exposing the fundamental hardware transfer modes (memory-, port-, and switch-mapped input and output, one-sided and asynchronous) while hiding synchronization and staying portable; a domain-specific language in which developers write custom algorithms with a global view of all GPUs, whose executor auto-inserts synchronization, fuses instructions, and optimizes; and a Collective API packaging tuned algorithms behind the same interface as NCCL. Across NVIDIA A100 and H100 and AMD MI300x, its collective kernels average 1.99, 2.08, and 1.43 times faster than NCCL, RCCL, and MSCCL, and LLM inference decodes run 1.11 to 1.31 times faster; overall a 1.7 times geomean (up to 5.4 times) for collectives and 1.2 times (up to 1.38 times) for inference. It is in production across Microsoft Azure AI services, was adopted by AMD's RCCL and the SGLang inference framework, and absorbed new hardware features (multimem, multi-node NVLink) in as little as 2 to 16 person-weeks.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

MSCCL++ is a mature, production-hardened answer to a real dilemma in AI systems. Multi-GPU communication is a large and growing share of AI cost, but the general-purpose libraries that make it easy cannot reach peak performance, so experts hand-write custom communication that is fast yet error-prone, non-portable, and obsolete the moment new hardware arrives. The paper’s thesis is that a single layer cannot deliver performance, portability, and productivity together, but three layers can. The Primitive API exposes the fundamental hardware transfer modes — including the one-sided, asynchronous capabilities prior primitives hid — while managing the cross-device synchronization and staying portable across vendors, giving experts the power they otherwise hand-code. The DSL lets developers write workload-specific algorithms with a global view of all GPUs while a compiler handles synchronization, fusion, and optimization, and crucially retains asynchrony so algorithms can overlap communication with computation. The Collective API packages the results behind the same interface as the standard library, so ordinary users benefit with no effort. The evidence is strong and unusually practical: collective speedups averaging around two times over the vendor libraries, real inference gains, and — most telling — production deployment across Microsoft Azure, adoption by AMD’s own RCCL and by the SGLang inference framework, and new hardware features absorbed in a couple of person-weeks. Read the scope precisely. The top layer is only as good as the algorithms experts put into it; peak performance still needs DSL or primitive-level work; the primitives are organized by transfer mode, so genuinely new transfer models still need new building blocks; asynchronous abstractions are more powerful but harder to reason about; and the end-to-end inference gains are modest since communication is only part of the picture. The durable idea is that when a domain forces a choice between an easy-but-slow library and a fast-but-brittle hand-built stack, the right resolution is layered abstractions that let each user pay exactly the effort their performance target demands — and that the lowest layer should expose the hardware as it really is, not a lowest-common-denominator view of it.

Background

Why layered abstractions that expose the hardware keep winning

It helps to see MSCCL++ within a recurring tension in performance-critical systems: general-purpose abstractions make software portable and productive but sacrifice the last, decisive increments of performance, while hand-tuned code captures that performance but is brittle, unportable, and unmaintainable — and the durable resolution is almost always a layered design that lets a user descend to exactly the level of control they need. Here the layers run from a hardware-exposing primitive interface, through a language for expressing algorithms, up to a drop-in standard library, and the key design choice is that the bottom layer reveals the real hardware transfer modes rather than hiding them. The heterogeneous-FPGA framework work covered elsewhere on this site embodies the same instinct in a neighboring domain, unifying diverse and fast-evolving accelerators behind layered abstractions so that portability and reduced engineering effort do not come at the price of exploiting each device’s features. The common thread is that hardware is diversifying and evolving faster than any single fixed interface can track, so the software that endures is built in layers whose lowest level is honest about the hardware and whose higher levels progressively trade control for convenience. Such designs pay off in three ways at once, as MSCCL++ shows: experts get the performance they would otherwise hand-code, ordinary users get it for free, and the whole stack absorbs new hardware in weeks rather than through a rewrite. As accelerators and interconnects keep proliferating, this layered, hardware-honest approach to abstraction is likely to be the way high-performance systems stay both fast and maintainable.

If you remember only three things

1. Multi-GPU communication is a major and growing bottleneck in AI — communication kernels are 10 to 40 percent of many LLM inference tasks — and GPUs coordinate through collective operations like AllReduce (sum every GPU's partial result and give all GPUs the total), implemented by vendor libraries such as NVIDIA's NCCL and AMD's RCCL. But to reach peak performance, practitioners abandon these general-purpose libraries and hand-write custom communication code (even NVIDIA's own inference stack does), because a one-size-fits-all library cannot tune latency versus bandwidth to each workload, cannot mesh reductions with parallelism as well, and lags fast-evolving hardware. Hand-writing is error-prone, non-portable, and expensive, forcing developers through immature, poorly-documented hardware and subtle synchronization and consistency issues across GPUs, CPUs, and network cards.
2. MSCCL++ is a design methodology of three layered abstractions that together provide performance, portability, and productivity. A Primitive API exposes the fundamental hardware transfer modes — memory-mapped, port-mapped, and switch-mapped input and output, with one-sided and asynchronous semantics — where prior primitives (like NCCL's synchronous send-and-receive) masked them, while hiding synchronization and staying portable across vendors and generations. A domain-specific language lets developers write custom communication algorithms with a global view of all GPUs, and its executor auto-inserts synchronization, fuses instructions, removes redundant memory accesses, and retains asynchrony so algorithms can overlap communication with computation. A Collective API packages a library of DSL-written, tuned algorithms behind the same interface as NCCL, so ordinary users adopt them with no effort.
3. Across NVIDIA A100 and H100 and AMD MI300x, MSCCL++ collectives average 1.99, 2.08, and 1.43 times faster than NCCL, RCCL, and MSCCL, and LLM inference decodes run 1.11 to 1.31 times faster (overall a 1.7 times geomean, up to 5.4 times, for collectives and 1.2 times for inference). It runs in production across Microsoft Azure AI services, was adopted as the default by AMD's RCCL and by the SGLang inference framework, and absorbed new hardware features (multimem, multi-node NVLink) in only 2 to 16 person-weeks — evidence the abstractions are robust. Scope: peak performance still needs expert DSL or primitive work, primitives are organized by transfer mode, asynchronous abstractions are harder to reason about, and inference gains are modest since communication is only part of the workload. The durable lesson is that when a domain forces a choice between an easy-but-slow library and fast-but-brittle hand-built code, layered abstractions whose lowest level honestly exposes the hardware let each user pay exactly the effort their performance target demands.

Check yourself

Check your understanding

1. Why do performance-minded practitioners abandon standard libraries like NCCL and hand-write custom communication code?
2. What are the three layers of MSCCL++, and what does each provide?
3. Why does the Primitive API expose the hardware's transfer modes rather than hide them, unlike prior interfaces?
4. What evidence shows MSCCL++'s abstractions are effective and robust?

Sources

The paper is “MSCCL++: Rethinking GPU Communication Abstractions for AI Inference” by Changho Hwang, Peng Cheng, Roshan Dathathri, Abhinav Jangda, Saeed Maleki, Madan Musuvathi, Olli Saarikivi, Aashaka Shah, Ziyue Yang, Binyang Li, Caio Rocha, Qinghua Zhou, Mahdieh Ghazimirsaeed, Sreevatsa Anantharamu, and Jithin Jose, published at ASPLOS 2026, DOI 10.1145/3779212.3790188. The authors are affiliated with Microsoft.

This explainer is based on the full text of the paper, obtained from the authors’ arXiv version (arXiv:2504.09014), which corresponds to the CC-BY version of record. The problem framing (GPU communication as a bottleneck accounting for 10 to 40 percent of LLM inference, collective operations such as AllReduce, ReduceScatter, and AllGather, the vendor libraries NCCL, RCCL, and MSCCL, the reasons practitioners write custom communication code — latency-versus-bandwidth tuning, computation within collectives, and rapidly evolving hardware — and the resulting engineering costs and the limitations of NCCL’s synchronous send-and-receive primitives), the MSCCL++ design (the three-layer methodology; the Primitive API exposing the memory-mapped, port-mapped, and switch-mapped data-transfer modes with one-sided and asynchronous semantics while managing synchronization and staying portable; the domain-specific language with a thread-block-based global view and its executor that inserts synchronization, fuses instructions, and optimizes while retaining asynchrony for compute-communication overlap; and the Collective API providing the same interface as NCCL), and the results (collective kernels averaging 1.99, 2.08, and 1.43 times faster than NCCL, RCCL, and MSCCL on NVIDIA A100 and H100 and AMD MI300x; Primitive-API kernels 3 percent faster than their DSL versions; LLM inference decodes 1.11 and 1.31 times faster in vLLM and SGLang; a 1.7 times geomean and up to 5.4 times for collectives and 1.2 times and up to 1.38 times for inference; production use across Microsoft Azure; adoption by RCCL and SGLang; and the 16 and 2 person-week efforts to support multimem and multi-node NVLink) are all drawn directly from the paper. MSCCL++ is open source. Background explanations of GPUs, collective communication, interconnects, and layered abstractions are standard and provided to make the paper’s contribution understandable.