Fine-grained and Non-intrusive LLM Training Monitoring via Microsecond-level Traffic Measurement

Yibo Xiao, Hao Zheng, Haifeng Sun, Qingkai Meng, Jiong Duan, Xiaohe Hu and 3 others

When a giant LLM training job runs on thousands of GPUs, a single slow node can silently drag the whole run down, but the usual monitors only see coarse "operator-level" timing that cannot tell a slow node from a healthy one — and getting finer detail means editing the training code, which a cloud provider cannot do to a tenant's job; this paper (Nanjing University, National University of Singapore, and Infrawaves — Xiaoyi Bo, Chen Tian, and colleagues) presents Pulse, which measures the network card's RDMA traffic at microsecond resolution, with no code changes, and turns that into per-operator detail fine enough to pinpoint the culprit node.

The argument in one glance

Problem

A single slow node can silently stall a whole LLM training run

Training a large language model runs for weeks across thousands of GPUs spread over many machines, and those machines must stay tightly in step: at each step every GPU computes, then they all exchange data over the network (a "collective communication" like all-reduce) before moving on together. Because the job is long, large, and synchronized, anomalies are frequent — a network card that slows down, a GPU that runs hot, a link that flaps — and worse, one slow node has a cascading effect: since everyone waits for the slowest, a single straggler drags the entire run down. Operators therefore want to monitor the job continuously and, when throughput drops, quickly find which node is at fault. The trouble is that seeing the problem clearly enough to localize it is hard. The tightly synchronized structure means the obvious symptoms look the same everywhere, and the tools that could reveal the difference are exactly the ones a cloud provider is not allowed to use on a customer's job. So an expensive run can limp along, or halt, while operators struggle to point at the responsible machine.
What people did

Monitor each communication operator by instrumenting the training code

The standard online approach adds monitoring hooks inside the training framework or the collective-communication library (CCL, such as NCCL), recording when each communication operator starts and ends on every rank. This "operator-level" (OP-level) data — a start timestamp and an end timestamp per operator — is lightweight to collect and gives a useful picture of where time goes, so it is what most production monitors gather. Offline benchmarking is the other common tactic: pause the job, run a fixed test, and see which node is slow.
Why it fell short

Operator-level timing is too coarse, and instrumentation is off-limits in the cloud

Operator-level data cannot localize the most important anomalies. Consider a communication straggler: because all ranks in a collective finish together, every rank reports an almost identical operator duration, so the straggler is invisible at OP-level — the numbers give no hint which node was slow. Worse, OP-level timing folds the waiting and the computation into the communication operator's duration, so a compute problem can masquerade as a communication one. The paper shows that telling these cases apart needs sub-operator-level detail: the transmission rate inside an operator, sampled at microsecond resolution, where a straggler shows up as a lower rate partway through. But collecting that traditionally means instrumenting the training code or the CCL, and a public-cloud provider running a tenant's job cannot modify the tenant's code or libraries. Offline benchmarking, meanwhile, stops the job and cannot catch anomalies that only appear under the real workload. What was missing was a way to get microsecond-level, per-operator detail without touching the training code at all.
This paper's idea

Measure the network card's RDMA traffic at microsecond resolution, with no code changes

Pulse is a fine-grained, non-intrusive monitoring system for LLM training. Its key move is to stop watching the software and start watching the wire: it measures the RDMA traffic on each network card (RNIC) at microsecond resolution, per queue-pair, and converts that raw traffic into the transmission rates of communication operators. Because it reads the card, not the code, it needs no instrumentation of the training framework or CCL — ideal for a cloud provider. It rests on three parts. First, high-precision microsecond RDMA measurement on the card, using a three-layer design: lightweight aggregation runs on-path in the packet pipeline, while the heavier measurement runs off-path on the card's embedded microprocessors (on hardware like the NVIDIA BlueField-3), keeping measurement off the packet critical path and scaling to thousands of flows with negligible overhead. Second, application-agnostic flow-to-operator association: Pulse hooks the NCCL and RDMA APIs to learn each operator's peer and data volume, then an operator-segmentation algorithm combines that expected volume with time intervals to mark each operator's start and end on the rate curves, yielding both OP-level and sub-OP-level data. Third, lightweight localization: rather than shipping raw microsecond traces, Pulse extracts two key metrics per operator that capture its network performance. On a 64 H200 GPU testbed reproducing twelve anomaly scenarios, Pulse pinpoints the anomalous node in 10 of 12 (group-level for the rest), while state-of-the-art methods reach node-level localization in only 4 and misdiagnose 2; Pulse reaches over 90 percent precision and 100 percent recall, with about a 6 second average diagnosis latency, supports up to 2000 concurrent RDMA flows per card, and imposes negligible training overhead.

Why this problem is worth solving

Large LLM training runs are enormously expensive and easily derailed by one slow machine, so quickly finding the culprit directly saves money and time.

A frontier training run occupies thousands of GPUs for weeks, and because the machines are tightly synchronized, everyone advances only as fast as the slowest. That makes the whole run acutely sensitive to a single bad node: one slow network card or overheating GPU cascades into stalled throughput across the entire job. When that happens, every hour spent guessing which machine is at fault is an hour of thousands of idle, costly accelerators. Fast, accurate localization — pointing at the responsible node quickly and confidently — is therefore worth a great deal, both to cut wasted compute and to keep long jobs from silently degrading. This is exactly the kind of high-stakes, fine-grained diagnosis that becomes more valuable as training scales, because the cost of a missed or misdiagnosed anomaly grows with the size of the job.

The setting adds a hard constraint that rules out the usual fine-grained tools: in the cloud, you cannot change the tenant’s code.

The natural way to get detailed timing is to instrument the training framework or the collective-communication library so it records what each operator is doing. But a large share of training now runs on public-cloud infrastructure, where the provider operates the machines and network yet does not own the tenant’s training code or libraries — and cannot modify them. So the very technique that would give fine-grained visibility is off the table for the party best placed to run the monitoring. Any practical solution must therefore be non-intrusive: it has to obtain per-operator, microsecond-level detail without touching the training code, the framework, or the CCL. That constraint is what pushes the design away from software instrumentation and toward measuring something the provider does control — the network card and the traffic flowing through it — which is the opening Pulse exploits.

What people did before, and where it broke

Instrument the training code or CCL to record operator timing. Add hooks inside the framework or NCCL to log each operator’s start and end. Where it broke. It yields only coarse operator-level data and requires modifying the training code or library — impossible for a cloud provider running a tenant’s job.
Rely on operator-level (OP-level) start and end timestamps. Use each operator’s duration to spot slowdowns. Where it broke. In a synchronized collective all ranks show almost identical operator durations, so a communication straggler is invisible, and computation time gets folded into the communication operator’s duration.
Benchmark the cluster offline. Pause the job and run a fixed diagnostic test to find slow nodes. Where it broke. It stops the real run and cannot catch anomalies that only appear under the actual workload’s traffic pattern.
Sample raw microsecond rate curves and ship them for analysis. Collect fine-grained rate data everywhere for localization. Where it broke. The raw microsecond data is enormous, so collecting and moving it directly introduces significant overhead at scale.
The gap. No monitor delivered microsecond-level, per-operator detail without instrumenting the training code, at an overhead low enough to run continuously on thousands of flows.

How Pulse works

Turning raw network-card traffic into per-operator detail that localizes the culprit

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Watch the wire instead of the software. Pulse’s founding decision is to stop asking the training code what it is doing and instead measure the traffic on the network card, then reconstruct per-operator behavior from that traffic. Reading a system’s behavior from a lower, application-agnostic layer — so you learn what is happening without modifying or trusting the application above — is the same approach the work on cost-efficient distributed tracing (Mint) covered elsewhere on this site takes, capturing rich observability without asking every application to cooperate. Pulse applies it to LLM training by measuring RDMA traffic on the card, which is precisely why it can be non-intrusive: the provider controls the card even though it cannot touch the tenant’s code.

Push the measurement work into the network, off the critical path. Rather than let measurement slow down packet processing, Pulse splits it into lightweight on-path aggregation inside the card’s pipeline and heavier off-path measurement on the card’s embedded microprocessors, so the fast path stays fast. Moving processing into the network device itself, so the data plane does useful work without paying for it on the hot path, is the same principle the work on in-network aggregation for key-value streams covered elsewhere on this site is built on, offloading effort to the card rather than the host or the wire. Pulse uses it to measure thousands of flows at microsecond resolution with negligible overhead, which a host-side or on-path-only design could not sustain.

Find the one faulty node when problems cascade through a synchronized system. The whole aim of Pulse is to attribute a job-wide slowdown to the specific machine responsible, in a setting where a single straggler drags everyone down together. Pinpointing which component actually caused a system-wide performance problem, rather than just observing that things are slow, is the same challenge the work on Shapley-value influence quantification for performance debugging (ShapleyIQ) covered elsewhere on this site tackles, isolating the true culprit from a tangle of correlated symptoms. Pulse does this for training by extracting two key per-operator metrics from microsecond rate curves, which expose the straggler that operator-level durations hide.

Following a straggler through Pulse

Catching a communication straggler that operator-level timing cannot see

A trace, drawn from the paper, of how Pulse turns raw RDMA traffic into a localization of the slow node.

Step 1 of 5

A collective runs and every rank reports the same duration

During a collective like all-reduce, one rank is a straggler, but because the ranks finish together, all of them report an almost identical operator duration. At operator level the straggler is invisible — nothing in the start and end timestamps reveals which node was slow.

This is exactly the case that defeats operator-level monitoring.

Why it works

Sub-operator-level rates reveal what operator-level durations hide. In a synchronized collective all ranks report identical operator durations → but at microsecond resolution a straggler transmits at a lower rate inside the operator → so the fine-grained view exposes the culprit the coarse view cannot.
Measuring the card needs no code changes. Instrumenting the training framework or CCL is impossible for a cloud provider → but reading RDMA traffic on the network card watches the wire, not the software → so the monitoring is fully non-intrusive and deployable on tenant jobs.
The three-layer design keeps overhead negligible. Measuring thousands of flows could slow packet processing → but lightweight on-path aggregation plus off-path measurement on the card’s microprocessors stays off the packet critical path → so Pulse scales to up to 2000 flows per card without hurting training.
Two metrics beat raw traces for collection cost. Shipping raw microsecond rate curves everywhere is enormous → but extracting two key per-operator metrics that capture network performance → gives accurate localization with minimal data-collection overhead.
API hooking makes flow data operator-aware. Raw traffic is flow-level and application-agnostic → but hooking NCCL and RDMA to learn each operator’s peer and volume, then segmenting the rate curves → turns anonymous traffic into per-operator, application-agnostic monitoring.

What it costs (trade-offs and scope)

It only sees inter-node collective traffic. Pulse relies on per-flow RDMA measurement, so proprietary scale-up links like NVLink, which do not expose per-flow transmission visibility, are invisible — anomalies confined to intra-node communication fall outside its view.
It does not support every collective mode. Pulse currently cannot monitor collectives using CollNet or NVLS, because CollNet needs specialized hardware and NVLS is designed for intra-node communication.
Its parallelism identification is tuned to mainstream frameworks. The method assumes parallelism implemented in the style of Megatron or DeepSpeed, and may need adaptation when a user’s parallelism deviates from those mainstream frameworks.
It depends on capable network cards. The design assumes mainstream RNICs with an embedded microprocessor and programmable pipeline (NVIDIA ConnectX-6 Dx, BlueField-3); the off-path measurement offload relies on that hardware being present.
It captures anomalies through their network signature. Pulse localizes by traffic behavior, so problems that leave a clear mark on communication rates are its strong suit; a purely computational fault is caught only insofar as it shows up in the timing and rate of communication.

The evaluation

Monitoring LLM training to localize an anomalous node

Property	Operator-level code instrumentation	Offline benchmarking	Pulse
Needs training-code changes
Detail collected
Localizes a communication straggler
Runs during the live job
Localization accuracy (paper testbed)

A comparison of ways to monitor LLM training for anomaly localization. By measuring RDMA traffic on the network card at microsecond resolution (with no code instrumentation), converting it into per-operator transmission rates via API hooking and operator segmentation, and localizing from two key per-operator metrics, Pinpoints the anomalous node in 10 of 12 scenarios on a 64 H200 GPU testbed — where state-of-the-art methods reach node-level localization in only 4 and misdiagnose 2 — at over 90 percent precision and 100 percent recall. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ copy hosted at Nanjing University, of the ASPLOS 2026 paper). The system described is the paper’s: Pulse is a fine-grained, non-intrusive LLM-training monitoring system that enables anomaly localization by measuring per-queue-pair RDMA traffic on network cards at microsecond resolution and converting it into the transmission rates of communication operators, requiring no instrumentation of the training code or collective-communication library. It has three parts: (1) high-precision microsecond RDMA measurement on mainstream RNICs (NVIDIA ConnectX-6 Dx and BlueField-3) via a three-layer design — lightweight on-path aggregation in the packet pipeline plus off-path measurement offloaded to the card’s embedded microprocessors — scaling to thousands of flows with negligible overhead; (2) application-agnostic flow-to-operator association that hooks NCCL and RDMA APIs to infer each operator’s peer and data volume, plus an operator-segmentation algorithm combining expected volume and time interval to mark operator start and end, producing OP-level and sub-OP-level data; (3) lightweight localization that extracts two key metrics per operator (for built-in and custom collectives) instead of raw microsecond rates. The paper reports that, on a 64 H200 GPU testbed reproducing twelve anomaly scenarios, Pulse achieves node-level localization in 10 of 12 (group-level otherwise), while state-of-the-art methods reach node-level localization in only 4 and misdiagnose 2, and that Pulse reaches over 90 percent precision and 100 percent recall with about a 6 second average diagnosis latency, supports up to 2000 concurrent RDMA flow measurements per NIC, and imposes negligible training overhead. Stated limitations: it monitors only inter-node collectives (not scale-up links like NVLink), does not support CollNet or NVLS collectives, and its parallelism identification assumes mainstream frameworks (Megatron or DeepSpeed). Exact mechanisms, the two metrics, and all quantitative results should be read from the paper.

Background

Why big training jobs keep needing new ways to see inside themselves

It helps to see Pulse within a broad pattern: as training clusters grow, the gap widens between how much can go wrong and how well operators can see what is going wrong, and closing that gap is a recurring theme in large-scale machine-learning systems. A run that spans thousands of accelerators for weeks is, statistically, almost always experiencing or about to experience some hardware or performance anomaly, and because the machines are synchronized, a local fault becomes a global slowdown. Yet the very scale that makes anomalies common also makes them hard to observe: coarse monitoring hides the culprit, and fine monitoring is too expensive or too intrusive to run continuously. The response, again and again, is to find some cheap, always-on signal that can be mined for the detail operators need — here, the network card’s own traffic. This concern for keeping large training infrastructure observable and healthy as it scales is the same terrain the work on the design and operation of shared machine-learning clusters covered elsewhere on this site occupies, grappling with how to run big training fleets reliably. Pulse’s contribution is to show that microsecond-level, per-operator visibility need not require touching the training code at all: by measuring RDMA traffic on the card and reconstructing operator behavior from it, monitoring can be both fine-grained and non-intrusive, which is exactly what a cloud provider needs to diagnose a tenant’s job.

If you remember only three things

1. Large-scale LLM training is anomaly-prone (long, large, tightly synchronized), and one slow node cascades into a job-wide stall, but the usual online monitors gather only operator-level data — each communication operator's start and end timestamps — which is too coarse to localize the culprit. In a synchronized collective all ranks show almost identical operator durations, so a communication straggler is invisible, and computation time gets folded into the communication operator's duration. Getting finer detail traditionally requires instrumenting the training code or the collective-communication library, which a public-cloud provider running a tenant's job cannot do.
2. Pulse gets microsecond-level, per-operator detail without any code changes by measuring the network card's RDMA traffic instead. It measures per-queue-pair RDMA rates at microsecond resolution using a three-layer on-card design (lightweight on-path aggregation in the packet pipeline plus off-path measurement on the card's embedded microprocessors, as on the BlueField-3), so measurement stays off the packet critical path and scales to thousands of flows. It hooks the NCCL and RDMA APIs to learn each operator's peer and data volume, then an operator-segmentation algorithm marks each operator's start and end on the rate curves, giving both operator-level and sub-operator-level data. For localization it extracts two key metrics per operator rather than shipping raw microsecond traces.
3. On a 64 H200 GPU testbed reproducing twelve anomaly scenarios, the paper reports Pulse pinpoints the anomalous node in 10 of 12 (group-level otherwise), while state-of-the-art methods reach node-level localization in only 4 and even misdiagnose 2; Pulse achieves over 90 percent precision and 100 percent recall, with an average diagnosis latency of about 6 seconds, supports up to 2000 concurrent RDMA flow measurements per network card, and imposes negligible training overhead. Its scope: only inter-node collective traffic (not scale-up links like NVLink), no CollNet or NVLS collectives, and parallelism identification tuned to mainstream frameworks like Megatron or DeepSpeed.

Check yourself

Check your understanding

1. Why is operator-level (OP-level) monitoring too coarse to localize a communication straggler in LLM training?
2. What is the key idea that lets Pulse get fine-grained, per-operator detail without instrumenting the training code?
3. How does Pulse measure thousands of RDMA flows at microsecond resolution without slowing down training?
4. What does the paper report about Pulse's localization accuracy compared with state-of-the-art methods on its 64-GPU testbed?

Sources

The paper is “Pulse: Fine-Grained and Non-Intrusive LLM Training Monitoring via Microsecond-Level RDMA Traffic Measurement” by Xiaoyi Bo, Chen Tian, and colleagues (Nanjing University, National University of Singapore, and Infrawaves), published at ASPLOS 2026, DOI 10.1145/3779212.3790163. An author copy is hosted at Nanjing University.

This explainer is written from the paper’s full text (the authors’ hosted copy). Pulse is a fine-grained, non-intrusive monitoring system for LLM training that enables anomaly localization by measuring per-queue-pair RDMA traffic on network cards at microsecond resolution and converting it into the transmission rates of communication operators, with no instrumentation of the training code or collective-communication library. It combines three parts: high-precision microsecond RDMA measurement on mainstream RNICs (NVIDIA ConnectX-6 Dx and BlueField-3) via a three-layer design — lightweight on-path aggregation in the packet pipeline plus off-path measurement offloaded to the card’s embedded microprocessors — that scales to thousands of flows with negligible overhead; application-agnostic flow-to-operator association that hooks NCCL and RDMA APIs to infer each operator’s peer and data volume, plus an operator-segmentation algorithm combining expected volume and time interval to mark operator start and end, producing operator-level and sub-operator-level data; and lightweight localization that extracts two key metrics per operator rather than raw microsecond rates. On a 64 H200 GPU testbed reproducing twelve anomaly scenarios, the paper reports node-level localization in 10 of 12 (group-level otherwise), versus 4 of 12 for state-of-the-art methods (which misdiagnose 2), with over 90 percent precision, 100 percent recall, about a 6 second average diagnosis latency, support for up to 2000 concurrent RDMA flow measurements per NIC, and negligible training overhead. Stated limitations include monitoring only inter-node collectives (not scale-up links like NVLink), no support for CollNet or NVLS collectives, and parallelism identification tuned to mainstream frameworks (Megatron or DeepSpeed). Numbers are the paper’s own; exact mechanisms, the two metrics, and full results should be confirmed in the paper.