EXIST: Enabling Extremely Efficient Intra-Service Tracing Observability in Datacenters

Xinkai Wang, Xiaofeng Hou, Chao Li, Yuancheng Li, Du Liu, Guoyao Xu and 6 others

When a cloud service is slow, engineers need to see the exact order of function calls inside it to find the culprit, but recording that detail normally slows the service by 5 to 18 percent — unacceptable in a busy datacenter, and worst exactly when the service is already struggling; this work from Shanghai Jiao Tong and Alibaba builds EXIST, which uses the processor's built-in tracing hardware and toggles it once per core instead of once per context switch, dropping the overhead to about a tenth of a percent while recovering full coverage by repeating short traces across the cluster, and it runs today in Alibaba's datacenters.

The argument in one glance

Problem

Seeing inside a slow service is powerful but costly

When a cloud service misbehaves, engineers want intra-service tracing: a step-by-step record of the function and instruction flow inside that one service, so they can find the exact sequence that went wrong. But recording that detail slows the traced service by 5 to 18 percent, which is far too much to run in a busy production datacenter.
What people did

Insert tracepoints into the running code

Existing intra-service tracing tools embed tracepoints into the application, in the kernel or the program itself, that fire as the code runs and write out a chronological trace. This is flexible and widely used for debugging.
Why it fell short

The overhead is worst exactly when it matters

Making the application itself do the recording steals its time. And the cost peaks in the situations engineers most want to observe: a shared node runs many services and switches between them thousands of times a second, and anomalies strike when the machine is already near its limits, where even a 2 percent tracing cost can cascade into a more than 10 percent slowdown. So tracing gets turned off precisely when it is needed most.
This paper's idea

Trace with hardware, toggled per core, completed across the cluster

EXIST uses the processor's built-in tracing hardware, so recording barely touches the application. It toggles the tracer once per core the service runs on, not once per context switch, collapsing the costly control operations. It sizes each buffer to actual usage, and because each node captures only a short window, it repeats tracing across replicas to rebuild full coverage. Overhead drops to about a tenth of a percent, and it runs in Alibaba datacenters.

Why this problem is worth solving

Finding the cause of a slowdown needs more than statistics.

There are two ways to watch a running system, and they answer different questions. Statistical observability, such as a flame graph built by sampling, tells you the averages: what fraction of time each function used, how often each ran. That is useful for a summary, but it cannot tell you which event caused which, because it throws away the order things happened in. Chronological observability, or tracing, records the actual sequence of events, so it can reveal cause and effect. Tracing itself comes at two granularities. Inter-service tracing, like the widely deployed Zipkin, records the calls between services — service A called B called C — and can point to the culprit service that was slow. It is cheap, around one percent overhead. But once you know which service is slow, you still need to know why, and for that you need intra-service tracing: a chronological record of the function and instruction flow inside that single service. The paper’s running example is a service where a function named Get, called right after a Flush, occasionally runs abnormally long and blocks the program. Statistics would show only that Get sometimes takes a while; inter-service tracing would show only that the whole service was slow. Intra-service tracing shows the exact culprit sequence — Flush then a stalled Get — which is what an engineer needs to fix it.

The reason this has stayed impractical is a cost that peaks at the worst moment.

Today’s intra-service tracing tools work by instrumentation: they embed extra tracepoints into the kernel or the application that fire as the code runs. Because the application itself is now doing the recording, every traced event steals time from the real work, and the measured overhead runs 5 to 10 percent on average, up to 18 percent in the worst case. That would be tolerable if it stayed constant, but it does not — it gets worse in exactly the conditions engineers most need to observe. First, sharing: a datacenter node runs many co-located services and the operating system switches between them thousands of times a second, and much of the tracing cost is paid at each of those switches. Second, stress: performance anomalies tend to appear when a machine is already running near its limits, above ninety percent utilization, and there the system is fragile. The paper measures that under such load, even a two percent tracing overhead on a single service can cascade into more than a ten percent end-to-end response-time degradation, because each disturbed request delays the ones behind it. So a single-digit overhead is not merely inconvenient; it makes tracing unsafe to run precisely when an anomaly is happening. Modern processors offer a way out: built-in hardware tracing that records the instruction flow from the side, with far less intrusion. But that hardware forces a three-way trade-off among speed, memory, and how much it can capture, and prior systems built on it never reached the needed efficiency.

What people did before, and where it broke

Statistical sampling (flame graphs and similar). Periodically sample the call stack and report averages and proportions. Where it broke. It is cheap but discards the order of events, so it can show that a function was sometimes slow but never why — it cannot reason about causality.
Software instrumentation for intra-service tracing. Embed tracepoints in the kernel or application that fire as the code runs. Where it broke. The application does the recording itself, so it costs 5 to 18 percent, and that cost is worst under the sharing and stress where anomalies happen, so operators cannot leave it on.
Prior hardware-tracing systems (for reverse debugging or full tracing). Use the processor’s built-in tracer for other goals. Where it broke. Hardware tracing forces a trade-off among speed, memory use, and coverage, and these systems picked other corners — tiny buffers with frequent tracer operations, or full continuous capture — so none reached the per-mille overhead a datacenter needs.
The gap. No system used off-the-shelf hardware tracing to deliver intra-service tracing at a tenth of a percent overhead, staying cheap under sharing and stress while still covering the behavior of interest. Building that, by rethinking how the tracer is controlled and how coverage is gathered across a cluster, is what EXIST does.

How EXIST works

From intrusive tracing to a tenth-of-a-percent, cluster-completed trace

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Let the hardware do the recording instead of the application. The reason software tracing is expensive is that the traced program itself runs the extra tracepoint code, so every event it records is time taken from real work. Modern processors include a hardware tracer that captures the instruction flow from the side, without the program executing any tracing code, which is the only way to reach a tenth of a percent overhead. Handing a job the software used to do on its critical path over to hardware that was built to do it cheaply is the same move the userspace-storage work covered elsewhere on this site makes, letting the memory-protection hardware translate and check storage accesses so the operating system stack drops out of the fast path.

Pay the costly control operation per resource, not per event. Even with hardware tracing, the conventional approach starts and stops the tracer at every context switch, and on a busy node those switches number in the thousands per second, so the control operations, not the recording, become the real overhead. EXIST’s key move is to toggle the tracer once per core the service is scheduled onto during a tracing period, turning a per-context-switch cost into a per-core one and collapsing it. Cutting how often an expensive control operation must fire, by tying it to a coarse resource rather than to every fine-grained event, is the same discipline the lazy-storage-barrier work covered elsewhere on this site follows, issuing its costly ordering operation only when it is truly forced rather than once for every thread.

Trade complete coverage on one node for cheap coverage across many. Keeping the overhead tiny means each node can only capture a short window of execution, not the whole run. EXIST accepts that, and recovers completeness a different way: the same service runs on many replicas across the cluster, and the same anomaly tends to recur, so repeating short cheap traces across instances and combining them reconstructs the full behavior. Assembling a complete picture from many cheap partial observations, instead of paying once for an exhaustive one, is the same instinct the early-stopping inference work covered elsewhere on this site follows, gathering just enough evidence and stopping the moment a statistical test says the answer is settled rather than always running the full computation.

Following an anomaly being traced

Debugging a stalled Get with EXIST

Tracing one anomaly shows how the pieces combine into a cheap, complete answer.

Step 1 of 5

Inter-service tracing names the culprit service

A recommendation service intermittently stalls. Cheap inter-service tracing points to it as the slow service, but only says that this service is slow, not why inside it.

Inter-service tracing records calls between services, not the flow within one.

Why it works

Statistics hide causality. Because sampling reports only averages and proportions, such as how often each function ran → it cannot say which event caused which → so explaining an anomaly needs a chronological trace of the real execution order.
Software instrumentation is too intrusive. Because inserting tracepoints makes the application itself run the recording code → every traced event steals time from the workload → so hardware tracing, which records from the side, is the only route to a tenth of a percent overhead.
Per-context-switch control is the true cost. Because the conventional tracer is toggled at every context switch, and busy nodes switch thousands of times a second → each toggle fires a costly interrupt that dominates the overhead → so toggling once per core instead collapses that cost.
Stress multiplies the harm. Because anomalies strike when the cluster is near its limits, where even a two percent tracing cost cascades into more than ten percent end-to-end slowdown → a single-digit overhead is unusable exactly when tracing is needed → so only per-mille overhead makes tracing safe to leave on.
Repetition rebuilds coverage. Because one node can only trace a short window cheaply, while the same anomaly recurs across many replicas → combining short traces from many instances → so the full behavior is reconstructed without any single node paying the full cost.

What it costs (trade-offs and scope)

Each node sees only a short window. The per-mille overhead is bought by tracing brief spans, on the order of milliseconds, rather than a whole execution. Completeness depends on the anomaly recurring across replicas so that repetition can fill it in; a rare, non-repeating event on one node may be missed.
It leans on cluster-scale replication. The coverage strategy assumes many instances of the same service run across the cluster. It fits large datacenters well, but offers less to a single-instance or small deployment where there is nothing to repeat across.
It depends on processor hardware tracing. EXIST is built on off-the-shelf hardware tracing features and the kernel hooks to control them, so it is tied to processors that provide such tracing and requires kernel extension to manage them.
It reconstructs, so it is not exhaustively exact. EXIST reports over ninety percent accuracy against an exhaustive reference, which means it is highly faithful but not perfect; the cheapness comes with a small loss of completeness that matters for corner cases.
Buffers and coverage need tuning. Sizing per-core buffers to usage and choosing tracing periods and repetitions are decisions the system makes from application information; they work well in the evaluated setting but are parameters that must be managed.
It finds the sequence, not the fix. EXIST makes the anomalous execution visible cheaply; interpreting it and repairing the underlying bug or misconfiguration remains the engineer’s job, though seeing the exact culprit sequence is most of the battle.

The evaluation

Three ways to trace what happens inside a service

Property	Software instrumentation	Prior hardware tracing
Intrudes on the application		Less
Tracer control cost	Not applicable
Average overhead
Coverage strategy		Full, costly
Safe to run under datacenter stress		No

Three ways to capture the execution flow inside a single service, evaluated on standard benchmarks and real Alibaba e-commerce applications and deployed in Alibaba clusters. Software instrumentation embeds tracepoints so the application records itself, costing 5 to 18 percent, worst under the sharing and stress where anomalies occur. Prior hardware-tracing systems intrude less but still toggle the tracer at every context switch and target other trade-offs, staying in the single-digit to double-digit percent range. EXIST records with the processor's hardware tracer, controls it once per core rather than per context switch, sizes buffers to usage, and rebuilds coverage by repeating short traces across replicas. On the SPEC benchmark suite its slowdown is 0.4 to 1.5 percent, and it cuts tracing overhead by 2 to 10 times over prior methods (for example 3.5, 4.4, and 6.6 times on compute benchmarks and 6.4, 7.3, and 12.2 times on online services), reaching about 1 percent overhead with over 90 percent accuracy against an exhaustive tracing reference.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

EXIST is an engineering result aimed at a real, measured pain: intra-service tracing is invaluable for debugging cloud anomalies, but its cost has kept it off in production, and worst under the exact stress where anomalies live. The contribution is a set of well-chosen trade-offs that push the overhead down to a tenth of a percent. It records through the processor’s hardware tracer so the application is barely disturbed; it toggles that tracer once per core instead of once per context switch, which is the key insight, since the control operations were the dominant cost on busy nodes; it sizes buffers to usage; and it accepts short per-node spans, recovering full coverage by repeating cheap traces across a service’s many replicas. The results are strong and, importantly, come from a real deployment in Alibaba clusters, with 2 to 10 times lower overhead than prior methods and over ninety percent accuracy. Read it with its scope. The low cost is bought by capturing short windows and leaning on cluster-scale replication, so a rare, non-repeating event on a lone instance can be missed, and small single-instance deployments gain less. It depends on processors that offer hardware tracing and on kernel hooks to drive them, and its accuracy, while high, is a reconstruction rather than an exhaustive record. The durable lesson is that an observability tool is only useful if it is cheap enough to leave running, and that the path to cheap enough can be moving cost from software to hardware and from per-event to per-resource.

Background

Why cheap observation changes what a datacenter can manage

It helps to see EXIST beside a broader truth: you can only manage what you can observe, and observation that is expensive distorts the very thing it measures. A datacenter operator wants to understand what its services are actually doing, but the act of looking closely has traditionally slowed them down, so operators looked coarsely and guessed. The cooperative-scheduling work covered elsewhere on this site attacks a sibling problem from the inside, where a guest operating system schedules badly because it cannot cheaply see the hypervisor’s live internal state, and the fix is to expose that state through a light, deliberately coarse channel so decisions improve without heavy cost. In both cases the enabling move is to make the needed visibility nearly free, so it can be on all the time rather than reserved for special investigations. The broad point is that the ceiling on how well a large system can be run is often set by how cheaply its internals can be watched. Drop the cost of observation far enough and a whole class of management and debugging that was previously too disruptive to attempt becomes routine, which is why shaving an observability tool from several percent to a tenth of a percent is not a minor optimization but a change in what is possible.

If you remember only three things

1. To debug why a cloud service is slow, engineers need intra-service tracing: a chronological record of the function and instruction flow inside one service, which reveals the culprit sequence (for example a Get that stalls right after a Flush) that statistics and inter-service tracing cannot. But existing intra-service tracing embeds tracepoints into the running code and costs 5 to 18 percent, and that cost is worst under the sharing and stress where anomalies actually happen — on a busy node even a 2 percent tracing overhead can cascade into more than a 10 percent end-to-end slowdown, so operators turn tracing off exactly when they need it.
2. EXIST reaches a tenth of a percent overhead with three cooperative components. Its tracing controller records through the processor's built-in hardware tracer (barely touching the application) and, crucially, toggles the tracer once per core the service runs on rather than once per context switch, collapsing the control operations that dominated the cost on busy nodes. Its memory allocator sizes each core's trace buffer to actual usage to hold more useful traces in limited memory. Its coverage optimizer accepts that each node captures only a short window and rebuilds full coverage by repeating short traces across the service's many replicas in the cluster.
3. Deployed in Alibaba datacenters and tested on standard benchmarks and real e-commerce applications, EXIST slows the SPEC suite by only 0.4 to 1.5 percent, cuts tracing overhead by 2 to 10 times over prior methods (about 0.9 to 1.1 percent overhead versus 3 to 16 percent), and reaches over 90 percent accuracy against an exhaustive tracing reference. Its limits: each node sees only a short window, so completeness relies on the anomaly recurring across replicas and on large-scale replication; it depends on processor hardware tracing and kernel hooks; and it reconstructs rather than records exhaustively. The lesson is that an observability tool is only useful if it is cheap enough to leave on.

Check yourself

Check your understanding

1. Why is intra-service tracing worth its cost, when cheaper options exist?
2. What is the single biggest source of overhead that EXIST attacks, and how?
3. EXIST keeps overhead tiny by tracing only short windows on each node. How does it still achieve good coverage?
4. A team runs a single instance of a service (no replicas) and wants to catch a one-off, non-repeating glitch. What does EXIST's design imply?

Sources

The paper is “EXIST: Enabling Extremely Efficient Intra-Service Tracing Observability in Datacenters” by Xinkai Wang, Xiaofeng Hou, Chao Li, Yuancheng Li, Du Liu, Guoyao Xu, Guodong Yang, Liping Zhang, Yuemin Wu, Xiaopeng Yuan, Quan Chen, and Minyi Guo, published at ASPLOS 2025, DOI 10.1145/3676641.3716283. The authors are at Shanghai Jiao Tong University and the Alibaba Group. EXIST is deployed in Alibaba clusters.

This explainer is written from the full text of the paper, obtained from co-author Chao Li’s Shanghai Jiao Tong University publications page, which matches the ASPLOS 2025 proceedings and is distributed under a Creative Commons Attribution license. Everything above comes from the paper’s own sections: the introduction distinguishing statistical from chronological observability and inter-service from intra-service tracing, with the Get-after-Flush culprit example and the 5 to 18 percent overhead figures for existing intra-service tracing; the motivation showing tracing overhead worsening under shared and stressed conditions (frequent context switches and the finding that a roughly 2 percent single-service overhead can cause more than 10 percent end-to-end degradation near ninety percent utilization) and the inherent three-way trade-off of hardware tracing among time efficiency, space overhead, and data coverage; the design of the three cooperative components — the operation-aware tracing controller that reduces tracer control operations from the number of context switches to the number of processing cores and records switches as a five-tuple, the usage-aware memory allocator that sizes the traced core-set and per-core buffers (from about 4 to 128 megabytes) by application usage, and the repetition-aware coverage optimizer that gathers coverage through tracing periods and repetitions across the cluster; and the evaluation deployed in Alibaba clusters on the SPEC benchmark suite, online benchmarks (Memcached, Nginx, MySQL), and five real Alibaba e-commerce applications, reporting a 0.4 to 1.5 percent SPEC slowdown, overhead reductions of 3.5, 4.4, and 6.6 times on compute benchmarks and 6.4, 7.3, and 12.2 times on online services over the baselines, an average overhead of about 0.9 to 1.1 percent versus 3 to 16 percent for prior methods, and over 90 percent accuracy against an exhaustive tracing reference.