Chips Need DIP: Time-Proportional Per-Instruction Cycle Stacks at Dispatch

Silvio Heverton Campelo de Santana, Joseph Rogers, Lieven Eeckhout and Magnus Jahre

To speed up code you have to know which instructions waste time and why, and modern profilers build a per-instruction breakdown by sampling the processor as instructions leave its out-of-order window at commit — but that only shows problems getting instructions out, not the equally common problems getting them in at dispatch, and the hardware that samples at dispatch today misattributes over a quarter of the time; this paper (NTNU and Ghent) builds DIP, a tiny hardware profiler that correctly charges each instruction's dispatch delay to the right cause, cutting profiling error from 27 percent to 5 percent and helping a real benchmark run 8.5 percent faster.

The argument in one glance

Problem

Profiling at commit shows how instructions leave the core, but not how they get in

As accelerators take over the easy-to-speed-up work, ordinary CPU single-thread performance becomes relatively more important, because by Amdahl's law the code that cannot be accelerated increasingly dominates runtime. Making that code fast means finding the handful of hot instructions that waste the most time and understanding why. Profilers do this by statistical sampling — occasionally peeking at the core — and present the result as a per-instruction cycle stack: for each hot instruction, a breakdown of its total time contribution across the events that caused it. The best existing profilers sample at the commit stage, where instructions leave the processor's out-of-order execution window. That reveals egress problems — for instance a long-latency load stuck at the head of the reorder buffer, blocking everything behind it. But a modern core also has to get instructions into the window in the first place, at the dispatch stage, and problems there — a pipeline flush starving the back-end, or the front-end failing to supply instructions — show up as high dispatch latency. Because the out-of-order window decouples getting in from getting out, a commit-time profile simply cannot see these ingress problems, so it often fails to explain why the code is slow.
What people did

Use the dispatch-tagging support that real CPUs already ship

Real processors do offer a way to look at dispatch: dispatch-tagging, implemented as AMD's Instruction-Based Sampling, Arm's Statistical Profiling Extension, and IBM's Random Instruction Sampling. These features tag an instruction as it dispatches and record information about it, so a developer can in principle attribute dispatch-time problems to specific instructions.
Why it fell short

Dispatch-tagging is not time-proportional, so it badly misattributes the blame

A profile is only trustworthy if it is time-proportional — if the time charged to each instruction is proportional to its real impact on runtime — because otherwise it adds systematic bias on top of unavoidable sampling noise. Dispatch-tagging is not time-proportional, for two reasons. First, it ignores dispatch parallelism: several instructions dispatch in the same cycle, in different dispatch slots, but dispatch-tagging does not account for the individual slots, so it cannot correctly divide a cycle's cost among them. Second, and worse, it invalidates any sample that lands on a mis-speculated instruction — throwing away exactly the wrong-path work whose cost the developer needs to see. The result is that dispatch-tagging misattributes a large fraction of cycles: measured against a ground-truth reference, it averages 26.9 percent profiling error and reaches over 55 percent. So the one dispatch-time signal real hardware provides is too biased to reliably explain ingress problems, and there was no accurate way to build a time-proportional per-instruction cycle stack at dispatch.
This paper's idea

Attribute each dispatch cycle to the right instruction and cause, cheaply

This paper proposes DIP, dispatch-time profiling, which builds time-proportional per-instruction cycle stacks at dispatch. Its key observation is that every dispatch slot, in every cycle, is in exactly one of four fundamental states: Base, the unavoidable dispatch time of a correct-path instruction; Stall, waiting on a back-end resource like a full reorder buffer; Front-End, when the front-end fails to supply instructions; and Misspeculation, the cost of mispredicted branches, exceptions, and serialization. DIP samples the core and, using these states, charges each sampled dispatch cycle to the instruction whose dispatch latency the core is actually exposing that cycle, and records the architectural cause. A tiny table in the performance-monitoring unit tracks the per-slot state, cause, and instruction address, costing just 49 bytes and about 1 percent overhead. The result cuts profiling error from 26.9 percent to 5.2 percent, reveals that most benchmarks suffer both ingress and egress problems, and guides a real 8.5 percent speedup.

Why this problem is worth solving

The whole value of a profile is that it points you at the right instruction and the right reason, and a commit-only view is blind to half of what matters.

A modern out-of-order processor has an execution window, the reorder buffer and issue queues, that instructions must enter and leave. They enter at dispatch, leaving the in-order front-end for the out-of-order back-end — this is ingress. They leave at commit, retiring in program order once complete — this is egress. Performance problems arise at both ends. An egress problem is when instructions cannot leave fast enough, for example a long-latency cache miss stuck at the head of the reorder buffer, which fills up and stalls the whole core; a commit-time cycle stack captures these well. An ingress problem is when instructions cannot enter fast enough, for example a branch misprediction flushing the pipeline and starving the back-end, or the front-end simply failing to fetch and decode quickly enough; these show up as certain instructions taking a long time to dispatch. Crucially, the out-of-order window decouples the two: a core can be perfectly good at retiring instructions while starving for new ones, or vice versa, so an ingress problem is often invisible in an egress-based profile and the reverse. This is why a per-instruction cycle stack at commit must be complemented by one at dispatch — together they pinpoint which instructions cause ingress and which cause egress problems, and they disagree often enough that you need both.

The catch is that building an accurate dispatch profile is subtler than it looks, and today’s hardware gets it wrong.

Instruction-level profiling relies on statistical sampling because there are far too many instructions to watch continuously: the profiler occasionally captures a sampling unit and treats it as representative of the time since the last one. The quality bar for such a profile is time-proportionality, meaning the weight assigned to each instruction is proportional to its true contribution to runtime. A time-proportional profile suffers only random sampling noise, which shrinks with more samples; a non-time-proportional one additionally carries systematic bias, which does not. Existing commit-time profilers achieve time-proportionality because the execution-time impact of an instruction is exposed at commit, and they charge each sample to the instruction whose impact the core is exposing that cycle. Dispatch-tagging, the hardware that samples at dispatch, fails this bar on two counts. It does not account for individual dispatch slots, so it cannot properly divide a cycle among the several instructions that dispatch together, and it invalidates samples that hit mis-speculated instructions, discarding the wrong-path cost that ingress analysis most needs. Both are sources of systematic bias, which is why dispatch-tagging misattributes so many cycles. Making a dispatch profile genuinely time-proportional, at low hardware cost, is the technical problem.

What people did before, and where it broke

Commit-time cycle stacks (TEA, TIP). Sample at commit and attribute execution time proportionally, producing accurate per-instruction cycle stacks at commit. Where it broke. They capture only egress; ingress problems, which manifest as high dispatch latency, are invisible in a commit-time view.
Dispatch-tagging (AMD IBS, Arm SPE, IBM RIS). Tag instructions at dispatch to attribute dispatch-time behavior. Where it broke. Not time-proportional — it ignores dispatch parallelism and discards samples on mis-speculated instructions — so it averages 26.9 percent error and cannot be trusted.
Commit-time samplers without proportional attribution (Intel PEBS). Sample at commit but with biased attribution policies. Where it broke. The biased attribution makes it non-time-proportional, so even at commit it carries systematic error.
Improving dispatch-tagging with branch records (DT-M, a bridge this paper builds). Use the last-branch record to charge misprediction cost to the responsible branch, plus account for other mis-speculation. Where it broke. It helps (16.8 percent error) but still ignores dispatch parallelism, so it is not time-proportional and stays far worse than a proper solution.
The gap. No profiler produced time-proportional per-instruction cycle stacks at dispatch, accurately attributing each dispatch cycle to the right instruction and architectural cause at low cost. Doing so, and showing that both dispatch and commit profiles are needed, is what DIP contributes.

How DIP works

From four dispatch states to an accurate, cheap dispatch profile

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

See that a dispatch profile must be complemented, not replaced — you need both ends of the window. DIP’s framing correction is that the community’s accurate commit-time cycle stacks, though excellent, only ever explain egress, and that ingress problems are just as real and, because the out-of-order window decouples the two ends, systematically hidden from a commit view. So the goal is not a better single profile but a second, complementary one at dispatch, and the paper shows empirically that this is the common case, with the large majority of benchmarks suffering both ingress and egress problems at once. Diagnosing why an out-of-order core stalls by attributing the trouble to the specific instructions responsible, so a developer knows exactly where to look, is the same critical-instruction-finding goal the work on critical-slice prefetching covered elsewhere on this site pursues, isolating the handful of instructions whose latency actually gates performance rather than treating the stall as a diffuse fog.

Make the attribution time-proportional by decomposing every dispatch slot into four states. The technical crux is that dispatch-tagging is biased because it neither splits a cycle across the several instructions dispatching in parallel nor keeps the wrong-path cost. DIP fixes both by observing that a dispatch slot is always in exactly one of four states, and charging the sampled cycle to the instruction whose latency the core is exposing in that slot, recording whether the delay is a stall, a front-end starvation, or mis-speculation. This is what makes the profile proportional to real impact instead of merely counting tagged instructions. Building a principled, low-overhead hardware profiler that attributes cost accurately and portably to guide optimization is the same instrumentation-quality goal the work on a portable, efficient framework for fine-grained value profiling covered elsewhere on this site sets, insisting that a profile be faithful enough to act on rather than suggestive.

A large fraction of dispatch trouble is the front-end and mis-speculation — the very things a dispatch view exposes. Two of DIP’s four states, Front-End and Misspeculation, capture exactly the ways a core fails to feed its execution window: the front-end cannot fetch and decode fast enough, or a mispredicted branch flushes the pipeline and the recovery costs dispatch time. These are precisely the ingress problems a commit-time profile misses, and DIP’s ability to charge them to the right instruction is what makes a dispatch profile worth having. Pinpointing where the front-end and branch machinery fail to supply the back-end, so that supply bottleneck can be fixed, is the same front-end-focused diagnosis the work on exposing shadow branches covered elsewhere on this site performs, attacking the instruction-supply side of the pipeline that a naive throughput view overlooks.

Following a dispatch cycle through DIP

Charging a stalled dispatch cycle to the right instruction and cause

A trace, following the paper's design, of how DIP attributes a sampled dispatch cycle, and how it differs from dispatch-tagging.

Step 1 of 5

A sampling unit is taken at a dispatch cycle

The profiler fires a sample at a particular cycle. Several instructions may be dispatching that cycle, each in its own dispatch slot, and some slots may be stalled or on the wrong path.

Dispatch-tagging would tag just one instruction and could not split the cycle across the parallel slots.

Why it works

The out-of-order window hides ingress from an egress view. A commit-time profile shows only how instructions leave the window → but the window decouples entry from exit, so a core can retire well while starving for instructions → so ingress problems are invisible at commit and need a dispatch profile to be seen.
Four states cover every dispatch slot exactly once. A dispatch cycle is complex, with parallel slots in different conditions → but every slot is provably in one of Base, Stall, Front-End, or Misspeculation → so DIP can attribute each slot’s time to the right instruction and cause with a complete, non-overlapping classification.
Per-slot attribution makes the profile time-proportional. Dispatch-tagging tags one instruction and ignores dispatch parallelism → but DIP charges each of the several parallel slots to the instruction whose latency it exposes → so the time each instruction receives is proportional to its true impact, removing that bias.
Keeping mis-speculated samples captures the cost that matters. Dispatch-tagging discards samples on wrong-path instructions, exactly the ingress cost of interest → but DIP tracks each sample to commit or squash and charges a flush to its cause → so mis-speculation dispatch time is measured rather than thrown away.
A per-slot table is enough, so the hardware is tiny. Full accounting of every instruction and cycle would be far too expensive to build → but statistical sampling plus a small per-slot Dispatch Information Table reproduces the golden reference approximately → so DIP needs only 49 bytes and about 1 percent overhead.

What it costs (trade-offs and scope)

It is a hardware proposal, not a feature of today’s CPUs. DIP requires the Dispatch Information Table in the performance-monitoring unit; current processors ship dispatch-tagging instead. The paper offers DT-M, an intermediate that uses existing last-branch records to improve dispatch-tagging, but DT-M still ignores dispatch parallelism and is much less accurate than DIP.
The attribution is microarchitecture-specific. The four-state golden reference and the set of stall causes are derived for a specific out-of-order core (a RISC-V BOOM design); porting DIP to another microarchitecture means re-deriving the per-slot attribution and cause set for that core.
It complements rather than replaces commit-time profiling. DIP produces dispatch-time cycle stacks; egress problems still require a commit-time profile, so full instruction-level analysis needs both, which the paper argues is the common case.
It remains a sampled approximation. DIP is far more accurate than dispatch-tagging, but statistical sampling leaves residual random error — an average of 5.2 percent against the golden reference, and up to about 21 percent on the hardest benchmark — so it guides rather than perfectly measures.
Finding the fix is still developer work. DIP identifies the instruction and cause of an ingress bottleneck, as with the load causing memory-ordering exceptions in fotonik3d, but the actual optimization — avoiding a register spill there — is human insight the profile enables, not automates, and gains depend on the workload.

The evaluation

Building a per-instruction cycle stack at dispatch

Property	Dispatch-tagging (IBS/SPE/RIS)	DT with mis-speculation (DT-M)	DIP
Time-proportional at dispatch
Handles dispatch parallelism
Handles mis-speculation cost
Average profile error
Hardware cost	Existing

From the paper's evaluation, implementing dispatch-tagging and DIP in a RISC-V BOOM out-of-order core and profiling 22 SPEC CPU2017 benchmarks against a time-proportional golden reference. DIP cuts average profile error from dispatch-tagging's 26.9 percent to 5.2 percent at 49 bytes and about 1 percent overhead; 18 of the 22 benchmarks exhibit both ingress and egress issues simultaneously, and using DIP's dispatch cycle stack to remove a memory-ordering exception improved the fotonik3d benchmark by 8.5 percent.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The contribution is a small, accurate piece of profiling hardware and a reframing of what instruction-level analysis needs. DIP argues that per-instruction cycle stacks at commit, which capture how instructions egress from the out-of-order window, must be complemented by cycle stacks at dispatch, which capture the equally common ingress problems that a commit view hides — and it makes the dispatch profile trustworthy by attributing every dispatch slot, in every cycle, to one of four fundamental states (Base, Stall, Front-End, Misspeculation) and to the instruction whose latency is being exposed, while keeping the mis-speculated samples that dispatch-tagging discards. In a small table it cuts profiling error from 26.9 percent to 5.2 percent, and it demonstrates value by guiding an 8.5 percent speedup. Read it within scope. DIP is a hardware proposal, not available on current CPUs, which ship the less accurate dispatch-tagging; its four-state attribution is derived for a specific out-of-order microarchitecture and must be re-derived elsewhere; it complements rather than replaces commit-time profiling, so both are needed; it remains a sampled approximation with residual error; and it locates bottlenecks but leaves the fix to the developer. Within those bounds it convincingly shows that half of a core’s performance story lives at dispatch, and gives the hardware to read it.

Background

Why instruction-level performance analysis is having a moment

It helps to see why careful, per-instruction profiling matters more as time goes on. For decades, faster hardware arrived automatically with each process generation, so mediocre code still got faster for free. That era is ending: Moore’s law and Dennard scaling have run out, and performance now has to be earned through better software and better use of the hardware. At the same time, specialized accelerators handle more and more of the parallel, regular work, which paradoxically makes the leftover general-purpose CPU code the bottleneck — by Amdahl’s law, the part you cannot accelerate comes to dominate. The payoff for optimizing that code can be enormous, with reports of order-of-magnitude speedups from careful tuning, but you can only optimize what you can see, and seeing means attributing wasted time to specific instructions and specific microarchitectural causes with enough fidelity to act. That is why faithful, low-overhead profiling infrastructure has become foundational, and why a profiler that turns raw hardware behavior into an actionable diagnosis is so valuable. Building the tooling that reveals where a program’s time actually goes, precisely enough to drive a fix, is the same enabling role the work on guiding memory optimization for accelerated applications covered elsewhere on this site plays, and DIP extends that role to the dispatch side of the out-of-order core, where a large and previously unreadable share of the trouble turns out to hide.

If you remember only three things

1. To optimize the increasingly performance-critical general-purpose code that accelerators leave behind, developers need per-instruction cycle stacks that attribute wasted time to specific instructions and causes, but the accurate commit-time profilers only capture how instructions egress from the out-of-order execution window (for example a long-latency load blocking the reorder buffer) and miss ingress problems — pipeline flushes or front-end starvation that show up as high dispatch latency — which the window's decoupling of entry from exit systematically hides; and the dispatch-tagging hardware that real CPUs ship (AMD IBS, Arm SPE, IBM RIS) is not time-proportional because it ignores that several instructions dispatch per cycle and discards samples on mis-speculated instructions, averaging 26.9 percent profiling error.
2. DIP builds time-proportional per-instruction cycle stacks at dispatch by observing that every dispatch slot in every cycle is in exactly one of four fundamental states — Base (a correct-path instruction dispatching normally), Stall (blocked on a back-end resource such as a full reorder buffer, issue buffer, or load/store queue), Front-End (the front-end failing to supply instructions), and Misspeculation (mispredictions, exceptions, and serialization) — and charging each sampled dispatch cycle, per slot, to the instruction whose latency the core is exposing while recording the architectural cause, and, unlike dispatch-tagging, tracking each sampled instruction by its reorder-buffer index so mis-speculation cost is attributed to the causing instruction rather than thrown away.
3. Implemented as a small Dispatch Information Table in the performance-monitoring unit costing 49 bytes and about 1 percent overhead, DIP cuts average profile error from 26.9 percent to 5.2 percent on 22 SPEC CPU2017 benchmarks in a RISC-V BOOM core; the paper is first to show both dispatch and commit cycle stacks are needed (18 of 22 benchmarks have simultaneous ingress and egress issues) and uses DIP to remove a memory-ordering exception in fotonik3d for an 8.5 percent speedup — with the caveats that DIP is a hardware proposal not yet in shipping CPUs, its four-state attribution is microarchitecture-specific, it complements rather than replaces commit-time profiling, and it remains a sampled approximation that locates bottlenecks but leaves the fix to the developer.

Check yourself

Check your understanding

1. Why does a per-instruction cycle stack built at commit often fail to explain why code is slow?
2. Why is the dispatch-tagging hardware in real CPUs not good enough?
3. What is DIP's key idea for making a dispatch profile accurate?
4. What did DIP achieve, and what are its limits?

Sources

The paper is “Chips Need DIP: Time-Proportional Per-Instruction Cycle Stacks at Dispatch” by Silvio Campelo de Santana, Joseph Rogers, Lieven Eeckhout, and Magnus Jahre (Norwegian University of Science and Technology and Ghent University), published at ASPLOS 2026 (Volume 2), DOI 10.1145/3779212.3790139. It is released under a Creative Commons Attribution license.

This explainer is written from the full text of the authors’ version. Everything above comes from the paper’s own sections: the introduction on why single-thread CPU performance grows relatively more important as accelerators proliferate (Amdahl’s law), the reliance of instruction-level analysis on statistical sampling and the notion of time-proportional profiles that avoid systematic bias, and per-instruction cycle stacks; the distinction between egress (commit) and ingress (dispatch) issues and why the out-of-order execution window decouples them so that commit-time cycle stacks (as in TEA and TIP) cannot see ingress problems, motivating per-instruction cycle stacks at dispatch; the limitations of dispatch-tagging (AMD IBS, Arm SPE, IBM RIS) — not accounting for individual dispatch slots and invalidating samples on mis-speculated instructions — making it non-time-proportional; the DIP design, including the four fundamental dispatch-slot states (Base, Stall, Front-End, Misspeculation), the golden-reference attribution policies specific to the BOOM core, the Dispatch Information Table in the performance-monitoring unit with its State, Cause, ROB Index, and Address fields and multi-hot-encoded causes, the tracking of sampled instructions until commit or squash to attribute mis-speculation cost, and the 49-byte storage and roughly 1 percent overhead; the intermediate DT-M that leverages the last-branch record; and the evaluation on 22 SPEC CPU2017 benchmarks reporting dispatch-tagging’s 26.9 percent average error (up to 55.1 percent), DT-M’s 16.8 percent (up to 34.6 percent), DIP’s 5.2 percent (up to 20.9 percent), the finding that 18 of 22 benchmarks exhibit simultaneous ingress and egress issues, and the fotonik3d case study where removing a memory-ordering exception via avoiding a register spill improves performance by 8.5 percent.