Chips save energy by lowering their clock when a workload is waiting on memory rather than computing, and the circuitry for changing frequency has got fast enough to do this every microsecond — but the usual method is to measure what just happened and assume the next interval resembles it, which on a graphics processor is wrong, because at that timescale consecutive intervals differ by 37 percent; this paper (AMD) finds that the same code re-executed behaves the same to within 10 percent, and builds a predictor indexed by where each thread is in the program rather than by what happened last.
The argument in one glance
-
Problem
Running fast is only worth it some of the time
Power rises roughly with the square of supply voltage and again with the clock frequency that voltage permits, so slowing a processor down saves energy steeply. Whether slowing down costs anything depends entirely on what the code is doing — a program stalled waiting for memory finishes at almost the same moment regardless of clock speed, while a program doing dense arithmetic finishes proportionally later. Workloads alternate between these constantly, so a chip that could follow the alternation would save a great deal for nothing.
-
What people did
Measure the interval that just finished and reuse it
Established control schemes work reactively. Observe how the last time interval behaved, estimate what frequency would have been best for it, and apply that to the next interval. This is simple, needs only counters the chip already has, and works well when workload behaviour changes slowly relative to how often the frequency can be adjusted.
-
Why it fell short
At a microsecond, the last interval tells you almost nothing
Circuit improvements have brought frequency transitions down to nanoseconds, so intervals of a microsecond are now possible — but the paper measures that on a graphics processor, how much a workload cares about frequency changes by 37 percent on average between consecutive microsecond intervals, against 12 percent at a hundred microseconds. Reacting to the previous interval is therefore reacting to something largely unrelated, and the faster hardware goes mostly unused.
-
This paper's idea
Index the prediction by position in the program, not by recency
This paper (Srikant Bharadwaj, Shomit Das, Kaushik Mazumdar, Bradford Beckmann and Stephen Kosonocky) observes that while consecutive intervals differ wildly, the same stretch of code re-executed behaves consistently — repeat iterations starting from the same instruction address differ by only 10 percent. So it records how sensitive each region of code was, indexed by program counter, and predicts the next interval from where each thread is about to be.
Why this problem is worth solving
The quantity being predicted needs a name and a definition before anything else makes sense.
POWER
scales roughly with the square of supply voltage multiplied by clock frequency, and since a
lower voltage permits only a lower frequency, the two move together and the saving from
slowing down is steep. What it costs is another matter. Model the number of instructions
completed in a fixed interval as a straight line in frequency, and the slope of that line is
what the paper calls SENSITIVITY. A high slope means the work is compute-bound — more clock
directly means more instructions. A slope near zero means the work is waiting on memory, so
clock speed changes little and the frequency can be dropped almost for free. The paper
verifies the linear model holds across the range a modern regulator can reach, roughly one to
three gigahertz, though it would break down at extremes. Sensitivity is therefore a
quantitative name for what a program phase is, and choosing a frequency amounts to estimating
it.
The second thing to establish is why this is harder on a graphics processor than a processor.
A
graphics processor executes thousands of threads at once, grouped into WAVEFRONTS that advance
in lock-step, each with its own program counter. A compute unit picks among whichever
wavefronts are ready, so the instructions retired in any given interval are an arbitrary
mixture drawn from many wavefronts at different points in their own progress. That produces
variation at three levels. The exact sequence executed in an interval differs depending on the
frequency itself, since frequency changes which wavefronts get ahead. The mixture differs
between consecutive intervals because wavefronts advance at different rates. And each wavefront
passes through phases of its own. Control schemes designed for a single-threaded processor
assume a coherent program moving through recognisable phases, and none of those three
variations exists there.
What people did before, and where it broke
-
Reactive control at millisecond intervals. Measure the interval just finished, pick a frequency for the next. At the timescales this was designed for — hundreds of milliseconds, later a millisecond — workload behaviour genuinely persists across intervals.
-
The gap. The persistence assumption is what circuit improvements invalidated. Once intervals shrink to a microsecond, the measured variation between consecutive intervals rises to 37 percent, and reacting to the last one is close to guessing.
-
Extend single-thread models to a compute unit. Prior work on graphics processors took the analytical performance models developed for processors and applied them at the level of a whole compute unit.
-
The gap. A compute unit is not one thread. Its instruction stream is an arbitrary interleaving of many wavefronts at unrelated points in their execution, so a model expecting a coherent program sees noise. Measured here, even a reactive scheme with a perfect estimate of the current interval reaches only 63 percent prediction accuracy.
-
Trigger on a fixed number of instructions instead of a fixed time. An alternative is to re-evaluate every so many instructions rather than every so many nanoseconds.
-
The gap. Instruction rates vary enormously on a graphics processor, so a fixed instruction window produces either long stretches with no opportunity to transition or bursts of transitions close together, which is its own problem for the regulators.
-
Assume behaviour is stable and just transition faster. The natural reading of faster circuits is that existing policies simply run more often.
-
The gap. This is the assumption the paper attacks head-on, showing variation grows as intervals shrink. Faster transitions without better prediction leave most of the available efficiency on the table.
How a frequency gets chosen every microsecond
One interval of fine-grained frequency control
-
-
new
-
new
-
new
-
new
-
new
The idea
What varies wildly in time is stable in program space. The measurement that reframes the problem is a pair of numbers. Between consecutive microsecond intervals, how much a workload cares about frequency changes by 37 percent on average, which is why reactive control fails there. But when the same wavefront reaches the same instruction address again — a later iteration of the same loop — its sensitivity differs from the earlier visit by only 10 percent. The conclusion is that sensitivity is a property of the code being executed rather than of the moment, and that the reason it looks chaotic in time is that thousands of wavefronts at unrelated points are being averaged together. Index by program counter instead and the chaos resolves. Discovering that the signal everyone was using is structurally the wrong one, and that a structural property predicts far better than recency, is exactly the finding of the mobile app-caching work covered elsewhere on this site, where what a returning app will need is predictable from distance to the collector roots and not at all from what was recently touched.
Estimate per wavefront, where the old models actually apply. The second move recovers something previous work discarded. Analytical models relating stalls to frequency sensitivity were developed for single-threaded processors and fail when applied to a whole compute unit, because a compute unit’s instruction stream is an interleaving of many independent threads. But an individual wavefront is very close to an in-order thread — it has a program counter, it commits in order, and it stalls waiting for loads. So the models are applied at wavefront level, where their assumptions hold, and the results are aggregated. That aggregation is only possible because of how sensitivity was defined: it is the slope of instructions against frequency, and slopes add, so a domain’s sensitivity is the sum of its compute units’ and a compute unit’s is the sum of its wavefronts’. The definition was chosen to make the composition work. Measuring a quantity at the level where it is actually meaningful, rather than where it is most convenient to observe, is the same corrective as the cloud power characterisation work covered elsewhere on this site, which found provisioning built on assumed draw rather than measured draw.
The predictor is a small table and the measurement is already there. What the mechanism costs is worth stating because it bounds how seriously to take it. The signal is stall time — cycles a wavefront spent waiting for memory — which the hardware can count without new instrumentation. The store is a table of 128 entries, indexed by the top bits of the program counter so that each entry covers about four instructions and the table spans roughly five hundred, which the paper justifies by noting most kernels loop over a few hundred instructions and measuring a hit rate above 95 percent. Updates happen after an interval ends and are off the critical path. The area accounting is honest about what dominates — the reported 4.59 percent of chip area is almost entirely the voltage regulators and domain-crossing logic that fine-grained control requires at all, not the predictor. Being explicit that the enabling hardware costs far more than the clever part is the same accounting discipline as the datacenter frequency work covered elsewhere on this site, which is similarly careful about where its headroom actually comes from.
Predicting from where the code is
How a program-counter-indexed table replaces reacting to the last interval
One wavefront through one interval, from measurement to the prediction it enables later.
Step 1 of 5
Note where this wavefront started
At the beginning of the interval, the wavefront's program counter is recorded. This address, not the wall-clock moment, is what the measurement about to be taken will be filed under.
The address is truncated to its upper bits so that one table entry covers roughly four consecutive instructions. Finer indexing was measured to add nothing, since sensitivity does not change meaningfully over a handful of instructions.
Why it works
- Sensitivity belongs to the code, not the clock. The same wavefront revisiting the same instruction address varies by only 10 percent → indexing by program counter gives a stable signal where indexing by time gives a 37 percent one → prediction accuracy reaches 81 percent against 63 for a reactive scheme with perfect measurement.
- Single-thread models are correct at wavefront granularity. A wavefront has a program counter, commits in order, and stalls on loads → the established stall-based model applies where its assumptions hold → the accuracy previous work lost by modelling a whole compute unit is recovered.
- The metric was defined so it composes. Sensitivity is the slope of instructions against frequency, and slopes add → a domain’s value is the sum over its compute units and each unit’s the sum over its wavefronts → per-wavefront estimates aggregate without any modelling of how they interleave.
- The signal needs no new hardware to observe. Stall cycles are already counted → measurement costs nothing → the mechanism is implementable at a timescale where anything requiring software involvement would be far too slow.
- A small table suffices because kernels loop. Most kernels cycle through a few hundred instructions → 128 entries covering about five hundred instructions achieve above 95 percent hit rate → the predictor is a few hundred bytes rather than a structure competing with the caches.
The evaluation
| Property | What was measured | The result | What it tells you |
|---|---|---|---|
| Prediction accuracy at the finest interval | |||
| Energy efficiency achieved | |||
| Whether it helps at today's timescales | |||
| How variable the workloads actually are | |||
| What it costs to build |
Check yourself
Check your understanding
1. Why does reactive frequency control break down as intervals shrink to a microsecond?
2. What makes a program counter a better index for predicting sensitivity than recency?
3. Why does the mechanism estimate sensitivity per wavefront rather than per compute unit?
4. The reported area overhead is 4.59 percent of the chip. Predict what most of that is.
Sources
Predict; Don’t React for Enabling Efficient Fine-Grain DVFS in GPUs. Srikant Bharadwaj, Shomit Das, Kaushik Mazumdar, Bradford M. Beckmann, Stephen Kosonocky. ASPLOS 2023, Volume 4, pages 253 to 267. DOI 10.1145/3623278.3624756. The mechanism is named PCSTALL. Full text read from the authors’ preprint, arXiv 2205.00121.