Using Analytical Performance/Power Model and Fine-Grained DVFS to Enhance AI Accelerator Energy Efficiency

Turning an accelerator's clock down saves power and costs speed, but how much of each depends entirely on what the chip is doing at that instant — and neural network operators alternate between arithmetic-bound and memory-bound work every few milliseconds, far faster than anyone could previously change frequency; this paper (Nanjing University, Huawei and Peng Cheng Laboratory) works out analytically how execution time bends with frequency, builds a power model that accounts for temperature, and uses hardware that can change clocks in one millisecond to pick a frequency per operator during GPT-3 training.

Why this problem is worth solving

The lever is well understood; the constraint on using it is not where people assume.

The chip is divided in a way that governs both the physics and the payoff.

What people did before, and where it broke

• Set one frequency for the whole application. The commonest approach in the literature, and easy to deploy. Where it broke. A training run contains matrix multiplications that want maximum clock and activation functions that barely notice it. One setting must compromise between them for the entire run.
• Adjust per phase of the application. More sophisticated, and used by several published strategies. Where it broke. The control cycles are still longer than several seconds. Operators execute in milliseconds, so a phase contains thousands of them with different needs.
• Argue for shorter control intervals in simulation. Work exists making exactly this case. Where it broke. It stayed in simulation, because the hardware to test it was not available — a frequency change on the measured graphics processor takes about 15 milliseconds.
• Model performance statistically from hardware counters. Feed counter readings to a learned model and predict runtime. Where it broke. It gives you a number without telling you why, so it cannot say which frequency an operator will tolerate or where the behaviour changes character.
• Assume memory stall time does not depend on the core clock. A simplification prior analytical work has made. Where it broke. It is wrong in the regime that matters. Transfers between the core and uncore domains are limited by the core clock right up until the uncore bandwidth saturates, so stall time does depend on frequency below that point — and the saturation point is exactly where the interesting behaviour is.
• Build power models without a temperature term. Standard practice across the power-modelling literature. Where it broke. Leakage current below the switching threshold varies with temperature, and a chip under load heats up. Omitting it folds a temperature effect into the activity coefficient, where it scales wrongly with frequency.
• The gap. Nobody had asked what execution time actually does as frequency changes, derived from how the hardware works rather than fitted; and nobody had hardware fast enough to act on the answer at the granularity where operators differ.

How a frequency schedule gets built

The idea

Execution time bends, and the bend is derivable rather than fitted. The analytical core is short. An operator’s load and store throughput is the smaller of two things: what the core clock can issue, and what the uncore’s bandwidth can supply. Below the saturation point the first binds and throughput rises with frequency; above it the second binds and throughput is flat. Convert that to cycles and add the fixed overhead of initiating a transfer, and cycle count becomes a maximum of linear functions of frequency — which composes, across the various ways loads, stores and computation can overlap, into a convex piecewise linear function whose slope steepens as frequency rises. That conclusion is what justifies fitting a simple curve of a particular shape rather than an arbitrary one, and it corrects a simplification in prior work that treated memory stall time as independent of the core clock. Reasoning from the structure of the hardware and letting the fitting handle only what is left, rather than asking a model to learn the whole relationship, is the same discipline the accelerator performance forecasting work covered elsewhere on this site follows.

Control latency is the whole story, and the paper proves it by making its own hardware worse. The reason this had not been done is not that nobody wanted per-operator frequency control; it is that a frequency change on the measured graphics processor takes about 15 milliseconds, which is longer than many operators run. The accelerator here exposes a frequency-setting operation that completes in one millisecond, dispatched onto its own stream and synchronised against the compute stream with ordinary event primitives, so the switch lands where intended without changing any user code. The convincing part is the ablation: artificially delaying that operation by 14 milliseconds, to imitate the slower hardware, cuts core power savings from 13.44 percent to 7.07 percent while making the performance loss slightly worse. Coarsening the switching interval to a second does the same thing — fewer switches, less saving, and more slowdown, not less. That reacting on a timescale slower than the workload changes is the thing that ruins fine-grained frequency control is exactly the finding the graphics-processor DVFS work covered elsewhere on this site builds its whole design around.

A cheap model in the loop is what makes the search affordable at all. Once frequency can be chosen per operator, the space of schedules is enormous: thousands of operators, nine available frequencies, and a performance budget that must be respected globally rather than locally. The alternative to modelling is to measure — pick a schedule, run it, see what happened — but a single training iteration of the largest model here takes 11 seconds, so a search that evaluates thousands of candidates that way is out of the question. With models, the same search scores twenty thousand candidate schedules in seconds. The models earn their keep not by being especially accurate but by being callable inside a loop. Replacing compile-and-run evaluation with a surrogate cheap enough to query thousands of times is the same manoeuvre the Bayesian compiler autotuner covered elsewhere on this site relies on.

Following one training iteration through the pipeline

This is a meaningful practical property. A single run at a given frequency yields data for every operator at once, so building models for thousands of operators does not require thousands of experiments.

Why it works

• The bend has a physical cause, so its shape is known in advance. Transfer throughput is the minimum of what the core can issue and what the uncore can supply → so below saturation it grows with frequency and above it is flat, making cycle count a maximum of linear terms → which is convex and piecewise linear, telling you what family of curve to fit instead of leaving it to guesswork.
• Two runs are enough to model thousands of operators. Every operator in the graph executes during a single training run → so profiling one run at one frequency yields a data point for all of them simultaneously → which is why fitting models for over 5,000 operators required running seven networks at a handful of frequencies rather than thousands of separate experiments.
• Operators differ enough that per-operator control has something to exploit. A matrix multiplication trades 6.9 percent of its performance for 7.9 percent power → an activation function trades 2 percent for 5 percent or more → so a single frequency for both is strictly worse than choosing separately, and the gap is what the whole approach harvests.
• Switching quickly matters more than switching cleverly. With a one-millisecond switch the strategy achieves 13.44 percent core power savings → adding 14 milliseconds of delay, imitating slower hardware, drops that to 7.07 percent while making performance slightly worse → because operators in a low-frequency stretch keep running fast, and those in a high-frequency stretch keep running slow, until the change lands.
• Modelling beats measuring because of the loop, not the accuracy. A single iteration of the largest workload takes 11 seconds to run → so evaluating candidate schedules on hardware would take hours for a search of any size → whereas the models let a genetic algorithm score twenty thousand candidates in seconds, which is what makes searching a space this large practical.

What it costs (trade-offs and scope)

• The temperature term, which the paper presents as its power-modelling contribution, is worth 0.35 percentage points. Setting the temperature coefficient to zero raises average error from 4.62 percent to 4.97 percent. The authors report this themselves and explain it: the temperature-dependent component is only 3 to 8 watts, about 10 to 20 percent of core power. They argue the derivation is still worth having as a small amount of code, which is fair, but the headline framing oversells it.
• The headline power number is for one part of the chip. Core power falls 13.44 percent; whole-chip power falls 4.95 percent. The reason is structural — only the core domain supports frequency adjustment at all, and the memory and auxiliary processor do not. The authors note the same limitation applies to a comparable graphics processor.
• More than half the operators were excluded from the performance evaluation. Operators running under 20 microseconds account for 58.3 percent of the total and were dropped as too variable. The justification is sound, since they contribute 0.9 percent of execution time, but the reported 1.96 percent average error describes the remaining 41.7 percent.
• The curve family was chosen for numerical convenience. One candidate was rejected because it overflowed in Python and required clamping a parameter, damaging its fit. Another was rejected because fitting was roughly 24 times slower on a real network. Neither rejection is a claim about which form is physically right.
• One vendor, one platform. Everything runs on a Huawei accelerator with its own software stack, and the one-millisecond frequency-setting operation is a property of that stack. The authors state that the classification and preprocessing steps are specific to it and would need re-tuning elsewhere.
• The performance model does not extend to out-of-order hardware. The authors say so directly: the timeline analysis assumes an execution model without multiple in-flight memory accesses or speculation, so it does not transfer to processors like CPUs.
• The search offers no guarantee. A genetic algorithm with a hand-chosen population size, mutation rate and generation count finds good schedules, not optimal ones.
• The baseline is maximum frequency. All savings are measured against running everything at the top of the range, which flatters the result relative to whatever default a production system would otherwise use.
• The inference result is not really about frequency scaling. Dropping every operator to a middle frequency during language model inference gives 25.06 percent core power savings for 2.48 percent performance loss — far better than the training numbers — but the authors explain why: inference on their device is host-bound, so the accelerator is idle waiting for work, and slowing it merely fills that idle time. That is a dispatch bottleneck being absorbed, not frequency scaling succeeding.
• Returns diminish quickly past a small performance budget. Going from a 2 percent target to 10 percent takes chip power savings from 5.56 to 11.9 percent while performance loss grows from 1.59 to 8.59 percent.

The evaluation

Check yourself

Sources

The paper is “Using Analytical Performance/Power Model and Fine-Grained DVFS to Enhance AI Accelerator Energy Efficiency” by Zibo Wang, Yijia Zhang, Fuchun Wei, Bingqiang Wang, Yanlin Liu, Zhiheng Hu, Jingyi Zhang, Xiaoxin Xu, Jian He, Xiaoliang Wang, Wanchun Dou, Guihai Chen and Chen Tian, published at ASPLOS 2025, pages 1118-1132, DOI 10.1145/3669940.3707231. The work was done on Huawei Ascend hardware by authors at Nanjing University, Huawei and Peng Cheng Laboratory.

This explainer is written from the full text, read from the authors’ copy hosted at Nanjing University after the publisher’s copy was unavailable. Everything above comes from the paper’s own sections: the background on frequency scaling and the memory hierarchy abstraction of core and uncore domains; the timeline analysis deriving load and store throughput as a minimum of core-limited and uncore-limited rates, the saturation frequency, the resulting cycle equation, and the case analysis across scenarios where loads and stores are independent or dependent and overlap with computation or do not, concluding that cycle count is a convex piecewise linear function of frequency; the practical obstacles to solving that function directly and the comparison of three candidate fitting forms with their accuracy and fitting-time trade-offs; the power model covering the frequency-voltage relationship on the platform, the decomposition into load-dependent and load-independent dynamic power plus static leakage, the temperature dependence of subthreshold leakage, the iterative solution for temperature, and the measured accuracy with and without the temperature term; the strategy generation covering bottleneck classification from pipeline utilisation, the four preprocessing steps, the genetic algorithm’s initialisation, scoring function and operators, and the mechanism for dispatching frequency-setting operations on a separate stream with event synchronisation; and the evaluation covering performance and power model accuracy, the end-to-end table across models and performance targets, the comparative experiments on switching latency and switching interval, and the discussion of model-free alternatives, the uncore limitation, generalisation to other hardware, and inference scenarios.