Squeezing Operator Performance Potential for the Ascend Architecture

Yuhang Zhou, Zhibin Wang, Guyue Liu, Shipeng Li, Xi Lin, Zibo Wang and 12 others

Huawei's Ascend AI chip gives programmers far more manual control than a GPU over which compute unit runs what and how data moves between its many buffers, which unlocks speed but makes it painfully hard to see why an operator is slow; the standard tool for that, the roofline model, breaks on Ascend because naively copying the GPU approach produces 180 tangled charts and, worse, misreads hardware that shares a transfer engine as underused when it is actually saturated; this work from Nanjing University and Huawei rebuilds the roofline around the idea of a component — a unit whose instructions run one after another while different components run in parallel — so it correctly pinpoints the real bottleneck, and applied to eleven production models it delivered 1.07 to 2.15 times end-to-end speedups.

The argument in one glance

Problem

Ascend gives manual control but hides why operators are slow

Huawei's Ascend AI chip lets programmers manually manage its compute units, buffers, and data transfers — powerful, but easy to misuse. To optimize, you first need to know which part of the hardware is the bottleneck, and that diagnosis is hard.
What people did

Use the roofline model, as on CPUs and GPUs

Apply the standard roofline model — plot arithmetic intensity against achieved performance to see whether a kernel is memory-bound or compute-bound — extending it to Ascend the way it was extended from CPUs to GPUs.
Why it fell short

Naive roofline is unreadable and misdiagnoses Ascend

Ascend's many precision and transfer combinations produce 180 tangled roofline charts. Worse, the naive model assumes transfers or precision operations that actually run sequentially on a shared unit run in parallel, so it reports hardware as underutilized when it is fully saturated — misdirecting optimization.
This paper's idea

Rebuild the roofline around components (component-based roofline)

Define a component as a unit whose instructions run serially while different components run in parallel — physically each compute or transfer unit. A component-based roofline shows whether an operator is bound by a component or underutilized, and splits underutilization into insufficient parallelism versus inefficient components, pinpointing the real bottleneck. Applied to eleven production models, it gave 1.07 to 2.15 times speedups.

Why this problem is worth solving

Domain-specific AI accelerators are now everywhere, and getting real performance out of them is a distinct skill from writing the model.

As deep learning has grown, companies have built domain-specific architectures — chips designed specifically for AI rather than general computation — with Google’s TPU and Huawei’s Ascend as leading examples, and Ascend runs influential models including large LLMs. What sets Ascend apart from a general-purpose GPU is how much it exposes to the programmer. It has distinct compute units with assigned roles: a scalar unit for control logic, and high-performance vector and matrix (called Cube) units for the heavy numerical work. It has customized on-chip memory buffers tailored to AI access patterns, and it allows flexible data movement — cross-layer transfers with asymmetric bandwidth, rather than a rigid layer-by-layer flow — so a developer can pick transfer paths that best suit a workload. Movement is handled by memory transfer engines, and the programmer explicitly controls instruction pipelining to overlap compute and transfer. Where a GPU hides much of this behind an automatic runtime, Ascend requires the user to manage it by hand. That manual control is exactly what unlocks Ascend’s performance potential — and exactly what makes it easy to leave that performance on the table.

The consequence is that optimization on Ascend lives or dies on accurate bottleneck analysis, and the field’s standard tool for that does not fit.

The roofline model is the standard way to reason about why a computation is slow. In its basic form it plots, for a kernel, its arithmetic intensity (the ratio of arithmetic operations to bytes moved) on one axis against its achieved performance on the other, with two ceilings: a sloped line for memory bandwidth and a flat line for peak compute. Where a kernel lands tells you whether it is memory-bound (limited by data movement) or compute-bound (limited by arithmetic), which points to what to optimize; hierarchical roofline extends this to multiple cache levels and precisions on CPUs and GPUs. On Ascend, though, the mainstream models do not work, and even naively extending the GPU approach fails for two reasons. First, visualization explodes: Ascend has around nine precision-compute units and twenty transfer paths, and comparing every precision against every transfer as GPU analysis would yields on the order of 180 roofline combinations, an unreadable tangle of points. Second, and more damaging, the naive model gives wrong answers, because it assumes operations run in parallel when Ascend actually runs many of them sequentially on a shared unit — so it can report hardware as badly underutilized when it is in fact fully saturated, sending the developer to optimize the wrong thing.

What people did before, and where it broke

Apply the CPU or GPU roofline model directly. Use DRAM or hierarchical roofline, the mainstream tools, to classify Ascend kernels as memory- or compute-bound. Where it broke. They were designed for architectures very different from Ascend and do not capture its distinct compute units, buffers, and transfer engines, so they simply do not fit.
Naively extend the roofline to every precision and transfer. Build a roofline for each precision-compute unit against each transfer path, as hierarchical roofline expands from DRAM roofline. Where it broke. Around nine precisions times twenty transfers gives roughly 180 roofline combinations — a crowded, unreadable chart that defeats the roofline’s key virtue of being easy to visualize.
Assume parallel execution when reading utilization. Treat separate transfers or precision operations as running concurrently, as the naive model does. Where it broke. On Ascend, transfers within one memory transfer engine run sequentially, and mixed-precision operations in one compute unit run sequentially; a unit fully occupied by two sequential transfers taking 67 and 33 percent of the time is wrongly reported as 67 and 33 percent underutilized, when it is actually saturated.
Optimize by intuition and trial and error. Without a fitting model, tune parameters, buffers, and pipelines by hand and hope. Where it broke. Ascend’s causes of slowness are complex and interacting, so without accurate bottleneck identification the effort is unguided and often wasted.
The gap. No performance model captured Ascend’s real execution semantics — serial within a unit, parallel across units — so bottleneck analysis was either unreadable or wrong. Providing that, with the component abstraction, is what this work does.

How the component-based roofline works

From tangled, misleading rooflines to component-based bottleneck analysis

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Model the hardware with the abstraction that matches how it actually runs. The central move is to introduce the notion of a component: a unit within which instructions execute one after another, serially, while different components execute in parallel with each other, with each component corresponding physically to one of Ascend’s compute or memory-transfer units. This single abstraction captures the execution semantics that the GPU-style roofline missed — serial within, parallel across — and it is what lets the model reason correctly about the chip. Choosing an abstraction that faithfully reflects the underlying hardware’s real structure and parallelism, so that analysis built on it is sound rather than approximate, is the same move the deep-learning tensor-program work covered elsewhere on this site makes, exposing the accelerator’s true organization to the programmer through a model that matches the hardware instead of papering over it with a convenient but inaccurate one.

A model that respects real execution stops misdiagnosing the bottleneck. Because a component’s instructions are serial, the component-based roofline correctly sees that two transfers sharing one memory-transfer engine, or two precision operations sharing one compute unit, are not running in parallel — so a unit fully occupied by sequential work is recognized as saturated (a real bottleneck) rather than misreported as underutilized. It then separates the two genuinely different failure modes: insufficient parallelism between components, where units sit idle waiting because work is nearly serialized, versus an inefficient component, where a unit is active but running below its capability. Diagnosing the true cause of poor performance precisely, rather than guessing from a model whose assumptions the hardware violates, is the same discipline the GPU memory-optimization guidance work covered elsewhere on this site is built on, attributing a slowdown to its actual source so that optimization effort is aimed where it will help instead of at a symptom.

Distill years of hard-won practice into reusable strategies. Beyond the model, the paper turns two years of optimizing real operators into a transferable method: use the component-based roofline to find the bottleneck component, inspect the pipeline and code to pin down the root cause, and apply the optimization that matches that cause, with case studies spanning the cube, vector, and scalar units and all the transfer engines, and a summary mapping common bottlenecks to the optimizations that best address them. Converting accumulated, quantitative experience with a specific accelerator into explicit, actionable guidelines that others can follow is the same contribution the data-streaming accelerator analysis work covered elsewhere on this site offers, taking careful measurement of a real accelerator and packaging it into guidance that turns one team’s expertise into a repeatable optimization process.

Following an optimization through the component-based roofline

Diagnosing and fixing a slow Ascend operator

A trace of the paper's roofline-guided optimization workflow, following its component model and case-study method.

Step 1 of 5

An operator runs slower than expected

A deep-learning operator — a matrix multiply, an attention block — underperforms on Ascend. The developer needs to know which of the compute units and transfer engines is holding it back before touching the code.

Ascend exposes compute, buffers, transfers, and pipelining to manage by hand.

Why it works

The component abstraction matches Ascend’s execution. Because instructions within a unit run serially while different units run in parallel → modeling each unit as a component with that exact semantics mirrors the real hardware → so the analysis reflects what actually happens rather than a convenient approximation.
Serial-aware modeling fixes the misdiagnosis. Because the naive model assumed parallel execution and mislabeled saturated shared units as underutilized → treating a component’s work as serial correctly identifies a fully-occupied unit as the bottleneck → so optimization is aimed at the real constraint, not a phantom.
Splitting utilization separates distinct failures. Because underutilization can come from either idle waiting or slow work → decomposing it into active-time ratio and execution efficiency distinguishes insufficient parallelism from an inefficient component → so the developer knows whether to improve overlap or improve a unit.
Components tame the visualization explosion. Because analysis is organized around a manageable set of components rather than every precision-by-transfer pair → the roofline stays readable instead of collapsing into 180 tangled combinations → so the model keeps the roofline’s core virtue of being easy to interpret.
A repeatable workflow turns analysis into speedups. Because the model pinpoints the bottleneck component and cause → inspecting the pipeline and code to find and fix the root cause becomes a directed process → so accumulated practice reliably converts diagnosis into real end-to-end acceleration.

What it costs (trade-offs and scope)

It is specific to the Ascend architecture. The component abstraction and the model are built around Ascend’s compute units, buffers, and transfer engines. The general principle — model serial-within, parallel-across execution — could transfer, but the concrete model is not a drop-in for other accelerators.
It targets operator compute performance, not the whole system. The paper explicitly scopes itself to the computational performance of operators; input/output and communication bottlenecks in large-scale distributed training are orthogonal and not addressed here.
It still requires expertise to act on. The model pinpoints the bottleneck component and cause, but diagnosing the root cause then means inspecting pipeline status and operator code and choosing the right optimization — the analysis guides skilled work rather than automating it away.
The gains depend on the operator and model. The reported speedups span a wide range (computation-time speedups of about 1.08 to 2.70 times, end-to-end 1.07 to 2.15 times), reflecting that some operators had far more slack than others; a well-tuned operator will show little improvement.
The evidence is practice-based, on Huawei’s own workloads. The results come from two years optimizing eleven real models and integrating 41 operators into the Ascend operator library, which is strong practical validation but is drawn from one organization’s models and hardware rather than an independent benchmark suite.
The model captures compute and transfer, within its abstraction. By focusing on components as compute and transfer units, effects outside that framing — for instance certain cross-layer or system-level interactions beyond the modeled components — are handled only insofar as they map onto the component view.

The evaluation

Ways to analyze operator performance on Ascend

Property	CPU/GPU roofline	Naive Ascend roofline
Fits Ascend's architecture
Readable visualization	Yes, but wrong target
Handles serial-on-shared-unit execution
Distinguishes underutilization causes	No	No
Demonstrated optimization outcome	Not applicable	Misleads effort

Ways to analyze operator performance on Huawei's Ascend AI accelerator, as compared in the paper. The mainstream CPU and GPU roofline models do not fit Ascend, whose distinct compute units, customized buffers, and flexible transfer engines differ sharply from those architectures. Naively extending the roofline to compare every precision-compute unit against every transfer path yields around 180 combinations, an unreadable chart, and worse, it assumes operations run in parallel when Ascend runs many of them serially on a shared unit, so it reports a fully-saturated memory-transfer engine (67 and 33 percent of the time on two sequential transfers) or a busy compute unit doing mixed precision as merely 67 and 33 percent utilized, misdirecting optimization. The component-based roofline defines a component as a unit whose instructions run serially while different components run in parallel, each physically a compute or transfer unit, so it correctly identifies whether an operator is bound by a component (a real bottleneck) or underutilizing the hardware, and it decomposes underutilization into a component active-time ratio and an execution efficiency, attributing it to insufficient parallelism between components or to an inefficient component. Paired with a practice-derived workflow — find the bottleneck component, inspect the pipeline and code for the root cause, apply the matching optimization, with case studies across the cube, vector, and scalar units and all transfer engines — it was used over two years to optimize eleven real models (including the 100-billion-parameter PanGu-alpha), integrating 41 operators into the Ascend operator library and yielding computation-time speedups of about 1.08 to 2.70 times and end-to-end speedups of 1.07 to 2.15 times.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This is a pragmatic, experience-driven paper whose value is as much in the reusable method as in the model. The setting is a real industrial pain point: Huawei’s Ascend, a leading domain-specific AI accelerator, deliberately exposes far more of the hardware to the programmer than a GPU does — distinct scalar, vector, and matrix compute units, customized buffers, flexible cross-layer transfers driven by memory-transfer engines, and explicit instruction pipelining — which is the source of both its performance and its difficulty. Optimizing on such a chip hinges on accurately identifying the bottleneck, and the field’s standard tool, the roofline model, does not fit: extending it naively from the GPU produces around 180 tangled precision-by-transfer charts and, more seriously, misdiagnoses the hardware, because it assumes parallel execution where Ascend actually serializes transfers within a memory engine and precision operations within a compute unit, reporting saturated units as underutilized. The paper’s contribution is the component abstraction — a unit whose instructions run serially while different units run in parallel — and a roofline built on it that correctly tells whether an operator is bound by a component or underutilized, and in the latter case whether the cause is insufficient parallelism or an inefficient component. Around this it packages a repeatable optimization workflow and case studies covering every unit type, distilled from two years of practice, with strong results: eleven real models optimized, 41 operators upstreamed, and 1.07 to 2.15 times end-to-end speedups. Read the scope carefully. The model is specific to Ascend, though its serial-within, parallel-across principle generalizes; it targets operator compute performance and explicitly leaves out input/output and communication bottlenecks in distributed training; it guides skilled developers rather than automating optimization; the speedups vary widely because some operators had much more slack than others; and the evidence, while substantial, comes from one organization’s models and hardware. The durable idea is that a performance model is only as good as its fidelity to how the hardware really executes, and that on an accelerator built around explicit, manually-managed parallelism, getting that execution model right — serial within a unit, parallel across units — is what turns performance analysis from misleading into actionable.

Background

Why each new AI accelerator needs its own performance lens

It helps to see this work within the broader proliferation of domain-specific AI accelerators, where the industry has moved away from general-purpose GPUs toward a growing variety of custom chips — TPUs, Ascend, and many others — each with its own compute units, memory hierarchy, and execution model. The full-stack accelerator search work covered elsewhere on this site reflects the design side of this same trend, exploring the vast space of possible domain-specific accelerator designs to match hardware to workloads, which is precisely what produces chips like Ascend with their idiosyncratic, manually-managed structure. The common thread is that specialization buys performance at the cost of generality, and one thing that does not transfer across these divergent architectures is the performance analysis: a roofline or profiler tuned for a GPU can be actively misleading on an accelerator whose execution semantics differ, as this paper shows for Ascend. So every serious accelerator ends up needing its own performance lens — a model faithful to how that specific hardware runs — before its potential can be squeezed out. As the accelerator landscape keeps diversifying, the pattern this paper exemplifies, of building an execution-faithful performance model and a practice-derived optimization workflow for a particular chip, is likely to recur for each new architecture that programmers are asked to hand-tune.

If you remember only three things

1. Huawei's Ascend is a domain-specific AI accelerator (like Google's TPU) that, unlike a GPU, gives the programmer manual control over its distinct compute units (scalar for control, vector and matrix/Cube for heavy math), its customized on-chip buffers, its flexible cross-layer data transfers driven by memory-transfer engines (MTEs), and its instruction pipelining. This control unlocks performance but makes it easy to lose performance to poor algorithms, task or buffer assignments, transfer paths, or synchronization — and fixing any of these first requires knowing which part of the hardware is the bottleneck. The standard tool, the roofline model (which plots arithmetic intensity against achieved performance to reveal whether a kernel is memory- or compute-bound), does not fit Ascend. Naively extending the GPU approach fails two ways: comparing every precision-compute unit against every transfer yields around 180 tangled roofline combinations, and the model assumes parallel execution when Ascend runs many transfers and mixed-precision operations serially on a shared unit, so it reports a fully-saturated unit (e.g. two sequential transfers taking 67 and 33 percent of the time) as underutilized, misdirecting optimization.
2. The paper introduces a component-based roofline model. A component is a unit whose instructions execute serially, while different components execute in parallel, each physically a specific compute or memory-transfer unit — matching Ascend's real execution semantics. The model shows whether an operator is bound by a component (that unit is saturated and is the true bottleneck) or is underutilizing the hardware; for underutilization it decomposes utilization into a component active-time ratio and an execution efficiency, attributing the loss to either insufficient parallelism between components (near-serial execution leaving units idle) or an inefficient component (active but running below capacity). This correctly recognizes a saturated shared engine as the bottleneck instead of misreading it as underused, and it keeps the roofline readable by organizing analysis around a manageable set of components rather than every precision-by-transfer pair.
3. Around the model the paper packages a repeatable, practice-derived workflow — use the component-based roofline to find the bottleneck component, inspect pipeline status and operator code to pin the root cause, and apply the matching optimization — with case studies spanning the cube, vector, and scalar units and all MTE transfer engines, plus a summary mapping common bottlenecks to their best fixes. Over two years it was used to optimize eleven real production models (vision, NLP, recommendation, and LLMs including the 100-billion-parameter PanGu-alpha), integrating 41 optimized operators into the Ascend operator library, with computation-time speedups of about 1.08 to 2.70 times and end-to-end speedups of 1.07 to 2.15 times. Scope: it is specific to Ascend (though the serial-within, parallel-across principle generalizes), targets operator compute performance rather than distributed I/O and communication, guides skilled developers rather than automating optimization, and its evidence comes from one organization's models. The durable lesson is that a performance model is only as useful as its fidelity to how the hardware actually executes.

Check yourself

Check your understanding

1. Why is optimizing operators on Ascend harder than on a GPU?
2. Why does naively extending the GPU roofline model to Ascend fail?
3. What is a "component" in the component-based roofline model, and why does it matter?
4. What did the component-based roofline achieve, and what are its limits?

Sources

The paper is “Squeezing Operator Performance Potential for the Ascend Architecture” by Yuhang Zhou, Zhibin Wang, Guyue Liu, Shipeng Li, Xi Lin, Zibo Wang, Yongzhong Wang, Fuchun Wei, Jingyi Zhang, Zhiheng Hu, Yanlin Liu, Chunsheng Li, Ziyang Zhang, Yaoyuan Wang, Bin Zhou, Wanchun Dou, Guihai Chen, and Chen Tian, published at ASPLOS 2025, DOI 10.1145/3676641.3716243. The authors are at Nanjing University, Peking University, Shandong University, and Huawei Technologies.

This explainer is based on the full text of the paper, obtained from an author’s copy hosted on the Nanjing University group page of Chen Tian. The problem framing (Ascend as a representative domain-specific AI accelerator with distinct scalar, vector, and Cube compute units, customized buffers such as the L0 A/B/C and unified buffers, flexible cross-layer and asymmetric-bandwidth transfers via memory-transfer engines, and explicit instruction pipelining, all under manual programmer control; and why the mainstream DRAM and hierarchical roofline models plus their naive extension fail on Ascend), the three concrete failures of the naive roofline (around 180 precision-by-transfer combinations complicating visualization; misdiagnosis due to sequential transfers within one memory-transfer engine reporting a saturated engine as 67 and 33 percent underutilized; and misdiagnosis due to sequential mixed-precision operations within one compute unit), the component-based roofline design (the component abstraction of serial execution within a unit and parallel execution across units, each component a physical compute or transfer unit; distinguishing operators bound by a component from underutilized ones; and decomposing underutilization into component active-time ratio and execution efficiency, attributed to insufficient parallelism or an inefficient component), the optimization workflow and case studies across the cube, vector, and scalar units and the transfer engines, and the results (two years of practice optimizing eleven real models including the 100-billion-parameter PanGu-alpha, 41 operators integrated into the Ascend operator library, computation-time speedups of about 1.08 to 2.70 times and end-to-end speedups of 1.07 to 2.15 times, with the stated scope of operator compute performance excluding distributed input/output and communication) are all drawn directly from the paper. Background explanations of domain-specific accelerators, the roofline model, arithmetic intensity, and memory- versus compute-bound analysis are standard and provided to make the paper’s contribution understandable.