Optimizing Deep Learning Inference Efficiency through Block Dependency Analysis

Machine learning compilers speed up neural-network inference by optimizing across operators, but to do that safely they must know which computations depend on which — and they analyze those dependencies at the fine grain of individual elements and whole operators, which makes the dependency web tangled enough to block many cross-operator optimizations and hides chances to run independent work in parallel, leaving the GPU underused; this paper (Chinese Academy of Sciences and collaborators — Zhanyuan Di, Leping Wang, En Shao, Zhaojia Ma, Ziyi Ren, Feng Hua, Lixian Ma, Jie Zhao, Guangming Tan, Ninghui Sun) presents BlockDepend, a compiler that instead analyzes dependencies at the granularity of blocks by inspecting the lower-level phases of compilation, untangling the relationships between operators and exposing parallelism that element- and operator-level analysis missed, enabling memory and GPU-utilization optimizations that run 1.71 times faster than NVIDIA TensorRT and 2.88 times faster than AMD MIGraphX.

The argument in one glance

Problem

Optimizing across operators needs to know the dependencies, and the usual view is too fine

A machine learning compiler turns a neural network into fast code, and much of its speedup comes from inter-operator optimization: rearranging, fusing, or overlapping the network's operators (the matrix multiplies, convolutions, activations, and so on) so the hardware does less redundant work and stays busy. To do any of this safely, the compiler must know the data dependencies — which computations must finish before which others can start, because they consume their results. Get the dependencies wrong and you break correctness; see them as more tangled than they really are and you leave speed on the table. So the quality of inter-operator optimization rests directly on how the compiler analyzes dependencies, and the level at which it looks turns out to matter a great deal.
What people did

Analyze data dependencies at the element and operator levels

Traditional machine learning compilers perform static data-dependency analysis at two granularities: the element level, tracking dependencies between individual tensor elements, and the operator level, treating each operator as a unit and asking which operators feed which. This is the natural way to reason about a computation graph, and it is what mainstream compilers build their inter-operator passes on.
Why it fell short

Element- and operator-level analysis is both too tangled and too coarse

The paper argues these two levels each fail in opposite ways. Element-level dependencies are so numerous and intricate that the relationships between operators come out complex and tangled, and that complexity hinders the compiler from applying efficient inter-operator optimizations — it cannot see a clean enough picture to act on. Operator-level analysis, meanwhile, is too coarse: by treating a whole operator as one indivisible node, it overlooks computations inside and across operators that are actually independent and could run in parallel, so those parallelization opportunities are missed and the GPU's resources sit underused. Neither granularity gives the compiler what it needs, which is a view that is simple enough to optimize against yet fine enough to reveal hidden parallel work.
This paper's idea

Analyze dependencies at the granularity of blocks, from the lower compilation phases

BlockDepend is a machine learning compiler framework built around block-level dependency analysis — a middle granularity between individual elements and whole operators. Instead of reasoning only over the high-level computation graph, it examines the lower-level phases of compilation, where operators have been broken down into blocks of computation, and extracts dependency information at that block level. Seeing dependencies as relationships between blocks does two things at once: it simplifies the tangled web that element-level analysis produces, giving the compiler a clean enough picture to apply inter-operator optimizations, and it uncovers hidden parallelization opportunities that operator-level analysis hid inside its coarse nodes. With that block-level view, BlockDepend applies targeted optimization strategies that improve memory-access efficiency and raise GPU utilization. Across various workloads it reports speedups of 1.71 times over NVIDIA TensorRT and 2.88 times over AMD MIGraphX.

Why this problem is worth solving

Machine learning compilers are what stand between a trained model and fast execution, and their leverage comes almost entirely from how well they understand the computation.

A neural network, as written in a framework, is a graph of operators — matrix multiplications, convolutions, normalizations, activations — each consuming tensors and producing tensors. Running that graph naively, one operator at a time with each writing its full output to memory before the next reads it, wastes enormous effort: the hardware stalls waiting on memory, and much of the work could be combined or overlapped. A machine learning compiler exists to avoid that waste. It rewrites the graph — fusing adjacent operators so intermediate results never touch slow memory, reordering work to keep the processor busy, and mapping computation onto the accelerator’s many parallel units. Every one of these transformations is only legal if it respects the data dependencies: a fused or reordered computation must still produce each value before anything that consumes it. So dependency analysis is not a side detail; it is the foundation the whole optimization edifice stands on. If the analysis is imprecise or presents the dependencies in an unwieldy form, the compiler must be conservative, and conservative means slow — it forgoes fusions it cannot prove safe and parallelism it cannot see. That is why the granularity at which a compiler analyzes dependencies quietly determines how much performance it can ever extract, and why revisiting that granularity is worth a paper.

The specific stakes here are memory traffic and GPU occupancy, the two things that decide inference speed on modern accelerators.

On a modern GPU, a neural network’s inference speed is rarely limited by raw arithmetic; it is limited by two things. The first is memory access: moving tensors between the GPU’s compute units and its memory is slow relative to computing on them, so any optimization that keeps intermediate data in fast on-chip storage instead of writing it out and reading it back — which is what operator fusion achieves — directly cuts the dominant cost. The second is utilization: a GPU has thousands of parallel lanes, and if the work handed to it does not expose enough independent computation to fill them, most of the chip sits idle even while the model is nominally running. Both of these are governed by how the compiler schedules and combines operators, and both depend on the dependency analysis. Fusion needs the analysis to be clean enough to prove that two operators can be merged; filling the parallel lanes needs the analysis to be fine enough to reveal computations that are independent and can run at the same time. When the analysis is too tangled, fusions are missed and memory traffic stays high; when it is too coarse, parallelism is missed and the GPU stays underutilized. Improving the analysis therefore attacks exactly the two bottlenecks that matter, which is why a better dependency granularity can translate into large end-to-end speedups.

What people did before, and where it broke

Element-level dependency analysis. Track dependencies between individual tensor elements to know precisely what feeds what. Where it broke. The relationships are so numerous and intricate that the resulting picture of how operators relate is tangled, and that complexity hinders the compiler from applying efficient inter-operator optimizations.
Operator-level dependency analysis. Treat each operator as a single unit and reason about dependencies between whole operators. Where it broke. Treating an operator as indivisible is too coarse, so computations that are actually independent — inside or across operators — are overlooked, and those parallelization opportunities go unused, leaving GPU resources underutilized.
Aggressive operator fusion without a better dependency view. Push harder on fusing operators to cut memory traffic. Where it broke. Fusion is limited by what the dependency analysis can prove safe and simple; on a tangled element-level picture or a coarse operator-level one, the compiler cannot see enough clean, independent structure to fuse and parallelize as much as the hardware would allow.
Rely on vendor inference libraries. Use highly tuned engines such as NVIDIA TensorRT or AMD MIGraphX that apply their own graph optimizations. Where it broke. These are strong baselines, but they still rest on element- and operator-level dependency reasoning, so they inherit the same blind spots that a different analysis granularity could remove.
The gap. No compiler analyzed dependencies at an intermediate, block-level granularity drawn from the lower phases of compilation — one that is simultaneously simple enough to enable inter-operator optimization and fine enough to expose the parallelism that operator-level analysis hides.

How BlockDepend works

Analyzing dependencies at the block level to drive inter-operator optimization

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Pick a granularity of analysis that is neither too fine nor too coarse. The organizing insight of BlockDepend is that the level at which a compiler analyzes data dependencies is itself a design choice with a sweet spot. Element-level analysis is maximally precise but produces a dependency web so intricate that it obscures the structure the optimizer needs; operator-level analysis is simple but so coarse that it hides real independence inside its nodes. Blocks — the units operators are broken into during the lower phases of compilation — are the intermediate granularity that captures the useful structure of both without either’s failing. Choosing the right intermediate representation at which to reason, so that the compiler sees a computation at exactly the level that exposes its optimization opportunities, is the same principle the work on an intermediate representation for optimized tensor computations on GPUs covered elsewhere on this site is built on, meeting the hardware where a well-chosen abstraction makes the transformations tractable. BlockDepend applies that principle specifically to dependency analysis, arguing the block level is where dependencies become both simple and revealing.

Simpler dependencies unlock more inter-operator fusion. The first payoff of the block-level view is that it untangles the relationships between operators. When dependencies are expressed between blocks rather than between countless individual elements, the compiler can actually see which operators are safe to fuse, reorder, or overlap, and fusion is what keeps intermediate tensors in fast memory instead of paying to write them out and read them back — the dominant cost of inference. Combining adjacent operators so their intermediate results never spill to slow memory, driven by an analysis clean enough to prove the combination safe, is the same memory-motivated fusion discipline the work on in-near-memory operator fusion covered elsewhere on this site pursues, collapsing a chain of operations to cut the traffic between compute and memory. BlockDepend’s contribution is that the block-level dependency picture is what makes more of that fusion visible and legal in the first place.

Finer structure exposes the parallelism that fills the GPU. The second payoff is that blocks reveal independent work the operator-level view concealed. A modern GPU has thousands of parallel lanes, and if the compiler cannot see enough independent computation, most of them idle; by breaking operators into blocks and analyzing their dependencies, BlockDepend surfaces computations that can run at the same time and schedules them to raise utilization. Recovering a finer-grained structure of a computation in order to expose parallelism that a coarse view had hidden, and then dispatching that parallel work to keep the hardware busy, is the same move the work on recovering program structure to accelerate task-parallel workloads covered elsewhere on this site is built on, reconstructing the independence the surface form obscured. BlockDepend does it for neural-network operators, turning the recovered block-level independence into higher GPU occupancy and, with it, its reported speedups over vendor inference engines.

Following a model through BlockDepend

From a computation graph to block-level-optimized inference code

An illustrative trace, based on the paper's abstract, of how BlockDepend uses block-level dependency analysis. Internal details beyond the abstract were not available for this draft.

Step 1 of 5

Start from the neural network's operators

The compiler receives the model as a graph of operators connected by tensors. Inter-operator optimization — fusing, reordering, and parallelizing across these operators — is where the speedups live, but every such transformation must respect the data dependencies among the computations.

The abstract frames the whole problem as inter-operator optimization gated by dependency analysis.

Why it works

Block granularity beats element granularity on clarity. Element-level dependencies are so numerous they tangle the operator relationships → but grouping computation into blocks collapses that web into a simple picture → so the compiler can finally see which inter-operator optimizations are safe to apply.
Block granularity beats operator granularity on parallelism. Treating an operator as one indivisible node hides the independent work inside it → but analyzing blocks exposes those independent computations → so parallelism the coarse view missed becomes available to fill the GPU.
Cleaner dependencies mean more fusion, less memory traffic. Memory movement dominates inference cost and fusion needs a provably safe dependency view → but the block-level picture is clean enough to justify more fusions → so more intermediates stay in fast memory and traffic drops.
Exposed parallelism means higher GPU utilization. A GPU idles when the compiler cannot see enough independent work → but block-level analysis surfaces concurrent computations to dispatch → so more of the chip’s parallel lanes stay busy during inference.
Working from the lower compilation phases is what makes blocks visible. The high-level graph only shows whole operators → but the lower-level phases have already broken operators into blocks → so analyzing there is what gives BlockDepend the granularity its optimizations need.

What it costs (trade-offs and scope)

This is a draft built from the paper’s abstract. The full paper was behind a publisher access wall for this draft (no arXiv version, no open-access or author copy located, and the ACM Digital Library returned a bot-verification challenge), so this account is assembled from the paper’s complete abstract plus standard machine-learning-compiler background. The exact definition of a block, which lower-level compilation phases are inspected, how the block-level dependencies are computed and represented, the specific optimization strategies, and the evaluation setup are not reproduced here and should be read from the paper.
The reported speedups are the abstract’s own. The 1.71 times over NVIDIA TensorRT and 2.88 times over AMD MIGraphX are the abstract’s results across unspecified various workloads; the models, input sizes, hardware, and precise measurement methodology behind those numbers are not detailed in this draft.
Block-level analysis adds compiler-side complexity. Descending into lower compilation phases and extracting block-level dependencies is more analysis machinery than an operator-level pass, so the approach trades additional compile-time reasoning for run-time speed; whether that trade is always worthwhile is not addressed by the abstract.
The benefit depends on there being structure to expose. The gains rest on operators containing hidden independent work and on operator relationships being simplifiable at the block level; workloads whose operators are already simple or already well fused would offer the approach less to find.
It targets inference on GPUs. BlockDepend is presented for deep-learning inference and evaluated against GPU inference engines; training, other accelerators, or workloads outside this setting are beyond the abstract’s stated scope, and how the idea generalizes is not discussed there.

The evaluation

Analyzing dependencies to optimize DNN inference

Property	Element-level analysis	Operator-level analysis	BlockDepend (block-level)
Dependency picture
Inter-operator optimization
Parallelism found
GPU utilization
Reported outcome (per abstract)

A qualitative comparison based on the paper's abstract; the paper's main body, including the exact definition of a block, the lower-level phases inspected, the optimization strategies, and the evaluation setup, was not available for this draft. The 1.71-times and 2.88-times figures are the abstract's reported results.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This is an honest draft assembled from the paper’s complete abstract plus standard background on machine learning compilers and GPU inference; the full text was not accessible (no arXiv preprint, no open-access or author copy found, and the ACM Digital Library version returned a bot-verification challenge), so the internal design is not reproduced. What is solid and comes directly from the abstract: inter-operator optimization in DNNs relies on accurate data-dependency analysis; traditional machine learning compilers perform static dependency analysis at the element and operator levels, which leads to two limitations — complex dependencies that hinder efficient inter-operator optimizations, and overlooked parallelizable computations that underutilize GPU resources; BlockDepend is a machine learning compiler framework that addresses these through block-level dependency analysis; by examining the lower-level phases of compilation it extracts block-level dependency information, simplifying complex relationships between operators and uncovering hidden parallelization opportunities; this enables targeted optimization strategies that enhance memory-access efficiency and improve GPU utilization; and it achieves speedups of 1.71 times over NVIDIA TensorRT and 2.88 times over AMD MIGraphX across various workloads. The explanations of why element-level analysis is tangled, why operator-level analysis is too coarse, and why memory efficiency and GPU utilization are the payoff are standard compiler background and the natural reading of the abstract, not claims about BlockDepend’s specific internals. Read the exact block definition, the compilation phases inspected, the optimization strategies, and the evaluation in the paper itself; treat this page as a faithful account at the level of the abstract, not of the implementation.

Background

Why the granularity of analysis keeps mattering

It helps to see BlockDepend as one more instance of a theme that runs throughout compilers and program analysis: the level at which you analyze a program decides what you can optimize. Look too finely and you drown in detail that obscures the useful structure; look too coarsely and you smear away the very independence and locality you were hoping to exploit. This tension recurs because there is rarely a single natural granularity — a computation can be viewed as elements, as blocks, as operators, as whole subgraphs, and each view reveals some opportunities while hiding others. The art is choosing, for the transformation you want to perform, the granularity at which that transformation becomes both legal to prove and profitable to apply. Deep-learning compilers feel this acutely because their computations are enormous and regular, so a granularity that captures the regularity without exploding into detail can expose large, uniform chunks of parallel and fusible work. The broader practice of building a framework that operates over deep neural networks at a chosen, uniform level of abstraction so that analysis and transformation apply cleanly across many operators is the same design instinct the work on a unified instrumentation framework for deep neural networks covered elsewhere on this site follows, standardizing the level at which the network is examined. BlockDepend argues that for dependency analysis in inference, the block level is that right level.

If you remember only three things

1. Machine learning compilers make neural-network inference fast largely through inter-operator optimization — fusing, reordering, and parallelizing across operators — and every such transformation is gated by data-dependency analysis. Traditional compilers analyze dependencies at the element level (which produces a tangled web that hinders inter-operator optimization) and the operator level (which is so coarse it overlooks independent computations and leaves the GPU underutilized). Neither granularity gives the compiler a view that is both simple and revealing.
2. BlockDepend introduces block-level dependency analysis: by examining the lower-level phases of compilation, where operators are broken into blocks, it extracts dependency information at an intermediate granularity. This simplifies the complex relationships between operators, enabling more inter-operator optimization, and simultaneously uncovers hidden parallelization opportunities that operator-level analysis missed, enabling targeted strategies that improve memory-access efficiency and GPU utilization.
3. Across various workloads BlockDepend reports 1.71 times faster inference than NVIDIA TensorRT and 2.88 times faster than AMD MIGraphX. This explainer is a draft written from the abstract without the paper body, so the exact definition of a block, the lower-level phases inspected, the optimization strategies, the hardware, and the evaluation setup should be confirmed against the paper. The approach adds compiler-side analysis complexity, depends on there being structure to expose, and targets GPU inference specifically.

Check yourself

Check your understanding

1. Why does the granularity of dependency analysis limit a machine learning compiler?
2. What is BlockDepend's core idea?
3. How does block-level analysis improve both memory efficiency and GPU utilization?
4. What should a reader keep in mind about this explainer, and what did the paper report?

Sources

The paper is “Optimizing Deep Learning Inference Efficiency through Block Dependency Analysis” by Zhanyuan Di, Leping Wang, En Shao, Zhaojia Ma, Ziyi Ren, Feng Hua, Lixian Ma, Jie Zhao, Guangming Tan, and Ninghui Sun (Institute of Computing Technology, Chinese Academy of Sciences, and collaborators), published at ASPLOS 2025 (Volume 2), DOI 10.1145/3676641.3716264. The system is named BlockDepend. It is licensed CC BY.

This explainer is a draft. The paper’s main text was not accessible at the time of writing (there is no arXiv preprint, no open-access or author copy was located, and the ACM Digital Library PDF returned a bot-verification challenge), so it is written from the paper’s complete abstract together with standard background on machine learning compilers, operator fusion, and GPU inference. The abstract supplies the problem and approach: inter-operator optimization in DNNs relies on accurate data-dependency analysis; traditional machine learning compilers perform static data-dependency analysis at the element and operator levels, leading to two key limitations — complex dependencies that hinder efficient inter-operator optimizations, and overlooked parallelizable computations that underutilize GPU resources; BlockDepend is a machine learning compiler framework that addresses these through block-level dependency analysis; by examining the lower-level phases of compilation it extracts crucial block-level dependency information, simplifying complex relationships between operators and uncovering hidden parallelization opportunities; this allows targeted optimization strategies that enhance memory-access efficiency and improve GPU utilization; and experiments demonstrate speedups of 1.71 times over NVIDIA TensorRT and 2.88 times over AMD MIGraphX across various workloads. The explanations of why element-level analysis is tangled, why operator-level analysis is too coarse, and why memory efficiency and GPU utilization are the payoff are standard compiler background and the natural reading of the abstract, not claims about BlockDepend’s specific internal design. The paper’s exact block definition, the compilation phases it inspects, its dependency representation, its optimization strategies, its hardware configuration, and its evaluation setup are not reproduced here and should be read from the paper. No numbers beyond the abstract’s have been introduced.