DPACS: Hardware Accelerated Dynamic Neural Network Pruning through Algorithm-Architecture Co-design

Dynamic pruning decides per input which parts of a neural network to skip, and papers routinely report cutting three quarters of the arithmetic with almost no accuracy loss — but almost nobody builds the hardware, and this paper (University of Hong Kong) does, finding that real speedup tracks the arithmetic reduction only until about 2.5 times and then flattens near 3 to 4 no matter how much more you prune, because once the compute goes away what remains is memory traffic that the algorithms were never counting.

The argument in one glance

Problem

Most of the network is irrelevant to most inputs

A trained network applies its full weight to every image it sees, but the parts that matter differ per image — the region containing the object, the channels that respond to its features. Exploiting that is attractive because it costs nothing at training time and adapts automatically: the network activates only the paths the current input actually needs. The literature on this is substantial, with many schemes reporting that three quarters of the arithmetic can be skipped for a fraction of a percent of accuracy.
What people did

Learn a mask that says what to skip, and count what you saved

The standard construction adds small decision layers to the network that emit a binary mask at run time — which pixels to compute, which channels to keep — trained end to end alongside the original weights. Success is then measured by multiply-accumulate reduction, the ratio of arithmetic operations the mask removed, plotted against the accuracy that survived. It is a clean metric, it is comparable across papers, and it is what almost every dynamic pruning result reports.
Why it fell short

The arithmetic was never the whole cost

Three things the count does not include. Deciding what to prune requires computing the mask, which is arithmetic that partly cancels the arithmetic saved. The resulting sparsity is decided at run time, so the compression and encoding schemes that make statically pruned networks fast cannot be applied, and irregular access patterns are expensive in hardware. And as the arithmetic shrinks, memory traffic stops being hidden behind it and becomes the limit — which is precisely what the metric cannot see, since it counts operations rather than bytes.
This paper's idea

Design the mask so that the hardware stays regular

This paper (University of Hong Kong — Gao, Zhang, Qi and So) builds the full stack and lets the hardware dictate the algorithm. The masks are shared: one spatial mask across every layer of a residual block, one channel mask across the adjacent channel dimensions of consecutive layers. That sharing is not a simplification but the load-bearing idea, because it makes the pruned computation regular again — a dense operation on fewer channels, driven by one set of indices. Measured end to end, that yields 1.1 to 3.9 times speedup, and reveals where the reported numbers stop meaning anything.

Why this problem is worth solving

Two kinds of pruning exist and they differ in what the hardware is allowed to know in advance, which turns out to matter more than the accuracy difference.

STATIC pruning removes weights offline, before deployment. Because the resulting pattern is fixed, the weights can be compressed and encoded ahead of time into formats built for sparse hardware, and an accelerator can be designed around exactly that layout. DYNAMIC pruning instead decides during inference, per input, which is strictly more powerful — it can skip spatial regions as well as weights, activating only the paths this particular image needs, and different inputs genuinely take different paths. But nothing can be prepared in advance. The mask is not known until the previous layer has run, so no offline encoding is possible, and any compression must happen on the fly at a cost that eats the savings. The decision units themselves are also new arithmetic that did not exist before, working against the arithmetic they eliminate

The evaluation habit of the field is what conceals the actual difficulty.

The standard metric is MULTIPLY-ACCUMULATE REDUCTION — how many arithmetic operations the mask removed, plotted against accuracy retained. It is a reasonable target while training an algorithm, because it is what the mask directly controls. As a predictor of speed it holds only while arithmetic is the bottleneck. Measured on real hardware here, speedup tracks the arithmetic reduction closely below about 2.5 times and then FLATTENS around 3 to 4 times however much more is pruned. The reason is that the prunable arithmetic was only ever part of the run time. Loading input features so the mask generator can decide what to prune is unavoidable traffic that pruning does not reduce, weights still have to arrive, and once the arithmetic that was hiding those transfers disappears they become the whole cost

What people did before, and where it broke

Prune weights offline and encode them for sparse hardware. Compressed formats let accelerators skip zeros efficiently, which works well because the pattern is fixed.
But it cannot adapt to the input. Every image pays the same cost regardless of how much of the network it actually needs.
Decide at run time with a learned mask and report the arithmetic saved. This is what most dynamic pruning work does, and the reductions are genuinely large.
But the arithmetic count is not the run time. Measured end to end, speedup detaches from the count above about 2.5 times and flattens near 3 to 4.
Prune at fine granularity for the best accuracy. Removing individual weights or channels wherever they are least useful preserves the most accuracy per operation removed.
But it scatters the surviving weights through memory. Fetching them becomes random access, and the accelerator spends its time on addresses rather than arithmetic.
Build a scheduler that steers the processing elements to the active locations. Prior hardware work adds a controller that dispatches work to wherever the mask says there is something to do.
But it leaves the irregularity in place, and every engine in the pipeline has to discover independently what to work on.
The gap. The algorithms were designed to maximise a count, and the hardware was asked afterwards to cope with whatever pattern that produced.

Letting the accelerator dictate what the pruning algorithm is allowed to do

Train a mask generator and report the arithmetic it removed

Add small decision layers that emit binary masks at run time, train them end to end with the network, and evaluate by multiply-accumulate reduction against accuracy. This is the established approach and the source of the algorithm, but its metric is what the rest of the work calls into question.
↓ →
Make the decision unit cheap by construction new

Generate the spatial mask with a pointwise 1x1 convolution and a threshold, chosen because at each pixel it is a single dot product of the feature vector with a weight vector, decidable on the spot. A larger kernel would need a line buffer and the area that comes with it.
↓ →
Share one spatial mask across every layer of a block new

Rather than letting each layer decide independently, use one mask for the whole residual block. That amortises the decision cost, and more importantly gives every compute engine in the pipeline the same set of active pixel indices, so input and output active locations are identical throughout.
↓ →
Share one channel mask across adjacent channel dimensions new

Apply the same channel mask to the output channels of one layer and the input channels of the next. Now no index matching between activations and weights is needed anywhere, and the computation collapses into an ordinary dense operation over a smaller number of channels.
↓ →
Group channels coarsely and reorder memory to match new

Let one mask bit switch a block of consecutive channels rather than a single one, and lay the weights out in memory by that grouping. Surviving weights then arrive in large burst transfers rather than scattered reads, and the mask generator own fully-connected layer shrinks by the same factor.
↓ →
Compute the next block channel mask before it is needed new

Place the channel mask generator at the end of the previous block rather than the start of the current one, feeding it the already spatially-pruned output. The mask is then known before the block runs, so weights for pruned input channels are never loaded at all.

Implemented on an FPGA at 187.5 megahertz with 8-bit arithmetic, evaluated on four ResNet variants and MobileNetV2 on ImageNet plus a custom network on CIFAR-10, across nine combinations of spatial and channel pruning ratios.

The idea

Share the mask and the sparsity stops being irregular. The move that makes the whole system work is deciding that one mask governs many layers, and it pays off twice in different ways. For spatial pruning, a single mask across all layers of a residual block means every compute engine in the pipeline is driven by the same set of active pixel indices, and the active input locations are identical to the active output locations — which is what lets the accelerator push an index once, reuse it later as an output index, and run a sliding-window convolution over scattered data at all. For channel pruning, the same mask governs the output channels of one layer and the input channels of the next, and the consequence is sharper still: with both sides masked identically, there is nothing to match up between the surviving activations and the surviving weights, so the computation becomes an ordinary dense operation over fewer channels. Sparse work has been turned back into dense work on a smaller tensor, which is the one thing a systolic-style accelerator is actually good at. Choosing a slightly less expressive algorithm because the regularity it buys is worth more to the hardware than the expressiveness was worth to the accuracy is the same make-the-sparsity-structured argument covered elsewhere on this site where an accelerator wins by insisting the workload arrive in a shape it can consume.

The number the field reports stops predicting the number that matters. The most useful measurement in the paper is a scatter plot of theoretical arithmetic reduction against measured speedup, and its shape is the whole argument. Below about 2.5 times the two track each other closely, which is why the metric earned its reputation. Above that the measured speedup flattens near 3 to 4 times no matter how much more arithmetic the algorithm claims to have removed. The explanation is that the prunable arithmetic was only ever a portion of the run time. Input features still have to be read so the mask generator can decide what to prune — traffic that pruning does not reduce and which the paper identifies as a bottleneck in its own right, to the point of adding a bypass path for it. Weights still have to arrive. And once the arithmetic that was hiding those transfers has gone, they are what is left. The authors are direct about the implication: a metric that is a fine training objective is not a performance claim, and only a full-stack implementation reveals the difference. Finding that a long-trusted proxy tracks reality precisely up to the point where the optimisation starts working, and then stops, is the same the-proxy-expires-exactly-when-you-need-it correction covered elsewhere on this site where a measure agrees with the truth in the regime nobody cared about.

The version with the better speedup is the slower machine. Two arrangements of the convolution hardware are built, and comparing them produces the paper’s most quietly instructive result. The serial arrangement loads each spatial position individually, so skipping a position removes both its arithmetic and its on-chip traffic, and it reaches up to 3.9 times. The parallel arrangement loads every position in a convolution window regardless of the mask, so skipping saves arithmetic but wastes the traffic and leaves processing elements idle, and it reaches only about 2.0 times. And yet the parallel arrangement has better absolute latency in most configurations, losing only at very high pruning ratios. Speedup is a ratio against your own unpruned baseline, so a design that wastes more when unpruned reports a larger improvement while remaining slower. The same reasoning explains a second asymmetry the authors draw out: the serial design benefits more from spatial pruning and the parallel one more from channel pruning, because the parallel design is more traffic-bound and channel pruning is what reduces weight traffic. Reporting a ratio alongside the absolute number it is a ratio of, so that a flattering baseline cannot be mistaken for a fast system, is the same mind-the-denominator discipline covered elsewhere on this site where a headline improvement is put next to the handicap it is recovering from.

Following one image through a pruned block

What happens to a residual block when the mask decides most of it is unnecessary

Measurements from an FPGA implementation at 187.5 megahertz with 8-bit arithmetic, on ResNet variants and MobileNetV2.

Step 1 of 5

Decide which pixels matter, cheaply

As the input feature map streams in, a pointwise convolution takes the dot product of each pixel feature vector with a learned weight vector and thresholds it. Above zero, the pixel and its coordinates are pushed into the pipeline; otherwise the buffered vector is discarded. The kernel is 1x1 specifically so this decision needs no line buffer and no lookahead.

this generator must read all the dense input in order to discard most of it, so it can itself become the bottleneck in traffic-bound layers
for that case there is a bypass, used when a mask from an earlier block at the same resolution is reused instead

Why it works

One spatial mask governs every layer in a block → active input positions equal active output positions throughout the pipeline → so an index can be pushed once and reused, which is what makes sliding-window convolution over scattered data possible.
The same channel mask covers adjacent output and input channel dimensions → no matching between surviving activations and surviving weights is needed → so the pruned computation is just a dense operation on fewer channels.
One mask bit switches a group of consecutive channels rather than one → weights can be laid out so survivors are contiguous → turning scattered fetches into large burst transfers and shrinking the mask generator itself.
The next block channel mask depends only on this block spatially-pruned output → it can be computed at the end of this block → so weights for pruned channels are never loaded rather than loaded and ignored.
The spatial decision is a pointwise dot product at a single position → it needs no line buffer and no lookahead → keeping the cost of deciding well below the cost of what the decision skips.

What it costs (trade-offs and scope)

Sharing masks costs accuracy. Reusing one mask across all blocks at a resolution saves input traffic but increases the penalty when the generator picks the wrong locations, which the authors state and measure.
The mask generator must read everything it discards. It streams the full dense input to decide what to keep, making it a bottleneck in traffic-bound layers and requiring a bypass path.
Speedup and latency disagree. The arrangement reporting up to 3.9 times is slower in absolute terms than the one reporting 2.0 in most configurations, which only shows up because both are reported.
Coverage is partial by design. Pruning is applied only to residual blocks, and channel pruning only to the later stages, on the grounds that those have the most redundancy.
One FPGA at 187.5 megahertz. A competing design on a custom chip is faster in absolute throughput on one network, which the authors concede while noting their approach covers more models.

The evaluation

The design space is swept rather than sampled: four ResNet variants plus MobileNetV2 on ImageNet and a custom network on CIFAR-10, nine combinations of spatial and channel pruning ratio, two spatial mask sharing schemes, and two arrangements of the convolution hardware. The baseline is the same dataflow accelerator with the pruning hardware removed, which is the comparison that makes speedup meaningful. Alongside that sits the comparison the paper is really making — theoretical arithmetic reduction against measured speedup — and a survey of prior dynamic pruning work noting which results come from real hardware and which from simulation.

Property	Static pruning with offline encoding	Dynamic pruning evaluated by operation count	Dynamic pruning co-designed with hardware
Adapts to the input
Weight layout
Matching activations to weights
What is reported
Behaviour as pruning increases

End-to-end speedup ranges from 1.1 to 3.9 times over the same accelerator without pruning. ResNet-101 at a modest pruning setting retains 76.8 percent top-1 accuracy, half a point below baseline, with 1.78 times fewer operations and 1.6 times faster inference at 59.6 milliseconds; pushed harder it reaches 2.63 times speedup at 73.94 percent. Measured speedup tracks arithmetic reduction below about 2.5 times and flattens near 3 to 4 above it.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The two hardware arrangements should be read together rather than separately. The serial convolution design reports up to 3.9 times speedup and the parallel one about 2.0, yet the parallel design is faster in absolute latency in most configurations, losing only at very high pruning ratios — because speedup is a ratio against your own unpruned baseline, and a design that wastes more when unpruned flatters itself when pruned. That the paper reports both is to its credit and is exactly what lets a reader see it. Several design choices also trade accuracy for traffic rather than being free: sharing one spatial mask across every block at a resolution saves loading input features but increases the cost of a generator picking the wrong locations, and the authors measure the degradation rather than asserting it away. Coverage is deliberately partial, with pruning applied only to residual blocks and channel pruning only to later stages where redundancy is higher, so the reported ratios are over a subset of each network. And the comparison against prior work is more careful than most: one competing design on a custom chip is conceded to have better absolute throughput on one network, and another comparison is flagged as resting on simulation results whose real-world reproducibility is unclear. Everything here is one FPGA at 187.5 megahertz with 8-bit arithmetic, so the absolute latencies are of that platform rather than of the idea.

Background

When the metric everyone reports stops tracking the thing everyone wants

This paper sits in the site’s machine learning systems and accelerators clusters and rhymes with several ideas covered elsewhere on this site. Its central finding — that a long-trusted proxy tracks reality closely right up to the point where the optimisation starts working, then detaches — is the same correction the performance-analysis and warehouse-scale allocator pages make when a familiar measure turns out to describe a regime nobody is in any more, and it produces the same conclusion that only a full-stack implementation settles the question. Its resolution, deliberately choosing a coarser and less expressive form of sparsity because the regularity it buys is worth more than the expressiveness it costs, is the same structured-sparsity bargain the sparse accelerator and matrix multiplication pages strike, and the same instinct the address mapping page follows when it picks a parameter so that the hardware primitive does the work. Its observation that the system becomes bound by memory traffic exactly as the arithmetic is optimised away echoes the pages where removing one bottleneck simply promotes the next one. And its care in reporting a ratio next to the absolute number it is a ratio of belongs with the pages that put a headline improvement beside the handicap it is recovering.

If you remember only three things

1. Dynamic pruning decides per input which spatial locations and which channels a network actually needs, which is strictly more powerful than removing weights offline because different images genuinely take different paths. The literature reports large reductions in arithmetic at small accuracy cost, but evaluates by counting operations removed — and that count omits three things. Computing the mask is itself arithmetic. The sparsity is decided at run time, so the compression and encoding that make statically pruned networks fast cannot be applied. And as arithmetic shrinks, the memory traffic it was hiding becomes the limit.
2. Building the full stack makes the gap measurable, and the shape is the argument: speedup tracks the arithmetic reduction closely below about 2.5 times and then flattens near 3 to 4 however much more the algorithm claims to remove, because loading input features so the mask generator can decide what to prune is traffic that pruning does not reduce. The fix is to let the hardware dictate the algorithm through mask sharing. One spatial mask across a whole residual block means every engine in the pipeline sees the same active positions, so an index pushed on the way in can be reused on the way out. One channel mask across adjacent output and input dimensions means nothing has to be matched between surviving activations and surviving weights, and the pruned computation becomes an ordinary dense operation on fewer channels.
3. Two further moves attack traffic directly: one mask bit switches a group of consecutive channels so surviving weights arrive in bursts rather than scattered reads, and the next block channel mask is computed at the end of the current block from its already spatially-pruned output, so pruned weights are never fetched at all. End to end this yields 1.1 to 3.9 times over the same accelerator without pruning, with ResNet-101 keeping 76.8 percent accuracy at 1.6 times faster. The most instructive result is a comparison the paper did not have to publish: the hardware arrangement reporting 3.9 times is slower in absolute latency than the one reporting 2.0, because speedup is measured against your own baseline.

Check yourself

Check your understanding

1. Dynamic pruning papers report large reductions in multiply-accumulate operations, yet measured speedup flattens near 3 to 4 times however much more is pruned. Predict why.
2. The design shares one channel mask across the output channels of one layer and the input channels of the next. Predict what that buys.
3. The channel mask generator is placed at the end of the previous block rather than the start of the current one, even though the two are functionally identical. Predict why.
4. Two convolution hardware arrangements are built. One reports up to 3.9 times speedup, the other about 2.0. Predict what the comparison actually shows.

Sources

Yizhao Gao, Baoheng Zhang, Xiaojuan Qi and Hayden Kwok-Hay So. “DPACS: Hardware Accelerated Dynamic Neural Network Pruning through Algorithm-Architecture Co-design.” ASPLOS 2023, Volume 2. The University of Hong Kong. DOI 10.1145/3575693.3575728.
Full text of the paper was used for this explainer, including the static and dynamic pruning background and the survey of sparse accelerators in Section 2, the dynamic pruning block with its spatial and channel mask designs, coarse-grained channel grouping, mask precomputation and training method in Section 3, the sparse dataflow architecture with its elastic line buffer and processing element variants in Section 4, and the hardware speedup results, the comparison of operation count against measured speedup, the resolution mask study and the comparison with prior work in Section 5.
The authors released their implementation publicly.