BitRed: Taming Non-Uniform Bit-Level Sparsity with a Programmable RISC-V ISA for DNN Acceleration

Yanhuan Liu, Wenming Li, Kunming Zhang, Yuqun Liu, Siao Wen, Lexin Wang and 6 others

When a neural network multiplies numbers, many of the individual bits are zero, and an accelerator can skip the zero bits to save work — but different numbers have wildly different counts of nonzero bits, so spreading them across parallel hardware lanes leaves some lanes swamped while others idle; this paper (Institute of Computing Technology, Chinese Academy of Sciences) argues the fix is not another fixed circuit but a programmable one, so it builds BitRed, whose processing unit exposes the acceleration as a few custom RISC-V instructions that let software balance the bit-level work dynamically, reportedly running up to 9.4 times faster than the prior best bit-level accelerator.

The argument in one glance

Problem

Skipping the zero bits saves work, but the nonzero bits are spread unevenly

A neural network spends most of its time multiplying numbers, and each number, written in binary, is often mostly zeros. Because a multiplication can be decomposed into a sum of shifted partial products, one for each nonzero bit of an operand, an accelerator can skip the zero bits entirely and do work only for the nonzero ones. This bit-level sparsity is a real opportunity: the fewer nonzero bits, the less work. The catch is that bit-level sparsity is non-uniform and dynamic. Different values carry very different numbers of nonzero bits — one value might have a single one, the next might have six — and which bits are set depends on the data, changing from input to input. So when an accelerator spreads many values across its parallel processing lanes, some lanes get swamped with nonzero-bit work while others sit nearly idle, and the whole array runs at the speed of its busiest lane. This load imbalance is the central obstacle to actually cashing in bit-level sparsity on parallel hardware.
What people did

Use the bit-interleaving paradigm, as in state-of-the-art accelerators

The leading response, embodied by accelerators such as Bitlet, is the bit-interleaving paradigm: rather than processing each number's bits in isolation, it distills and interleaves the nonzero bits across many numbers so they can be fed through the hardware together, evening out the parallelism and exploiting the sparsity more effectively than a naive bit-serial design.
Why it fell short

A rigid datapath cannot follow the shifting, uneven distribution of work

The abstract's critique is that bit-interleaving, though promising, is fundamentally constrained by being a fixed hardware design. It is built as a monolithic datapath, wired one way, and it still suffers severe inter-channel load imbalance: the work handed to different channels remains uneven, and a rigid circuit has no way to redistribute it as the distribution shifts from layer to layer, model to model, and input to input. Because the sparsity pattern is dynamic, a datapath optimized for one distribution of nonzero bits is wrong for another, and there is no knob to turn. The hardware is committed at design time to one strategy for balancing and reducing the bit-level work, so when the workload does not match that strategy, channels stall and the promised speedup erodes. What is needed is a way to make the balancing decision adaptable at runtime rather than frozen into the silicon.
This paper's idea

Turn the rigid hardware into programmable RISC-V instructions that schedule the work

BitRed embodies what the abstract calls a "programmable adaptive bit-interleaving" philosophy. Instead of a monolithic circuit, its core is an Adaptive-Sparse Processing Unit that deconstructs the acceleration into a small set of orthogonal RISC-V instruction-set extensions: one for pre-processing (cal.pre), one for adaptive distillation with dynamic load balancing (cal.adis), and one for an efficiency-optimal reduction (cal.red). By exposing these as programmable instructions, BitRed transforms a rigid hardware problem into a flexible scheduling problem — software can now arrange and balance the bit-level work to match whatever the current distribution demands, rather than relying on a fixed datapath. The abstract argues this makes the design fundamentally more adaptable and extensible, and reports large gains: up to 9.4 times the speed of Bitlet and competitiveness with high-end GPUs.

Why this problem is worth solving

Bit-level sparsity is a genuine source of efficiency, but only if the uneven work it produces can be spread across parallel hardware.

Inside a neural network, weights and activations are numbers stored in binary, and a multiply-accumulate can be built up as a sum of shifted copies of one operand, one copy for each bit of the other operand that is set to one. Any bit that is zero contributes a zero partial product and can simply be skipped, so an accelerator that works at the level of individual bits does less work when the operands have fewer nonzero bits — this is bit-level sparsity, and it is abundant because typical values have many zero bits. The difficulty, which the abstract names as central, is that this sparsity is non-uniform and dynamic. The count of nonzero bits varies sharply from one value to the next, and it depends on the actual data flowing through the network, so it shifts across layers, models, and inputs. A parallel accelerator has many lanes working at once, and if it assigns values to lanes without accounting for their differing nonzero-bit counts, the lanes finish at wildly different times: the array can only advance as fast as its most heavily loaded lane, so the idle lanes represent wasted throughput. Turning bit-level sparsity from a theoretical saving into real speedup therefore hinges on balancing this uneven, data-dependent work across the hardware.

The best existing answer improves the balance but bakes its strategy into fixed silicon.

The state-of-the-art approach, the bit-interleaving paradigm exemplified by accelerators like Bitlet, attacks the imbalance by not processing each number’s bits separately. Instead it distills the nonzero bits out of many numbers and interleaves them, packing bit-level work together so the hardware lanes are kept busier and the parallelism is more fully exploited than in a straightforward bit-serial machine. This is a real advance over processing values one at a time. But the abstract identifies a fundamental limit: it is a rigid datapath. The interleaving and reduction logic is a fixed circuit designed around one way of balancing and combining the bit-level work, and it still leaves severe inter-channel load imbalance because the actual distribution of nonzero bits does not match the fixed strategy across the range of real workloads. Since the sparsity is dynamic, no single hardwired balancing scheme is right for every layer and input, and a monolithic design offers no way to change course at runtime. The opportunity the paper pursues is to keep the benefit of interleaving while making the balancing decision programmable, so it can adapt to the workload instead of the workload having to match the hardware.

What people did before, and where it broke

Naive bit-serial acceleration. Process each value bit by bit, skipping its zero bits. Where it broke. With each value handled in isolation, the differing nonzero-bit counts leave lanes badly imbalanced, so parallel throughput is throttled by the busiest lane.
The bit-interleaving paradigm (for example Bitlet). Distill and interleave nonzero bits across many values to keep lanes busier. Where it broke. It is a rigid, monolithic datapath, and it still suffers severe inter-channel load imbalance because a fixed circuit cannot match the dynamic, non-uniform distribution of nonzero bits.
Dense accelerators and GPUs. Ignore bit-level sparsity and process full-width numbers on general-purpose parallel hardware. Where it broke. They leave the bit-level savings entirely unclaimed, doing work for the many zero bits, which the abstract’s comparison targets.
The gap. No design kept the benefit of bit-interleaving while making the load-balancing and reduction strategy adaptable at runtime rather than frozen into a fixed datapath. Recasting the acceleration as programmable instructions that schedule the bit-level work is what BitRed is described as contributing.

How BitRed works

Recasting bit-level acceleration as a set of programmable instructions

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Keep the interleaving, but make the balancing programmable. The reframing at the heart of BitRed, as the abstract presents it, is that the bit-interleaving paradigm’s weakness is not the interleaving but the rigidity: a fixed datapath commits to one balancing-and-reduction strategy that cannot track a dynamic, non-uniform sparsity pattern. So BitRed keeps the goal of distilling and interleaving nonzero bits, but implements it as an Adaptive-Sparse Processing Unit whose behavior is directed by software rather than frozen in wiring, exposing the acceleration through composable instructions. Exploiting sparsity while refusing to lock into a single fixed dataflow, so the hardware can bend to the shape of the data instead of forcing the data to fit the hardware, is the same flexibility-first stance the work on a multi-dataflow sparse matrix multiplication accelerator covered elsewhere on this site takes, treating adaptability across sparsity patterns as the thing worth designing for.

Deconstruct the accelerator into orthogonal ISA extensions, and let scheduling do the rest. The technical move the abstract describes is to break the monolithic accelerator into a small set of orthogonal RISC-V instruction-set extensions — pre-processing, adaptive distillation with dynamic load balancing, and efficiency-optimal reduction — that compose. This turns what had been a rigid hardware problem into a flexible scheduling problem: the order and arrangement of these instructions can be chosen to balance the bit-level work for whatever distribution is present, and new precisions or patterns can be handled by scheduling rather than a hardware redesign, which is why the abstract calls the design extensible. Specializing a processor with custom instruction-set extensions so that a demanding, irregular computation becomes a matter of issuing the right instructions rather than building a bespoke fixed unit is the same programmable-specialization philosophy the work on a stream instruction-set architecture and processor specialization for sparse computation covered elsewhere on this site is built on, trading a monolithic accelerator for composable, programmable primitives.

Push bit-level efficiency far enough to rival dense GPUs. The payoff the abstract reports is that operating at the granularity of individual bits, and balancing that work adaptively, extracts enough savings to be competitive not just with prior bit-level accelerators but with high-end general-purpose GPUs, across precisions from floating-point down to eight-bit, within a few square millimeters of area and a few hundred milliwatts of power. Wringing performance out of arithmetic by working at the level of bits and narrow numeric representations, rather than always paying for full-width operations, is the same low-level efficiency instinct the work on a lightweight accelerator for narrow-integer arithmetic covered elsewhere on this site pursues, recognizing that a great deal of the work in these numbers is spent on bits that need not be computed at all.

Following a computation through BitRed

Balancing the bit-level work of a layer

An illustrative trace, at the level the abstract describes, of how BitRed's programmable instructions handle non-uniform bit-level sparsity. Specific internal details beyond the abstract should be confirmed against the full paper.

Step 1 of 5

A layer's values have uneven nonzero-bit counts

The operands of a layer, in binary, carry very different numbers of nonzero bits, and which bits are set depends on the input, so the bit-level work is non-uniform and would imbalance a naive parallel array.

A fixed bit-interleaving datapath would still leave channels unevenly loaded.

Why it works

Skipping zero bits does less work when values are sparse. A multiply is a sum of partial products, one per set bit → but zero bits contribute nothing and can be skipped → so working at the bit level does proportionally less work as the operands’ nonzero-bit counts fall.
The imbalance is the real barrier, and it is dynamic. Different values have very different nonzero-bit counts that depend on the data → but a parallel array runs at the speed of its busiest lane → so unless the uneven work is balanced at runtime, most of the sparsity saving is lost to stalled lanes.
A fixed datapath cannot match a shifting distribution. A monolithic bit-interleaving circuit commits to one balancing strategy → but the sparsity pattern changes across layers, models, and inputs → so a rigid design is mismatched much of the time, which is why it still suffers severe inter-channel imbalance.
Programmable instructions turn balancing into scheduling. Exposing pre-processing, adaptive distillation, and reduction as orthogonal instructions makes their arrangement a software choice → but software can re-arrange them per workload → so the balancing adapts to the current distribution instead of the workload having to fit the hardware.
Orthogonality makes the design extensible. Because the three instructions are independent and composable → but new precisions or sparsity patterns can be handled by scheduling them differently → so the accelerator extends to new cases without a hardware redesign, and the abstract reports versatility across precisions from floating-point to eight-bit.

What it costs (trade-offs and scope)

This account is based on the abstract only. The full paper was not accessible when this page was written, so the mechanisms are described at the level the abstract states; the microarchitecture of the Adaptive-Sparse Processing Unit, the precise semantics of cal.pre, cal.adis, and cal.red, the meaning of “PDP-optimal”, and the scheduling policy are not detailed here and should be read from the paper.
Programmability adds instruction and scheduling overhead. Turning a fixed datapath into programmable instructions introduces the cost of issuing and scheduling those instructions; the abstract argues the flexibility is worth it, but the balance between programmable overhead and the imbalance it removes is a detail to confirm in the full paper.
The gains depend on exploitable bit-level sparsity. The benefit comes from values having many skippable zero bits with an uneven distribution to balance; workloads with dense, high-entropy values or already-balanced bit work would offer less for BitRed to exploit.
The comparisons are to specific baselines and a specific setting. The reported figures are normalized to an SCNN baseline and measured against Bitlet, BitWave, an NVIDIA A100, and a Jetson Orin, at a 28-nanometer process; the benchmark set, the precise conditions, and how the advantage varies across models and precisions are things to verify in the full paper.
Reported area and power are point figures. The roughly five-square-millimeter area and few-hundred-milliwatt power numbers are for the described implementation and precisions; how they scale with larger designs, other technology nodes, or integration into a full system is not established from the abstract.

The evaluation

Exploiting non-uniform bit-level sparsity

Property	Naive bit-serial	Fixed bit-interleaving (Bitlet)
How bit-level work is handled
Adapts to dynamic sparsity
Extensible to new precisions	Limited
Reported speedup (vs SCNN baseline)	Baseline family	Prior state of the art
Versus high-end GPUs	n/a	n/a

From the paper's abstract, normalized to an SCNN baseline. BitRed reports up to 9.4 times the speed of the bit-interleaving accelerator Bitlet and 5.6 times the bit-serial state-of-the-art BitWave, up to 7.6 times higher inference efficiency than Bitlet, roughly 5.072 square millimeters of area with power from about 550 milliwatts (floating-point) down to 458 milliwatts (eight-bit) at 28 nanometers, and up to 18.9 times and 13.5 times the efficiency of an NVIDIA A100 and a Jetson Orin. Baselines, benchmarks, and conditions are in the full paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This page is written from the paper’s abstract; the full text was not accessible at the time of writing, so it stays at the level the abstract supports and fills in only standard, clearly-labeled background about bit-level sparsity and bit-serial acceleration. The claim structure is: bit-level sparsity offers real savings but is non-uniform and dynamic, causing severe load imbalance on parallel hardware; the leading bit-interleaving accelerators like Bitlet help but are locked into a rigid datapath with severe inter-channel imbalance; and BitRed responds with a “programmable adaptive bit-interleaving” design whose Adaptive-Sparse Processing Unit exposes the acceleration as orthogonal RISC-V instruction-set extensions for pre-processing, adaptive distillation with dynamic load balancing, and efficiency-optimal reduction, turning a rigid hardware problem into a flexible scheduling problem and reporting up to 9.4 times Bitlet’s speed and competitiveness with high-end GPUs. Treat the microarchitecture, the exact instruction semantics, the meaning of “PDP-optimal”, the benchmarks, and the baselines as things to verify against the full paper, since they are not established from the abstract alone; the computation trace in the step-through is illustrative. Within these limits, the core idea is clear: make the balancing of uneven bit-level work a programmable, schedulable decision instead of fixed silicon.

Background

Why load balancing is the recurring enemy of sparse accelerators

It helps to see the pattern BitRed belongs to. Sparsity — whether zeros in a matrix, in a tensor, or, as here, in the bits of individual numbers — is one of the great opportunities in accelerating machine learning, because skipping the zeros means doing less work. But sparsity is almost always the enemy of parallel hardware’s core assumption: that the work can be divided evenly across many identical units running in lockstep. Real sparsity is irregular and data-dependent, so when you hand different pieces of a sparse problem to different processing elements, they finish at different times, and the fast ones wait for the slow ones. This load-imbalance problem recurs at every level of sparse acceleration, and much of the field’s ingenuity goes into redistributing irregular work so the hardware stays busy. Extracting fine-grained parallelism from an irregular workload while keeping the processing elements evenly loaded, rather than letting a few overloaded units bottleneck the rest, is the same challenge the work on exploiting fine-grained parallelism in a graph-mining accelerator covered elsewhere on this site confronts, where uneven work is the thing standing between available parallelism and realized speed. BitRed’s answer, per its abstract, is to make the balancing programmable, so the accelerator can chase the moving target of dynamic bit-level sparsity instead of being outrun by it.

If you remember only three things

1. Bit-level sparsity — many bits of a neural network's numbers are zero and can be skipped, since a multiply is a sum of one shifted partial product per nonzero bit — is a real efficiency opportunity, but it is non-uniform and data-dependent, so different values carry very different amounts of nonzero-bit work, which severely imbalances the parallel lanes of an accelerator that then runs only as fast as its busiest lane; the leading bit-interleaving accelerators such as Bitlet distill and interleave nonzero bits to help, but are locked into a rigid datapath that still suffers severe inter-channel imbalance and cannot adapt to the shifting distribution.
2. BitRed proposes a "programmable adaptive bit-interleaving" design whose core, an Adaptive-Sparse Processing Unit, deconstructs the acceleration into a small set of orthogonal RISC-V instruction-set extensions — cal.pre for pre-processing, cal.adis for adaptive distillation with dynamic load balancing, and cal.red for an efficiency-optimal (PDP-optimal) reduction — so that balancing and combining the bit-level work becomes a flexible software scheduling problem rather than a fixed circuit, which the abstract argues makes the design more adaptable and extensible across precisions and patterns.
3. The abstract reports, normalized to an SCNN baseline, up to 9.4 times the speed of Bitlet and 5.6 times the bit-serial state-of-the-art BitWave, up to 7.6 times higher inference efficiency than Bitlet, about 5.072 square millimeters of area with power scaling from roughly 550 milliwatts at floating-point to 458 milliwatts at eight-bit precision at 28 nanometers, and up to 18.9 and 13.5 times the efficiency of an NVIDIA A100 and a Jetson Orin; this page is written from the abstract because the full text was unavailable, so the microarchitecture, exact instruction semantics, the meaning of "PDP-optimal", the benchmarks, and the baselines should be confirmed against the full paper, and the durable idea to retain is that making the balancing of uneven bit-level work programmable, rather than fixed in silicon, lets the accelerator follow dynamic sparsity.

Check yourself

Check your understanding

1. Why does bit-level sparsity cause load imbalance on parallel hardware?
2. What is the limitation of the bit-interleaving paradigm that BitRed targets?
3. What is BitRed's core approach?
4. What did BitRed report achieving, and what should you keep in mind about this summary?

Sources

The paper is “BitRed: Taming Non-Uniform Bit-Level Sparsity with a Programmable RISC-V ISA for DNN Acceleration” by Yanhuan Liu, Wenming Li, Kunming Zhang, Yuqun Liu, Siao Wen, Lexin Wang, Tianyu Liu, Haibin Wu, Zhihua Fan, Xiaochun Ye, Dongrui Fan, and Xuejun An (Institute of Computing Technology, Chinese Academy of Sciences), published at ASPLOS 2026 (Volume 2), DOI 10.1145/3779212.3790132. It is released under a Creative Commons Attribution license.

This explainer is written from the paper’s abstract; the full open-access text was not accessible at the time of writing. The claims traceable to the abstract are: that the non-uniform and dynamic nature of bit-level sparsity poses a critical load-imbalance challenge for parallel hardware accelerators; that the bit-interleaving paradigm, represented by state-of-the-art accelerators like Bitlet, shows promise but is fundamentally constrained by a rigid datapath and severe inter-channel load imbalance; that BitRed embodies a new “programmable adaptive bit-interleaving” philosophy whose core is an Adaptive-Sparse Processing Unit that deconstructs the acceleration process into a set of orthogonal RISC-V instruction-set extensions for pre-processing (cal.pre), adaptive distillation with dynamic load balancing (cal.adis), and PDP-optimal reduction (cal.red); that this ISA-based approach transforms a rigid hardware problem into a flexible scheduling problem and provides a more adaptable and extensible solution; and that empirical studies, normalized to an SCNN baseline, report up to 9.4 times speedup over Bitlet and 5.6 times over the latest bit-serial state-of-the-art BitWave, up to 7.6 times higher inference efficiency than Bitlet on representative models, an area of 5.072 square millimeters and scalable power from 550.43 milliwatts (floating-point 32) to 495.12 milliwatts (16-bit) and 457.90 milliwatts (8-bit) at 28-nanometer TSMC, high versatility across precisions, and up to 18.9 times and 13.5 times higher than an NVIDIA A100 and a Jetson Orin 32GB. Background explanations of bit-level sparsity, bit-serial computation, and load imbalance in sparse accelerators are standard domain context provided to make the abstract’s claims understandable, and the computation trace in the step-through is illustrative. The microarchitecture, the exact instruction semantics, the meaning of “PDP-optimal”, the benchmarks, the baselines, and the measurement conditions should be confirmed against the full paper, which is available through the ACM Digital Library at the DOI above.