BiSon-e: a lightweight and high-performance accelerator for narrow integer linear algebra computing on the edge

A 1990s number-theory trick lets one 64-bit multiplication do the work of ten small ones — BiSon-e adds four instructions to a RISC-V core so the trick actually pays off, using 0.07% more chip area and no new arithmetic unit at all.

Why this problem is worth solving

Moving inference from the cloud to the device is attractive for privacy, latency and energy, and the standard way to make it fit is

The conventional escape is a

What people did before, and where it broke

• Just widen everything to 8 bits. Simple and universally supported. It throws away the memory saving of sub-byte formats and does nothing for arithmetic cost.
• Add narrow SIMD instructions. Prior RISC-V work adds multiply-accumulate units and bit-manipulation instructions for quantised networks. But when the units are tuned for 8-bit operands, performance stops scaling as data narrows, and extra instructions are needed for 4-bit and 2-bit cases.
• Add dot-product units with custom instructions. A follow-up from the same group improved on that, cutting a convolution layer’s runtime by 5.3 and 8.9 times for 4-bit and 2-bit data. Effective, but tied to the deep learning domain and considerably larger.
• Add a vector processing unit. The most general answer, and genuinely fast — the VPU in this paper’s own system-on-chip beats BiSon-e at 8-bit. It also occupies roughly 600 times the area.
• Build an application-specific accelerator. A state-of-the-art mobile neural network accelerator with 192 processing elements takes 36 square millimetres at 65nm and supports only 8-bit data. Faster, far bigger, and useless for anything but its one workload.
• Use binary segmentation in software. The mathematics has been known since the 1990s and applied theoretically to polynomial multiplication, complex multiplication, discrete Fourier transforms and more. Nobody had built hardware around it, and on a stock instruction set the packing overhead dominates: the paper measures inner product gains of only 1.3 to 1.5 times, rising to 2.5 to 3.4 times even with bit-manipulation instructions.
• The gap. A technique that should give an order of magnitude gives a factor of one and a half, because standard architectures cannot cheaply manipulate values at non-standard bit widths. The arithmetic was never the problem.

How BiSon-e makes the packing disappear

The idea

One multiplication can carry many, exactly. Binary segmentation is the observation that a machine word is a polynomial in base two, so several small integers written into disjoint bit fields of one word multiply as a unit. Multiply two packed words and every pairwise product appears, already summed where the exponents coincide — which is precisely an inner product if you reverse one of the vectors first, and precisely a convolution if you do not. The one requirement is the clustering width: each slot must be wide enough for the largest partial result plus enough guard bits that carries never cross a boundary, which for an inner product means the two input widths plus the logarithm of the vector length. Get that right and the result is exact, not approximate. The same manoeuvre — exploiting the gap between the small values a workload actually needs and the wide words a machine natively provides, with guard bits sized to make overflow impossible — reappears in zero-knowledge proof compilers packing several results into one field element to check them with a single constraint, covered elsewhere on this site.

The bottleneck was never the arithmetic. The paper’s design-space exploration is the most useful part of it, because it takes a technique everyone assumed was good and measures exactly where it fails. On a 64-bit machine the theoretical reduction in arithmetic operations runs from 3 times for 15-bit data up to 19 times for 1-bit data on inner products, and up to 90 times on convolutions. Implemented in software those numbers collapse to about 1.5 times, and the profiling says why: building the packed operands requires shifting and masking each element to a width no instruction understands, and on the convolution side, extracting the outputs consumes roughly half the runtime with the overlap-add stitching taking another quarter. So the architecture’s target is not the multiply, which existing hardware already does well. It is everything wrapped around the multiply. Discovering that the useful work was already cheap and the marshalling around it was the real cost is the same diagnosis that motivates fusing tensor operations to eliminate the intermediate results between them, covered elsewhere on this site.

Reuse the multiplier you already have. The design decision that makes BiSon-e worth reading is what it declines to build. There is no new arithmetic unit, no new register file, no new datapath — just a combinational stage that reshapes operands on the way into the core’s ordinary integer multiplier and another that selects bits on the way out, controlled by four new instructions. The consequence is an area cost of 0.003704 square millimetres at 65nm, under 0.07% of the layout, against a narrow-SIMD unit that the authors synthesised for comparison and found ten times larger, and a vector unit roughly six hundred times larger. It also gives capabilities the fixed-width alternatives lack for free: because the clustering width is computed from the two operand widths, mixed-precision operation is automatic, and because the packed form is the storage form, data stays compressed in memory. Buying capability by rearranging what surrounds an existing functional unit rather than adding one is a recurring move in edge accelerator design, and the trade-offs of pushing quantised transformer inference onto small accelerators are covered elsewhere on this site.

Following an inner product through the hardware

For the first configuration in the paper, each 64-bit register holds eight 8-bit elements, and the hardware will form packed operands of three 21-bit slots. Three iterations therefore consume one pair of registers.

Why it works

• The wasted bits in a wide word become parallelism. An 8-bit value in a 64-bit register leaves most of the register idle → packing three of them at a 21-bit spacing uses it → and at 1 bit, ten elements fit, which is why speedup grows as data narrows instead of flattening out.
• Guard bits make the packing exact. The clustering width includes room for the largest partial sum and the carries → no value can spill into its neighbour’s slot → so this is compression of the computation, not approximation of it, and no accuracy is traded away.
• Fusing the marshalling removes the tax that killed the software version. Packing and extraction happen inside the operand and result paths → the software bottleneck the profiling identified simply does not execute → lifting the inner product from about 1.5 times to 4.4 times at 8-bit and 22.6 times at 1-bit.
• The compressed format is the computed format. Data stays packed in memory and is consumed packed → fewer memory transfers and no conversion passes → which is why gains come from both arithmetic and data movement rather than arithmetic alone.
• Reusing the multiplier keeps the cost near zero. No new datapath, register file or functional unit → 1210 cells and under 0.07% of layout area → so the comparison against a vector unit is not performance against performance but performance against six hundred times the silicon.

What it costs (trade-offs and scope)

• At 8 bits, the vector unit wins. BiSon-e runs 1.8 times slower than the VPU on 8-bit AlexNet. It draws level at 4-bit and pulls ahead by 1.9 times at 2-bit. Since 8-bit is the most widely deployed quantisation today, the technique is betting on the field moving further down.
• Concurrency at 8 bits is modest. A 64-bit register holds only three 21-bit slots, so a single operation covers three elements. The dramatic numbers belong to 1-bit and 2-bit data, where ten and roughly six elements fit.
• The memory-footprint claim needs care. The paper’s own analysis notes that only 1-bit clusters hold more elements than plain byte allocation would. The saving comes from keeping data compressed in memory and consuming it compressed, not from the clustered form being denser than a byte array.
• A dedicated accelerator is still faster. The paper compares honestly against a prior custom-functional-unit design that runs 1.1 to 1.4 times faster than BiSon-e across widths — while occupying 11.6 times the area. BiSon-e wins on performance per unit area, by 8.4 to 11.2 times, not on raw speed.
• Programming is through intrinsics. The toolchain was extended with the custom instructions and the benchmarks call them through intrinsics in hand-written library code. There is no compiler that discovers these opportunities automatically.
• Performance is simulated. Runtimes come from gem5 with a 5-stage in-order RV64IM pipeline and energy from McPAT. The area and power figures are stronger evidence, coming from real synthesis and place-and-route in TSMC 65nm and GlobalFoundries 22nm, with the 65nm layout described as production-ready.

The evaluation

Check yourself

Sources

The paper is “BiSon-e: A Lightweight and High-Performance Accelerator for Narrow Integer Linear Algebra Computing on the Edge” by Enrico Reggiani, Cristóbal Ramírez Lazo, Roger Figueras Bagué, Adrián Cristal, Mauro Olivieri and Osman Sabri Unsal, published at ASPLOS 2022, pages 56-69, DOI 10.1145/3503222.3507746. The authors were at the Barcelona Supercomputing Center and Sapienza University of Rome.

This explainer is written from the full text, read from the open-access deposit in the Sapienza University institutional repository. All figures above come from the paper’s own sections 3 through 5: the theoretical arithmetic reductions of up to 19 times for inner products and roughly 90 times for convolutions, the software speedups of 1.3 to 1.5 times and 2.5 to 3.4 times with bit-manipulation instructions, the BiSon-e kernel speedups of 4.4 to 22.6 times for inner products and 3.5 to 61 times for convolutions, the AlexNet and VGG-16 layer results, the VPU comparisons and the roughly 600-fold area ratio, and the area and power figures of 0.003704 square millimetres and 1.089 milliwatts at 65nm with 1210 cells. Readers comparing against the original should note that the abstract’s headline speedups of 5.6, 13.9 and 24 times are maxima taken across individual layers of both networks and do not correspond to any single figure reported in section 5.3, where the highest values are 20.5 times for AlexNet convolutional layers and 18.5 times for VGG-16 end to end.