Carat: Unlocking Value-Level Parallelism for Multiplier-Free GEMMs

When a neural network runs on batched, very-low-precision data, the same weight gets multiplied by the same handful of distinct input values over and over — with a 4-bit input there are only sixteen possible products, yet a normal accelerator recomputes them thousands of times; this paper (UW-Madison) proposes computing each unique product once by simple repeated addition instead of multiplication, and letting every input pick out the product it needs by the timing of a pulse, yielding a multiplier-free matrix-multiply engine that matches a systolic array in the same area at lower energy and pulls far ahead when scaled up.

The argument in one glance

Problem

When data is low-precision and batched, accelerators multiply the same values over and over

Matrix multiplication is the workhorse of neural-network inference, and specialized accelerators like systolic arrays exist to do it fast and efficiently. Two trends are reshaping the data these accelerators chew on. First, precision is falling — from 32-bit floating point toward 16-bit and now 8-bit formats — which means each number carries only a few bits of mantissa, so there are very few distinct values it can take. Second, inference is increasingly batched, so a great many inputs stream through at once. Put together, these mean that when a weight is multiplied against a whole batch of inputs, the inputs only take a small number of unique low-precision values, so the same product of that weight and that value gets recomputed again and again. Concretely, multiplying a weight by a length-1024 vector of 4-bit inputs yields at most sixteen distinct products, yet a conventional accelerator computes 1024 of them — sixty-four times more multiplications than are actually necessary. The redundant work, and the full hardware multipliers that perform it, are pure waste that grows with batch size and shrinking precision.
What people did

Build faster, denser multiplier arrays, or reuse similar computations

The standard response is to keep improving the multiply-accumulate hardware: pack more multipliers into a systolic array, or lower precision so each multiplier is smaller. A more recent line reuses computation across inputs that are similar, detecting when two inputs would produce nearly the same result and computing it once.
Why it fell short

These still pay for multipliers and miss the reuse that low precision makes obvious

Denser multiplier arrays still perform every one of those redundant multiplications and still pay the area and energy of full multipliers, which dominate a multiply-accumulate unit. Lowering precision shrinks the multipliers but does nothing to remove the redundancy — it actually increases it, since fewer unique values means more repeats. Computation-reuse accelerators that exploit input similarity capture some of the overlap, but they reason about approximate similarity between whole inputs rather than the exact, structural fact that low-precision data can only take a handful of values, so they leave much of the reuse on the table and add their own detection overhead. What none of these do is confront the underlying observation head-on: with low-precision batched data, the set of possible products is tiny and known, so the right move is to compute each unique product exactly once and let every input simply pick the one it needs — without a multiplier at all.
This paper's idea

Compute each unique product once by adding, and let inputs subscribe to it by timing

Carat introduces value-level parallelism: rather than multiplying each input by the weight, it computes the small set of unique products once and has every input select, or subscribe to, the product it needs. Two ideas make this multiplier-free. For value reuse, the unique products of a weight and each possible mantissa value are generated by accumulation — the product for value one is the weight, for value two is that plus the weight again, and so on — so a running accumulator sweeps through all unique products using only addition, never a multiplier. For subscription, each input is encoded in temporal coding: its value is represented by the timing of a spike, so when the accumulator reaches an input's value, that input's spike fires and captures the current product. Many inputs subscribe in parallel, each grabbing its product at the right cycle. The resulting accelerator, per the authors' materials, matches a systolic array's throughput in the same area at lower energy, and pulls well ahead when scaled to multiple nodes.

Why this problem is worth solving

The redundancy is not incidental; it is created and amplified by the exact trends driving modern inference.

A general matrix multiplication multiplies each weight against many inputs. When the inputs are low-precision — an eight-bit floating-point format has only a few mantissa bits, and four-bit integers only sixteen possible values — the set of distinct values an input can take is tiny. And when inference is batched, there are far more inputs than distinct values, so by the pigeonhole principle the same input value recurs constantly. The product of a given weight with a given input value is therefore computed over and over, once for every input that shares that value, even though it is the same number every time. The paper quantifies this with a scalar-vector multiplication: a weight times a length-1024 vector of four-bit inputs has at most sixteen unique products, so computing all 1024 wastes sixty-four times the necessary multiplications. This untapped potential for value reuse grows precisely as the field pushes toward lower precision and larger batches, which is what makes it worth building hardware around. The redundant multiplications are not a minor inefficiency; they are the bulk of the work, and they are performed by full multipliers, the most expensive part of a multiply-accumulate unit.

The deeper opportunity is that this redundancy points at a way to remove the multiplier entirely.

A hardware multiplier is large and power-hungry compared to an adder, so accelerators spend much of their area and energy on multiplication. But there is a classical fact this work exploits: a product of a weight by an integer value is just that weight added to itself that many times, and the sequence of products for values zero, one, two, and so on is a running sum — each is the previous product plus one more copy of the weight. So if you only need the set of products of one weight with all the small values a low-precision input can take, you can generate that whole set with an accumulator that repeatedly adds the weight, never invoking a multiplier. The challenge is then no longer how to multiply, but how to let each of many inputs, in parallel, pick out the one product from that swept sequence that corresponds to its own value. Solving that selection cheaply is what turns the accumulation trick from a curiosity into a complete, multiplier-free matrix-multiply engine, and it is where the paper’s temporal-coding idea comes in.

What people did before, and where it broke

Denser or lower-precision multiplier arrays. Pack more multiply-accumulate units into a systolic array, and shrink each multiplier by lowering precision. Where it broke. They still perform every redundant multiplication and still pay for full multipliers; lowering precision even increases the redundancy by shrinking the set of unique values.
Computation reuse via input similarity. Detect inputs that are similar and compute their shared result once. Where it broke. They reason about approximate similarity between whole inputs rather than the exact, structural fact that low-precision data takes only a few values, so they miss much of the reuse and add detection overhead.
Bit-serial and unary accelerators. Trade the multiplier for serial or unary processing of bits. Where it broke. They still process each input’s computation separately rather than sharing the unique products across all inputs, so they do not capture value-level reuse across a batch.
The gap. No accelerator computed each unique product of a weight and a low-precision value exactly once and let all inputs subscribe to those products in parallel without a multiplier. Doing so through accumulation and temporal-coded subscription is what Carat contributes.

How Carat works

Turning batched low-precision matrix multiply into shared accumulation plus subscription

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Recognize that low-precision batched data has few unique values, and reuse the products. Carat’s founding observation, from the authors’ materials, is that the twin trends of falling precision and rising batch size conspire to make value reuse enormous: the products a weight forms with a batch of inputs are drawn from a tiny set, yet conventional accelerators recompute each one for every input. The whole design flows from computing each unique product once and letting inputs share it — the paper calls this value-level parallelism, a new axis of reuse alongside the usual spatial and temporal parallelism of accelerators. Exploiting the structural redundancy in a neural-network computation to avoid work that need not be done, rather than merely doing that work faster, is the same skip-the-unnecessary instinct the work on hardware-accelerated dynamic neural-network pruning covered elsewhere on this site pursues, removing computation the data reveals to be redundant instead of paying for it.

Replace the multiplier with an accumulator that sweeps the products. The mechanism that makes reuse cheap is to generate the unique products by addition. Because the products of a weight with successive integer values form a running sum, an accumulator that repeatedly adds the weight visits every unique product in sequence, so Carat needs no hardware multiplier at all — it transforms multiplication into accumulation. This is what makes its processing elements so small, since a multiplier is far larger and hungrier than an adder. Doing away with expensive full-width multipliers by processing values through a leaner, serial style of arithmetic is the same efficiency move the work on taming bit-level sparsity with programmable serial arithmetic covered elsewhere on this site makes, trading the monolithic multiplier for cheaper primitives that build the result up incrementally.

Let inputs select their product by the timing of a spike. The elegant part is subscription: how does each of many inputs, in parallel, grab the one swept product that matches its value, without a comparator forest or a multiplier? Carat encodes each input in temporal coding, where a value is represented by when a spike occurs, and a counter advances in lockstep with the product-sweeping accumulator; an input’s spike therefore fires exactly at the cycle when the accumulator holds that input’s product, capturing it. Many inputs, each with its own converter and register, subscribe independently during one shared sweep, which is precisely the value-level parallelism. Unlocking a previously unexploited dimension of parallelism on a deep-learning accelerator, so that one pass of the hardware serves far more of the work at once, is the same goal the work on exploiting instruction-level parallelism on deep-learning accelerators covered elsewhere on this site pursues, finding a new axis along which the accelerator can do more per unit of hardware.

Following a scalar-vector multiply through Carat

Multiplying one weight by a batch of low-precision inputs, without a multiplier

A trace, following the authors' slides, of how a Carat processing-element column computes a weight times a vector of inputs by accumulation and temporal subscription.

Step 1 of 5

A weight must multiply many low-precision inputs

A processing-element column receives one weight and a vector of inputs, each a low-precision value with only a few possible mantissa values. Conventionally this would be many multiplications, most of them recomputing the same products.

For four-bit inputs there are only sixteen distinct products of the weight and a value.

Why it works

Few unique values mean the products can be shared. Conventional hardware multiplies each input independently → but low-precision batched data takes only a few values, so the products are a tiny recurring set → so computing each unique product once and sharing it removes most of the redundant multiplication.
Products of a weight and successive values form a running sum. Generating each product from scratch would need a multiplier → but the product for the next value is just the previous product plus the weight → so an accumulator sweeping by repeated addition produces all unique products with no multiplier at all.
Temporal coding turns value selection into timing. Each input must pick its product from the swept sequence, which naively needs comparison or indexing → but encoding a value as the timing of a spike, with a counter advancing alongside the accumulator, makes an input’s spike coincide exactly with its product → so subscription is a cheap enable at the right cycle.
Independent converters give parallel subscription. One swept sequence must serve many inputs at once → but each input has its own temporal converter and register → so all inputs subscribe simultaneously during a single accumulation sweep, which is the value-level parallelism.
Multiplier-free processing elements are tiny. A multiplier dominates the area and energy of a multiply-accumulate unit → but Carat’s processing elements contain only adders, counters, converters, and registers → so they take a small fraction of the array’s area and energy, freeing budget for more of them under an equal-area comparison.

What it costs (trade-offs and scope)

This account is based on the abstract, poster, and slides. The full paper text was not accessible when this page was written, so the mechanism is described from the authors’ companion materials; additional optimizations and design-space analysis the paper contains, which the slides explicitly reference, should be read from the paper, and finer numbers confirmed there.
The benefit grows with batch size and low precision, and shrinks without them. Value-level parallelism pays off because a large batch of low-precision inputs contains far more inputs than unique values; small batches or higher-precision data expose fewer unique-value repeats, so the reuse and thus the advantage diminish.
The sweep spends cycles proportional to the value range. Generating the unique products by accumulation takes a number of cycles set by how many distinct values the precision allows; this is cheap for very low precision but grows as precision rises, bounding how high a precision the approach suits.
The iso-area throughput edge is modest; the larger wins are conditional. On the authors’ evaluation, iso-area throughput is roughly on par with a systolic array with a small energy advantage, and the larger throughput and energy gains appear when scaled to multiple nodes or measured at the array level; the whole-accelerator, single-node picture is more incremental.
It targets matrix multiplication with this data profile. Carat is a GEMM accelerator built around low-precision batched inference; workloads outside that profile, or computations that are not dominated by such matrix multiplies, are not the target, and the comparison baselines and formats are specific to the study.

The evaluation

Computing a batched low-precision GEMM

Property	Binary systolic array	Computation-reuse accelerator
How products are computed
Exploits value-level reuse
Processing-element cost
Iso-area throughput and energy	Baseline	Comparable
Scaled to multiple nodes	Baseline	n/a

From the paper's abstract and the authors' materials, on the MLPerf benchmark suite against a binary systolic array, a unary systolic array, and a computation-reuse-based accelerator. Iso-area, Carat is roughly on par with a systolic array in throughput (about 1.02 times) with better energy efficiency (about 1.06 times), reaching about 3.2 times throughput and 4.3 times energy efficiency scaled to multiple nodes; at the array level it reports about 1.23 and 2.17 times the energy efficiency of the systolic array and the reuse-based accelerator, with the Carat processing element contributing only about 7 percent of array area and 10 percent of array energy. Full details and baselines are in the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This page is written from the paper’s abstract together with the authors’ poster and slide deck; the full paper text was not accessible at the time of writing, so it stays close to those authoritative materials and fills in only standard background about low-precision arithmetic and accumulation. The core idea is clear and well-supported by them: low-precision batched data has few unique input values, so a weight’s products with them are a tiny recurring set; Carat computes each unique product once by accumulation rather than multiplication, and lets each input subscribe to its product through temporal coding, where a spike’s timing selects the product during a shared sweep, giving a multiplier-free GEMM accelerator. On the authors’ evaluation it matches a systolic array’s throughput in equal area at lower energy, with larger gains at the array level and across multiple nodes, and its processing elements are a small fraction of area and energy. Read it within scope. The benefit depends on large batches and low precision; the sweep’s cost grows with precision; the single-node iso-area edge is modest while the bigger numbers are array-level or multi-node; and the optimizations and design-space analysis the slides reference as being in the paper should be confirmed there, along with exact figures and baselines. Within these limits, the idea is an elegant reframing: compute the few products once, and select them by timing.

Background

Why arithmetic efficiency is where accelerator gains now come from

It helps to see why replacing a multiplier with an accumulator is a big deal. For years, accelerator progress came from packing more multiply-accumulate units onto a chip and running them faster, but that path is bounded by area and power: the multiplier at the heart of each unit is expensive, and there are only so many you can afford. So attention has shifted to doing the arithmetic more cleverly — using lower precision, exploiting sparsity, reusing computation, and rethinking how numbers are represented and combined — because a cheaper way to compute the same result multiplies across every one of the millions of operations a network performs. Low precision has been central to this shift, since eight-bit and smaller formats slash the cost of each operation while modern networks tolerate the reduced accuracy. Squeezing more inference out of a fixed hardware budget by embracing aggressive low precision and the arithmetic tricks it enables is the same terrain the work on eight-bit transformer inference for edge accelerators covered elsewhere on this site occupies, treating numeric efficiency as the lever. Carat pushes that lever in a distinctive direction, observing that once precision is low enough, the products themselves become so few that the honest thing to do is compute each once and hand it out — no multiplier required.

If you remember only three things

1. Matrix-multiply accelerators waste enormous work on redundant products, because the trends toward low-precision, batched data mean inputs take only a few unique values — a weight times a length-1024 vector of four-bit inputs has at most sixteen unique products but is computed 1024 times, sixty-four times more multiplications than necessary — and this redundant work is performed by full hardware multipliers, the most expensive part of a multiply-accumulate unit.
2. Carat proposes value-level parallelism: compute each unique product once and let every input subscribe to the one it needs. It is multiplier-free through two ideas — value reuse, where the unique products of a weight and successive values form a running sum so a single accumulator sweeps through them all by repeated addition with no multiplier; and subscription via temporal coding, where each input's value is encoded as the timing of a spike so that, as a counter advances alongside the accumulator, the input's spike fires exactly when the accumulator holds its product and captures it, with many inputs subscribing in parallel during one shared sweep.
3. Carat arranges these small accumulator-based processing elements into an outer-product array; on the MLPerf suite the authors report it matches a binary systolic array's throughput in equal area at about 1.06 times the energy efficiency, reaching about 3.2 times throughput and 4.3 times energy efficiency scaled to multiple nodes and about 1.23 and 2.17 times the array-level energy efficiency of the systolic array and a computation-reuse accelerator, with the processing element only about 7 percent of array area; this page is written from the abstract, poster, and slides because the full paper text was unavailable, so the additional optimizations and exact figures should be confirmed against the paper, and the durable idea is to compute the few products once and select them by the timing of a pulse.

Check yourself

Check your understanding

1. Why do matrix-multiply accelerators waste so much work on low-precision batched data?
2. What is value-level parallelism, and how does Carat generate the unique products without a multiplier?
3. How does each input select its product, in parallel and without a multiplier?
4. What did Carat achieve, and what should you keep in mind about this summary?

Sources

The paper is “Carat: Unlocking Value-Level Parallelism for Multiplier-Free GEMMs” by Zhewen Pan, Joshua San Miguel, and Di Wu (University of Wisconsin-Madison), published at ASPLOS 2024 (Volume 2), DOI 10.1145/3620665.3640364. It is released under a Creative Commons Attribution license and received an IEEE Micro Top Picks Honorable Mention and a Distinguished Artifact Award.

This explainer is written from the paper’s abstract together with the authors’ poster and slide deck; the full paper text was not accessible at the time of writing. The claims are traceable to those authoritative materials: that hardware architectures for general matrix multiplication are well studied for deep neural networks, and that trends toward batched, low-precision data (such as the FP8 format) create growing untapped potential for value reuse; that Carat proposes value-level parallelism, whereby unique products are computed only once and different inputs subscribe to (select) their products via temporal coding, transforming multiplication into accumulation and performing matrix multiplications with efficient multiplier-free hardware; the illustration that a weight times a length-1024 vector of four-bit inputs has only sixteen unique products, sixty-four times fewer than computed by a conventional accelerator; the mechanism from the slides of generating the unique products by an accumulator that repeatedly adds the weight, converting each input to temporal coding via a temporal converter driven by a counter so that an input’s spike fires at the cycle equal to its value and captures the current product, and subscribing many inputs in parallel; the Carat architecture as an outer-product array of accumulator-based processing elements with a memory hierarchy and peripheral logic like existing accelerators, evaluated on the MLPerf benchmark suite against a binary systolic array, a unary systolic array, and a computation-reuse-based accelerator; and the reported results that iso-area throughput and energy efficiency improve by about 1.02 times and 1.06 times over a systolic array and about 3.2 times and 4.3 times when scaled to multiple nodes, with array-level energy efficiency about 1.23 and 2.17 times that of the systolic array and the reuse-based accelerator, iso-flops throughput about 2.02 times the systolic array, and the Carat processing element contributing about 7 percent of array area and 10 percent of array energy. Background on low-precision arithmetic and accumulation is standard domain context, and the further optimizations and design-space analysis referenced in the slides, along with exact figures and baselines, should be confirmed against the full paper, available through the ACM Digital Library at the DOI above.