Energy Efficient Convolutions with Temporal Arithmetic

A digital chip spends most of its energy on the multiply-accumulate math of convolution and on turning analog camera pixels into numbers first; this work from UC Santa Barbara instead computes in the time domain, where a value is the moment a wire flips high, and applies a negative-logarithm trick so that multiplication collapses into a mere delay — letting a camera's own staggered pixel readout feed a tiny race-logic circuit that convolves images for about eight times less energy than the state of the art, with no analog-to-digital conversion at all.

The argument in one glance

Problem

Convolution near a sensor burns energy on conversion and multiplication

Convolution underlies image filters and vision neural networks, and when it runs right at the camera its energy goes to two things: converting each analog pixel to a digital number, and the multiply-accumulate math that mixes pixels with filter weights. Multiplication in particular is costly in hardware. As transistor scaling slows and sensor arrays grow denser, there is real pressure to find a cheaper way to compute.
What people did

Compute in the time domain with race logic

Temporal computing offers an unusually efficient alternative: encode a value in when a wire's edge arrives rather than in its voltage level. The style called race logic uses just a few timing operations — earliest arrival, latest arrival, delay, and inhibit — and because its wires rarely switch, it consumes very little energy. It has been used for shortest paths, decision trees, and sorting.
Why it fell short

Race logic could not do arithmetic, and multiplication was the wall

For all its efficiency, race logic had never been shown to do general arithmetic. Multiplication was the obstacle: multiplying a value encoded as a time would mean delaying it by the current time over and over, which takes impractically long and awkward circuitry. Negative numbers and subtraction had no natural representation either. Without multiplication and subtraction there could be no multiply-accumulate, so convolution — the very operation you would most want near a sensor — was out of reach.
This paper's idea

Encode values as negative-log delays so multiplication becomes a delay

This work encodes each value as a delay equal to its negative logarithm — a "delay space" — under which multiplication turns into the addition of delays, that is, a single delay element. Addition becomes a soft-min function that can be approximated using only race logic's own min, max, and delay operations, and negative numbers are handled by splitting each value in two. A camera's staggered pixel readout feeds this circuit directly, with no analog-to-digital conversion, for about eight times less energy than the state of the art.

Why this problem is worth solving

To see why this is a breakthrough rather than a curiosity, it helps to understand how computing with time actually works.

In ordinary digital logic a wire carries a value in its voltage level, and each extra bit needs its own physically separate wire. Temporal computing does something different: it encodes a value in the timing of a single edge — the moment the wire transitions from low to high. Race logic is the discipline built on this idea, and it needs only a handful of operations on those edges. Taking the earliest of two arriving edges computes a minimum (and is just an OR gate on rising edges); taking the latest computes a maximum (an AND gate); a delay element adds a fixed amount of time to an edge; and an inhibit gate needs only two transistors. These operations are logically complete, so in principle they can compute anything, and they have a near-minimal activity factor — the wires almost never toggle, since each carries just one edge — which is why race logic can be astonishingly energy efficient. It has been used to great effect for shortest-path problems, decision trees, and sorting networks, all of which map naturally onto minima, maxima, and delays. The catch is that these are exactly the operations that do not obviously include ordinary arithmetic, and that gap is what kept race logic away from the multiply-accumulate math at the heart of convolution.

The specific thing that blocked arithmetic, and the trick that unblocks it, is worth stating precisely.

If you treat an edge’s delay as directly, linearly encoding a value, then delaying an edge adds to that value — addition is easy. But multiplication is a disaster: to multiply a value by another you would have to delay it by the current time as many times over as the multiplier, which takes impractically long and needs circuitry that does not really exist in race logic. Multiplication is the core of every multiply-accumulate, so this single obstacle kept convolution off the table. The paper’s key move is to change what a delay means. Instead of a delay encoding the value directly, it encodes the negative natural logarithm of the value: a value maps to an edge that arrives after a delay of minus-log of that value, a representation the authors call delay space. Because the logarithm turns products into sums, multiplying two values in the ordinary number space becomes simply adding their two delays — and adding delays is exactly what a delay element does. The hardest operation collapses into the cheapest one. Addition of the original values, by the same transform, becomes a function called negative log-sum-exponential (nLSE), which looks daunting but, as the next sections show, can be built from the min, max, and delay operations race logic already has.

What people did before, and where it broke

Digital near-sensor processing. Convert each pixel with an analog-to-digital converter, then do the multiply-accumulate in ordinary digital logic. Where it broke. Both the per-pixel conversion and the digital multipliers are energy-hungry, and with slowing transistor scaling and ever-denser sensor arrays, that energy is the bottleneck.
Race logic for structured problems. Encode values in edge timing and compute with earliest arrival, latest arrival, delay, and inhibit. Where it broke. It handled minima-and-maxima problems like shortest paths and sorting beautifully, but had no efficient general arithmetic — no multiplication, no negative numbers — so it could not do convolution.
Linear temporal (unary) encodings. Let a delay encode a value directly and proportionally. Where it broke. Multiplication requires delaying by the current time repeatedly (impractical), and a linear scale gives a narrow dynamic range, unable to handle very large and very small values together the way real image and network computations need.
Processing-in-pixel analog compute. Push convolution into analog circuitry embedded in the pixel array itself. Where it broke. It reduces conversion cost but crams compute circuitry into the pixel array, cutting pixel density and speed, and its energy advantage shrinks for the small strides typical of real filters.
The gap. No approach delivered general, low-energy convolution in the time domain: something that could multiply and subtract with race logic’s efficiency, span a wide dynamic range, and take data straight from the sensor without a costly conversion. Turning multiplication into a delay via a negative-log encoding is exactly what closes that gap.

How temporal convolution works

From a costly digital MAC to a delay that multiplies

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Change the representation so the hard operation turns into an easy one. Race logic is starved of arithmetic because multiplication is impossible when a delay encodes a value directly. Encoding the negative logarithm instead makes a product of values equal to a sum of delays, and a sum of delays is a single delay element — the operation that was the wall becomes the cheapest primitive in the system. Reducing a hard, native operation to a simple, well-understood one that the hardware can already do cheaply is the same move the concurrency data-structure work covered elsewhere on this site makes, reducing an awkward reachability question to the textbook problem of suffix minima so an efficient standard structure can solve it directly.

Put the important contributions first, so late ones can be dropped. In delay space a larger value maps to a shorter delay, so the most significant contributions to a convolution arrive earliest and the least significant arrive last — which means computation can be truncated at any time, trading a little accuracy for time and energy, without reordering anything. Arranging a computation so the decisive evidence comes first and the tail can be cut off once enough has accumulated is exactly the principle the hyperdimensional early-termination work covered elsewhere on this site is built on, letting the parts that matter most settle the outcome before the rest is even processed.

Reuse one small circuit over time instead of replicating it. A multiply-accumulate sums many inputs, and building a separate adder for each would be huge; instead the design loops each partial result back into the same block with a shifting time reference, an act of recurrence that trades extra hardware for a little extra time. Sharing one physical instance of a computation across many uses rather than duplicating the circuitry is the same instinct the circuit-deduplication work covered elsewhere on this site follows, collapsing many identical copies of a hardware block into a single reused one to shrink the design.

Following a convolution through the delay-space engine

From a camera pixel to a convolved output

Tracing one convolution shows conversion, multiplication, and accumulation all done in time.

Step 1 of 5

A pixel becomes a delay, with no converter

As the sensor reads a pixel, a starved-inverter voltage-to-time converter turns its voltage directly into a rising edge whose delay is the pixel's negative-log value. The higher the pixel voltage, the shorter the delay — the value inversion delay space needs, and no analog-to-digital converter at all.

A standard converter aims for a linear voltage-to-time map; here the starved inverter deliberately gives the negative-log shape.

Why it works

The negative log turns multiplication into a delay. Multiplying two timing values directly is impractical → but encoding each value as its negative-log delay makes the product of values the sum of their delays → so the hardest operation becomes a single delay element, essentially free.
The soft-min is buildable from what race logic has. Delay-space addition is a soft-min function → but that function is associative, commutative, and bounded by the plain minimum → so it can be approximated arbitrarily well using only the min, max, and delay gates race logic already provides.
Edges barely switch, so energy is tiny. Race logic carries each value as a single rarely-toggling edge → so its activity factor is near-minimal and its switching energy very small → so convolving in delay space costs about eight times less energy than the digital state of the art.
The camera serializes the inputs for free. A rolling shutter reads pixel rows out one interval at a time → so the inputs are already staggered in time to match a recurrence that loops results through one circuit → so accumulation needs no per-input adder and rides the sensor’s own timing.
The sensor produces delay space directly. A starved inverter’s delay falls as pixel voltage rises, roughly as a negative log → so the sensor itself emits delay-space values → so the expensive analog-to-digital conversion is removed from the pipeline entirely.

What it costs (trade-offs and scope)

It is approximate, for error-tolerant work. The delay-space add and subtract are approximations whose accuracy is set by the number of terms and the circuit noise; the target is image filters and feature extraction that tolerate small errors, not exact arithmetic.
Accuracy, energy, and noise trade off. A larger time unit buys accuracy but costs linearly more energy, with diminishing returns past a few nanoseconds, and more approximation terms paradoxically raise sensitivity to random jitter — so a minimum time scale and long inverter chains are needed to keep noise controlled.
Converting back to digital eats the savings. The win comes from keeping results in the time domain; at the ultra-low-energy design points, adding a time-to-digital converter erases the advantage, which favors chaining many operations temporally before ever converting.
The filter is hard-wired. This architecture bakes the filter weights in as fixed delays — ideal for a fixed edge detector or blur, but not a programmable accelerator; a follow-on design adds programmability.
It is a near-sensor, positive-input design. Sensor outputs are non-negative and negative weights are handled by splitting values; genuinely negative inputs would need more kernels, and the evaluation is on small image-filter convolutions at the sensor, not general networks.
The results are from simulation. Energy and area come from an architectural simulator with SPICE delay-line models at a 65-nanometer predictive node, not fabricated silicon.

The evaluation

Three ways to convolve near a camera sensor

Property	Digital (ADC plus MAC)	Processing-in-pixel
How a value is represented	Binary voltage levels	Analog charge in the pixel
Cost of a multiplication		Analog multiply
Analog-to-digital conversion		Reduced, but in-pixel
Energy and placement	Baseline

Three ways to convolve near a camera sensor, evaluated with an architectural simulator plus SPICE delay-line models at a 65-nanometer node on OpenCV filters (Sobel, pyrDown, GaussianBlur) over Imagenette images. The delay-space design performs the whole pipeline — sensor readout through convolution — in the time domain, reaching throughputs up to about 71 million frames per second and consuming over 2.5 times less energy than a state-of-the-art processing-in-pixel engine on a small-stride filter (roughly eight times less than the prior state of the art overall) at similar accuracy, while remaining robust to sensor noise and, unlike processing-in-pixel, fully separable from the pixel array. The advantage holds when results stay in the time domain; converting back to digital at the lowest-energy points erodes it.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This is an elegant piece of computer architecture: a single change of representation — encoding values as negative-log delays — turns multiplication, the operation that had kept race logic away from arithmetic, into a plain delay, and from there a whole convolution pipeline runs in the time domain with no analog-to-digital conversion, for roughly eight times less energy than the state of the art. The design is also thoughtfully engineered, from approximating the soft-min with existing gates to reusing one circuit through recurrence to reading delay space straight off a starved-inverter sensor. Read the result with its scope. It is approximate and aimed at error-tolerant vision work, not exact math; accuracy, energy, and noise trade against one another, and more approximation terms can counterintuitively worsen noise sensitivity, so the design must be tuned carefully. The energy win depends on keeping data in the time domain — a conversion back to digital at the lowest-energy points can erase it. The filter weights are hard-wired (a later design adds programmability), inputs are non-negative near-sensor pixels, and the numbers come from simulation with SPICE models rather than fabricated silicon. The durable lesson is that the cost of an operation is not fixed by the operation but by the representation you compute it in — and that choosing a representation in which the expensive step becomes trivial can be worth more than any amount of optimizing the expensive step directly.

Background

Why unconventional computing is worth the trouble

It helps to see this work against the pressure that motivates it: the energy cost of computing, especially for the vision and machine-learning workloads spreading to every sensor and edge device, has become the binding constraint, and conventional digital scaling can no longer relieve it on its own. That is what makes it worth reaching for radically different substrates — computing in the timing of edges, in analog charge, in memory, in light — each trading the comfortable generality of digital logic for a large efficiency gain on some class of problems. The retrospective on deep-learning hardware covered elsewhere on this site argues the same point at the level of whole systems, that the era of simply adding arithmetic throughput is ending and the real ceiling is now energy and data movement, which is precisely the ceiling temporal computing sidesteps by barely switching its wires at all. The broad lesson is that when a resource like energy becomes the true limit, the field stops merely refining the dominant paradigm and starts exploring the physics of computation itself — looking for representations and substrates where the operations you need most happen to be the ones nature makes cheap.

If you remember only three things

1. Convolution near a camera sensor spends most of its energy converting each analog pixel to digital and performing multiply-accumulate, with multiplication especially costly. Temporal computing (race logic) offers an ultra-low-energy alternative by encoding a value in the timing of a single edge rather than a voltage, using only earliest-arrival (minimum), latest-arrival (maximum), delay, and inhibit operations, whose wires barely switch. But race logic had never done general arithmetic: multiplying a time-encoded value would require delaying it by the current time repeatedly, and negative numbers had no representation, so convolution was out of reach.
2. The paper encodes each value as a delay equal to its negative logarithm ("delay space"). Because logarithms turn products into sums, multiplication becomes the addition of delays — a single delay element — and addition of the original values becomes a soft-min function (nLSE), subtraction its sibling (nLDE). The soft-min is associative, commutative, and bounded by the plain minimum, so it is approximated arbitrarily well from race logic's own min, max, and delay gates; negative weights are handled by splitting each value into positive and negative parts. Accumulation reuses one circuit via recurrence (looping partial results back with a reference-shifting delay), and a starved-inverter voltage-to-time converter emits delay-space values straight from the pixel while the camera's rolling-shutter readout serializes the inputs — eliminating the analog-to-digital converter.
3. Evaluated with an architectural simulator plus SPICE delay-line models at a 65-nanometer node on OpenCV filters over Imagenette images, the design reaches up to about 71 million frames per second and consumes over 2.5 times less energy than a state-of-the-art processing-in-pixel engine on a small-stride filter (about eight times less than the prior state of the art overall) at similar accuracy, staying robust to sensor noise and, unlike processing-in-pixel, fully separable from the pixel array. Its limits: it is approximate (for error-tolerant vision), trades accuracy against energy and noise (with more terms raising jitter sensitivity), loses its edge if results are converted back to digital, hard-wires the filter weights, targets non-negative near-sensor inputs, and is evaluated in simulation.

Check yourself

Check your understanding

1. Why could race logic not perform the multiplication at the heart of convolution?
2. What is "delay space" and why does it make multiplication cheap?
3. How does the architecture accumulate a convolution and get data from the sensor without an ADC?
4. Which statement best captures the results and limitations of this work?

Sources

The paper is “Energy Efficient Convolutions with Temporal Arithmetic” by Rhys Gretsch, Peiyang Song, Advait Madhavan, Jeremy Lau, and Timothy Sherwood, published at ASPLOS 2024, DOI 10.1145/3620665.3640395, and named an IEEE Micro Top Pick. The authors are at the University of California, Santa Barbara (with Advait Madhavan affiliated with NIST and the University of Maryland).

This explainer is written from the full text of the work. The ACM Digital Library page was protected by an anti-bot challenge that blocked automated retrieval, but the lead author’s open-access PhD dissertation (“Designing and Enabling Temporal Architectures for Neural Networks,” UC Santa Barbara, on eScholarship) contains this paper as its Chapter 2 — the dissertation states explicitly that the chapter’s content previously appeared in the ASPLOS 2024 proceedings under this DOI. Everything above is drawn from that chapter: the framing of near-sensor convolution energy (analog-to-digital conversion plus multiply-accumulate) and of race logic and its four primitives; the delay-space encoding via the negative logarithm and its consequences (multiplication becoming a delay, addition becoming the negative log-sum-exponential and subtraction the negative log-difference-exponential); the approximation of those functions from min, max, and delay with optimized max-terms (seven to eight terms sufficing), the split representation for negative values, and the time-shifted, single-delay-chain hardware; the recurrence architecture for reference-frame synchronization and compute-on-arrival; the rolling-shutter convolution architecture with the starved-inverter voltage-to-time converter, inverter-chain delay elements, and hard-coded convolution engine; and the evaluation with the architectural simulator and 65-nanometer SPICE models — the approximation-accuracy and noise analysis (power-supply-induced and random jitter), the OpenCV benchmarks (Sobel, pyrDown, GaussianBlur) with their area, energy, throughput (up to about 71 million frames per second), and accuracy figures, and the comparison to a processing-in-pixel architecture showing over 2.5 times energy savings on a small-stride filter and roughly eight times less energy than the prior state of the art overall.