TinyForge: A Design Space Exploration to Advance Energy and Silicon Area Trade-offs in tinyML Compute Architectures with Custom Latch Arrays

Tiny AI chips for always-on gadgets — a smart doorbell, a hearing aid — live or die on two numbers, how much energy each inference costs and how much silicon it takes; this work from Stanford observes that a tinyML network is small enough to keep all its weights on the chip, avoiding the costly trips to off-chip memory that dominate energy, and builds tinyForge, a design-space explorer plus a hand-designed custom latch memory and bit-serial compute, that keeps every weight on-chip and cuts energy per inference by 7.5 times versus off-chip and by up to 1.8 times more, at up to 3.7 times less area, than conventional on-chip designs.

The argument in one glance

Problem

Tiny AI chips are dominated by the energy of fetching weights from off-chip memory

A tinyML accelerator is judged on energy per inference and silicon area. Most designs and design tools were built for large cloud models that must stream weights from off-chip DRAM or Flash every inference, and those accesses cost far more energy than the arithmetic — so on a tiny device, memory energy dominates.
What people did

Reuse cloud-style accelerators and design tools

Apply the existing accelerator architectures and design-space exploration frameworks, which optimize a memory hierarchy that fetches each layer from off-chip memory and rely on analytical cost models.
Why it fell short

They assume off-chip access and inaccurate cost models

These tools assume every layer is read from off-chip memory each inference, which prevents low-energy operation; they use analytical models that misestimate real optimized hardware; and they never explore storing all weights on-chip in register files next to the compute — exactly what a small tinyML model makes possible.
This paper's idea

Keep all weights on-chip, and co-explore memory and compute (tinyForge)

tinyForge is a design-space exploration that co-optimizes memory, compute, fabric, and dataflow to find energy-area Pareto-optimal tinyML accelerators. It introduces a hand-designed custom latch array reaching close-to-SRAM density (8 square microns per byte, 11 femtojoules per byte read energy), and pairs bit-serial compute with tuned combinational and systolic scaling. Storing all weights on-chip cuts energy per inference by 7.5 times versus off-chip, and the custom latch arrays with bit-serial compute cut up to 1.8 times energy, 2.2 times latency, and 3.7 times area.

Why this problem is worth solving

The devices tinyML targets are so constrained that two numbers govern the whole design.

tinyML is the practice of running deep neural networks on the smallest and cheapest computing devices — always-on sensors, wearables, smart-home gadgets — at extremely low power, from microwatts to milliwatts. The models are small by design, roughly 50 to 500 kilobytes, and run at low clock frequencies, from about 1 to 100 megahertz. For a chip to be deployable at this scale, it has to hit tight targets: an inference must complete within milliseconds and cost only microjoules to millijoules of energy, and because the cost per unit cannot exceed a few tens of cents, the accelerator should occupy less than about 2 square millimeters of silicon in a low-cost process. So the two metrics that decide whether a tinyML accelerator is viable are energy per inference and silicon area, and they trade off against each other. A design that saves energy by adding hardware can blow the area budget; one that saves area can cost energy. Understanding and navigating that trade-off is the whole game, and it is different from the concerns of the large accelerators that dominate the literature.

The reason existing designs waste energy is that they keep fetching weights from far away.

A neural network runs by multiplying inputs against its learned weights, layer after layer. Large accelerators, built for cloud models of tens to hundreds of megabytes, cannot hold all those weights on the chip, so they store them in off-chip memory — DRAM or Flash — and stream each layer in as it is needed, on every single inference. The catch is that reading a value from off-chip memory costs far more energy than the multiply that uses it, so for a small always-on device the energy is dominated by these memory accesses, not by the arithmetic. tinyML models, though, are small enough that all of their weights can fit on the chip, and because the device runs the same network continuously, those weights can simply stay there and be reused. That is a fundamentally different opportunity, and prior design-space exploration tools miss it: they assume every layer is fetched from off-chip memory, they lean on analytical cost formulas that misrepresent real optimized hardware, and they never explore the option of storing all the weights in fast register files placed right next to the compute elements.

What people did before, and where it broke

Use cloud-scale accelerator architectures. Adopt designs optimized for large models needing high throughput, storing weights off-chip. Where it broke. They ignore the tinyML regime and omit off-chip memory access energy, which is exactly what dominates a tiny device’s energy budget.
Chase peak compute efficiency alone. Optimize in-memory-computing arrays for the highest operations per second per watt of the compute array. Where it broke. Peak compute numbers ignore the memory hierarchy, the resulting accesses, and array under-utilization, which together determine real energy per inference.
Explore designs with analytical models assuming off-chip weights. Use existing design-space tools that map workloads onto architectures fetching each layer from off-chip memory. Where it broke. The analytical models misestimate optimized hardware, and assuming off-chip access forecloses the biggest tinyML win — keeping all weights on-chip next to the compute.
The gap. No tool co-optimized memory, compute fabric, and dataflow for the tinyML regime with accurate hardware models, exploring on-chip weight storage and custom dense memories. Building exactly that is what tinyForge does.

How tinyForge works

From streaming weights off-chip to keeping the whole network next to the compute

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Keep the whole network on-chip, right next to the compute. tinyForge’s central insight is that because a tinyML model is small and reused continuously, all its weights can be stored on the chip and never fetched from off-chip memory, eliminating the accesses that otherwise dominate energy — worth a 7.5 times reduction in energy per inference over an off-chip design. Removing the costly, repeated movement of data from far away by holding it close to where it is used is the same principle the warehouse-scale memory-traffic work covered elsewhere on this site is built on, recognizing that the transfers a system does not make are where the savings are and arranging for data to stay where it will be reused rather than shuttling it back and forth.

Compute one bit at a time to shrink the arithmetic. Rather than bit-parallel multipliers that process a full number at once, tinyForge explores bit-serial compute, which processes one bit per cycle with a far simpler and smaller multiplier; combined with tuned combinational and systolic scaling, this cuts the area per multiplier by 4.8 times and the energy per multiply-accumulate by 2.3 times. Trading full-width, expensive arithmetic for a leaner, lower-precision-style compute primitive that the workload can tolerate, in order to save energy and area, is the same bet the edge-training work covered elsewhere on this site makes, running its computation in a reduced numeric style to unlock the efficiency of the hardware rather than paying for full-width operations everywhere.

Hand-design a custom memory that beats the synthesized one. To make on-chip storage next to the compute practical, tinyForge introduces a custom latch array — a bespoke circuit reaching close-to-SRAM density and low read energy, where an automatically synthesized latch array is many times less dense, and it is validated in a silicon prototype. This density is what lets the entire network sit in register files inside the compute array. Building a targeted, custom hardware block that outperforms the default automatically-generated design in efficiency is the same move the secure-undervolting work covered elsewhere on this site makes, reworking a specific piece of the circuit by hand so that it does its job at far lower cost than the generic implementation would allow.

Following an accelerator design through tinyForge

Designing an accelerator for an image-classification workload

A trace of how tinyForge searches for a Pareto-optimal tinyML accelerator, following the paper's method.

Step 1 of 5

Start from a workload and two goals

Take a tinyML workload — image classification from the MLPerf Tiny benchmark — and the goal of minimizing both energy per inference and silicon area, under a small area budget.

Energy and area are the two metrics that matter.

Why it works

Off-chip memory access dominates energy. Because reading weights from off-chip DRAM or Flash costs far more energy than the multiply that uses them → an accelerator that fetches weights every inference is dominated by memory energy → so eliminating off-chip access is the single largest win.
tinyML models are small enough to keep on-chip. Because a tinyML network is only tens to hundreds of kilobytes and is reused continuously → all its weights can fit and stay on-chip next to the compute → so the costly off-chip fetches can be removed entirely, saving 7.5 times the energy.
Bit-serial compute is leaner than bit-parallel. Because processing one bit at a time uses a far simpler multiplier → the area per multiplier and the energy per multiply-accumulate drop sharply → so the compute array shrinks and uses less energy for a small latency cost.
A custom memory cell beats the synthesized one. Because a hand-designed custom latch array reaches close-to-SRAM density where a synthesized one is many times less dense → the whole network can be held in register files inside the compute array → so weights sit right next to the multipliers with minimal data movement.
Joint search beats optimizing parts in isolation. Because memory, compute, fabric, and dataflow interact and the space is astronomically large → co-optimizing them with a genetic algorithm finds Pareto-optimal designs in minutes → so the best trade-offs emerge that point-by-point tuning would miss.

What it costs (trade-offs and scope)

On-chip storage costs area for larger models. Keeping all weights on-chip wins clearly for small models, but memory-bound workloads with larger models (more than about 270 kilobytes, such as Visual Wake Words and Anomaly Detection) incur significant area overhead when stored in register files, so the energy win comes at an area cost.
The custom latch array has layout limits. The custom latch array’s bespoke circuit layout lacks some of the automated layout conveniences of standard cells, which contributes to area overhead for the larger, memory-bound workloads.
SRAM sometimes wins the low-area region. The custom-latch-array design gives the best energy-area trade-off in the higher-area part of the Pareto front, but a conventional on-chip SRAM design with in-memory compute is better in the low-area region, so neither dominates everywhere.
A general-purpose chip pays a modest penalty. A single architecture optimized across all the benchmark workloads, rather than one chip per workload, costs only a 2 to 18 percent energy penalty but needs 1.4 to 3 times the area to accommodate the range of model sizes.
The results are from modeling, not a full taped-out accelerator. The custom latch array itself was validated in a silicon prototype, but the system-level energy, area, and latency numbers come from post-synthesis characterization in a 28-nanometer process and design-space modeling, with the in-memory-compute figures taken from prior published work.
It targets the tinyML regime specifically. The design points and conclusions are shaped by small models, low frequencies, and tight area budgets; they do not carry over to large, high-throughput accelerators, which face a different balance of memory and compute.

The evaluation

Three tinyML accelerators by where the weights are stored

Property	Off-chip weights	On-chip SRAM
Where the weights live
Energy per inference
Data movement		On-chip only
Density of the weight store	Not applicable
Best suited to		Broad range, low-area region

Three tinyML accelerator styles distinguished by where the network weights are stored, as studied by tinyForge. Storing weights off-chip (the approach cloud-scale accelerators use) makes energy per inference memory-dominated. Storing all weights on-chip in SRAM with digital in-memory compute cuts energy per inference by 7.5 times versus off-chip. Storing the whole network in a hand-designed custom latch array inside the compute array (8 square microns per byte density, 11 femtojoules per byte read energy, validated in a silicon prototype) can do better still in the higher-area region — for image classification, 1.6 times lower energy at equal area or 3.7 times lower area at equal energy versus the SRAM design — while the SRAM design wins the low-area region. Across the study, the custom latch arrays and bit-serial compute yield up to 1.8 times lower energy, 2.2 times lower latency, and 3.7 times smaller area. The paper is honest that memory-bound workloads with larger models incur area overhead when stored in register files, and that a single general-purpose architecture across all workloads costs only a 2 to 18 percent energy penalty but 1.4 to 3 times the area.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

tinyForge is a careful, quantitative piece of hardware-software co-design aimed at a real regime the mainstream had overlooked. Its foundational observation is simple and correct: because tinyML models are small and reused continuously, all their weights can live on-chip, and since off-chip memory access dominates the energy of a tiny device, keeping the weights on-chip is worth a large, 7.5 times energy reduction. From there the paper does three things well. It builds a design-space exploration that co-optimizes memory, compute fabric, and dataflow with accurate post-synthesis hardware models rather than crude analytical formulas, and searches the space efficiently with a genetic algorithm. It shows that bit-serial compute plus tuned combinational and systolic scaling shrink the arithmetic substantially. And it contributes a hand-designed custom latch array, validated in silicon, dense enough to hold the whole network right beside the multipliers. The results are attractive — up to 1.8 times lower energy, 2.2 times lower latency, and 3.7 times smaller area — and, importantly, the paper is candid about the trade-offs. On-chip register-file storage wins for small models but costs area for larger, memory-bound workloads; the custom latch array wins the higher-area part of the Pareto front while conventional SRAM wins the low-area part; and a single general-purpose chip across all workloads is nearly as efficient but larger. Read the scope precisely: the custom latch array is validated in silicon, but the system-level numbers come from post-synthesis characterization and modeling in a 28-nanometer process, and the whole analysis is tuned to the tinyML regime of small models, low frequencies, and tight area, not to large accelerators. The durable idea is that when a workload is small enough to change what is possible — here, small enough to keep entirely on-chip — the right response is to rethink the memory hierarchy and the compute together, not to shrink a design meant for a different scale.

Background

Why co-designing around data locality keeps winning

It helps to see tinyForge within a recurring theme in computer systems: the biggest efficiency gains come not from faster arithmetic but from moving data less, and from co-designing the hardware around where the data lives. Here the decisive move is keeping the entire network on-chip so that weights never make the expensive trip from off-chip memory, and then packing them into a custom dense memory placed right next to the compute. The traversal-synthesis work covered elsewhere on this site pursues a related discipline from the software side, planning how computation walks over structured data so that the work and the state stay as lean as the problem allows, rather than paying for movement and bookkeeping that a smarter arrangement avoids. The common thread is that a system performs best when its structure is designed around the shape and size of its data — when the data is small enough to keep close, keep it close; when it must be traversed, traverse it frugally. tinyForge adds a second lesson worth carrying: the value of exploring an enormous design space with a principled optimizer instead of intuition or exhaustive search, co-optimizing choices that interact rather than tuning them one at a time. As specialized accelerators proliferate across every scale of device, both instincts — design around data locality, and search the space of designs intelligently — are likely to matter more, not less.

If you remember only three things

1. tinyML accelerators run deep neural networks on tiny, cheap, power-starved devices, where the two decisive metrics are energy per inference and silicon area (budgets on the order of microjoules and a couple of square millimeters). Most accelerator designs and design tools were built for large cloud models that must stream weights from off-chip DRAM or Flash every inference, and those off-chip accesses cost far more energy than the arithmetic, so they dominate a tiny device's energy. Existing design-space tools assume off-chip access, use inaccurate analytical cost models, and never explore keeping all weights on-chip next to the compute — which a small tinyML model makes possible.
2. tinyForge is a design-space exploration that co-optimizes memory, compute fabric, and dataflow to find energy-area Pareto-optimal tinyML accelerators, using a parameterized template, accurate post-synthesis 28-nanometer hardware estimates, a dataflow optimizer, and a genetic algorithm that finds the Pareto front in under 1000 evaluations (about 20 minutes) instead of the ten-to-the-sixteenth an exhaustive search needs. It contributes a hand-designed custom latch array reaching close-to-SRAM density (8 square microns per byte, 11 femtojoules per byte read energy, validated in a silicon prototype), dense enough to hold the whole network in register files inside the compute array, and it pairs bit-serial compute with tuned combinational and systolic scaling to cut area per multiplier by 4.8 times and energy per multiply-accumulate by 2.3 times.
3. Storing all weights on-chip cuts energy per inference by 7.5 times versus off-chip, and the custom latch arrays with bit-serial compute cut up to 1.8 times energy, 2.2 times latency, and 3.7 times area beyond conventional on-chip designs (evaluated on the MLPerf Tiny benchmark and BERT-Tiny). The paper is candid about scope: on-chip register-file storage costs area for larger memory-bound models, the custom latch array wins the higher-area Pareto region while SRAM wins the low-area region, a general-purpose chip across all workloads costs only a 2 to 18 percent energy penalty but 1.4 to 3 times the area, and the system-level numbers come from post-synthesis modeling (the latch array itself is silicon-validated). The durable lesson is that when a workload is small enough to change what is possible, the right response is to co-design the memory hierarchy and compute together around data locality, not to shrink a design meant for a different scale.

Check yourself

Check your understanding

1. Why does memory, not arithmetic, dominate the energy of a conventional tinyML accelerator?
2. What opportunity do tinyML models offer that large cloud models do not, and how does tinyForge exploit it?
3. What are the custom latch array and bit-serial compute, and why do they help?
4. The paper reports that neither the custom-latch-array nor the SRAM design is best everywhere. What does this show?

Sources

The paper is “TinyForge: A Design Space Exploration to Advance Energy and Silicon Area Trade-offs in tinyML Compute Architectures with Custom Latch Arrays” by Massimo Giordano, Rohan Doshi, Qianyun Lu, and Boris Murmann, published at ASPLOS 2024, DOI 10.1145/3620666.3651328. The authors are at Stanford University.

This explainer is based on the full text of the paper, obtained as the open-access (CC-BY) version of record (retrieved from an archived copy of the publisher’s open-access PDF). The problem framing (the tinyML regime of small models, low frequencies, microwatt-to-milliwatt power, tight energy and area budgets, and low unit cost; the dominance of off-chip memory access energy; and the limitations of existing design-space exploration tools that assume off-chip access and use analytical cost models) the contributions (the custom latch array with its close-to-SRAM density and low read energy validated in a silicon prototype; the parameterized hardware template; the exploration of synthesized standard-cell compute versus digital in-memory compute, bit-serial versus bit-parallel, and combinational versus systolic scaling; the dataflow optimizer built on ZigZag; and the genetic-algorithm multi-objective optimization), and the results (the 7.5 times energy reduction from on-chip versus off-chip weight storage; the 4.8 times area and 2.3 times multiply-accumulate energy reductions from bit-serial compute and scaling; the up-to 1.8 times energy, 2.2 times latency, and 3.7 times area reductions; the image-classification iso-area and iso-energy comparisons; the memory-bound workload area overheads; the general-purpose versus single-purpose penalty of 2 to 18 percent energy and 1.4 to 3 times area; and the design-space search efficiency) are all drawn directly from the paper, evaluated on the MLPerf Tiny Inference Benchmark and the BERT-Tiny model. Background explanations of neural-network layers, memory hierarchies, in-memory computing, bit-serial arithmetic, and Pareto optimization are standard and provided to make the paper’s contribution understandable.