Tandem Processor: Grappling with Emerging Operators in Neural Networks

Soroush Ghodrati, Sean Kinzer, Hanyang Xu, Rohan Mahapatra, Yoonsung Kim, Byung Hoon Ahn and 6 others

Neural-network accelerators are built almost entirely around matrix multiply (GEMM), because more than 99% of a network's raw operations are matmuls and convolutions — but real models are threaded with a growing zoo of NON-matmul operators (activations, pooling, softmax, transpose, element-wise math) that are only about 15% of the operator count yet, once the matmul unit is heavily optimized, come to dominate runtime (up to 73% in newer models); this paper (UC San Diego, KAIST, Google DeepMind, Illinois) adds a specialized-yet-programmable companion SIMD processor that handles all the non-matmul work and orchestrates the whole network in tandem with the matmul unit, and its key insight is that the cost of a programmable unit is the plumbing — load/store, address arithmetic, loop control — not the math, so it replaces the register file with software-managed scratchpads, computes strided addresses in hardware with iterator tables, and runs nested loops in hardware with a "code repeater", while keeping a small general set of ALU primitives that composes every operator — beating a TPU-style vector unit by 2.6x, an on-chip RISC-V design by 47.8x, and matching an A100 GPU at equal compute.

The argument in one glance

Problem

Neural accelerators are built for matmul, but networks are not only matmul

A neural processing unit (NPU) is a chip specialized to run deep networks fast, and for a decade its design has centered on GEMM — general matrix multiply, the matmuls and convolutions that make up more than 99% of a network's raw arithmetic operations. That focus was reasonable: optimize the thing that dominates the operation count, and you win. But a real network is not a pure sequence of matmuls. Threaded between the GEMM layers is a growing zoo of NON-GEMM operators — element-wise add and multiply, activation functions like ReLU and GeLU and Softmax, pooling and other reductions, transposes and reshapes. These have to run too, and they have to hand data back and forth with the GEMM unit.
What people did

Bolt on something to handle the non-GEMM operators

Existing NPUs handle the non-GEMM operators one of four ways. (1) Fall back to the off-chip CPU — fully programmable, run anything. (2) Add fixed dedicated hardware blocks — one for ReLU, one for MaxPool, one for clip, and so on. (3) Put a small general-purpose core on-chip, such as the RISC-V core in Gemmini, to run the non-GEMM layers and control things. (4) Add a general-purpose vector (SIMD) unit, as Google's TPU does with its VPU, to process the element-wise work in parallel. Each of these can, in principle, do the job.
Why it fell short

Each option fails one of three requirements — and the bottleneck is shifting onto this tail

Grade the four against what non-GEMM execution actually needs: to run IN TANDEM with the GEMM unit (no off-chip round-trips), to be both efficient AND programmable, and to orchestrate the whole network. The off-chip CPU pays PCIe round-trips and integer/float conversions — not in tandem. Dedicated blocks are not programmable and do not scale: a new operator needs a new block, and anything unsupported still falls back off-chip. The on-chip RISC-V core has a single ALU, too weak for large tensors, so it becomes the bottleneck. The vector unit works with the GEMM unit, but its register file and generic load/store and loop control burn most of its time on plumbing rather than compute. And this matters more every year: as the GEMM unit gets ever more optimized, Amdahl's law shifts the bottleneck onto the non-GEMM tail, which is only about 15% of the operator NODES yet grows to dominate runtime — up to 73% of it in newer models.
This paper's idea

A specialized-yet-programmable companion that specializes the plumbing, not the math

This paper (UC San Diego, KAIST, Google DeepMind, Illinois — Ghodrati et al.) proposes the Tandem Processor: a companion SIMD processor that handles ALL the non-GEMM layers and orchestrates the end-to-end network, running in tandem with the GEMM unit and eliminating the need for a separate CPU. Its central insight is that a programmable non-GEMM unit is slow because of PLUMBING — load/store, address arithmetic, loop control — not because of the arithmetic itself (which is cheap, and memory-bound). So it specializes the plumbing while keeping the compute general: it replaces the register file and cache with software-managed on-chip scratchpads, computes strided scratchpad addresses in hardware via iterator tables (a dedicated pipeline stage, no address-calc instructions), runs nested loops in hardware via a "code repeater" (no branch logic), and keeps a small union of ALU primitives that composes every operator. It overlaps non-GEMM work with GEMM work at tile granularity via double-buffering and fluid ownership of the GEMM unit's output buffer. The result: 2.6x over a TPU-style vector unit, 47.8x over an on-chip RISC-V design, matching an A100 GPU at equal compute — at 2.7 W, open-sourced in the GeneSys project.

Why this problem is worth solving

For most of the last decade, “make the accelerator faster” meant “make the matrix-multiply faster,” and it worked — but it worked so well that the remaining time is now spent elsewhere. Understanding the split between the two kinds of work is the key. A

GEMM (general matrix multiply) covers the matmuls and convolutions at the heart of a neural network — the dense multiply-accumulate work that dominates the raw operation count (over 99%) and that systolic-array accelerators are built to crush. A NON-GEMM operator is everything else the network needs between those matmuls: element-wise arithmetic (add, multiply, exponent), activation functions (ReLU, LeakyReLU, GeLU, Sigmoid, Softmax), reduction operators (MaxPool, AveragePool, depth-wise convolution, ReduceMean), data-layout transforms (transpose, reshape, concatenate), and type conversions (cast, bit-shift). They are wildly diverse in what they compute and how they map inputs to outputs (one-to-one for element-wise, many-to-one for reductions), and — crucially — they have GROWN in variety and in how tightly they interweave with GEMM as models have evolved from early vision networks to today’s language models

is the distinction that reframes accelerator design. The non-GEMM operators were long treated as a rounding error because they are a small fraction of the operation count; the paper’s measurements show they are not a rounding error in TIME. They are only about 15% of the operator nodes, yet they consume up to 73% of end-to-end runtime in newer networks — because once the GEMM unit is heavily optimized, whatever is left over becomes the bottleneck.

The reason a naive “just add a fast vector unit” does not fix it is that the non-GEMM operators have a fundamentally different performance character from GEMM. They are

limited by data movement rather than arithmetic. A useful way to see this is OPERATIONAL INTENSITY — the number of arithmetic operations performed per byte of data moved. GEMM has high operational intensity (it reuses each loaded value in many multiply-accumulates), so it is compute-bound and rewards more arithmetic hardware. Most non-GEMM operators are the opposite: more than 80% are element-wise, touching each value once or twice, so they perform very little compute per byte and spend their time waiting on memory. On a roofline plot they sit in the memory-bound region, far from the compute roof. The consequence for design is decisive: throwing more ALUs at them does nothing; what limits them is the cost of getting data in and out and stepping through it — the load/store instructions, the address calculations, and the loop control

. That is why the fix cannot be “more compute.” It has to be “less plumbing” — and identifying plumbing, not arithmetic, as the real cost is what makes the Tandem Processor’s design fall out cleanly.

What people did before, and where it broke

Off-chip CPU fallback. Send non-GEMM layers to the host CPU over PCIe. Ultimate programmability, runs anything — but pays for round-trip data transfer and integer/float conversions, and cannot execute in tandem with the GEMM unit, so the accelerator stalls waiting.
Dedicated on-chip blocks. A ReLU unit, a MaxPool unit, a clip unit, tightly coupled to the GEMM unit. Efficient and in-tandem for the operators they cover — but not programmable and not scalable: each new operator needs new silicon, and any unsupported or emerging operator still falls back to the off-chip CPU.
On-chip RISC-V core (e.g. Gemmini). A general-purpose core runs the non-GEMM layers and controls resources, avoiding off-chip trips. Programmable, but its single ALU is far too weak for the large tensors non-GEMM layers touch, so the core itself becomes the bottleneck; and datatype-cast and layout-transform overheads remain.
General-purpose vector/SIMD unit (e.g. TPU’s VPU). Parallel, works alongside the GEMM unit. But it inherits register-file and cache semantics, so it drowns in memory instructions: the register-file load/store, the per-access address arithmetic, and the loop control eat most of its runtime, and it lacks execution control.
The gap. No option satisfies all three requirements at once — in tandem with GEMM, balanced efficiency and programmability, and orchestrating the whole network. That is the gap the Tandem Processor is built to close.

How the Tandem Processor grapples with non-GEMM

Diagnose the plumbing, specialize it, keep the ALU general, and run in tandem

Diagnose where the time really goes

Non-GEMM operators are only ~15% of operator nodes but grow to dominate runtime (up to 73% in newer models), and they are memory-bound. Measuring a programmable unit shows the time is spent on PLUMBING: register-file load/store is 41% of non-GEMM runtime, address calculation 59%, and loop control 70% — the ALU sits idle waiting. The insight: specialize the plumbing, not the math.
↓ →
Replace the register file with software-managed scratchpads new

Because non-GEMM tensors have statically-known shapes laid out in strided patterns, the Tandem Processor drops the vector register file and cache entirely and streams tensors from off-chip directly into single-level on-chip scratchpads ("Namespaces"), populated and drained by a Data Access Engine at tile granularity — no register-file shuffling.
↓ →
Compute strided addresses in hardware with iterator tables new

Each scratchpad access is a compact (Scratchpad ID, Iterator Index) pair: the ID selects an iterator table, the index points to a stored (Offset, Stride) tuple. A dedicated pipeline stage computes offset-plus-stride addresses in PARALLEL with compute, so strided addressing fits in a single instruction and adds no runtime overhead.
↓ →
Run nested loops in hardware with the Code Repeater new

Non-GEMM layers are nested loops of primitive ops with pre-determined iteration counts. A software-managed "Code Repeater" in the fetch stage is configured once with the counts and up to eight nesting levels, then drives the loops in hardware — removing branch prediction and loop overhead, and combining loop control with the strided address calculation.
↓ →
Keep a small general union of ALU primitives new

Rather than one dedicated instruction per operator (which does not scale) or fully generic arithmetic (which wastes hardware), it implements a UNION of simple primitives — add, multiply, multiply-accumulate, min/max, shift, sign, absolute, compare, conditional-move — that composes every operator (GeLU from a few multiplies plus a sign, an absolute, and a minimum). INT32 precision suffices for all of them, and the compiler lowers complex ops to integer primitives.
↓ →
Run in tandem with the GEMM unit and orchestrate the network new

It overlaps non-GEMM work on one tile with GEMM work on the next via double-buffering and tile-granularity coordination, and takes FLUID OWNERSHIP of the GEMM unit's output buffer instead of copying data out. Synchronization instructions handshake between the two units, and the Tandem Processor itself orchestrates the whole DNN — no separate CPU needed.

Implemented in RTL (GlobalFoundries 65nm and FreePDK 15nm at 1 GHz), evaluated on seven networks from VGG-16 to BERT and GPT-2. The GEMM unit is a 32-by-32 systolic array; the Tandem Processor is a 32-lane INT32 SIMD unit with 128 KB of scratchpad. Open-sourced in the GeneSys project.

The idea

The cost of a programmable unit is the plumbing, not the math — so specialize the plumbing. This is the paper’s pivot. A vector unit running memory-bound operators spends almost nothing on arithmetic; its time goes to moving values through a register file, computing an address for every access, and running the loop that walks the tensor. The measured overheads are stark: register-file load/store is 41% of non-GEMM runtime, address calculation 59%, loop control 70%. So instead of the usual accelerator move — add more or fancier compute — the Tandem Processor leaves the ALU simple and attacks the three plumbing costs directly: scratchpads replace the register file, iterator tables compute strided addresses in a pipeline stage, and a code repeater runs nested loops in hardware. This the-real-cost-is-the-plumbing diagnosis is one the site keeps returning to under a different name — the observation that data movement and control, not arithmetic, dominate memory-bound work, the same reason the dynamic-tiling and graph-processing pages redesign around access patterns rather than FLOPs. Reifying strided addressing and loop nesting as explicit hardware structures is what turns those overheads to near zero.

Refuse the choice between specialized and programmable. The four existing approaches line up on a false dichotomy: dedicated blocks are efficient but rigid, while CPUs, RISC-V cores, and vector units are flexible but inefficient. The Tandem Processor rejects the choice by splitting where each property is needed — it is SPECIALIZED exactly in the memory-access and control logic (scratchpads, iterator tables, code repeater), where specialization buys efficiency, and GENERAL exactly in the ALU (a small union of primitives), where generality buys coverage of the operator zoo. That is the reject-the-false-dichotomy, hybridize discipline the site rewards elsewhere — the on-chip-memory-allocation page fusing a fast heuristic with an exact solver, the accelerator- mapping page combining simulated annealing with an ILP — where the win comes from applying each tool only to the sub-problem it fits, rather than forcing one tool to do everything. Programmability where the diversity lives, specialization where the overhead lives.

Run alongside, not after — overlap in time and keep data local. A companion that finishes the non-GEMM work only after the GEMM unit idles would just move the bottleneck. So the Tandem Processor is designed to overlap: while the GEMM unit computes tile N+1, the Tandem Processor consumes tile N, coordinated at tile granularity through double-buffering, and it takes fluid ownership of the GEMM unit’s output buffer so the intermediate data never has to be copied between units. This overlap-and-share-in-time discipline — pipeline two units so neither waits, and communicate at a fine granularity that keeps data on chip — is the same one the concurrent-graph-traversal and streaming- dataflow pages use to hide latency behind useful work, and the fuse-to- keep-activations-on-chip idea the tiling literature relies on. Tandem execution, not sequential hand-off, is what lets specializing the non-GEMM path actually speed up the whole network.

Turning a matmul chip into a full-network engine

From the neglected 15% to an end-to-end win

Adding a non-GEMM companion to a 32-by-32 systolic GEMM unit. Numbers from the paper's RTL evaluation on seven networks.

Step 1 of 5

The problem hiding in 15% of the nodes

Non-GEMM operators are a small fraction of the operator count but grow to dominate runtime as the GEMM unit is optimized, and they are memory-bound — element-wise work that touches each value once or twice. Optimizing GEMM harder does nothing for them; they need a memory-optimized design of their own.

only ~15% of operator nodes are non-GEMM, yet they take up to 73% of end-to-end runtime in newer models
more than 80% of non-GEMM operators are element-wise; on a roofline they sit in the memory-bound region

Why it works

Non-GEMM operators are memory-bound and only ~15% of operator nodes, yet grow to dominate runtime as the GEMM unit is optimized → the leftover non-GEMM tail becomes the Amdahl bottleneck → a memory-optimized companion for it, not more GEMM compute, is what speeds up the whole network.
A programmable unit’s time goes to load/store, address arithmetic, and loop control, not arithmetic → specializing those three (scratchpads, iterator tables, code repeater) removes the dominant overhead → the ALU stops waiting and the memory-bound work runs near its data-movement limit.
Non-GEMM tensors have statically-known shapes and loop counts → strided addresses and nested loops can be pre-configured and computed in hardware in parallel with compute → strided addressing and loop control cost no extra instructions at runtime.
A small union of ALU primitives composes every operator → an emerging operator is a new program rather than new silicon → the design scales to the growing operator zoo without the rigidity of dedicated blocks.
The companion overlaps non-GEMM on one tile with GEMM on the next and takes ownership of the output buffer → neither unit waits on the other and intermediate data stays on chip → specializing the non-GEMM path translates into an end-to-end speedup instead of a moved bottleneck.

What it costs (trade-offs and scope)

INT32 is a fixed commitment for the non-GEMM ALU. It is accurate enough for every operator studied and avoids a cast on the GEMM-to-non-GEMM boundary, but it is a design choice; when the GEMM unit runs at lower precision, a datatype cast is still needed on the non-GEMM-to-GEMM boundary.
The efficiency relies on static structure. Iterator tables and the Code Repeater are pre-configured from statically-known tensor shapes and loop counts, and loop nesting is capped at eight levels; workloads with data-dependent shapes would lean harder on the compiler.
The compiler does heavy lifting. It must tile without splitting GEMM reduction dimensions (or the GEMM unit produces partial results that stall the Tandem Processor), pick tile sizes that fit the scratchpad yet hold whole patches (for example an entire 5-by-5 depth-wise window), and relax dependencies via loop fission and interchange. Complexity moves from hardware into the compiler.
It is a companion, not a standalone chip. It is co-designed with a GEMM unit and takes fluid ownership of that unit’s output buffer, so it is not a drop-in accelerator on its own — it presumes the systolic partner it runs in tandem with.
Specializing the plumbing still costs area and energy. In the Tandem Processor itself the ALU logic is 56.6% of area and the nested-loop and address-calculation logic is about 40% of energy — the specialization is not free; it pays off by keeping the far more expensive GEMM unit continuously fed.

The evaluation

RTL implementation synthesized in GlobalFoundries 65nm and FreePDK 15nm at 1 GHz, a cycle-accurate simulator validated against the RTL (within 5%), and on-chip memory energy from CACTI-P. Seven networks (VGG-16, ResNet-50, MobileNetv2, Yolov3, EfficientNet, BERT, GPT-2) at batch size one, comparing against off-chip CPU fallback, dedicated blocks, an on-chip RISC-V design (Gemmini), a TPU-style vector unit (VPU), and Jetson Xavier, RTX 2080 Ti, and A100 GPUs.

Four ways to run the non-GEMM operators

Property	Off-chip CPU / dedicated blocks	On-chip RISC-V / vector unit	Tandem Processor
Programmable for new operators
Runs in tandem with GEMM
Where the time goes
Scales to the operator zoo
End-to-end result

No prior class runs in tandem with the GEMM unit AND stays programmable AND avoids drowning in plumbing. The Tandem Processor specializes the memory access and control while keeping the ALU general, beating the on-chip RISC-V design by 47.8x, the TPU-style vector unit by 2.6x (1.4x less energy), and matching an A100 GPU at equal compute while drawing 2.7 W.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The headline numbers are ratios against specific baselines under stated conditions, so read them in context. The 2.6x over a TPU-style vector unit is modeled from Google’s VPU patent rather than measured silicon; the 47.8x over Gemmini and 3.5x/2.7x over CPU-fallback and dedicated- block baselines use the same GEMM unit, frequency, and on-chip memory across designs for fairness; and the A100 comparison is iso-TOPs (the Tandem design scaled up to match) — it matches A100 with TensorRT and is 4.0x over A100 CUDA, with a 3.4x advantage on non-GEMM operators alone, not a claim of beating a datacenter GPU outright. The efficiency assumes statically-known tensor shapes and loop counts (so iterator tables and the Code Repeater can be configured ahead of time), caps loop nesting at eight levels, commits the non-GEMM ALU to INT32, and leans on a compiler that must tile carefully and relax dependencies. And it is a companion, co-designed with a systolic GEMM unit whose output buffer it takes over, not a standalone part. Within that scope the message is clean and general: for memory-bound work the bottleneck is the plumbing — load/ store, address arithmetic, loop control — so specialize THAT and keep the ALU programmable, and a small overlooked fraction of the operator graph stops being the thing that caps the whole accelerator.

Background

Chase the shifted bottleneck, specialize the plumbing, and refuse the false choice

This paper sits in the site’s accelerator-and-ML-systems cluster and rhymes with several ideas covered elsewhere on this site. Its starting move is an Amdahl argument: optimize the obvious 99% (GEMM) hard enough and the bottleneck shifts onto the neglected tail (non-GEMM, only ~15% of nodes but most of the runtime) — the same chase-the-real-bottleneck, measure-where-time-actually-goes discipline that drives the pages which insist on profiling the true cost rather than the assumed one, and a reminder that the thing worth accelerating is whatever is left after you accelerate the obvious thing. Its core technique is to recognize that for memory-bound work the cost is the plumbing, not the arithmetic, and to reify strided addressing and loop nesting as explicit hardware (iterator tables, a code repeater) — the same data-movement-and-control- dominate insight, and the same make-the-implicit-explicit move, that the dynamic-tiling and graph-processing pages use when they redesign around access patterns instead of FLOPs. Its structure refuses the false dichotomy between specialized and programmable by splitting where each belongs — specialized memory access, general ALU — exactly the hybridize- two-tools discipline the on-chip-memory-allocation and accelerator- mapping pages use when they fuse a heuristic with a solver, or annealing with an ILP. Its integration overlaps non-GEMM with GEMM in time and keeps intermediate data on chip through fluid buffer ownership, the same share-work-in-time and fuse-to-stay-local ideas the concurrent-graph and streaming-dataflow pages rely on to hide latency. And its ALU is a union of primitives that composes every operator rather than a block per operator — the same build-a-small-composable-primitive-set-then-compose discipline behind the graph-compiler and control-flow-attack pages’ abstractions. Together they make the Tandem Processor a study in attacking the bottleneck that specialization itself creates.

If you remember only three things

1. Neural accelerators are built around GEMM (matmul and convolution) because it is more than 99% of a network's raw operations, but real networks are threaded with a growing zoo of NON-GEMM operators — activations, pooling, softmax, transpose, element-wise math — that are memory-bound and, once the GEMM unit is heavily optimized, come to dominate runtime: only about 15% of operator nodes yet up to 73% of end-to-end time in newer models. Existing options each fail a requirement: off-chip CPU (not in tandem), dedicated blocks (not programmable), on-chip RISC-V (too weak, becomes the bottleneck), or a vector unit (drowns in load/store and loop overhead).
2. The Tandem Processor is a specialized-yet-programmable companion SIMD processor whose key insight is that the cost of a programmable unit is the PLUMBING — load/store (41% of non-GEMM runtime), address calculation (59%), and loop control (70%) — not the math. So it specializes the plumbing while keeping the ALU general: software-managed scratchpads replace the register file, iterator tables compute strided addresses in a pipeline stage, a "code repeater" runs nested loops in hardware, and a small union of INT32 primitives composes every operator (so a new operator is a new program, not new silicon). It runs in tandem with the GEMM unit, overlapping tiles via double-buffering and taking fluid ownership of the output buffer.
3. This balanced design beats a TPU-style vector unit by 2.6x (1.4x less energy), an on-chip RISC-V design (Gemmini) by 47.8x, and CPU-fallback and dedicated-block baselines by 3.5x and 2.7x, while matching an A100 GPU at equal compute (4.0x over A100 CUDA, 3.4x on non-GEMM alone) at just 2.7 W and open-sourced in GeneSys. The transferable lessons: when you optimize the obvious majority, the bottleneck shifts to the neglected tail; for memory-bound work specialize the data movement and control, not the arithmetic; and refuse the false choice between specialized and programmable by applying each where it belongs.

Check yourself

Check your understanding

1. Non-GEMM operators are only about 15% of a network's operator nodes, yet the paper argues they are worth building special hardware for. Predict why such a small fraction matters so much.
2. The Tandem Processor keeps its ALU simple and instead specializes the load/store, address, and loop machinery. Predict why that is the right place to specialize for these operators.
3. The four existing approaches trade off programmability against efficiency. Predict how the Tandem Processor gets both instead of picking one.
4. A companion processor that handled non-GEMM work only after the GEMM unit finished each layer would gain little. Predict what the Tandem Processor does instead, and why it matters.

Sources

Tandem Processor: Grappling with Emerging Operators in Neural Networks (author PDF) — ASPLOS 2024 (full text read).
ACM DL: Tandem Processor (DOI 10.1145/3620665.3640365) — published version.
GeneSys project (open-source NPU generator and compiler) — RTL, compiler, and artifacts.