PICACHU: Plug-In CGRA Handling Upcoming Nonlinear Operations in LLMs

For a decade, accelerating a neural network meant accelerating matrix multiplication, because that was almost the entire cost — but in large language models the OTHER operations, the ones that reshape and normalise numbers rather than multiply matrices, have grown to take up to 46.3 percent of inference time, since they need exponentials and logarithms that hardware computes badly; building fixed circuits for today's handful of them fails because models keep inventing new ones, and approximating them in cheap integer arithmetic the way earlier models allowed destroys a large language model outright, taking perplexity from 6.75 to 70,000; this paper (NYU, ASU and Northeastern) instead plugs a small grid of reconfigurable tiles into the existing matrix engine, reusing its output memory, and computes each function by splitting the input into a part the number format already gives away for free and a remainder small enough for a four-term polynomial.

Why this problem is worth solving

The shift here is easy to miss because it inverts a long-settled assumption.

The architecture the paper reaches for is less familiar than a chip or an FPGA.

What people did before, and where it broke

• Dedicated circuits for the operations that matter. Accelerators ship with purpose-built softmax and normalisation units, and transformer accelerators add more. For the operations they implement these are excellent, and one such design matches this paper on GPT2-XL and OPT-13B.
• But the operation set keeps moving. LLaMA2-13B uses SwiGLU and RMSNorm. Faced with operations its circuits do not implement, that accelerator offloads them to an on-chip general-purpose core and its latency rises sharply. Fixed units encode the state of model design at tape-out.
• Replace each function with a cheap polynomial in integer arithmetic. This is well-established for earlier networks, with methods designed for BERT-scale models widely used, and it avoids hardware changes entirely.
• But language models will not tolerate it. Applied to LLaMA-7B with the linear layers left untouched, one established method takes perplexity from 6.75 to about 70,000. Another, better-behaved method still moves it from 6.75 to 9.67, roughly 43 percent worse. These are not tuning failures; the models are simply far more sensitive than those the methods were validated on.
• Approximate the functions with small neural networks. This generalises to new functions, which is the right instinct, but it requires training a network per function and has not been shown reliable inside a language model.
• Build a general processor for the non-matrix work. One prior design targets exactly this. But it approximates using the same integer methods above and its accuracy was never evaluated, which for operations this sensitive leaves the central question unanswered.
• The gap. Every approach optimised one corner: fixed circuits chose efficiency and lost flexibility, polynomial methods chose cheapness and lost accuracy, and the general processor chose coverage without establishing that its results were usable. Nothing had to satisfy all three at once, and nobody had checked whether an architecture already existed that could.

How a reconfigurable grid absorbs an unknown operation set

The idea

Ask what the operations have in common instead of what they are. The trap in this problem is that the list of nonlinear operations is genuinely open-ended — GeLU, GeGLU, SwiGLU, LayerNorm, RMSNorm, RoPE, and whatever appears next — so any design enumerating them is dated before it ships. The paper sidesteps this by characterising the CLASS rather than the members. All of them are built from a small set of mathematical primitives; all of them are loop-shaped, splitting cleanly into element-wise loops and reduction-then-element-wise loops; and all of them do many operations per byte fetched. That third property is the surprising one, because the standard roofline analysis, applied at the level of whole tensors, files these operations as memory-bound. Counting individual computations against individual memory accesses instead gives intensities from 5.3 up to 14.5 — each value is read once, worked on heavily, and written back. The conclusion about what hardware to use follows from the class, not the list. Discovering that an established model gives a misleading answer because it is applied at the wrong granularity is the same look-closer-at-the-measurement lesson covered elsewhere on this site where a metric everyone trusted stopped describing the situation it was invented for.

Let the number format do half the arithmetic. The algorithm underneath is the most elegant part and is worth stating plainly. To compute an exponential without an exponential unit, rewrite it as a power of two and split that exponent into a whole number and a fraction. The whole-number part costs nothing at all, because a floating-point number already stores a power of two in its exponent field — raising two to an integer is writing that integer into a field. The fractional part is confined to a range less than one wide, and over such a range a four-term polynomial is extremely accurate, which is exactly what the array’s adders and multipliers can evaluate. Logarithms split the same way into exponent and mantissa; sines and cosines range-reduce into a narrow interval first. One small hardware module performs the split, costing 1.7 percent of a tile’s area. The principle is general: divide the input so that one part is free in the representation you already have and the rest is small enough for a cheap approximation to be exact enough. Re-encoding a problem so that an expensive operation is replaced rather than optimised is the same change-the-representation move covered elsewhere on this site where the fix is not a faster circuit for the hard operation but an arrangement in which the hard operation is no longer needed.

Plug in rather than build alongside. The integration detail is easy to skip and does a lot of work. A conventional reconfigurable array wants its own memory; a systolic array already has input, weight and output memories of its own. Instead of adding a third, this design multiplexes the systolic array’s OUTPUT memory to serve as the array’s shared buffer — which is not merely thrifty but correct, because the matrix engine’s output is precisely what the nonlinear operation consumes. Element-wise operations can therefore begin the instant the matrix engine produces anything, overlapping the two engines with no tensor-size constraint at all, and even softmax’s first pass can overlap while the matrix engine still works. The nonlinear share of LLaMA2-7B runtime falls from 42.4 percent to 22.8 percent. Noticing that the resource a new component needs already exists next door in exactly the right form, rather than provisioning it again, is the same reuse-what-is-already-there instinct covered elsewhere on this site where the structure that appeared to be an obstacle turns out to be the thing the design should build on.

Computing an exponential with only add and multiply

this multiplication is a single ordinary floating-point multiply by a constant, so it costs one operation
nothing has been approximated yet, only rewritten, so no accuracy has been given up at this point

Why it works

• The operations change but their primitives do not → building the primitives rather than the operations covers what has not been invented yet → which is why SwiGLU and RMSNorm need no new hardware while a fixed-circuit design must fall back to a general-purpose core.
• Splitting an input leaves one part encoded free in the floating-point exponent and the other confined below one → a four-term polynomial is accurate over that narrow range → so perplexity moves by at most 0.05 where an established integer method moves it by about 70,000.
• The polynomials generate the same small graph shapes over and over, some in every kernel → giving each shape a functional unit that runs it in one cycle shortens the loop’s critical path → yielding 2.95 times the speed of a plain array of the same size, and up to 6.4.
• Narrow lanes can be chained for width or run independently for throughput → precision becomes a per-kernel decision → letting 16-bit integers give four-way vectorisation and 2.77 times more speed where the accuracy budget allows.
• The systolic array’s output memory already holds what the nonlinear operation reads → reusing it as the shared buffer removes both a memory and a copy → so element-wise work starts as soon as the matrix engine emits anything, cutting the nonlinear share of runtime from 42.4 percent to 22.8.

What it costs (trade-offs and scope)

• A bigger array mostly does not help. Doubling from 16 tiles to 32 yields under 1.4 times rather than 2, which the authors attribute to the compiler’s mapping ability and place outside this work’s scope. Their suggested workaround is to run two independent halves rather than one larger array.
• Vectorisation falls short of its theoretical gain. Four-way 16-bit vectorisation delivers 2.77 times on average rather than 4, because some instructions in the loop cannot be vectorised at all.
• It is an addition, not a replacement. The design accelerates the nonlinear operations beside a matrix engine that still does the linear work. End-to-end gains are therefore bounded by how much of the runtime was nonlinear — 6.74 times on that part, but 3.36 times overall.
• Some functions still need a lookup table. Operations that cannot be built from basic arithmetic, notably the Gaussian cumulative distribution inside GeLU, are stored as precomputed values. That works and is cheap, but it is a small fixed-function concession inside an argument about flexibility.
• Evaluation is by synthesis and simulation. Results come from a 45nm synthesis with modelled memory and an analytical performance tool, not silicon, and the GPU comparison requires scaling the design to match A100 throughput before comparing.

The evaluation

Accuracy was measured by replacing only the nonlinear operations, leaving all linear layers in 16-bit floating point, on GPT2-XL, OPT-6.7B, OPT-13B, LLaMA2-7B and LLaMA2-13B — reporting perplexity on WikiText2 and accuracy on five reasoning benchmarks. Hardware was written in synthesisable Verilog and synthesised in a 45nm library, with memory modelled separately and end-to-end behaviour estimated analytically. The comparison set covers a fixed-circuit accelerator, a general-purpose non-matrix processor, a laptop-class CPU and an A100 GPU, with the design scaled to match the GPU’s throughput before comparing.

Check yourself

Sources

• PICACHU: Plug-In CGRA Handling Upcoming Nonlinear Operations in LLMs (author PDF) — ASPLOS 2025; the full paper, read in full for this explainer.
• ACM Digital Library (DOI 10.1145/3676641.3716013) — published version.