oFFN: Outlier and Neuron-aware Structured FFN for Fast yet Accurate LLM Inference

A big part of an LLM's work is the feed-forward block, where after a ReLU most of the intermediate neurons output zero — so if you can predict which ones, you skip loading and computing them; but two messy facts get in the way, a few giant "outlier" activations and the fact that some neurons fire constantly while most rarely do, and prior work handled these separately; this paper (Sogang University and Santa Clara University — Geunsoo Song, Hoeseok Yang, Youngmin Yi) presents oFFN, which rearranges the FFN weights by rows and columns to predict sparsity accurately despite outliers and to route hot and cold neurons to their best-suited computation, reaching up to 2.01 times faster end-to-end inference.

The argument in one glance

Problem

Skipping zero neurons in an LLM is only as good as the prediction

A large fraction of a language model's compute and memory traffic lives in its feed-forward network (FFN) — the block inside each transformer layer that expands the hidden vector into a much larger set of intermediate "neurons", applies a nonlinearity, and contracts it back. A powerful way to speed this up is to exploit output activation sparsity: after the nonlinearity, many of those intermediate neurons output zero, and a zero contributes nothing to the result. If you can know in advance which neurons will be zero for the current token, you can skip loading their weights from memory and skip computing them entirely. To make this pay off, models are often "ReLU-fied" — fine-tuned to use the ReLU activation, which produces many exact zeros, so sparsity is high and worth exploiting. The whole scheme hinges on a predictor that guesses, cheaply and before the real computation, which output neurons will be active. And here is the tension: the prediction must be both fast (or it eats the savings) and accurate (or it either drops neurons that were actually needed, hurting model quality, or keeps too many, losing speed). Getting fast and accurate sparsity prediction is therefore the crux of efficient sparse LLM inference.
What people did

Predict sparsity, and separately cope with outliers and neuron skew

Existing approaches build a lightweight predictor of which FFN neurons will be active for each token, and use it to skip the rest. Alongside this, the community has noticed two awkward phenomena and dealt with them one at a time. One is activation outliers: a small number of activation values are far larger in magnitude than the rest and matter disproportionately for accuracy, so they must not be mishandled. The other is hot and cold neurons: a few neurons are active for almost every token (hot) while most are active only rarely (cold), a very skewed usage pattern. Prior work addressed these individually, or bolted fixes together, rather than treating them as parts of one structure.
Why it fell short

Handling outliers and hot/cold neurons piecemeal leaves speed on the table

Treating outliers and neuron skew as separate, unrelated problems misses that they interact, and that interaction is exactly what determines how well sparsity can be exploited. A predictor that ignores outliers can be thrown off by those few huge activation values and mispredict which neurons matter, forcing a choice between accuracy loss and conservative, less-aggressive skipping. And ignoring the hot/cold split wastes an opportunity: hot neurons, which fire constantly, and cold neurons, which fire rarely, have very different cost profiles, so computing them all the same way is suboptimal — the best way to handle a neuron that is almost always needed is not the best way to handle one that is almost never needed. On top of this, the ideal handling shifts with the batch size: as more tokens are processed together, the effective density of neuron activations changes, so a fixed strategy is wrong for at least some operating points. What was missing was a single, integrated structure that predicts sparsity accurately in the presence of outliers and, at the same time, sorts neurons by how hot or cold they are so each can be computed in the way that is cheapest for it — adapting even as the batch size changes.
This paper's idea

Rearrange the FFN by rows and columns to fuse both effects

oFFN — an Outlier-aware Structured FFN — starts from a realization the paper makes explicit: activation outliers and the hot/cold-neuron phenomenon are closely related, not independent, and so should be exploited together in one structure rather than patched separately. Its mechanism is a rearrangement of the FFN's weights along both the row and the column dimensions. This reordering does two things at once. First, it enables efficient and accurate prediction of which output neurons will be sparse, done in a way that accounts for the outliers instead of being confused by them. Second, it separates the hot neurons from the cold ones, so each group can be computed with the operation best suited to it — the frequently-firing hot neurons and the rarely-firing cold neurons no longer share a single, compromise computation path. Because the structure sorts neurons this way, oFFN can pick the optimal computation path for each neuron even as the batch size changes dynamically during serving. Compared with existing sparsity-prediction techniques, oFFN reports the fastest speed with negligible accuracy loss, delivering up to 2.01 times faster end-to-end inference than dense inference and up to 5.46 times acceleration in the FFN layers under autoregressive decoding in ReLU-fied LLMs.

Why this problem is worth solving

The feed-forward block is one of the largest, most repeated costs in running a language model, and activation sparsity is one of the few ways to cut it without changing what the model computes.

Every token an LLM generates passes through the feed-forward network of every layer, and that block holds a large share of the model’s parameters and arithmetic — and, crucially, of its memory traffic, which is what dominates the slow, one-token-at-a-time autoregressive decoding phase. If most of the intermediate neurons produce zero, then loading their weights and multiplying by them is pure waste: the answer would be identical if you skipped them. Exploiting that sparsity is therefore a rare kind of optimization — it removes work without approximating the result, provided the prediction of which neurons are zero is correct. That is why so much effort goes into ReLU-fying models to make the zeros plentiful and into predicting them accurately: it directly attacks the memory-bound bottleneck of decoding. The prize is faster, cheaper inference on the same hardware for the same model quality, which at the scale LLMs are served is enormously valuable. The reason it is hard, and worth a paper, is that the real activation patterns are messy — dominated by a few outliers and a steep hot/cold usage curve — so naive skipping either slows down or degrades quality.

The key insight is that the two messy phenomena are not separate nuisances but a single structure you can organize the computation around.

It is natural to treat activation outliers and hot/cold neurons as two different engineering headaches — one about a few oversized values, the other about skewed usage — and to patch each on its own. The paper’s central claim is that this is the wrong framing: the two are closely related, and seeing them together is what unlocks a better design. If outliers and neuron hotness line up in a structured way, then a single rearrangement of the weights can serve both goals at once — making the sparsity prediction robust to outliers and, in the same stroke, grouping neurons by how often they fire. That is powerful because it turns two problems into one lever. Once neurons are sorted into hot and cold groups, you can stop paying the same price for every neuron: hot neurons that are almost always active are cheapest to handle one way, cold neurons that are almost never active another, and the split can be tuned as the batch size (and thus the effective activation density) changes. Framing the problem as one integrated structure, rather than a pile of independent fixes, is what lets oFFN be simultaneously fast and accurate rather than trading one for the other.

What people did before, and where it broke

Predict which FFN neurons are active and skip the rest. Use a lightweight predictor on ReLU-fied models to skip the memory access and compute for neurons expected to output zero. Where it broke. The gains depend entirely on the predictor being fast and accurate, and real activation patterns make that hard.
Handle activation outliers on their own. Add machinery to protect the few disproportionately large activation values. Where it broke. Treated in isolation, it does not improve — and can complicate — the sparsity prediction that the speedup actually depends on.
Handle hot and cold neurons on their own. Notice that some neurons fire far more often than others and exploit that separately. Where it broke. Without integrating it into the prediction and the weight layout, the skew is not turned into different, cheaper computation paths per neuron.
Use one fixed computation strategy regardless of batch size. Apply the same sparse-execution scheme across serving conditions. Where it broke. The effective activation density changes with batch size, so a fixed strategy is suboptimal at some operating points.
The gap. No approach structurally integrated outliers and hot/cold neurons into one layout that predicts sparsity accurately and routes each neuron to its optimal computation, adapting as the batch size changes.

How oFFN works

Turning two messy activation phenomena into one structured FFN

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Exploit that most neurons are silent for any given token. oFFN builds on the foundation that, in a ReLU-fied LLM, the FFN’s output activations are highly sparse, so predicting and skipping the zero neurons removes real memory and compute cost. Organizing inference around the observation that only a small, input-dependent subset of a network’s neurons actually fire for a given input — and fetching or computing just those — is the same principle the work on fast on-device LLM inference through neuron co-activation linking (NeuraLink) covered elsewhere on this site is built on, treating neuron-level activation sparsity as the thing to exploit. oFFN focuses this on the FFN and, crucially, on making the prediction robust rather than just present.

Take the outliers seriously instead of letting them distort things. A recurring obstacle in efficient LLM inference is that a few activation values are enormous and disproportionately important, and any scheme that ignores them pays in accuracy; oFFN’s weight rearrangement is explicitly outlier-aware, keeping the sparsity prediction accurate in their presence. Designing the data layout and computation around the reality that a small set of outlier values must be handled specially, rather than assuming a uniform distribution, is the same concern the work on a metadata-augmented microscaling format for efficient low-bit LLMs (M2XFP) covered elsewhere on this site is built around, where respecting outliers is what preserves quality. oFFN brings that outlier-awareness to sparsity prediction rather than to numeric quantization.

Restructure the FFN computation so each neuron takes its cheapest path. Rather than computing every neuron the same way, oFFN reorders the weights so hot and cold neurons are separated and each is executed with the operation best matched to it, choosing the optimal path per neuron even as the batch size shifts. Reorganizing the arithmetic of an LLM’s dense layers to fit how the values actually behave, so that computation is spent where it pays off, is the same style of restructuring the work on value-level parallelism for efficient LLMs (Mugi) covered elsewhere on this site pursues, reshaping the computation to the data. oFFN’s reshaping is along the hot/cold-neuron axis, matching each neuron’s handling to how often it fires.

Following a token through oFFN

Computing one FFN layer for a token with oFFN

An illustration, drawn from the paper's abstract, of how oFFN's rearranged FFN serves a token. Internal mechanism details are the paper's.

Step 1 of 5

A token reaches the feed-forward block

During autoregressive decoding, the current token's hidden vector arrives at a transformer layer's FFN, whose weights oFFN has rearranged offline along both rows and columns into a structured layout.

The rearrangement is a one-time restructuring of the model, not per-token work.

Why it works

Skipping zero neurons removes real, not approximate, work. A dense FFN loads and multiplies every neuron’s weights → but zero outputs contribute nothing, so skipping predicted-zero neurons → so memory traffic and compute drop with no change to the correct result.
Outlier-aware layout keeps the prediction accurate. Outliers can mislead a naive predictor into wrong skips → but rearranging weights so prediction accounts for outliers → so oFFN can skip aggressively while keeping accuracy loss negligible.
Separating hot and cold neurons lets each be cheapest. Computing every neuron the same way ignores their very different firing rates → but grouping hot and cold neurons via the reordering → so each is executed with its optimal operation instead of a one-size compromise.
One structure serving both goals beats two bolted-on fixes. Handling outliers and neuron skew separately misses their relationship → but a single row-and-column rearrangement that addresses both → so the design is integrated and each part reinforces the other.
Adapting to batch size keeps it optimal across serving conditions. A fixed sparse strategy is wrong when activation density shifts → but choosing the per-neuron computation path for the current batch size → so oFFN stays fast whether decoding one token or many.

What it costs (trade-offs and scope)

It requires a ReLU-fied model. The high, exploitable output sparsity oFFN depends on comes from models fine-tuned to use ReLU; models with other activations produce fewer exact zeros, so the approach targets ReLU-fied LLMs specifically.
The weights must be rearranged, and prediction still costs something. oFFN restructures the FFN weight matrices by row and column and runs a sparsity predictor each step; the reordering is offline work and the predictor is lightweight by design, but the net win depends on the predictor being cheap and accurate.
The largest gains are in the FFN under decoding. The up-to-5.46-times figure is for FFN layers under autoregressive decoding; the end-to-end speedup is smaller (up to 2.01 times) because the FFN is only part of the model and other stages are unchanged.
Accuracy loss is small but not zero. The paper reports negligible accuracy loss, which means the skipping and restructuring do perturb the computation slightly; whether “negligible” is acceptable depends on the application’s tolerance.
Reported figures come from specific models and settings. The speedups are measured on particular ReLU-fied LLMs and configurations; the exact gains depend on the model, the sparsity level, the batch size, and the hardware.

The comparison

Exploiting FFN activation sparsity in ReLU-fied LLMs

Property	Dense FFN	Predict-and-skip (prior sparse inference)	oFFN
Zero neurons
Activation outliers
Hot vs cold neurons
Adapts to batch size
Speed and accuracy

A comparison of ways to run the feed-forward block of a ReLU-fied LLM. Dense computation wastes work on zero neurons; predict-and-skip methods save work but face an accuracy-speed tension worsened by outliers and neuron skew. oFFN rearranges the FFN weights by row and column to make sparsity prediction accurate despite outliers and to route hot and cold neurons to their best-suited operations, reporting the fastest speed with negligible accuracy loss — up to 2.01 times end-to-end and up to 5.46 times in FFN layers under decoding. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s abstract and public metadata, not its full text, which was not available at the time of writing. The high-level claims are the authors’: oFFN (the abstract expands the name as “Outlier-aware Structured FFN”) is a feed-forward-network design for fast, accurate LLM inference that exploits output activation sparsity in ReLU-fied models. The paper states that activation outliers and hot/cold neurons are closely related and that oFFN exploits both simultaneously by rearranging FFN weights along both the row and column dimensions, in order to (1) enable efficient and accurate prediction of output sparsity while accounting for outliers, and (2) separate hot from cold neurons so each is computed with its optimal operation, allowing an optimal per-neuron computation path even as batch size changes. The reported results are up to 2.01 times faster end-to-end inference versus dense inference and up to 5.46 times acceleration in FFN layers under autoregressive decoding in ReLU-fied LLMs, with negligible accuracy loss and the fastest speed among compared sparsity-prediction techniques. The background above on the transformer FFN, on output activation sparsity, on ReLU-fication, on sparsity predictors, on outliers, and on hot/cold neurons is standard context provided to make the approach intuitive. The Flow, the step-by-step walkthrough, and the “Why it works” reasoning are illustrations built from the abstract’s description, not detailed accounts from the paper — how the row-and-column rearrangement is actually constructed, what the per-neuron “optimal operations” are, and the full evaluation should be read from the paper. Do not treat specific procedural details here as established beyond what the abstract states.

Background

Why fast LLM inference keeps coming down to taming irregular structure

It helps to see oFFN within a broad pattern in efficient LLM inference. A language model, on any given token, only really needs a small, input-dependent slice of its enormous weight set — most neurons are silent, most values are unexceptional — so the path to speed is to compute only what matters. But that slice is irregular: a handful of outlier activations carry outsized importance, neuron usage is steeply skewed between a few hot and many cold, and the right thing to do shifts with batch size and serving conditions. The recurring lesson across this line of work is that raw skipping or raw quantization is not enough; the irregularity has to be made structured — reorganized into a layout the hardware can exploit predictably — or it degrades either speed or accuracy. oFFN is a clean instance: instead of patching outliers and neuron skew separately, it rearranges the FFN weights so both become one exploitable structure, predicting sparsity accurately and routing each neuron to its cheapest computation. This drive to reshape the messy, outlier-ridden statistics of an LLM into a regular form that fast, low-cost execution can rely on is the same terrain the work on practical low-bit W4A4KV4 LLM serving (COMET) covered elsewhere on this site occupies, taming irregular value distributions so aggressive optimization stays accurate. oFFN’s particular contribution is to show that, for the feed-forward block, outliers and hot/cold neurons are two faces of one structure — and that organizing the weights around it delivers the fastest sparse inference with negligible accuracy loss.

If you remember only three things

1. A big share of an LLM's inference cost is the feed-forward network (FFN) in each layer, and a strong way to cut it is output activation sparsity: in ReLU-fied models many of the FFN's intermediate neurons output zero for a given token, so a predictor can skip loading and computing them. The catch is that the prediction must be both fast and accurate, and two real-world phenomena make that hard — a few activation outliers that are much larger than the rest and matter disproportionately, and a steep split between hot neurons that fire for almost every token and cold neurons that fire rarely. Prior work handled these two issues individually or without integrating them structurally, leaving speed or accuracy on the table.
2. oFFN (Outlier and Neuron-aware Structured FFN) starts from the observation that activation outliers and hot/cold neurons are closely related, and exploits both at once by rearranging the FFN weight matrices along both the row and column dimensions. This single structural change enables efficient and accurate prediction of output sparsity that accounts for outliers (so skipping can be aggressive without hurting accuracy) and separates hot neurons from cold ones so each is computed with the operation best suited to it — and it can choose the optimal computation path per neuron even as the batch size changes during serving. Turning two separate problems into one exploitable structure is what lets oFFN be fast and accurate at the same time.
3. The paper reports oFFN achieves the fastest speed among sparsity-prediction techniques with negligible accuracy loss, up to 2.01 times faster end-to-end inference than dense and up to 5.46 times faster FFN layers under autoregressive decoding in ReLU-fied LLMs. The trade-offs and scope: it depends on ReLU-fied models for high sparsity, requires rearranging the weights and running a predictor each step, delivers its largest gains in the FFN under decoding (so end-to-end speedup is smaller), and accepts a small but non-zero accuracy change. This explainer is based on the paper's abstract; the construction of the rearrangement, the per-neuron operations, and the full evaluation should be confirmed in the paper.

Check yourself

Check your understanding

1. What is output activation sparsity in an LLM's feed-forward network, and why does exploiting it help?
2. What two phenomena complicate sparse FFN inference, and what does oFFN claim about them?
3. How does oFFN's mechanism work?
4. What does oFFN achieve, and what are the limits of the approach?

Sources

The paper is “oFFN: Outlier and Neuron-aware Structured FFN for Fast yet Accurate LLM Inference” by Geunsoo Song, Hoeseok Yang, and Youngmin Yi (Sogang University and Santa Clara University), published at ASPLOS 2026, DOI 10.1145/3779212.3790194.

This explainer is written from the paper’s abstract and public metadata; the full text was not available at the time of writing, so the page is marked a draft. As reported there, oFFN is an Outlier-aware Structured FFN that speeds up LLM inference by exploiting output activation sparsity in ReLU-fied models. The paper reveals that activation outliers and hot/cold neurons are closely related and exploits both simultaneously by rearranging FFN weights along the row and column dimensions — to predict output sparsity efficiently and accurately while accounting for outliers, and to separate hot from cold neurons so each is computed with its optimal operation, allowing an optimal per-neuron computation path even as batch size changes. The reported results are up to 2.01 times faster end-to-end inference versus dense and up to 5.46 times acceleration in FFN layers under autoregressive decoding, with negligible accuracy loss. Background on the transformer FFN, output activation sparsity, ReLU-fication, sparsity predictors, outliers, and hot/cold neurons is standard context. The specific numbers are the paper’s own; the construction of the rearrangement, the per-neuron operations, and the full evaluation should be confirmed in the paper.