RedFuser: An Automatic Operator Fusion Framework for Cascaded Reductions on AI Accelerators

Xinsheng Tang, Yangcheng Li, Nan Wang, Zhiyi Shu, Xingyu Ling, Junna Xing and 2 others

AI compilers speed models up by "fusing" neighboring operations into one GPU kernel so intermediate results never leave fast on-chip memory — but they choke on chains of reductions that depend on each other, like the softmax-then-matrix-multiply inside attention, so those get hand-written by experts (FlashAttention) instead; this paper (Alibaba Cloud — Xinsheng Tang, Qiang Liu, and colleagues) presents RedFuser, which formally analyzes such cascaded reductions, fuses the whole chain into one incremental loop automatically, and generates kernels 2 to 5 times faster than existing compilers.

The argument in one glance

Problem

Chained reductions keep spilling data to slow memory

AI compilers make models fast largely through operator fusion: merging several consecutive operations into a single GPU kernel so that the intermediate results flow through fast on-chip memory (registers and shared memory) instead of being written out to slow global memory and read back for the next operation. This works beautifully for element-wise operations, which are independent and easy to parallelize. But a huge amount of deep learning is built from reductions — operations that collapse an array into a single value through a loop, like a sum or a maximum. On a GPU these run as a reduction tree: split the data across thread blocks, reduce each piece locally, then launch a second kernel to combine the partial results. Even matrix multiplication (GEMM) is fundamentally a pile of reductions. The hard case is cascaded reductions: chains of reductions where each one depends on the output of the one before it. These are everywhere — the safe softmax followed by a matrix multiply inside an attention mechanism, softmax-then-top-k in mixture-of-experts routing, quantize-then-GEMM in low-precision math, normalization layers. Because each reduction needs the previous reduction's final result, the chain cannot simply be flattened, and it creates two painful bottlenecks: redundant memory accesses (each reduction re-reads its input from global memory) and limited parallelism (the reductions must run one after another, and memory transfers cannot be overlapped with compute to hide latency). Each stage also has to cache the full result of the previous stage, straining on-chip memory.
What people did

Let compilers fuse, or hand-write a kernel per pattern

There are two existing routes. The first is to rely on modern AI compilers (such as TVM or Dynamo) to fuse operators automatically. The second, when the compiler falls short, is for experts to hand-write a single fused kernel for a specific pattern: FlashAttention and FlashDecoding are famous examples that fuse the softmax and matrix-multiply of attention into one kernel using tiling and an "online softmax" trick, achieving remarkable performance. Both aim to avoid the memory traffic and serialization of running each reduction as its own kernel.
Why it fell short

Compilers cannot analyze the chain, and hand kernels do not generalize

Neither route solves the general problem. Existing AI compilers do not formally model cascaded reductions as a structured pattern, and they do not analyze the dependency and memory-access characteristics of a chain of reductions. As a result, they cannot generate a loop-level fused kernel that spans multiple reduction stages — they fall back on suboptimal fusion that still pays the redundant memory and serialization costs. The hand-crafted kernels, on the other hand, are excellent but narrow: FlashAttention's brilliance is tightly coupled to the attention pattern, built through expert-level GPU programming and manual scheduling, so its logic does not transfer to other cascaded reductions like the softmax-plus-top-k of mixture-of-experts routing or the quantize-plus-GEMM of FP8 computation. Every new pattern needs a new painstaking derivation by a specialist. So the field was caught between compilers that are general but cannot handle cascaded reductions well, and hand kernels that handle one pattern beautifully but cannot generalize. Given how prevalent these chained-reduction patterns are across deep learning, that gap leaves large performance on the table. What was missing was a framework that formally characterizes cascaded reductions and automatically generates fused, hardware-tuned kernels for the whole family, not just one instance.
This paper's idea

Formally fuse the whole chain into one incremental loop, automatically

RedFuser (Reduction Fuser) closes that gap with a formal methodology plus an automatic framework. The theoretical core is a way to analyze cascaded reductions and fuse them into a single reduction, one loop, by reasoning about the algebraic properties of the reduction operators (such as having an identity element and distributing over one another) to prove when fusion is valid and to derive the fused expression. Fusing the chain eliminates the redundant data loads and lets dependent reductions execute together rather than strictly in sequence. Because a single level of on-chip cache can only hold so much, fusing the whole chain naively would exceed capacity, so RedFuser introduces an incremental computation form — a generalization of the online-softmax idea — that updates partial results progressively as data streams through, so it never has to materialize a full intermediate. On top of this methodology, RedFuser is a lightweight two-stage framework: first a symbolic deduction engine automatically analyzes whether a chain of reductions is fusible and derives the fused expression and its incremental form; then hardware-aware code generation lowers that fused expression into a tile-based representation and emits a high-performance kernel. Evaluated on NVIDIA A10 and H800 GPUs across four workloads — multi-head attention, multi-latent attention, mixture-of-experts routing, and FP8 quantize-plus-GEMM — RedFuser achieves 2 to 5 times speedup over state-of-the-art AI compilers and matches or exceeds expert-optimized hand-written kernels. It is open-source.

Why this problem is worth solving

Cascaded reductions sit inside the most expensive parts of modern models, so automating their fusion pays off across a huge fraction of AI compute.

Modern models are enormous — leading large language models now exceed a trillion parameters — so the cost and speed of training and inference are first-order concerns, and much of that cost is spent in exactly the patterns cascaded reductions describe. Attention, the core of every transformer, is a softmax reduction feeding a matrix multiply; mixture-of-experts routing is a softmax feeding a top-k selection; low-precision inference is a quantization reduction feeding a GEMM; normalization layers are reductions too. These are not rare corner cases but the load-bearing operations of contemporary AI, run billions of times. When they are executed as separate kernels, they waste memory bandwidth re-reading data and lose parallelism to serialization, and on GPUs — where memory bandwidth and latency-hiding are everything — that waste directly slows the whole model. The reason automation matters, rather than just hand-tuning, is scale and diversity: there are many such patterns, new ones keep appearing (multi-latent attention, FP8 pipelines), and hand-writing a FlashAttention-quality kernel for each is a scarce-expert, slow, error-prone effort. A framework that automatically recognizes and fuses any supported cascaded reduction turns a bespoke, per-pattern craft into a compiler pass, which means the 2-to-5-times gains show up wherever these patterns appear, not just where an expert had time to optimize. That is why closing this gap is high-leverage.

The key realization is that a chain of dependent reductions can be treated as one reduction if you formalize the algebra and compute it incrementally.

The reason cascaded reductions resist ordinary fusion is the data dependency: the second reduction needs the first one’s complete result, so it seems you must finish one before starting the next, materializing the intermediate in memory. FlashAttention showed, for the specific case of attention, that this is not actually necessary — with the online-softmax trick you can stream tiles and keep a running, incrementally-updated summary, never storing the full intermediate. RedFuser’s insight is that this is not a one-off attention hack but an instance of a general algebraic structure. If the reduction operators have the right properties — an identity element, and one distributing over another — then a chain of them can be provably rewritten as a single fused reduction, and the running-summary trick generalizes to an incremental computation form that updates partial results as data flows. Making this formal is what unlocks automation: instead of a human reasoning out, per pattern, whether and how the reductions can be merged, a symbolic engine can check the algebraic conditions and mechanically derive the fused expression and its incremental form. And separating the formal fusion from the hardware-aware code generation is what makes it both general (the algebra does not care about the GPU) and fast (the code generator tiles it for the specific device). So the crux is turning a clever manual trick into a proved, checkable transformation, which is exactly what lets a compiler do automatically what only experts could do before.

What people did before, and where it broke

Run each reduction as its own kernel. Execute the reduction tree, launch a second kernel to combine partials, repeat for the next reduction. Where it broke. Each reduction re-reads its input from global memory and the stages run strictly in sequence, wasting bandwidth and parallelism.
Rely on AI compilers to fuse (TVM, Dynamo). Let the compiler merge operators automatically. Where it broke. They do not model cascaded reductions as a structured pattern or analyze the dependencies, so they cannot generate a fused kernel spanning multiple reduction stages.
Hand-write a fused kernel per pattern (FlashAttention). Have experts fuse a specific chain using tiling and online softmax. Where it broke. It is excellent but narrow — tied to the attention pattern, built by manual scheduling, and not transferable to other cascaded reductions like MoE routing or FP8 pipelines.
Cache the full intermediate between stages. Store each reduction’s complete result for the next stage to consume. Where it broke. On-chip cache capacity limits how long a reduction can be, straining memory and constraining fusion.
The gap. No framework both formally characterized cascaded reductions and automatically generated fused, hardware-tuned kernels for the whole family, so most chained-reduction patterns went un-fused or needed a bespoke expert kernel.

How RedFuser works

From per-stage reduction kernels to one automatically-fused incremental loop

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Fuse the chain so intermediates never touch slow memory. RedFuser’s foundation is operator fusion aimed at memory-bound reduction chains: merge the stages so intermediate results stay on-chip, eliminating the redundant global-memory reloads that dominate the cost. Opening up an optimization space that fuses memory-intensive operators to cut the global-memory traffic that bottlenecks them is the same enterprise the work on enabling a new multi-dimensional optimization space for memory-intensive fusion (AStitch) covered elsewhere on this site pursues, treating fusion as the lever for memory-bound performance. RedFuser extends fusion to the hard case of dependent reduction chains that general compilers could not fuse.

Stream tiles and keep a running summary instead of materializing intermediates. The heart of RedFuser’s methodology is the incremental computation form — a generalization of the online-softmax technique — that updates partial results as data flows, so a long reduction fed into another never has to store its full output. Reorganizing the dataflow of an attention-style softmax-and-matmul so that tiles are processed with running statistics rather than a materialized intermediate is the same principle the work on an optimized dataflow for mitigating attention bottlenecks (FLAT) covered elsewhere on this site is built on, keeping the expensive intermediate off memory. RedFuser turns that dataflow idea into a general, derivable form rather than an attention-specific schedule.

Turn a formal expression into a tuned kernel automatically. RedFuser separates a symbolic derivation of the fused reduction from hardware-aware, tile-based code generation, so a formal specification becomes an efficient GPU kernel without a human writing it. Compiling a high-level, formally-expressed tensor computation down to locality-aware, hardware-tuned code is the same undertaking the work on compiling Einstein summations to locality-aware dataflow (SIGMA) covered elsewhere on this site pursues, bridging a mathematical description to efficient hardware execution. RedFuser’s symbolic engine and tile-based generator are its bridge from the fused reduction expression to a high-performance kernel.

Following a cascaded reduction through RedFuser

Fusing a softmax-then-matmul chain automatically

A trace, drawn from the paper, of how RedFuser fuses and generates a kernel for a cascaded reduction.

Step 1 of 5

Identify a chain of dependent reductions

RedFuser detects a supported cascaded reduction — for example the safe softmax (a max reduction and a sum reduction) feeding a matrix multiply (a reduction along the contraction axis) inside attention.

Each stage depends on the previous stage's reduced result.

Why it works

Fusing the chain removes redundant global-memory reads. Running each reduction separately re-reads its input from slow memory → but merging the chain into one kernel keeps intermediates on-chip → so the dominant memory traffic disappears.
Incremental computation avoids materializing intermediates. A dependent reduction seems to need the full previous result → but updating running partial results as tiles stream → so a long reduction feeds the next without exceeding on-chip cache.
Algebraic conditions make fusion provable and automatic. Deciding fusibility by hand is per-pattern and error-prone → but checking identity and distributivity of the reduction operators → so a symbolic engine can mechanically prove and derive the fused expression.
Separating algebra from codegen gives generality and speed. A fixed hand kernel serves one pattern → but deriving the fused form abstractly, then generating tiled hardware code → so one framework covers many patterns while still tuning to the GPU.
Automation matches experts everywhere the pattern appears. Hand-written kernels only exist where a specialist optimized them → but automatically fusing any supported cascaded reduction → so 2-to-5-times gains apply broadly, not just to attention.

What it costs (trade-offs and scope)

It applies to supported cascaded reduction patterns. RedFuser fuses chains whose reduction operators satisfy the algebraic conditions it requires; reductions that do not fit those properties fall outside its automatic fusion.
The methodology rests on operator algebra. The proofs of fusibility and the incremental form depend on properties like identity and distributivity; a pattern where these do not hold cannot be fused this way.
It targets GPU-style accelerators and their memory hierarchy. The code generation is hardware-aware and tile-based for GPUs (evaluated on A10 and H800); other accelerators would need the generator adapted.
It optimizes fusion, not the algorithm. RedFuser makes an existing computation run faster by fusing and generating better kernels; it does not change what the model computes, so gains are bounded by how memory-bound and fusible the pattern is.
Reported figures are on specific workloads and baselines. The 2-to-5-times speedups and the parity with hand kernels are measured on four workloads against particular compilers (TVM, Dynamo) and GPUs; results depend on the pattern, shapes, and hardware.

The comparison

Optimizing a cascaded reduction (e.g. softmax then matmul)

Property	AI compiler auto-fusion	Hand-written fused kernel	RedFuser
Models cascaded reductions
Fuses across reduction stages
Generalizes to new patterns
Effort
Performance

A comparison of ways to optimize a cascaded reduction. AI compilers auto-fuse but do not model chained reductions, leaving performance on the table; hand-written kernels like FlashAttention are excellent but tied to one pattern and expert-built. RedFuser formally analyzes cascaded reductions, fuses the chain into one incremental loop, and generates tiled GPU kernels automatically — reaching 2 to 5 times the speed of existing compilers and matching hand-written kernels across attention, MoE routing, and FP8 pipelines. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ arXiv copy, 2603.10026, of the ASPLOS 2026 paper). The described system is the paper’s: RedFuser (Reduction Fuser) is an automatic operator-fusion framework for cascaded reductions on AI accelerators. The paper presents a formal methodology that fuses a chain of dependent reductions into a single reduction by reasoning about algebraic properties of the reduction operators (the paper discusses conditions such as the existence of an identity element and distributivity), eliminating redundant data loads and enabling parallel execution of dependent reductions, and it introduces an incremental computation form (generalizing the online-softmax technique) to progressively update partial results and avoid materializing full intermediates under cache-capacity limits. RedFuser is a two-stage framework: a symbolic deduction engine analyzes fusibility and derives the fused expressions and incremental forms, followed by hardware-aware code generation that lowers the fused expression into a tile-based representation and generates kernels. The paper evaluates it on NVIDIA A10 and H800 GPUs across four workloads (Multi-Head Attention, Multi-Latent Attention, MoE routing, and FP8 Quant plus GEMM), reporting 2 to 5 times speedup over state-of-the-art AI compilers (it names TVM and Dynamo) and performance matching or exceeding expert-optimized hand-written kernels; the code is open-source. The background above on operator fusion, GPU memory hierarchy, reductions and reduction trees, GEMM as reduction, and online softmax is standard context provided to make the approach intuitive. The step-by-step walkthrough is an illustration of the described mechanism. Exact algebraic formulations, the symbolic engine, the code generator, and the full evaluation should be read from the paper.

Background

Why deep-learning compilers keep chasing the kernels experts write by hand

It helps to see RedFuser within a long-running effort in deep-learning systems: the fastest code for a given operation is usually a hand-written kernel crafted by a GPU expert, and the goal of a DL compiler is to reach that quality automatically, for far more operations than any team could hand-optimize. The reason this matters is that models are enormous and evolve constantly, so the set of operations that need fast kernels is always growing — new attention variants, new precisions, new routing schemes — and hand-writing a peak kernel for each is a scarce, slow craft. The field’s progress comes from finding the right abstractions: formalizing what a class of computations is, reasoning about how to transform it, and then generating hardware-tuned code, so that an entire family gets optimized at once instead of one kernel at a time. Fusion is one of the most valuable such transformations, because so much of deep learning is memory-bound and the win from keeping intermediates on-chip is large. RedFuser is a clean instance for the stubborn case of cascaded reductions, which resisted automation because of their data dependencies until the online-softmax trick was generalized into a formal, derivable transformation. This drive to automate what expert kernel-writers do, by capturing a computation in a principled abstraction and generating optimized code from it, is the same terrain the work on an abstraction for automatic tensorized program optimization (TensorIR) covered elsewhere on this site occupies, turning hand-tuned tensor kernels into compiler output. RedFuser’s particular contribution is to bring cascaded reductions — attention, MoE routing, FP8 pipelines and more — into that automated regime, matching FlashAttention-class kernels without hand scheduling.

If you remember only three things

1. AI compilers speed models up through operator fusion — merging consecutive operations into one GPU kernel so intermediate results stay in fast on-chip memory instead of being written to and reread from slow global memory. This works well for independent element-wise operations, but not for cascaded reductions: chains of reduction loops (sum, max, matrix multiply) where each stage depends on the previous stage's result. These are ubiquitous — the safe softmax then matrix multiply in attention, softmax then top-k in mixture-of-experts routing, quantize then GEMM in low-precision math. Because each reduction needs the previous one's output, existing compilers cannot fuse the chain (they do not model or analyze the dependencies), so it runs as separate kernels that redundantly reread memory and serialize; the only good solutions are hand-written kernels like FlashAttention, which are excellent but tied to one pattern and require expert manual scheduling.
2. RedFuser closes this gap with a formal methodology plus an automatic framework. The methodology analyzes cascaded reductions by the algebraic properties of their reduction operators (such as having an identity and distributing over one another) to prove when a chain can be fused into a single reduction — one loop — eliminating redundant loads and letting dependent reductions run together. Because a fused chain would exceed on-chip cache, it derives an incremental computation form (generalizing online softmax) that updates running partial results as data streams, never materializing a full intermediate. The framework has two stages: a symbolic deduction engine that automatically checks fusibility and derives the fused expression and its incremental form, and hardware-aware code generation that lowers the expression into a tile-based representation and emits a tuned GPU kernel.
3. On NVIDIA A10 and H800 GPUs across four workloads — multi-head attention, multi-latent attention, MoE routing, and FP8 quantize-plus-GEMM — RedFuser achieves 2 to 5 times speedup over state-of-the-art AI compilers (TVM, Dynamo) and matches or exceeds expert-optimized hand-written kernels, all automatically; it is open-source. The trade-offs and scope: it applies to cascaded reductions whose operators satisfy its algebraic conditions, rests on operator algebra (identity, distributivity), targets GPU-style accelerators with tile-based code generation, and optimizes fusion rather than changing what the model computes. This explainer is based on the paper's full text.

Check yourself

Check your understanding

1. Why can existing AI compilers not fuse cascaded reductions well?
2. What is RedFuser's formal methodology, and why does it need an incremental computation form?
3. How is RedFuser structured as a framework, and why does that design help?
4. What does RedFuser achieve, and what are the limits of the approach?

Sources

The paper is “RedFuser: An Automatic Operator Fusion Framework for Cascaded Reductions on AI Accelerators” by Xinsheng Tang, Yangcheng Li, Nan Wang, Zhiyi Shu, Xingyu Ling, Junna Xing, Peng Zhou, and Qiang Liu (Alibaba Cloud Computing), published at ASPLOS 2026, DOI 10.1145/3779212.3790209. It is available on arXiv as 2603.10026, and the code is open-source at github.com/alibaba/redfuser.

This explainer is written from the paper’s full text (the authors’ arXiv copy). RedFuser is an automatic operator-fusion framework for cascaded reductions. It presents a formal methodology that fuses a chain of dependent reductions into a single reduction using algebraic properties of the reduction operators, plus an incremental computation form (generalizing online softmax) that avoids materializing full intermediates. The framework has a symbolic deduction engine that analyzes fusibility and derives the fused expressions, and hardware-aware code generation that emits tile-based GPU kernels. Evaluated on A10 and H800 GPUs across multi-head attention, multi-latent attention, MoE routing, and FP8 quantize-plus-GEMM, the paper reports 2 to 5 times speedup over state-of-the-art AI compilers and parity with expert hand-written kernels. Background on operator fusion, reductions, and online softmax is standard context. The numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.