The Configuration Wall: Characterization and Elimination of Accelerator Configuration Overhead

Josse Van Delm, Anton Lydike, Joren Dumoulin, Jonas Crols, Xiaoling Yi, Ryan Antonio and 3 others

A hardware accelerator has to be set up — its configuration registers written by the host CPU — before it can compute, and as accelerators get faster and more flexible, that setup time becomes a bigger and bigger share of the total, so a more configurable accelerator can end up slower overall; this paper (KU Leuven, Cambridge — Josse Van Delm, Marian Verhelst, Tobias Grosser, and colleagues) names this the configuration wall, builds a roofline-style model to tell when a system hits it, and gives an MLIR compiler abstraction that removes redundant configuration and hides the rest behind computation — for a 2 times geomean speedup.

The argument in one glance

Problem

Setting up an accelerator takes time, and faster accelerators make it matter more

To get the most performance per watt, systems offload compute kernels from the CPU to a dedicated hardware accelerator. But an accelerator does not just run: before it can compute, the host CPU must configure it — write its configuration registers to tell it what operation to perform and how — then launch the computation, then wait for it to finish. The cycles spent configuring and synchronizing are cycles where neither the CPU nor the accelerator is doing useful work; the paper calls this the configuration overhead. This overhead has grown as accelerators have become more complex and more flexible, because more flexibility means more configuration options to set. And here is the sting: the faster the accelerator computes, the larger a fraction of total time the fixed configuration cost becomes — a fast accelerator finishes its compute quickly, so the setup around it dominates. The paper names this the configuration wall. It exposes a genuine trade-off: adding configuration options makes an accelerator more useful, but every added option also, without careful optimization, directly cuts the achievable performance. Taken far enough, a more reconfigurable accelerator can make the whole system slower than a less flexible one. Existing performance models do not capture this trade-off, and existing compilers cannot optimize it away, so the configuration wall goes unmeasured and unaddressed.
What people did

Notice the overhead and patch it with ad-hoc, accelerator-specific fixes

Prior work has observed that configuration overhead hurts and proposed ways to reduce it — hand-tuned instruction sequences, accelerator-specific tricks to cut the number of configuration writes, or bespoke overlapping of setup with work. Each of these addresses the problem for a particular accelerator or system. The general shape of the response has been to recognize the cost where it appears and apply a targeted fix, rather than to model the phenomenon or handle it systematically in a compiler.
Why it fell short

Ad-hoc fixes do not generalize and offer no model of when the wall bites

The trouble with these point solutions is that they are neither universal nor explanatory. A trick that cuts configuration writes on one accelerator does not transfer to another with a different configuration interface, host processor, or interconnect, so every new design re-litigates the problem from scratch. More fundamentally, the field lacked a way to reason about the phenomenon: no model tells you whether a given system is configuration-bound, how far it is from the wall, or how much a proposed optimization would help. Classic performance models like the roofline capture when you are limited by compute throughput or by memory bandwidth, but they say nothing about configuration — so a system architect could not even place configuration overhead alongside the bottlenecks they already understand. Without such a model, optimization is guesswork, and without a general compiler mechanism, any fix must be hand-built per accelerator. What was missing was both a way to characterize the configuration wall — to know when and how badly a system hits it — and a general, automatic way for a compiler to move a program out of the configuration-bound region.
This paper's idea

Model configuration as a roofline, then dedupe and overlap it in the compiler

This paper supplies both the model and the mechanism. First, it extends the well-known roofline model to configuration. The classic roofline plots attainable performance against operational intensity (useful operations per byte of memory traffic), showing you are memory-bound when intensity is low. The new roofline instead plots performance against operation-to-configuration intensity — how many useful operations you get per unit of configuration — against a maximum configuration bandwidth, so you can see exactly when a system is configuration-bound and how far it sits from the wall. That model motivates two compiler optimizations. Configuration deduplication: when the compiler can prove that part of the configuration does not change between operations (for example, the same settings reused across loop iterations), it skips rewriting those registers, eliminating redundant configuration cycles. Configuration-computation overlapping: when the accelerator supports being reconfigured while it is still computing, the compiler configures the next operation during the current one, hiding the remaining configuration latency behind useful work. To do this generally, the paper introduces an MLIR compiler dialect called accfg that tracks each accelerator's configuration state and lets these passes apply across accelerators regardless of domain, host, or interconnect. Implemented in MLIR and evaluated on the open-source matrix-multiplication accelerators OpenGeMM and Gemmini (each tightly coupled to a RISC-V host), the optimizations deliver a geomean 2 times speedup by removing redundant configuration and hiding what remains.

Why this problem is worth solving

The configuration wall is a hidden tax that grows precisely as accelerators improve, so ignoring it undermines the very gains hardware acceleration is meant to deliver.

There is a counterintuitive dynamic at the heart of this problem. Configuration overhead is a largely fixed cost paid around each accelerator invocation — you write the registers, launch, and wait — and it does not shrink when the accelerator computes faster. But the compute time does shrink. So as accelerators get faster, the fixed setup cost becomes a larger and larger fraction of the total, exactly as in Amdahl’s law, where a fixed serial portion caps the benefit of speeding up the rest. This means the configuration wall bites hardest on the best accelerators: the more you improve the compute, the more the surrounding configuration dominates, until further hardware speedups buy almost nothing because the system is configuration-bound rather than compute-bound. It also means the wall is a moving target that worsens with each hardware generation. And it interacts with flexibility, which is the other thing designers keep adding: every new configuration knob that makes an accelerator more broadly useful also adds setup work. So a design team can make an accelerator both faster and more flexible and find the whole system slower, with the wasted time invisible to their compute-focused benchmarks. Because acceleration is the main lever left for performance per watt, a tax that silently eats its returns is worth measuring and removing.

The deeper contribution is giving this overlooked cost the same conceptual footing as the bottlenecks architects already reason about, which is what makes it optimizable rather than merely lamentable.

Engineers manage what they can measure and place in a familiar frame. The roofline model earned its influence by turning a complicated question — is my kernel limited by compute or by memory? — into a single picture: plot operational intensity, and where it lands relative to the compute ceiling and the memory-bandwidth slope tells you your bottleneck and what to do about it. Configuration overhead had no such frame, so it was easy to overlook and hard to reason about. This paper’s move is to give configuration the same treatment: define an operation-to-configuration intensity and a configuration bandwidth, and draw a roofline where a low intensity means you are configuration-bound. Suddenly the wall is not a vague complaint but a quantified regime with a boundary you can locate a system against, and the two ways off the wall become obvious from the geometry — raise the intensity by doing more useful work per configuration (deduplication), or raise the effective bandwidth by hiding configuration behind computation (overlapping). This is why the model matters as much as the optimizations: it explains when each optimization helps and by how much, turns an ad-hoc annoyance into a first-class performance axis, and lets a compiler target it systematically. Naming and modeling a bottleneck is often the step that makes it solvable.

What people did before, and where it broke

Classic roofline modeling. Characterize a system as compute-bound or memory-bound via operational intensity. Where it broke. It says nothing about configuration overhead, so a system architect cannot even see when configuration is the bottleneck, let alone how far off the wall they are.
Ad-hoc configuration-reduction tricks. Hand-tune instruction sequences to cut configuration writes on a specific accelerator. Where it broke. They are tied to one accelerator’s interface and host, so they do not transfer, and every new design must be re-optimized by hand.
Bespoke overlapping. Manually overlap setup with computation for a particular accelerator that supports it. Where it broke. It is not a general, automatic capability, and requires expert effort per system rather than a compiler pass anyone can reuse.
Compute-focused benchmarking. Measure an accelerator by its kernel throughput. Where it broke. This hides configuration overhead entirely, so designs that are fast in isolation but configuration-bound in practice look fine on paper.
The gap. There was neither a model to characterize when a system is configuration-bound nor a general compiler mechanism to move it out of that region, so the configuration wall stayed unmeasured and handled only case by case.

How the paper attacks the configuration wall

From an unmodeled, hand-patched overhead to a modeled, compiler-eliminated one

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Characterize the accelerator’s setup overhead as a first-class cost. The paper’s starting point is to measure and model configuration overhead — the cycles spent setting up and synchronizing an accelerator — rather than ignore it. Carefully quantifying the invocation and setup overheads that surround an accelerator, so their real performance impact is visible instead of hidden by kernel-only benchmarks, is the same accelerator-characterization instinct the work on a quantitative analysis of a data streaming accelerator (data streaming accelerator analysis) covered elsewhere on this site pursues, exposing the true cost of using an accelerator. This paper turns that characterization into a predictive roofline model of the configuration wall.

Give the compiler a real abstraction for the accelerator. To optimize configuration generally, the paper builds accfg, an MLIR dialect that represents accelerator configuration state so a compiler can reason about and transform it. Introducing a dedicated compiler intermediate representation that captures what a hardware accelerator needs, so ordinary compiler passes can target it, is the same MLIR-for-accelerators approach the work on an MLIR-based intermediate representation for hardware accelerators (HIR) covered elsewhere on this site is built on, giving accelerators a first-class place in the compiler. The accfg dialect is precisely such an abstraction, specialized to tracking configuration state across accelerators.

Hide unavoidable overhead behind useful work. For the configuration that cannot be removed, the paper overlaps it with computation — configuring the next operation while the accelerator runs the current one — so the setup latency disappears behind productive cycles. Overlapping a necessary non-compute cost with ongoing computation so that it is hidden rather than stalling the critical path is the same latency-hiding idea the work on overlapping communication with dependent computation via decomposition (overlapping communication with dependent computation) covered elsewhere on this site is built on, folding overhead into the compute it accompanies. This paper applies that overlap to accelerator configuration, driven automatically by the compiler where the hardware allows.

Following a matrix-multiply loop through the optimizations

From a configuration-bound loop to an overlapped, deduplicated one

A trace, drawn from the paper, of how accfg optimizes a program that repeatedly invokes a matrix-multiplication accelerator.

Step 1 of 5

A loop repeatedly configures and launches the accelerator

The program runs a loop where each iteration writes the accelerator's configuration registers, launches a matrix-multiplication macro-operation, and waits for it to finish. Much of each iteration is spent on the configure-and-wait steps, not on useful compute.

This is the configuration-bound starting point the roofline flags.

Why it works

A roofline shows exactly when configuration is the bottleneck. Compute-and-memory models miss configuration entirely → but plotting operation-to-configuration intensity against configuration bandwidth → so an architect can locate a system relative to the wall and target the right fix.
Deduplication raises useful work per configuration. Rewriting unchanged registers every operation wastes cycles → but proving parts of the configuration are identical and skipping them → so more useful operations happen per unit of configuration, moving right on the roofline.
Overlapping hides the configuration that remains. Necessary setup normally stalls the host and accelerator → but configuring the next operation during the current computation → so the remaining configuration latency is hidden behind productive work.
An MLIR dialect makes the passes general. Hand-tuned fixes are tied to one accelerator → but tracking configuration state in the accfg abstraction → so the same optimizations apply across accelerators, hosts, and interconnects.
The model predicts the speedup the passes deliver. Ad-hoc tuning cannot say how much it will help → but a quantitative roofline tying intensity and bandwidth to performance → so the measured 2 times geomean gain matches what the model forecasts.

What it costs (trade-offs and scope)

Overlapping needs hardware support. Configuration-computation overlapping only works on accelerators that can be reconfigured while computing (concurrent configuration); on strictly sequential accelerators, only deduplication applies, so part of the benefit is hardware-dependent.
Deduplication needs provable invariance. The compiler can only remove configuration writes it can prove are unchanged; configurations that vary every operation, or that the analysis cannot prove static, still incur their full cost.
It assumes a register-based, tightly-coupled model. The paper focuses on accelerators controlled through configuration registers by a tightly-coupled host without an OS; systems with more layers of hardware protocol or software abstraction perform the same operations but through additional layers the analysis must be extended to cover.
The abstraction is an MLIR-specific mechanism. The optimizations are realized as an MLIR dialect and passes, so adopting them means working within the MLIR compiler framework and modeling an accelerator’s configuration in accfg.
Gains depend on how configuration-bound the workload is. The 2 times geomean speedup reflects systems and workloads sitting near the configuration wall; a workload already compute- or memory-bound has less configuration overhead to remove, so the benefit varies with where a system sits on the roofline.

The comparison

Handling accelerator configuration overhead

Property	Classic roofline plus manual tuning	Ad-hoc per-accelerator fixes	This work
Models configuration overhead
Generality across accelerators
Removes redundant configuration
Hides remaining configuration
Reported outcome

Approaches to accelerator configuration overhead. The classic roofline models compute and memory but not configuration, and prior fixes were ad-hoc and tied to one accelerator, with no way to know when a system is configuration-bound. This work introduces a roofline over operation-to-configuration intensity to characterize the wall, and an MLIR dialect (accfg) with two general passes — deduplicating unchanged configuration and overlapping the rest with computation — that automatically move systems off the wall, for a geomean 2 times speedup on the open-source OpenGeMM and Gemmini accelerators. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ arXiv copy, 2511.10397, of the ASPLOS 2026 paper; code at github.com/kuleuven-micas/snax-mlir). The described work is the paper’s: it names and characterizes the configuration wall — the performance loss when a host CPU spends increasing time configuring and synchronizing an accelerator, which worsens as accelerators get faster and more flexible. Its two contributions are a new roofline model that plots attainable performance against operation-to-configuration intensity versus a maximum configuration bandwidth to determine when a system is configuration-bound, and a compiler abstraction — an MLIR dialect called accfg — that tracks accelerator configuration state and enables two optimization passes: configuration deduplication (skipping register writes the compiler proves are unchanged) and configuration-computation overlapping (configuring the next operation during the current computation on accelerators that support it). It is implemented in MLIR and evaluated on the open-source matrix-multiplication accelerators OpenGeMM and Gemmini, each tightly coupled to a RISC-V host, reporting a geomean 2 times speedup on concurrent-configuration systems by eliminating redundant configuration cycles and hiding the remainder. The paper focuses on register-based, tightly-coupled, OS-less accelerator control; more complex accelerators perform similar operations through additional protocol or abstraction layers. Background on hardware accelerators, the roofline model, and MLIR is standard context. The step-by-step walkthrough illustrates the described mechanism. Exact model definitions, the accfg passes, and full results should be read from the paper.

Background

Why specialized hardware keeps discovering that the overhead is the point

It helps to see the configuration wall as one instance of a recurring lesson in specialized computing. As general-purpose scaling has slowed, the industry has leaned on domain-specific accelerators for performance per watt, and a great deal of ingenuity has gone into making the compute cores themselves faster and more efficient. But time and again, once the core computation is fast, the surrounding costs — moving data to and from the accelerator, invoking it, synchronizing with the host, and here, configuring it — turn out to dominate. The faster and more specialized the accelerator, the more these overheads, which do not shrink with the compute, come to define real-world performance. The response has matured from noticing such overheads in specific systems to modeling them and building general compiler support to eliminate them, so that the accelerator’s theoretical speed can actually be realized end to end. This effort to make reconfigurable accelerators deliver their promise by taming the cost of reconfiguring and controlling them is the same terrain the work on a systematic framework for heterogeneous reconfigurable accelerator arrays (REVAMP) covered elsewhere on this site occupies, wrestling with the flexibility-versus-overhead trade-off of configurable hardware. This paper’s contribution is to give that trade-off a name and a model — the configuration wall and its roofline — and a general, compiler-driven way over it.

If you remember only three things

1. Systems offload compute to hardware accelerators for performance per watt, but an accelerator must first be configured by the host CPU — writing its configuration registers, launching, and synchronizing — and those cycles are wasted on setup rather than useful work. This configuration overhead is a largely fixed cost per invocation that does not shrink when the accelerator computes faster, so as accelerators get faster and more flexible (more configuration options), the setup becomes a growing fraction of total time, exactly as in Amdahl's law. The paper names this the configuration wall, and it exposes a real trade-off: adding configurability makes an accelerator more useful but, without optimization, directly cuts achievable performance, so a more reconfigurable accelerator can make the whole system slower. Existing performance models do not capture this and existing compilers cannot optimize it away.
2. The paper supplies both a model and a mechanism. It extends the roofline model to configuration: rather than plotting operational intensity (operations per byte) against memory bandwidth, it plots operation-to-configuration intensity (useful operations per configuration) against a maximum configuration bandwidth, so one can see exactly when a system is configuration-bound and how far from the wall. That model motivates two compiler optimizations, realized through a new MLIR dialect, accfg, that tracks each accelerator's configuration state: configuration deduplication, which skips rewriting configuration registers the compiler can prove are unchanged (for example across loop iterations), eliminating redundant cycles; and configuration-computation overlapping, which, on accelerators that can be reconfigured while computing, configures the next operation during the current one to hide the remaining setup latency behind useful work.
3. Because the optimizations live in an MLIR abstraction rather than hand-tuned per accelerator, they apply broadly across accelerators, hosts, and interconnects — and the roofline model predicts how much they help. Implemented in MLIR and evaluated on the open-source matrix-multiplication accelerators OpenGeMM and Gemmini, each tightly coupled to a RISC-V host, the passes deliver a geomean 2 times speedup by removing redundant configuration and hiding the rest. The trade-offs: overlapping requires hardware that supports concurrent configuration, deduplication only removes configuration the compiler can prove is invariant, the work targets register-based tightly-coupled accelerator control, and the benefit depends on how configuration-bound a workload already is. The broader value is giving an overlooked bottleneck a name, a model, and a general compiler path around it.

Check yourself

Check your understanding

1. What is the configuration wall, and why do faster accelerators hit it harder?
2. How does the paper adapt the roofline model to configuration overhead?
3. What are the paper's two compiler optimizations, and what abstraction enables them?
4. What did the paper achieve, and what is a key limitation of one of its optimizations?

Sources

The paper is “The Configuration Wall: Characterization and Elimination of Accelerator Configuration Overhead” by Josse Van Delm, Anton Lydike, Joren Dumoulin, Jonas Crols, Xiaoling Yi, Ryan Antonio, Jackson Woodruff, Tobias Grosser, and Marian Verhelst (KU Leuven and the University of Cambridge), published at ASPLOS 2026, DOI 10.1145/3760250.3762225. It is available on arXiv as 2511.10397, with code at github.com/kuleuven-micas/snax-mlir.

This explainer is written from the paper’s full text (the authors’ arXiv copy). The paper names and characterizes the configuration wall — the performance loss when a host CPU spends growing time configuring and synchronizing an accelerator, worsening as accelerators get faster and more flexible. It contributes a roofline model plotting attainable performance against operation-to-configuration intensity versus a maximum configuration bandwidth to identify when a system is configuration-bound, and an MLIR dialect, accfg, that tracks configuration state and enables two passes: configuration deduplication (skipping provably-unchanged register writes) and configuration-computation overlapping (configuring the next operation during the current computation where the hardware allows). Evaluated in MLIR on the open-source matrix-multiplication accelerators OpenGeMM and Gemmini, each tightly coupled to a RISC-V host, the paper reports a geomean 2 times speedup. Background on hardware accelerators, the roofline model, and MLIR is standard context. The numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.