SuperNoVA: Algorithm-Hardware Co-Design for Resource-Aware SLAM

For AR and VR to feel right, a headset must constantly map its surroundings and track itself in real time, but the accurate backend that corrects accumulated drift occasionally hits a huge, unpredictable computation — a loop closure — that blows the frame deadline and triggers motion sickness; this work from UC Berkeley co-designs the whole stack so a wearable can keep up, with an algorithm that spreads that expensive update across many frames to always hit the deadline, a runtime that virtualizes accelerators to exploit the problem's tree structure, and a low-power chip tailored to the sparse, dynamic math SLAM actually does.

The argument in one glance

Problem

AR headsets must map and track in real time, or you get sick

SLAM is how a device figures out where it is while building a map of an unknown space, and for AR and VR it has to run on a lightweight, battery-powered headset while staying accurate and hitting every frame. Missing a frame deadline, or tracking your position wrongly, makes virtual objects jump or drift against the real world, which over minutes of use induces motion sickness. So the computation must be both accurate and reliably fast on very constrained hardware — a genuinely hard combination.
What people did

Use an incremental solver that only updates what changed

The state-of-the-art online approach, ISAM2, is an incremental solver: instead of re-optimizing the whole map every frame, it updates only the small part affected by the newest measurements. Most frames touch only a little near the top of its computation tree, so it is usually fast and accurate. It represents the map as a factor graph and factorizes a large sparse matrix incrementally as the device moves.
Why it fell short

A loop closure triggers a huge, deadline-busting spike

The catch is loop closure. When the device revisits a place it has seen before, the solver must correct the accumulated drift, which updates a node deep in its tree and every ancestor above it — a large, unpredictable recomputation. Its latency spikes far past the frame deadline, so during the long solve the system falls back on a stale estimate and tracking error jumps. Fixed-function SLAM accelerators cannot rescue this, because the sparse structure of the update is only determined at runtime and their hardwired sizes cannot bound the latency of a big one.
This paper's idea

Co-design algorithm, runtime, and chip to always hit the deadline

SuperNoVA rebuilds the whole stack together. Its algorithm, RA-ISAM2, no longer relinearizes everything a loop closure touches at once; it estimates the cost of each candidate update and greedily takes the most important ones that still fit the remaining time, spreading the expensive correction across several frames. A runtime virtualizes a pool of accelerators and schedules the computation tree across them, hiding memory latency behind compute. And a small, low-power chip provides exactly the sparse-matrix and dynamic-memory machinery SLAM needs, so the device always meets its deadline while staying accurate.

Why this problem is worth solving

The workload at the center of this paper is one of the defining problems of embodied computing.

SLAM, for simultaneous localization and mapping, is the problem of a device figuring out where it is while building a map of a place it has never seen before, using only its own sensors. It is foundational for robots, self-driving cars, and augmented and virtual reality, where a headset must know its exact position and orientation to place virtual objects convincingly in the real world. A SLAM system has two halves. The frontend reads the sensors and turns them into constraints — this measurement says these two positions are a certain distance apart. Its cost is small and steady. The backend then solves for the most likely set of positions and map given all those constraints, and this is the hard part: its cost swings wildly from frame to frame depending on where the device has been and what it now sees. Taming that variability, so the backend reliably finishes within a frame time on a small device, is exactly the challenge this work takes on.

The single event that makes the backend so unpredictable is also the one that keeps the map honest.

A loop closure is the moment a device realizes it has returned to a place it visited before. It is enormously valuable, because it lets the system correct all the small errors that have quietly accumulated along the way — without it, a device wandering a building slowly drifts until its map no longer matches reality. But acting on a loop closure is expensive. Recognizing that a position now lines up with one seen long ago forces the solver to go back and adjust a large swath of the map at once, a computation far bigger than a normal frame’s, and one whose exact size cannot be predicted in advance. That is the crux of real-time SLAM: the very event that keeps the map accurate is also the one that threatens to blow the frame deadline, and handling it gracefully is what separates a smooth experience from a nauseating one.

What people did before, and where it broke

Local solvers (VIO, sliding-window smoothers). Keep only a fixed window of recent poses and discard older ones, marginalizing them out. Where it broke. The latency is bounded, but because it throws away history it never closes loops, so small errors accumulate into large drift and the map slowly diverges from reality.
Global solvers (full bundle adjustment, pose graph optimization). Optimize the whole map so loop closures can be handled and drift corrected. Where it broke. There is no latency bound — a loop closure is a huge solve, and while it runs the system must use a stale estimate, so the error spikes anyway and the correction arrives too late.
Incremental solvers (ISAM2). Update only the sub-map the newest measurements affect, staying fast and accurate on ordinary frames. Where it broke. A loop closure updates a node deep in the computation tree and every ancestor above it, producing exactly the latency spike that overruns the deadline; the very case that matters most is the one it handles worst.
Fixed-function SLAM accelerators. Characterize the workload, then build an FPGA or ASIC accelerator sized to it, as prior drone and VIO chips do. Where it broke. They work only for small, static problem sizes; the sparse structure of a loop-closure update is determined at runtime, and hardwired units cannot reshape to it or bound the latency of a large problem.
The gap. Each option sacrificed one of the three things AR needs at once: local solvers gave up accuracy, global and incremental solvers gave up the latency bound, and fixed accelerators gave up the flexibility to handle whatever the runtime throws at them. Nobody had co-designed an algorithm that reshapes its own work to fit a deadline together with hardware flexible enough to run whatever that reshaped work turns out to be. That full-stack combination is what large-scale real-time SLAM on a wearable actually requires.

How SuperNoVA works

Reshaping the backend to fit a frame, then running it on tailored hardware

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Reshape the algorithm’s own workload to fit the deadline, instead of hoping it finishes in time. The deepest move is to stop treating the loop-closure computation as a fixed lump that either fits the frame or does not. The standard incremental solver relinearizes every variable a loop closure affects in one shot, and when that overruns the budget it simply misses the deadline. RA-ISAM2 instead makes the amount of work a decision: it estimates the cost of each candidate update and admits only as many as the remaining frame time allows, spreading a big correction over several frames so no single one ever stalls. Deliberately doing less this instant, in a principled way, to guarantee a real-time deadline — and catching up over subsequent cycles — is the same discipline the cooperative-embodied-agent work covered elsewhere on this site follows, reshaping its computation to fit a real-time budget rather than letting an expensive step block the whole system.

Build the hardware around the workload’s real bottlenecks, not just its dense multiplies. It is tempting to think a SLAM accelerator just needs a fast matrix-multiply engine, since the numeric factorization is mostly GEMM. But much of the real cost is elsewhere: because the map’s sparse structure changes every frame, the system spends heavily on dynamically allocating workspace and scattering tiny sparse blocks into place, work a bare matrix engine leaves stranded on the general-purpose core. SuperNoVA adds a dedicated memory accelerator for the dynamic allocation and a sparse-index unit that packs many small block scatters into a single instruction, and as a result it runs the numeric part far faster than a prior accelerator that only sped up the GEMM. Recognizing that the true bottleneck of an irregular workload is the memory and bookkeeping a generic engine ignores, and building silicon for exactly that, is the same insight the GPU graph-learning work covered elsewhere on this site acts on, tailoring hardware to the irregular, memory-bound part of a workload that a stock engine serves poorly.

Virtualize a pool of accelerators and orchestrate them from software over the tree. Rather than freeze one accelerator sized to a guessed problem, SuperNoVA exposes a pool of programmable compute and memory accelerators and lets a software runtime conduct them, allocating accelerators to the parts of the computation tree that are ready, running independent branches in parallel, splitting the big nodes near the root across several accelerators, and overlapping memory moves with compute so nothing waits. Because the accelerators are virtualized and decoupled from the core, the same hardware bends to whatever shape a given frame produces. Orchestrating a set of virtualized hardware accelerators from software through a clean, disaggregated interface, so the mix of resources can be reassigned on the fly, is the same architecture the FPGA-middlebox work covered elsewhere on this site is built on, where software conducts a pool of accelerators rather than welding them into fixed hardware.

Following the reasoning through SuperNoVA

Surviving a loop-closure frame

Following the single hardest kind of frame shows how each layer keeps the deadline while preserving accuracy.

Step 1 of 5

The device revisits a place, and a naive solver panics

A loop closure fires: the headset recognizes a spot it saw long ago. A standard incremental solver would now have to relinearize a variable deep in its tree and refactorize every ancestor above it at once — hundreds of milliseconds of work, many times the 33-millisecond frame budget.

This is the latency spike that makes the otherwise-excellent incremental solver unusable for AR.

Why it works

Spreading a loop closure across frames turns a spike into a steady stream. A single loop closure updates a node deep in the tree and every ancestor at once, a burst that overruns the frame budget → but the algorithm estimates each update’s cost and admits only as much as the remaining time allows, deferring the rest → so the same total work is done over several frames and no single frame ever misses its deadline.
Ranking updates by how far a variable has drifted spends a tight budget where it counts. With too little time to fix everything, choosing arbitrarily would waste effort on updates that barely matter → but scoring each variable by the size of its pending correction puts the most-wrong ones first → so the frame’s limited compute buys the largest possible accuracy gain, and the deferred updates were the least urgent anyway.
Programmable accelerators absorb a structure that is only known at runtime. A loop closure reshapes the sparse matrix in a way no one can predict before it happens, which is fatal for fixed-function units sized in advance → but a pool of programmable, virtualized accelerators can be re-orchestrated on the fly to whatever tree the frame produces → so the same hardware bounds the latency of both a tiny update and a huge one.
Building in memory and sparse-scatter machinery beats a bare matrix engine. SLAM spends much of its time not on dense multiplies but on dynamically allocating workspace and scattering tiny sparse blocks → but a dedicated memory accelerator and a sparse-index unit take over exactly that bookkeeping → so the numeric part runs far faster than on a prior GEMM-only accelerator that left those costs on the general-purpose core.
Exploiting the tree’s parallelism and hiding memory behind compute fills a tiny chip. The elimination tree has many independent branches near the leaves and few large nodes near the root, and naive scheduling either idles hardware or thrashes the cache → but the runtime runs independent branches in parallel, splits the big root nodes across accelerators, and overlaps memory moves with compute → so a chip that is 40 percent of a single out-of-order core sustains real-time SLAM at 114 milliwatts.

What it costs (trade-offs and scope)

It runs in simulation, not on a real headset. Everything is FPGA-accelerated cycle-exact RTL simulation plus 16-nanometer synthesis, on replayed pose-graph datasets — two synthetic and two from a real AR capture. There is no fabricated chip or live AR demonstration, so real sensor noise and system integration are untested.
The headline gains are the best datasets. The 89.5-percent reduction over a CPU and 78.6 percent over a GPU are on the long AR trajectory; on the sparse M3500 dataset the gain over the GPU is under 1 percent and the total is 3 times slower than a server CPU, because small supernodes leave the accelerators underused. The benefit depends heavily on dataset structure.
“Always meets the deadline” is achieved by dropping work. When the map history grows too large, updates deep in it cannot fit the deadline, so the algorithm drops older measurements to stay on time — a real-time-versus-completeness trade the authors flag as a scalability limit, with a base-station offload proposed as future work.
The accuracy ceiling is the very solver it replaces. RA-ISAM2 is judged against the full incremental solver treated as an infinite-compute ideal. It approaches that ideal and beats the deployable baselines, but it is fundamentally a resource-constrained approximation of the solver it is built on.
Much of the machine is prior group work. The compute accelerator is built on an existing systolic-array GEMM design, and the virtualized, disaggregated accelerator integration reuses the group’s earlier work; the new parts are the SLAM-specific algorithm, the memory accelerator, the sparse-index unit, and the runtime.
It accelerates the backend only. The non-numeric setup runs on the CPU and the entire sensor frontend is out of scope, so end-to-end system latency still includes stages this design does not speed up.
Power is not yet a first-class constraint. The chip is very low power in absolute terms, but the resource-aware algorithm optimizes for latency and accuracy, not energy; an energy-aware version is explicitly left to future work.

The evaluation

Backend solvers for real-time SLAM

Property	Local	Global	Incremental (ISAM2)
Closes loops and corrects drift
Bounded latency (meets the deadline)
Adapts to available compute	No	No
Accuracy over a long AR run
Hardware	Generic CPU or GPU	Power-hungry big solve	Generic CPU or GPU

SLAM backend solvers, with SuperNoVA evaluated in FPGA-accelerated cycle-exact simulation and 16-nanometer synthesis across M3500, Sphere, and the LaMAR AR datasets against a 33.3-millisecond frame deadline. It cuts backend latency by 89.5 percent over an out-of-order CPU and 78.6 percent over an embedded GPU on the long AR trajectory, reduces maximum pose error by 89 percent versus a local-plus-global baseline while always meeting the deadline, and does so in 40 percent of one out-of-order core's area at 114 milliwatts.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

SuperNoVA is a strong full-stack co-design: by making an incremental SLAM solver reshape its own workload to fit each frame deadline, virtualizing a pool of accelerators to run the resulting computation tree, and building a chip around SLAM’s real bottlenecks — dynamic memory and sparse scatter, not just dense multiplies — it delivers real-time, accurate, large-scale SLAM at a fraction of a GPU’s power. Read the numbers with their scope. Everything is simulated and synthesized, not fabricated or demoed on a live headset. The headline speedups are on the favorable datasets; on a sparse benchmark the gain over a GPU is under one percent and it trails a server CPU. Its deadline guarantee is achieved by dropping older work when the history grows too large, a completeness trade it acknowledges, and its accuracy is measured against the full solver it approximates. Much of the underlying accelerator and virtualization machinery is prior group work, and it speeds up only the backend, not the sensor frontend or non-numeric setup. The durable contribution is the framing: co-design an algorithm that fits its work to a deadline with hardware flexible enough to run whatever that work becomes.

Background

Why real-time robotics is its own kind of hard

It is worth seeing why so much specialized engineering goes into a problem that, on a big machine, a standard library could just solve. A SLAM backend on a datacenter server is not hard; the difficulty is entirely in the constraints of the place it has to run. An AR headset has a hard, unforgiving deadline set by the human visual system — miss it and the world stutters and the wearer feels ill — and it has almost no power or silicon area to spend, because it sits on someone’s face. Those constraints are what turn a solved numerical problem into a systems research problem: you cannot simply buy more compute, so you have to co-design the algorithm and the hardware to extract real-time behavior from a tiny budget, reshaping the work itself to fit the clock. That is the same pressure that produces domain-specific accelerators throughout computing, and the mirror image of the transformer-training work covered elsewhere on this site, which co-designs across the stack to squeeze maximum throughput out of a vast datacenter cluster rather than a wearable — the two extremes of the same discipline. The broader pattern is that as computing moves into the physical world, where deadlines are set by physics and power by a battery, the winning designs are the ones that treat the constraint, not the algorithm, as the thing to design around.

If you remember only three things

1. SLAM lets a device map an unknown space while tracking itself, and AR/VR needs it accurate, real-time, and on a tiny power-limited headset — but the accurate incremental backend, ISAM2, occasionally hits a loop closure (the device revisits a known place and must correct accumulated drift) that updates a node deep in its computation tree and every ancestor, spiking latency far past the frame deadline and, when missed, causing tracking-error jumps and motion sickness; fixed-function accelerators cannot help because the sparse structure is only known at runtime.
2. SuperNoVA co-designs the whole stack: an algorithm, RA-ISAM2, scores each candidate update by how far a variable has drifted and greedily admits only the most important ones that fit the remaining frame time, spreading a loop closure across several frames so the deadline is always met; a runtime virtualizes a pool of programmable compute and memory accelerators and orchestrates the elimination tree across them, exploiting inter- and intra-node parallelism and hiding memory behind compute; and a tailored chip adds a memory accelerator for dynamic allocation and a sparse-index unit for block scatter, not just a GEMM engine.
3. In simulation and 16-nanometer synthesis across synthetic and real AR datasets, it cuts backend latency by 89.5 percent over an out-of-order CPU and 78.6 percent over an embedded GPU on the long AR trajectory, reduces maximum pose error by 89 percent versus a local-plus-global baseline while always meeting the 33.3-millisecond deadline, in 40 percent of one out-of-order core's area at 114 milliwatts — though it is simulated not fabricated, the gains are far smaller on sparse datasets, the deadline is met by dropping older work as history grows, and it accelerates only the backend.

Check yourself

Check your understanding

1. Why does the state-of-the-art incremental SLAM solver struggle with real-time AR?
2. What is the core idea of RA-ISAM2, SuperNoVA's algorithm?
3. Why isn't a fast matrix-multiply accelerator enough for SLAM, and what does SuperNoVA add?
4. Which statement best reflects the limits of SuperNoVA's results?

Sources

The paper is “SuperNoVA: Algorithm-Hardware Co-Design for Resource-Aware SLAM” by Seah Kim, Roger Hsiao, Borivoje Nikolic, James Demmel, and Yakun Sophia Shao, published at ASPLOS 2025, pages 1035 to 1051, DOI 10.1145/3669940.3707258. The authors are at the University of California, Berkeley.

This explainer is written from the full 17-page text. Everything above comes from the paper’s own sections: the introduction and background on SLAM’s frontend and highly variable backend, the deployment requirements for AR/VR (a 33.3-millisecond, 30-frames-per-second target and stringent power and area budgets), and the comparison of local, global, incremental, and resource-aware backend solvers; the algorithm section formalizing the nonlinear-least-squares backend, its Gauss-Newton and Cholesky factorization over an elimination tree of supernodes, the Hessian construction, factorization, and merge steps, ISAM2’s fluid relinearization, and the loop-closure latency spike; the SuperNoVA system’s three parts — the RA-ISAM2 algorithm that scores variables by update-step size and greedily selects updates that fit the deadline (with cached path costs and at most two visits per node), the hardware architecture with its Gemmini-based compute accelerator plus sparse-index unroller, its memory accelerator with DMA virtual channels, and ReRoCC disaggregated integration, and the runtime that allocates virtualized accelerators over the tree exploiting inter- and intra-node parallelism and heterogeneous overlap while respecting the shared cache; the methodology using Chisel, Chipyard, and FireSim with a 16-nanometer synthesis, the M3500, Sphere, and LaMAR CAB datasets, and six hardware and three algorithm baselines; and the evaluation reporting 89.5-percent and 78.6-percent backend latency reductions over an out-of-order CPU and an embedded GPU on the AR trajectory, runtime-parallelism breakdowns, a resource-aware algorithm that always meets the 33.3-millisecond deadline where the incremental baseline misses it by up to 68 percent, accuracy reductions of up to 89 to 99 percent in maximum and incremental RMS error over baseline solvers, a 40-percent-of-BOOM area and 114-milliwatt power figure, and the future-work discussion of finite scalability, base-station offload, and energy-awareness.