SlimSLAM: An Adaptive Runtime for Visual-Inertial Simultaneous Localization and Mapping

Armand Behroozi, Yuxiang Chen, Vlad Fruchter, Lavanya Subramanian, Sriseshan Srikanth and Scott A. Mahlke

SLAM lets a headset or robot track where it is by fusing camera and motion data, but doing it accurately is expensive and the software ships one fixed configuration tuned for the hardest case, wasting compute the rest of the time; this work from the University of Michigan and Meta builds SlimSLAM, a runtime that watches the SLAM algorithm's own internal health — how well it is tracking its visual landmarks — to dial back over-provisioned settings when the scene is easy, and separately trims quality just enough to hit real-time deadlines when the device is loaded, running up to 5.4 times faster at the same accuracy and enabling good SLAM on a Raspberry Pi.

The argument in one glance

Problem

Accurate SLAM is expensive, but the devices that need it are constrained

SLAM tracks where an agent is and maps its surroundings in real time — head tracking for AR and VR headsets, navigation for drones and robots. It must be both fast and accurate: a laggy or inaccurate headset causes motion sickness, a drifting drone crashes. But state-of-the-art accuracy is computationally heavy, while the platforms that need it most, like untethered headsets and small robots, are tightly limited in power and compute. SLAM algorithms expose dozens of tunable knobs — camera frame rate, number of visual landmarks tracked — that trade computation for quality, yet designers ship a single static configuration tuned for the worst-case input.
What people did

Adapt the knobs at runtime from a cheap signal like the agent's motion

Since a static worst-case configuration wastes compute on every easier input, the natural fix is to adapt the knobs while running. Prior adaptive schemes reach for a cheap, easy-to-measure external signal: the agent's velocity or how far it moved between frames. The intuition is that SLAM gets harder when the agent moves fast — motion blurs images and shifts the scene — so when motion is low, turn the knobs down to save computation.
Why it fell short

External proxies misjudge modern SLAM and adapt too bluntly

Motion is a poor proxy. Modern SLAM algorithms handle fast motion well, so the agent's velocity correlates weakly — below 0.17 — with what actually degrades tracking. Worse, these schemes turn all the knobs together from that one signal, never observing how the algorithm is actually doing. The result is a scheme that is either too conservative, leaving performance unclaimed, or too aggressive, causing tracking to fail — and SLAM is unforgiving, since once tracking is lost there is often no recovery.
This paper's idea

Watch the algorithm's own health, and separately meet the deadline

SlimSLAM adapts from the SLAM algorithm's own internal state instead of an external proxy. Its key observation is that these algorithms perceive the world through a sparse set of visual landmarks — key points — so it directly watches key point health: how long landmarks stay trackable, how many yield good pose updates, how near they are. This tells it not just when computation is over-provisioned but how, so it dials the right knob. A second, separate scheduler then trims quality just enough to meet real-time deadlines when the device itself is loaded.

Why this problem is worth solving

The heart of the opportunity is that SLAM’s cost depends enormously on the input, yet it is configured as if every input were the hardest, so it is worth seeing how much is left on the table.

Visual-inertial SLAM fuses two sensors. An inertial measurement unit reports acceleration and rotation hundreds of times a second, giving fast pose updates that drift within a fraction of a second. Cameras correct that drift by tracking how the scene moves: the algorithm finds distinctive patterns — key points, usually corners — in each frame, re-identifies them in the next, discards the ones that moved inconsistently, and triangulates their 3D positions to estimate the agent’s motion. A slower, heavier mapper runs occasionally on selected keyframes to build a longer-horizon map that limits drift over time. Every one of these stages has knobs: the camera frame rate, the maximum number of key points to track, whether to use one camera or a stereo pair, how many optimization iterations to run. Turning them up costs compute and improves accuracy; turning them down saves compute and risks error. The catch is that the right setting depends heavily on the scene — a slow walk through a feature-rich room needs far less than a fast turn through a bare corridor. But designers pick one static configuration for the worst case, so on an easy input the algorithm does vastly more work than it needs. The paper measures the gap: choosing the best static configuration per input sequence, with no quality loss, would cut computation by up to nearly twenty times. That is the waste an adaptive runtime could reclaim.

The reason reclaiming it is hard, and why prior attempts fell short, is that judging an input’s difficulty from the outside is unreliable and unforgiving.

To adapt safely, a runtime must know when the current settings are more than the scene requires — but predicting how a knob change will affect tracking quality on an unseen scene is genuinely hard, and SLAM punishes mistakes severely, because once tracking is lost there is usually no way to recover. Prior adaptive schemes tried to sidestep the difficulty by using a cheap external signal, the agent’s motion, as a stand-in for difficulty: move slowly, turn the knobs down. But this fails on two counts. First, modern SLAM handles fast motion well, so motion barely predicts real difficulty — the measured correlation is under 0.17 — and other cheap signals like image blur cost more to compute than the adaptation would save. Second, a single external signal cannot tell you which knob to turn; it forces a blunt, uniform adjustment of everything at once. SlimSLAM’s insight is that the algorithm already computes the exact signal that matters: since it perceives the world only through key points, the health of those key points — how many frames a landmark stays trackable, how many produce successful pose updates, how near they are to the camera — directly reflects how well SLAM is doing, for free, whatever the cause. Watching that internal state instead of an external proxy is what lets adaptation be both aggressive when it is safe and cautious when it is not, and lets it pick the specific knob that costs the least quality. That is what makes a state-driven runtime worth building.

What people did before, and where it broke

Ship one static configuration tuned for the worst case. Pick knob settings that keep tracking robust on the hardest expected input, and use them always. Where it broke. Every easier input then runs vastly over-provisioned — up to nearly twenty times more compute than needed at the same quality — which a power- and compute-constrained device simply cannot afford.
Adapt the knobs from the agent’s motion or velocity. Use a cheap external signal as a proxy for difficulty, turning settings down when the agent moves slowly. Where it broke. Modern SLAM tolerates fast motion, so velocity correlates below 0.17 with real tracking difficulty, making the scheme either too timid (wasting compute) or too aggressive (losing tracking, which is unrecoverable).
Control all parameters uniformly from that one signal. Treat the knobs as a single dial driven by the external metric. Where it broke. Different situations are over-provisioned in different ways — sometimes the frame rate is too high, sometimes too many key points are tracked — so a uniform adjustment cannot degrade the specific knob that costs the least quality.
Build a specialized SLAM accelerator, or drop expensive stages. Design an ASIC for efficiency, or remove the mapper to run faster. Where it broke. ASICs risk obsolescence over their long design cycles as SLAM keeps advancing, and dropping the mapper speeds things up but hurts accuracy, because mapping is what limits long-term drift.
The gap. Every approach either fixed the configuration for the worst case, judged difficulty from an unreliable external proxy, adjusted everything bluntly, or baked in a design that cannot follow the algorithm. What was missing was a runtime that reads the algorithm’s own internal health to know when and how it is over-provisioned, and that separately guarantees the real-time deadline — which is what adaptive SLAM on constrained devices needs.

How SlimSLAM works

Adapting SLAM from its own state, and meeting the deadline

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Adapt to the actual input instead of provisioning for the worst case. The entire waste comes from a static configuration sized for the hardest scene the device might ever face, applied uniformly to every easy scene it usually faces. SlimSLAM matches the work to the input frame by frame, reclaiming the up-to-nearly-twenty-times over-provisioning that a fixed setting leaves on the table, without dropping quality, because it only turns knobs down when the scene genuinely does not need them. Replacing a conservative, one-size-fits-all default with computation specialized to what the current situation actually requires is the same move the recommendation-compiler work covered elsewhere on this site makes, proving the framework’s blanket, worst-case safety work unnecessary in context and paring it down to just what is needed.

Measure the real signal from inside the algorithm, not a cheap proxy from outside. Prior schemes guessed difficulty from the agent’s motion because it was easy to measure, but on modern SLAM that proxy barely tracks what actually matters. SlimSLAM’s insight is that the algorithm already computes the true signal — the health of the key points it sees the world through — so difficulty can be read directly and freely, whatever its cause, whether motion, blur, or poor lighting. Reading a system’s real internal behavior rather than trusting an external stand-in that only approximates it is the same discipline the GPU-kernel study work covered elsewhere on this site follows, learning what actually governs behavior by observing the real thing instead of an assumed proxy.

Separate adapting to the input from adapting to the budget. These are two different problems: whether the scene is easy is about the input, while whether the device is falling behind is about load and battery the algorithm cannot control. SlimSLAM decouples them, using key point health to shed work the input does not need, and a distinct deadline-aware scheduler, over a disjoint set of knobs, to trim quality only when the compute budget demands it. Splitting one conflated control problem into two independently governed axes is the same decoupling the resource-resizing work covered elsewhere on this site performs, separating what you adjust from when you must adjust it so each can be reasoned about and bounded on its own.

Following a headset from an easy room into a hard turn

Adapting as the scene changes and the device loads up

Tracing one session shows both halves of SlimSLAM at work.

Step 1 of 5

A headset tracks calmly through a feature-rich room

The wearer walks slowly through a room full of corners and texture. Key points stay trackable for many frames and most yield successful pose updates, so SlimSLAM reads that the default frame rate and landmark count are far more than this easy scene requires.

The signals it reads — track length and successful updates — are already computed by the SLAM algorithm.

Why it works

Reading internal key point health beats guessing from external motion. The agent’s velocity correlates below 0.17 with real difficulty on modern SLAM, so motion-driven schemes misjudge the scene → but key point health directly measures how well the algorithm is actually tracking → so SlimSLAM cuts compute hard when it is truly safe and holds back when it is not, which a proxy cannot do reliably.
Adapting per frame reclaims the over-provisioning a static setting bakes in. One configuration tuned for the worst case runs heavy on every easy input → but matching the settings to each frame’s actual difficulty removes the slack → so easy scenes recover up to nearly twenty times of wasted compute with no loss in accuracy.
Different feedback signals pick the specific knob that costs least quality. A single external metric forces a blunt, uniform adjustment of everything → but track length drives the frame rate while depth drives landmark count and camera use → so SlimSLAM degrades exactly the parameter the situation over-provisions rather than sacrificing quality across the board.
A separate deadline scheduler handles what the input cannot explain. A device can fall behind from background load or a low battery, unrelated to scene difficulty → but a distinct scheduler predicts task times and trims a disjoint set of knobs against the budget → so SLAM still meets its real-time deadline, degrading gracefully instead of failing or lagging.
Observing only already-computed state keeps the runtime nearly free. Measuring difficulty could cost more than adaptation saves if it needed new sensing → but track length and depth are byproducts SLAM already produces → so the whole runtime adds under one percent overhead and, seeing only state rather than raw pixels, generalizes across environments.

What it costs (trade-offs and scope)

The deadline scheduler needs offline, per-hardware characterization. SlimSLAM profiles each parameter’s quality and performance effect on training inputs, which took a few days per platform, and because each device differs it must be re-characterized per hardware target; inaccurate characterization degrades quality and timeliness, though not fatally.
It is demonstrated on a single SLAM algorithm. The evaluation uses HybVIO; the design deliberately relies only on parameters and state variables the authors argue are common to key-point SLAM, but generality across algorithms is argued, not shown.
Depth-based savings arrive one use late. Because the algorithm learns a key point’s depth only after triangulating it, the mono-camera shortcut for far points cannot apply on that point’s first use, and HybVIO’s lack of stereo rectification limits inferring depth earlier.
Adaptation can be too slow when tracking is lost abruptly. On an input with hundreds of suddenly missing frames, timely adaptation tuned for other sequences reacts too late, and frame skipping still cannot beat a static oracle on every input; some scenarios, like a headset dataset with non-parallel stereo cameras, leave little room to adapt.
Graceful degradation still degrades. Under a tight budget SlimSLAM trims quality and, as a last resort, drops frames, so it is a best-effort tradeoff that meets deadlines by lowering accuracy, not free performance.
Onboarding a new algorithm takes user effort, and mapper adaptation is limited. The user must specify the task graph, tunable parameters, and prediction variables, and most savings come from the tracker (about five times) while the mapper is reduced only about twofold, with more mapper adaptation left to future work.

The evaluation

Three ways to configure SLAM on a constrained device

Property	Static worst-case config	Motion-proxy adaptive
What drives adaptation
Which knobs it turns	None
Risk profile
Real-time deadlines	Only if the worst case fits	Not explicitly guaranteed
Result	Caps quality on constrained hardware	Limited speedup at iso-quality

Three ways to run SLAM on a constrained device, evaluated on the HybVIO algorithm across the EuRoC (drone), TUM-VI Room (hand-held), and Monado VR (headset) datasets, on a desktop CPU and a 2-core Raspberry Pi 4. SlimSLAM's core adaptation reaches 5.4, 5.0, and 2.8 times speedups over the static default with under 0.005 meters average accuracy loss, beats two motion-proxy adaptive schemes by about 2.2 to 2.4 times at equal quality, cuts energy roughly 4 times, and lets the Raspberry Pi run in real time at twice the default's quality — because it reads the algorithm's own state instead of an external proxy and meets deadlines with a separate scheduler.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

SlimSLAM is a clean, well-aimed systems idea: SLAM ships a static configuration tuned for the worst case and wastes up to nearly twenty times the compute it needs on easy inputs, so adapt. Its cleverness is where it looks — not at an external motion proxy, which modern SLAM makes nearly useless (correlation under 0.17), but at the algorithm’s own key point health, the very signal it already computes, so difficulty is read directly and for free. Reading track length and depth lets it dial the right knob — frame rate, landmark count, stereo versus mono — and a separate deadline-aware scheduler trims quality only when the device itself falls behind. The results are strong: up to 5.4 times faster at equal accuracy, about 2.2 to 2.4 times better than motion-proxy schemes, roughly 4 times less energy, under 1 percent overhead, and real-time SLAM on a Raspberry Pi at twice the static quality. Read it with its scope. It is shown on one SLAM algorithm, with generality argued rather than demonstrated; the deadline scheduler needs a few days of per-hardware offline characterization, and inaccurate characterization degrades quality; depth-based savings arrive a use late; abrupt tracking loss can outrun the adaptation; and graceful degradation still lowers accuracy and can drop frames. The durable lesson is that the best signal for adapting a system is usually the one the system already computes about itself, not a convenient measurement from outside.

Background

Why best-effort systems on the edge live or die by adaptation

It helps to see SlimSLAM as part of a broad pattern in edge and embedded computing: when a demanding real-time task must run on a device that cannot always afford it, the answer is rarely a faster chip and usually a smarter tradeoff between quality and cost, made continuously as conditions change. SLAM is a textbook best-effort application — there is no single right amount of computation, only the most accuracy you can buy within the power, thermal, and latency budget you happen to have this instant, which shifts as the scene, the load, and the battery change. Provisioning for the worst case wastes the common case, while provisioning for the common case fails the worst, so the only real option is to adapt, and to adapt you must sense the right thing cheaply. The in-storage serverless-acceleration work covered elsewhere on this site fights a kindred battle on the other side of the constraint, squeezing a demanding workload into a tiny power budget by rethinking where the work happens rather than making it faster. The broad lesson is that as capable applications push onto constrained edge hardware, the decisive engineering is dynamic resource management — matching effort to need, moment by moment, under a budget that never stops moving — far more than raw peak performance.

If you remember only three things

1. SLAM must run fast and accurately for AR/VR head tracking and robot navigation, but accurate SLAM is compute-heavy while the target devices are power- and compute-constrained. SLAM algorithms expose dozens of knobs trading compute for quality, yet ship one static configuration tuned for the worst-case input, so they are hugely over-provisioned on easy inputs — per-sequence static tuning could cut compute up to nearly twenty times with no quality loss. Prior adaptive schemes tuned the knobs from an external motion proxy, but modern SLAM handles fast motion well so velocity correlates below 0.17 with real difficulty, and driving all knobs from one signal is too blunt, leaving these schemes either too timid or prone to unrecoverable tracking failure.
2. SlimSLAM adapts from the algorithm's own internal state rather than an external proxy: since key-point SLAM perceives the world only through sparse landmarks, it watches key point health — viable track length and successful pose updates to drive frame rate, and key point depth to drive landmark count and stereo-versus-mono choice (close scenes need fewer, higher-diversity observations; far points use one camera). This tells it both when and how it is over-provisioned, so it turns the specific knob that costs least quality. A separate deadline-aware scheduler, built from offline per-parameter characterization and per-task execution-time prediction models, trims a disjoint set of knobs to meet the real-time budget under load, dropping frames only as a last resort.
3. On HybVIO across the EuRoC, TUM-VI Room, and Monado VR datasets on a desktop and a 2-core Raspberry Pi 4, SlimSLAM-core reaches 5.4, 5.0, and 2.8 times speedups at under 0.005 meters average accuracy loss, beats two motion-proxy adaptive schemes by about 2.2 to 2.4 times at iso-quality, cuts energy roughly 4 times, adds under 1 percent overhead and about 2.2 kilobytes of memory, and lets the Raspberry Pi run in real time at twice the static quality while beating other SLAM algorithms that drop the mapper. Its limits: it is shown on one algorithm, needs a few days of per-hardware offline characterization (inaccuracy degrades but is not fatal), depth savings arrive a use late, abrupt tracking loss can outrun adaptation, and graceful degradation still lowers accuracy.

Check yourself

Check your understanding

1. Why does shipping one static SLAM configuration waste so much computation?
2. What is the key difference between SlimSLAM and prior motion-based adaptive schemes?
3. How does SlimSLAM use key point depth to save computation?
4. Which statement best reflects the scope and limits of SlimSLAM?

Sources

The paper is “SlimSLAM: An Adaptive Runtime for Visual-Inertial Simultaneous Localization and Mapping” by Armand Behroozi, Yuxiang Chen, Vlad Fruchter, Lavanya Subramanian, Sriseshan Srikanth, and Scott Mahlke, published at ASPLOS 2024, DOI 10.1145/3620666.3651361. The authors are at the University of Michigan and Meta Platforms.

This explainer is written from the full 16-page text. Everything above comes from the paper’s own sections: the introduction motivating adaptive SLAM (the up-to-19.6-times over-provisioning of static configurations and the below-0.17 correlation of external motion metrics with difficulty); the background on visual-inertial SLAM’s task graph (IMU integration, the key-point tracker with optical-flow re-identification and RANSAC outlier rejection, the multi-state constraint Kalman filter, and the heavier keyframe mapper with bundle adjustment) and on SLAM error (root-mean-square absolute trajectory error and graceful degradation); the SlimSLAM-core adaptation using viable track length and successful visual updates for frame-rate control and key point depth for landmark-count and stereo-versus-mono control; the full SlimSLAM scheduler with its offline task-graph characterization, per-task execution-time prediction models, and online greedy budget-fitting with frame dropping as a last resort; and the evaluation on HybVIO across EuRoC, TUM-VI Room, and Monado VR on a desktop and a 2-core Raspberry Pi 4, reporting core speedups of 5.4, 5.0, and 2.8 times with under 0.005 meters accuracy loss, 2.22-to-2.37-times gains over the Adapt-omega and PID-pd motion-proxy schemes, roughly 4-times energy reduction, under 1 percent runtime overhead and about 2.2 kilobytes of memory, and real-time Raspberry Pi operation at twice the default quality against Basalt and DM-VIO baselines.