AGS: Accelerating 3D Gaussian Splatting SLAM via CODEC-Assisted Frame Covisibility Detection

A robot exploring an unknown space runs SLAM to build a map and track where it is, and pairing SLAM with 3D Gaussian Splatting (representing the scene as millions of fuzzy 3D blobs) gives beautiful reconstructions — but it is far too slow, needing many training iterations per frame over a vast number of blobs, so a top GPU takes over 20 minutes; this paper (Shanghai Jiao Tong University — Houshu He, Zhuoran Song, and colleagues) presents AGS, an algorithm-hardware co-design that reads the motion data a video codec already computes to measure how much consecutive frames overlap, then spends few training iterations on easy overlapping frames and skips the blobs that never affect the image, reaching up to 17.12 times the speed of a mobile GPU with about 2.36 percent quality loss.

The argument in one glance

Problem

Beautiful 3D maps that a robot cannot build fast enough to use

Simultaneous Localization and Mapping, or SLAM, lets a robot or autonomous vehicle build a map of an unknown place while working out where it is inside that map, from a stream of camera frames. A recent leap pairs SLAM with 3D Gaussian Splatting (3DGS), which represents a scene as a huge collection of small fuzzy 3D blobs, called Gaussians, that are painted onto the image; this gives photorealistic, high-fidelity reconstructions. The problem is speed. For every incoming frame, a 3DGS-SLAM system runs many training iterations to fit the Gaussians to what the camera sees, and a scene contains a vast number of Gaussians. Two costs compound: the repeated per-frame optimization, and the sheer count of blobs. As a result, even a high-end A100 GPU can take more than twenty minutes to process a sequence, which is far too slow for a robot that must map and navigate in real time. So the field has a method that produces excellent maps but cannot deliver them at a useful rate.
What people did

Run more iterations on a faster GPU, or build a 3DGS accelerator

The straightforward responses are to throw more compute at it — run the many training iterations per frame on a powerful GPU — or to build a specialized 3DGS accelerator such as GSCore that speeds up the Gaussian rendering pipeline. Both aim to make each iteration and each Gaussian cheaper to process.
Why it fell short

Making each step cheaper still repeats the same work every frame

Speeding up individual iterations and Gaussians helps, but it does not touch the deeper waste. A SLAM camera moves smoothly, so consecutive frames overlap enormously — they see almost the same scene — yet the system re-optimizes from scratch every frame, spending the same large number of training iterations even when almost nothing changed. And within the scene, a large fraction of the Gaussians never actually affect the pixels of the rendered image, yet they are processed anyway, frame after frame. A faster GPU or a rendering accelerator still pays for both kinds of redundancy: unnecessary training iterations on easy, highly-overlapping frames, and computation on Gaussians that contribute nothing. What is missing is a way to notice, cheaply, how much a new frame overlaps the last one, and to use that to skip the redundant iterations and the non-contributory blobs — without adding expensive analysis of its own.
This paper's idea

Let the video codec measure frame overlap, then skip the redundant work

AGS is an algorithm-hardware co-design that accelerates 3DGS-SLAM by exploiting the fact that SLAM processes frames in a stream where adjacent frames overlap heavily. Its key trick is to measure that overlap almost for free. Video codecs — the compression hardware already present on most chips — perform motion estimation, searching for the most similar block between consecutive frames and producing motion vectors that quantify how much the content moved. AGS reads these codec-computed values as a cheap, quantitative measure of frame covisibility (how much two frames see in common). It then uses covisibility two ways. First, movement-adaptive tracking: for a high-covisibility frame the camera barely moved, so its pose is easy to estimate and AGS uses very few training iterations, spending more only on hard, low-covisibility frames — cutting redundant iterations. Second, Gaussian contribution-aware mapping: AGS marks some frames as key frames, does full mapping on them, and records which Gaussians were non-contributory (had no effect on the pixels); for the following high-covisibility non-key frames, it reuses that record to skip those same non-contributory Gaussians. On the hardware side, AGS adds a frame covisibility detection engine that pulls the intermediate data out of the video codec, plus a pose tracking engine and a mapping engine with schedulers so tracking and mapping run in parallel with balanced workloads. AGS reaches up to 17.12 times the throughput of a mobile GPU with only about 2.36 percent quality loss.

Why this problem is worth solving

Robots and autonomous vehicles need to understand three-dimensional space in real time, and 3D Gaussian Splatting made those maps beautiful — but only if they can be built fast enough to act on.

Simultaneous Localization and Mapping is the core capability that lets a machine operate in a place it has never seen: from a stream of camera frames, it simultaneously estimates the machine’s own position (tracking) and constructs a map of the surroundings (mapping). Traditional SLAM builds sparse or rough maps; the recent breakthrough is to represent the scene with 3D Gaussian Splatting, which models it as a large set of small three-dimensional Gaussians — think of soft, colored blobs — that are projected and blended onto the image plane to render a view. This produces reconstructions of exceptional fidelity, good enough for tasks that need to understand fine detail. The cost is throughput. To fit the Gaussians to each new frame, 3DGS-SLAM performs many training iterations per frame, essentially optimizing the scene repeatedly, and a detailed scene holds a vast number of Gaussians. Together these make it extremely heavy: processing a sequence can take a high-end A100 GPU more than twenty minutes. For a robot that must localize and map as it moves, that is unusable. So the value of solving this is direct: it is the difference between a gorgeous 3D map computed offline and one a machine can actually build while navigating.

The reason speeding up the pieces is not enough is that the real waste is redundancy across frames and across Gaussians.

A SLAM camera moves continuously and relatively smoothly, so consecutive frames are highly similar — they share most of their content, a property called frame covisibility. A high-covisibility frame contains almost the same view as the previous one, which means the camera pose barely changed and is easy to estimate, and the scene barely changed, so little new optimization is truly needed. Yet a standard 3DGS-SLAM system treats each frame with the same heavy recipe: the same many training iterations to estimate the pose and update the map, regardless of how little moved. That is the first redundancy — excessive iterations on easy frames. The second lives inside the map: a large portion of the Gaussians do not contribute to the rendered pixels at all (they are occluded, transparent, or off-screen), but they are still processed every frame, wasting work. Simply making each iteration or each Gaussian cheaper, with a faster GPU or a rendering accelerator, still performs all of these unnecessary iterations and touches all of these non-contributory blobs. To really accelerate the system you have to recognize the redundancy and skip it, and to do that you first need a cheap way to know, per frame, how much overlaps the previous one — which is exactly the measurement a video codec already produces.

What people did before, and where it broke

Run 3DGS-SLAM on a faster GPU. Execute the many per-frame training iterations and all the Gaussians on a powerful GPU. Where it broke. It makes each step faster but still performs every redundant iteration on easy frames and processes non-contributory Gaussians, so a high-end GPU can still take more than twenty minutes per sequence.
Build a 3DGS rendering accelerator (GSCore). Design specialized hardware to speed up the Gaussian rendering pipeline. Where it broke. It accelerates rendering per Gaussian but does not remove the cross-frame redundancy — the unnecessary training iterations and the repeated processing of blobs that never affect the image.
Fix the number of training iterations per frame. Use a set iteration budget for tracking and mapping on every frame. Where it broke. Easy, high-overlap frames need far fewer iterations while hard, low-overlap frames need more; a fixed budget wastes work on the easy ones and is blind to how much actually changed.
Treat all Gaussians equally during mapping. Process every Gaussian each frame. Where it broke. A large fraction of Gaussians are non-contributory, and which ones tend to persist across similar frames, so processing them all repeatedly is pure waste.
The gap. No system cheaply measured how much each frame overlaps the previous one and used that to skip the redundant training iterations and the non-contributory Gaussians, reusing hardware already on the chip.

How AGS works

Turning free codec motion data into skipped SLAM work

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Get the redundancy signal for free from hardware that is already there. The pivotal move in AGS is realizing that the expensive question — how much does this frame overlap the last one? — is already answered by the video codec, which does motion estimation to compress video and produces motion vectors that quantify inter-frame change. AGS reads those as a covisibility metric instead of computing its own, so the signal that drives all its savings costs almost nothing. Repurposing the codec that a chip already carries to accelerate a vision workload, rather than adding new analysis hardware, is the same move the work on video transformer acceleration via codec-assisted matrix condensing covered elsewhere on this site makes, letting the codec’s motion information mark what can be reused. AGS applies that idea to SLAM, turning codec output into a per-frame overlap measure.

Skip the computation that does not change the answer. Both of AGS’s algorithmic techniques are forms of skipping work that provably or very likely does not matter: few iterations for frames that barely changed, and no processing for Gaussians that do not affect any pixel. The heart of the method is identifying non-contributory computation and omitting it. Detecting the parts of a workload that contribute little to the output and dropping them to accelerate the whole is the same principle the work on detecting and omitting weak attentions for scalable transformer acceleration covered elsewhere on this site is built on, cutting the computations that do not shape the result. AGS carries this into 3DGS-SLAM by tracking which Gaussians are non-contributory on a key frame and skipping them on the similar frames that follow.

Co-design the algorithm and the hardware for a real-time SLAM budget. AGS is not just an algorithm; it pairs the covisibility-driven skipping with a hardware design — a covisibility detection engine reading the codec, and parallel tracking and mapping engines with schedulers — so the savings translate into real throughput on a robot’s constrained hardware. Jointly designing the SLAM algorithm and the accelerator so a robot can localize and map within a tight resource budget is the same undertaking the work on algorithm-hardware co-design for resource-aware SLAM covered elsewhere on this site pursues, shaping both sides together for real-time perception. AGS focuses that co-design on eliminating cross-frame and cross-Gaussian redundancy in the 3DGS variant.

Following a frame through AGS

Processing one new camera frame that barely moved

A trace, drawn from the paper, of how AGS uses codec-measured overlap to skip redundant SLAM work on a highly-similar frame.

Step 1 of 5

A new frame arrives, very similar to the last

The camera has moved only slightly, so the new frame overlaps the previous one heavily. A standard 3DGS-SLAM system would still run its full set of training iterations and process every Gaussian.

High overlap means most of the heavy per-frame work is unnecessary here.

Why it works

Codec motion data is a free covisibility signal. Measuring frame overlap directly would add cost → but the video codec already computes motion vectors for compression → so AGS reads overlap almost for free and spends its budget on savings, not measurement.
Easy frames genuinely need fewer iterations. A high-overlap frame means the camera pose barely changed → so few training iterations already converge on that frame → so cutting iterations there loses almost no accuracy while removing most of the work.
Non-contributory Gaussians persist across similar frames. A Gaussian that did not affect the pixels of one frame → and a next frame that is nearly identical to it → so that same Gaussian will almost certainly not affect the next frame either, making it safe to skip.
Recording once and reusing amortizes the analysis. Finding non-contributory Gaussians requires a full mapping pass → but AGS does that only on key frames and records the result → so the many following non-key frames reuse it without repeating the analysis.
Co-designed parallel engines convert savings into throughput. Skipped work only helps if hardware can exploit it → but AGS’s covisibility, tracking, and mapping engines run in parallel with balanced load → so the reduced computation becomes real speedup, up to 17.12 times a mobile GPU.

What it costs (trade-offs and scope)

It trades a small amount of quality for large speed. Skipping iterations and Gaussians is an approximation: AGS reports about 2.36 percent loss in reconstruction quality (measured as peak signal-to-noise ratio, an image-fidelity metric) versus a full 3DGS-SLAM baseline, while keeping tracking accuracy tight (within about half a centimeter). The gains are large, but they are not free of any quality cost.
It relies on high frame covisibility. The savings come from consecutive frames overlapping heavily; a workload with rapid, jerky motion or frequent scene cuts has low covisibility, so AGS would fall back to more iterations and skip fewer Gaussians, reducing the benefit.
It assumes an on-chip video codec and custom engines. AGS reads intermediate data from a hardware video codec and adds its own covisibility, tracking, and mapping engines; the full benefit needs that hardware, so it is an accelerator design rather than a drop-in software change for an arbitrary GPU.
Its behavior depends on tuned hyperparameters. The scheme uses thresholds and iteration counts to decide key versus non-key frames and how many iterations to run; the paper notes these settings affect performance and accuracy, so they must be chosen well for a given deployment.
The headline speedups are against specific baselines. The up-to-17.12-times figure is versus a mobile GPU (AGX Xavier), with 6.71 times over an A100 and 5.41 times over the GSCore accelerator on the server side; results depend on the baseline, the datasets, and the covisibility of the sequences used.

The evaluation

Running high-fidelity 3DGS-SLAM fast enough to use

Property	High-end GPU (A100)	3DGS accelerator (GSCore)	AGS
Removes redundant iterations on easy frames
Skips non-contributory Gaussians
Measures frame overlap cheaply
Throughput
Reconstruction quality

A comparison of ways to run 3DGS-SLAM. AGS uses codec-measured frame covisibility to skip redundant training iterations and non-contributory Gaussians, reaching up to 17.12 times the throughput of a mobile GPU (6.71 times over an A100, 5.41 times over the GSCore accelerator) with about 2.36 percent quality loss and tight tracking accuracy.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (arXiv:2509.00433). The mechanism described is the paper’s: AGS is an algorithm-hardware co-design framework for 3D Gaussian Splatting SLAM, based on the observation that SLAM streams frames whose neighbors have high covisibility. It extracts motion information already computed by a video codec (motion vectors from motion estimation) as a quantitative frame-covisibility metric via a frame covisibility detection engine. On the algorithm side it applies movement-adaptive tracking (adapting the number of training iterations to a frame’s covisibility, using a Droid-SLAM-based coarse pose estimation for fast convergence) and Gaussian contribution-aware mapping (full mapping on key frames that records non-contributory Gaussians, and selective mapping on high-covisibility non-key frames that skips them). On the hardware side it adds a pose tracking engine and a mapping engine with workload schedulers for parallel, balanced execution. The paper reports up to 17.12 times speedup over a mobile GPU (AGX Xavier) and 14.63 times over GSCore-Edge, 6.71 times over an A100 and 5.41 times over GSCore-Server, with large energy-efficiency gains (for example 22.58 times over A100 and 42.28 times over AGX Xavier) and about 2.36 percent average PSNR loss with tight tracking accuracy. Exact algorithm parameters, engine microarchitecture, datasets, and full results should be read from the paper.

Background

Why real-time 3D perception keeps hunting for redundancy to skip

It helps to place AGS in a broad and fast-moving effort to make rich three-dimensional perception run within a robot’s or headset’s power and time budget. Neural and Gaussian scene representations have made reconstructions astonishingly detailed, but their cost is enormous, and the workloads are streaming and real-time, which is both a burden and an opportunity. The burden is obvious: many iterations, many primitives, every frame. The opportunity is that streaming perception is full of redundancy — consecutive frames overlap, most of a scene is static, and much of the computation does not change the output — so the winning strategy is repeatedly to find that redundancy and skip it, ideally using signals and hardware you already have. This is why so much recent work co-designs the perception algorithm with the accelerator, matching the amount of computation to how much genuinely changed. The broader push to run SLAM adaptively, spending effort only where the scene demands it so it fits a real-time budget, is the same terrain the work on an adaptive runtime for visual-inertial simultaneous localization and mapping covered elsewhere on this site occupies, scaling work to the difficulty of the moment. AGS contributes a particularly elegant instance: it borrows the video codec’s motion analysis, already free on the chip, to decide what to skip, turning a compression tool into an accelerator for 3D mapping.

If you remember only three things

1. Combining SLAM with 3D Gaussian Splatting (representing a scene as a vast number of fuzzy 3D blobs) gives autonomous robots high-fidelity maps, but it is far too slow: every frame needs many training iterations and there are enormous numbers of Gaussians, so even an A100 GPU can take over twenty minutes on a sequence. The waste is redundancy — consecutive frames overlap heavily yet are re-optimized from scratch, and a large fraction of Gaussians never affect the rendered pixels but are processed anyway.
2. AGS is an algorithm-hardware co-design that measures frame overlap almost for free by reading the motion vectors a video codec already computes, using them as a frame-covisibility metric. It then applies movement-adaptive tracking (few training iterations on high-overlap frames, more on hard ones) and Gaussian contribution-aware mapping (record non-contributory Gaussians on key frames, then skip them on the similar non-key frames that follow), backed by a covisibility detection engine plus parallel tracking and mapping engines.
3. AGS reaches up to 17.12 times the throughput of a mobile GPU (6.71 times over an A100, 5.41 times over the GSCore accelerator) with only about 2.36 percent reconstruction-quality loss and tight tracking accuracy, plus large energy-efficiency gains. The benefit depends on high frame covisibility (smooth motion), assumes an on-chip codec and custom engines, and needs its thresholds tuned — but it turns a compression tool already on the chip into the driver of a real-time 3D mapping accelerator.

Check yourself

Check your understanding

1. Why is 3D Gaussian Splatting SLAM too slow for real-time robotics, and why does a faster GPU not fix it?
2. What is AGS's key trick for measuring how much consecutive frames overlap?
3. How does AGS use frame covisibility to cut work, once it has measured it?
4. Predict: a drone flies through a scene with sudden, jerky motions and frequent large viewpoint changes. How well would AGS accelerate it, and why?

Sources

The paper is “AGS: Accelerating 3D Gaussian Splatting SLAM via CODEC-Assisted Frame Covisibility Detection” by Houshu He, Naifeng Jing, Li Jiang, Xiaoyao Liang, and Zhuoran Song (Shanghai Jiao Tong University), published at ASPLOS 2026, DOI 10.1145/3760250.3762229.

This explainer is written from the paper’s full text (arXiv:2509.00433). AGS is an algorithm-hardware co-design framework that accelerates 3D Gaussian Splatting (3DGS) SLAM, whose throughput is limited by multiple training iterations per frame and a vast number of Gaussians. Its key insight is that SLAM processes frames in a stream where adjacent frames have high covisibility, and that a video codec already computes motion information (motion vectors from motion estimation) that quantifies inter-frame similarity. AGS extracts this via a frame covisibility detection engine and uses it for movement-adaptive tracking (adapting the number of training iterations to a frame’s covisibility, with a Droid-SLAM-based coarse pose estimation for fast convergence) and Gaussian contribution-aware mapping (full mapping on key frames that records non-contributory Gaussians, and selective mapping on high-covisibility non-key frames that skips those Gaussians). On the hardware side it adds a pose tracking engine and a mapping engine with workload schedulers for parallel, balanced execution. The paper reports up to 17.12 times speedup over a mobile GPU (AGX Xavier) and 14.63 times over GSCore-Edge, 6.71 times over an A100 GPU and 5.41 times over GSCore-Server, with energy-efficiency improvements such as 22.58 times over A100 and 42.28 times over AGX Xavier, at about 2.36 percent average PSNR loss and tight tracking accuracy. No numbers beyond the paper’s own have been introduced; exact algorithm parameters, engine microarchitecture, datasets, and full results should be confirmed in the paper.