Neo: Real-Time On-Device 3D Gaussian Splatting with Reuse-and-Update Sorting Acceleration

Changhun Oh, Seongryong Oh, Jinwoo Hwang, Yoonsung Kim, Hardik Sharma and Jongse Park

3D Gaussian Splatting renders photorealistic 3D scenes for AR/VR glasses, but every frame it must sort millions of translucent blobs by depth so they blend in the right order — and that sorting, redone from scratch each frame, floods the tiny device's memory bandwidth; this paper (KAIST and Google — Changhun Oh, Jongse Park, and colleagues) presents Neo, which notices the depth order barely changes between consecutive frames and just updates the small parts that moved, backed by a dedicated sorting accelerator, cutting memory traffic by over 90 percent.

The argument in one glance

Problem

Rendering Gaussian-splat scenes on AR glasses is stuck on sorting

3D Gaussian Splatting (3DGS) has become a leading way to render photorealistic 3D scenes: it represents a scene as millions of tiny translucent "Gaussian" blobs and, to draw a frame, projects them onto the screen and blends them together. For this to power immersive AR/VR on devices like Meta Ray-Ban Display or Google Android XR Glasses, it has to run in real time on very resource-constrained hardware, and at high resolution — which existing solutions struggle to do. A crucial and easily-overlooked step in the pipeline is sorting. Because the Gaussians are translucent, they must be blended in the correct order — from far to near — or the image comes out wrong, so before blending, every Gaussian that touches a screen tile has to be sorted by depth. And here is the catch: 3DGS re-sorts millions of Gaussians for every single frame. That sorting is enormously demanding on memory bandwidth — it streams huge amounts of data to and from memory each frame — and on a small device with limited bandwidth, it becomes the wall. A prior accelerator, GSCore, sped up the blending (rasterization) stage, but that only revealed sorting as the new dominant bottleneck. So the thing standing between Gaussian Splatting and smooth, high-resolution rendering on glasses is not the drawing itself but the constant, from-scratch re-sorting.
What people did

Sort all the Gaussians from scratch every frame, faster

The straightforward approach is to keep doing what 3DGS does — fully sort the Gaussians by depth for each frame — but make the sort faster, either on an edge GPU or with dedicated sorting hardware. Since sorting is a well-understood operation, one can throw an efficient sorting implementation at it and try to keep up with the frame rate. This treats each frame as an independent sorting problem to be solved as quickly as possible.
Why it fell short

From-scratch sorting keeps flooding the device's memory bandwidth

Sorting 3DGS differently from ordinary sorting, and doing it every frame, is what makes brute force fail. General-purpose sorting is done occasionally and its result reused across many operations; 3DGS instead reorders millions of Gaussians for every rendered frame, and existing sorting implementations do this with bandwidth-intensive memory accesses, streaming the whole set of Gaussians through memory each time. On a resource-constrained device, memory bandwidth is the scarcest resource, so this per-frame data movement, not the arithmetic, is what caps the frame rate — and it gets worse at higher resolution. Making the sort itself faster does not help much, because the bottleneck is the memory traffic the from-scratch sort generates, and that traffic is intrinsic to re-sorting everything every frame. Crucially, this brute-force view misses something: 3DGS renders frames sequentially as the camera moves, and between two consecutive frames the camera moves only a little, so the depth ordering of the Gaussians is almost the same from one frame to the next. Re-sorting from scratch throws that similarity away and recomputes an ordering that was nearly correct already. What was missing was a way to exploit the fact that most of the sort is the same as last frame's, so the device only pays for the small part that actually changed.
This paper's idea

Reuse last frame's order and update only what moved, in hardware

Neo attacks the sorting bottleneck by not re-sorting from scratch. Its core is a reuse-and-update sorting algorithm that exploits the temporal redundancy in Gaussian ordering across consecutive frames: it keeps the sorted depth ordering from the previous frame and, for the new frame, identifies the minimal changes — the few Gaussians whose relative depth order actually shifted as the camera moved — and updates only the affected segments of the ordering, rather than sorting everything anew. Because the order is nearly unchanged frame to frame, the number of updates is tiny compared to a full sort, so the redundant computation and, above all, the memory bandwidth pressure collapse. Neo pairs this algorithm with a hardware accelerator designed specifically for it: a sorting pipeline that integrates both partial (local) and global depth ordering and minimizes memory access overhead, so the reuse-and-update pattern runs efficiently on-device rather than being throttled by memory. Evaluated with a cycle-accurate simulator against a state-of-the-art edge GPU (NVIDIA Orin AGX) and the prior 3DGS accelerator GSCore, Neo achieves up to 12.4 times and 5.5 times higher throughput respectively, while reducing sorting-induced DRAM traffic by 94.6 percent and 81.4 percent — enough to make high-quality, low-latency 3D rendering practical on the glasses people would actually wear.

Why this problem is worth solving

Immersive AR/VR depends on rendering rich 3D scenes in real time on tiny wearable devices, and the sorting bottleneck is what currently prevents that.

3D Gaussian Splatting produces strikingly photorealistic, real-time-renderable 3D scenes, which makes it a natural fit for the augmented- and virtual-reality experiences that AR glasses promise. But the value of that fit is entirely conditional on running on the actual hardware: a technique that needs a workstation GPU is useless for glasses that must be light, low-power, and worn on a face. So whether 3DGS becomes a foundation for immersive computing, or stays a demo, comes down to whether it can hit real-time frame rates on resource-constrained devices, especially at the high resolutions immersion demands. When analysis shows that the sorting stage, not the drawing, is the wall, then removing that wall is precisely what unlocks the whole application class. And because the sorting cost scales with the number of Gaussians and the resolution, both of which keep rising, an approach that fundamentally reduces sorting’s demand rather than just speeding it up is what keeps on-device rendering viable as scenes get richer.

The key realization is that the expensive work is almost entirely repeated from the previous frame, so it should be updated, not redone.

Brute-forcing the sort treats every frame as a fresh, independent problem, which is the wrong model for rendering. Frames are not independent: they form a sequence produced as a camera moves smoothly through a scene, and a small camera motion changes the depth order of a scene’s Gaussians only slightly. That means the sorted ordering the device computes for one frame is almost identical to the one it just computed for the previous frame, differing in only a handful of places where two Gaussians swapped relative depth. Re-sorting from scratch recomputes the entire ordering to reproduce something it already had, which is why the memory traffic is so wasteful. The right move is to treat the previous frame’s ordering as a starting point that is nearly correct and to find and apply just the minimal set of changes needed to make it correct for the new frame. This is a huge reduction because the work now scales with how much the order actually changed — a tiny fraction — rather than with the total number of Gaussians. The subtlety is doing this efficiently: finding the minimal changes and updating a sorted structure in place, without incurring its own bandwidth cost, is what the accelerator is for. So the insight is simple — reuse the order and update the difference — but cashing it in requires an algorithm and hardware co-designed to make incremental updates cheap on a bandwidth-starved device.

What people did before, and where it broke

Render 3DGS on an edge GPU. Run the full projection-sort-rasterize pipeline on the device’s GPU. Where it broke. It cannot reach high frame rates at high resolution on resource-constrained hardware, especially because of sorting.
Accelerate the rasterization stage (GSCore). Build dedicated hardware for the blending step. Where it broke. It removes the rasterization bottleneck but leaves sorting as the new dominant one.
Sort all Gaussians by depth every frame. Re-sort the whole set for each rendered frame. Where it broke. This from-scratch sort streams millions of Gaussians through memory each frame, and its memory-bandwidth demand caps the frame rate on a bandwidth-limited device.
Make the from-scratch sort faster. Use a more efficient sorting implementation. Where it broke. The bottleneck is the memory traffic re-sorting everything generates, not the sort’s speed, and that traffic is intrinsic to redoing the whole sort each frame.
The gap. No approach exploited that the depth ordering is nearly identical between consecutive frames, so all of them paid to recompute an ordering that was already almost correct.

How Neo works

Updating the depth order instead of re-sorting it every frame

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Exploit that consecutive frames are nearly the same, and update rather than recompute. Neo’s foundation is temporal redundancy: the depth ordering barely changes between frames, so it reuses the previous ordering and applies only the small difference. Recognizing that adjacent items in a sequence are highly similar and reusing prior work instead of redoing it from scratch is the same principle the work on accelerating reinforcement learning by reusing similar rollouts across adjacent epochs (RhymeRL) covered elsewhere on this site is built on, turning repeated work into a small incremental update. Neo applies this to the per-frame sort in rendering, where the previous frame’s order is the reusable prior.

Attack the specific stage that has become the bottleneck. Neo targets sorting precisely because analysis showed it is the dominant memory-bandwidth bottleneck once rasterization is handled, and builds a specialized accelerator for it. Identifying the one stage of a pipeline that limits performance and designing hardware or a dataflow to relieve exactly that bottleneck is the same approach the work on an optimized dataflow for mitigating attention bottlenecks (FLAT) covered elsewhere on this site takes, concentrating effort on the stage that actually caps throughput. Neo’s reuse-and-update sorting pipeline is that targeted relief for the 3DGS sorting stage.

Make Gaussian Splatting practical on the hardware people actually use. Neo’s whole aim is to bring 3DGS rendering within the compute and memory budget of wearable AR/VR devices, rather than assuming powerful hardware. Engineering a Gaussian-Splatting system so it fits the constraints of the real, resource-limited devices it must run on is the same undertaking the work on host-offloaded large-scale 3D Gaussian Splatting training (GS-Scale) covered elsewhere on this site pursues, reshaping the workload to the hardware’s limits. Where that work makes training fit a consumer GPU’s memory, Neo makes rendering fit a wearable’s bandwidth by eliminating redundant sorting.

Following two frames through Neo

Rendering a new frame by updating the previous frame's sorted order

A trace, drawn from the paper, of how Neo avoids re-sorting between consecutive frames.

Step 1 of 5

Frame one is sorted and rendered

For the first frame, Neo sorts the Gaussians by depth per tile as usual so they can be blended front-to-back, and keeps that sorted ordering rather than discarding it.

The sorted order is retained as the reusable state for the next frame.

Why it works

Reuse makes the work scale with change, not scene size. A from-scratch sort processes all millions of Gaussians every frame → but keeping the previous order and updating only what shifted → so the work scales with the tiny number of order changes instead of the whole scene.
Less recomputation means far less memory traffic. The bottleneck is the bandwidth a full sort consumes streaming everything through memory → but updating only affected segments → so sorting-induced DRAM traffic drops by over 80 to 90 percent, relieving the actual bottleneck.
Temporal similarity is real and large. The reuse would not help if frames differed a lot → but smooth camera motion keeps consecutive depth orderings nearly identical → so the number of updates per frame is genuinely tiny.
Dedicated hardware makes incremental updates cheap. Doing in-place ordering updates on a general processor could reintroduce memory overhead → but a sorting pipeline integrating partial and global ordering with minimized memory access → so the reuse-and-update pattern runs efficiently within the device’s budget.
Targeting the true bottleneck is what raises frame rate. Speeding a non-bottleneck stage would not help → but attacking sorting, the dominant memory-bandwidth limit after rasterization → so throughput rises up to several-fold and real-time rendering becomes feasible.

What it costs (trade-offs and scope)

It depends on frames being similar. The gains come from consecutive frames sharing nearly the same depth order; a sudden large camera jump, a scene cut, or very fast motion would change much more of the order, shrinking the reuse benefit toward a full re-sort.
It requires keeping the previous order as state. Reuse-and-update maintains the sorted ordering across frames, which is state and structure a stateless from-scratch sort does not need, and that structure must be updatable in place efficiently.
It is a specialized accelerator, not a drop-in. Neo’s benefits rest on a hardware sorting pipeline co-designed with the algorithm; realizing them means building and using that accelerator, not running the idea on a conventional GPU.
It targets the sorting stage of 3DGS specifically. Neo relieves sorting once rasterization is already accelerated; the rest of the pipeline and other rendering methods are outside its scope, and the win is largest exactly where sorting dominates.
Reported figures are simulator-based and workload-dependent. The up-to-12.4-times and 5.5-times throughput and the DRAM-traffic reductions come from a cycle-accurate simulator against specific baselines (an edge GPU and GSCore) on particular neural-rendering scenes; the exact gains depend on the scenes, the motion, and the hardware assumptions.

The evaluation

Sorting for on-device 3D Gaussian Splatting rendering

Property	Edge GPU (Orin AGX)	Prior 3DGS accelerator (GSCore)	Neo
Sorting each frame
Exploits temporal redundancy
Sorting-induced DRAM traffic
Dedicated sorting hardware
Throughput

A comparison of ways to handle sorting in on-device 3DGS rendering. By exploiting temporal redundancy — reusing the previous frame's depth order and updating only the few Gaussians that changed — with a dedicated sorting accelerator, Neo cuts sorting-induced DRAM traffic by 94.6 percent and 81.4 percent versus an edge GPU and GSCore, and reaches up to 12.4 times and 5.5 times higher throughput, making real-time high-resolution on-device rendering practical. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ arXiv copy, 2511.12930, of the ASPLOS 2026 paper). The described system is the paper’s: Neo enables real-time on-device 3D Gaussian Splatting (3DGS) rendering by attacking the sorting stage, which its analysis identifies as the dominant bottleneck (due to high memory-bandwidth demand) once rasterization is accelerated. It introduces a reuse-and-update sorting algorithm that exploits temporal redundancy in Gaussian depth ordering across consecutive frames — reusing the previous frame’s ordering and updating only the minimal changed segments instead of re-sorting from scratch — and a hardware accelerator co-designed for this algorithm, with a sorting pipeline that integrates partial and global ordering while minimizing memory access. Evaluated with a cycle-accurate simulator (Neo synthesized with an ASAP 7-nanometer library) against an NVIDIA Orin AGX edge GPU and the GSCore accelerator, the paper reports up to 12.4 times and 5.5 times higher throughput respectively and reductions in sorting-induced DRAM traffic of 94.6 percent and 81.4 percent (the paper’s text also reports figures such as up to 10.0 times and 5.6 times, and 94.4 percent and 81.3 percent, for related measurements). The background above on 3DGS rendering, depth sorting for correct alpha blending, and the rasterization stage is standard graphics context provided to make the approach intuitive. The step-by-step walkthrough is an illustration of the described mechanism. Exact algorithm details, the accelerator microarchitecture, and the full evaluation should be read from the paper.

Background

Why real-time rendering keeps coming down to not redoing what you already did

It helps to see Neo within a broad pattern: interactive graphics and rendering produce a continuous stream of frames that are overwhelmingly similar to one another, and a great deal of the field’s progress comes from refusing to recompute, each frame, what barely changed since the last. Frames share geometry, lighting, and ordering; a camera nudges rather than teleports; most of the scene is static. Recomputing everything from scratch every frame is the simple model, but on constrained hardware it is ruinously wasteful, because the cost is dominated not by the new information but by re-deriving the old. The recurring answer is to hold onto results and update them incrementally — reprojecting, caching, or, as here, reusing a sorted order and patching the differences — so that the per-frame cost tracks the change rather than the total. What makes Neo a clean instance is that it identifies the exact operation, sorting, where this redundancy is both large and expensive on a memory-starved device, and then co-designs an algorithm and accelerator to cash it in. Underlying all of this is the same pressure that shapes so much edge computing: powerful workloads have to fit inside the tight memory and bandwidth budgets of small devices. This drive to run a demanding graphics-and-learning workload within the strict limits of resource-constrained on-device hardware is the same terrain the work on streaming weights for modern DNN workloads on mobile GPUs (FlashMem) covered elsewhere on this site occupies, bending a heavy workload to fit the device. Neo’s contribution is to show that, for on-device Gaussian Splatting, exploiting the near-identity of consecutive frames to update rather than re-sort — with hardware built for it — slashes memory traffic enough to make real-time, high-resolution rendering practical on the glasses people will wear.

If you remember only three things

1. 3D Gaussian Splatting (3DGS) renders photorealistic 3D scenes as millions of translucent Gaussian blobs, and for immersive AR/VR it must run in real time at high resolution on resource-constrained glasses. Because the Gaussians are translucent, they must be blended far-to-near, so every frame the pipeline sorts all the Gaussians that touch each tile by depth. This per-frame, from-scratch sort of millions of Gaussians is enormously demanding on memory bandwidth, and on a bandwidth-limited device it becomes the dominant bottleneck — especially after a prior accelerator (GSCore) sped up the rasterization stage, leaving sorting as the wall. Making the sort itself faster does not help, because the limit is the memory traffic re-sorting everything generates.
2. Neo's insight is that 3DGS renders frames sequentially as the camera moves smoothly, so the depth ordering of the Gaussians is almost identical between consecutive frames — only a few swap relative order. Its reuse-and-update sorting algorithm keeps the previous frame's sorted ordering and, for the new frame, identifies the minimal set of changes and updates only the affected segments, so the work (and memory traffic) scales with how much the order changed rather than with the total number of Gaussians. Neo pairs this with a dedicated hardware accelerator — a sorting pipeline integrating partial and global ordering with minimized memory access — so the incremental updates run efficiently on-device rather than being throttled by bandwidth.
3. Evaluated with a cycle-accurate simulator against an NVIDIA Orin AGX edge GPU and the GSCore accelerator, the paper reports Neo achieves up to 12.4 times and 5.5 times higher throughput respectively while reducing sorting-induced DRAM traffic by 94.6 percent and 81.4 percent, making real-time high-resolution on-device 3DGS practical. The trade-offs: the gains depend on consecutive frames being similar (fast motion or scene cuts shrink the benefit toward a full re-sort), it must keep the previous order as updatable state, it is a specialized accelerator rather than a drop-in, and it targets the sorting stage specifically. The reported numbers are simulator-based and depend on the scenes, the motion, and the hardware assumptions.

Check yourself

Check your understanding

1. Why is the sorting stage the bottleneck for on-device 3D Gaussian Splatting rendering?
2. What temporal-redundancy insight does Neo exploit?
3. How does Neo's reuse-and-update sorting work, and why does it cut memory traffic so much?
4. What does Neo achieve, and what are the limits of the approach?

Sources

The paper is “Neo: Real-Time On-Device 3D Gaussian Splatting with Reuse-and-Update Sorting Acceleration” by Changhun Oh, Seongryong Oh, Jinwoo Hwang, Yoonsung Kim, Hardik Sharma, and Jongse Park (KAIST and Google), published at ASPLOS 2026, DOI 10.1145/3779212.3790192. It is available on arXiv as 2511.12930.

This explainer is written from the paper’s full text (the authors’ arXiv copy). Neo enables real-time on-device 3D Gaussian Splatting rendering by targeting the sorting stage, which it identifies as the dominant memory-bandwidth bottleneck once rasterization is accelerated. It introduces a reuse-and-update sorting algorithm that exploits temporal redundancy in Gaussian depth ordering across consecutive frames — reusing the previous frame’s ordering and updating only the minimal changed segments instead of re-sorting from scratch — and a hardware accelerator co-designed for it, with a sorting pipeline integrating partial and global ordering while minimizing memory access. Evaluated with a cycle-accurate simulator against an NVIDIA Orin AGX edge GPU and the GSCore accelerator, the paper reports up to 12.4 times and 5.5 times higher throughput and reductions in sorting-induced DRAM traffic of 94.6 percent and 81.4 percent, making real-time high-resolution on-device rendering practical. Numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.