Nebula: Infinite-Scale 3D Gaussian Splatting in VR via Collaborative Rendering and Accelerated Stereo Rasterization

City-scale 3D scenes rendered as millions of Gaussian blobs are far too big for a VR headset to hold, so they must be rendered in the cloud — but streaming finished video to the headset is bandwidth-heavy and lossy, wrecking the motion-to-photon latency VR demands; this paper (Shanghai Jiao Tong and Rochester — He Zhu, Zheng Liu, Xingyang Li, Anbang Wu, Jieru Zhao, Fangxin Liu, Yiming Gan, Jingwen Leng, Yu Feng) presents Nebula, which splits the rendering pipeline so the cloud does the heavy level-of-detail search over the giant scene and streams only the resulting set of Gaussians — the intermediate result, not video — to the client, exploiting the near-total temporal overlap of that set across frames to send just what changed, while the headset shares almost all of its stereo-rendering work between the two eyes via triangulation, cutting cloud-to-client data by 19.25 times and delivering a 2.7 times motion-to-photon speedup.

The argument in one glance

Problem

A city-scale 3D scene is too big for a headset, but streaming it as video is too heavy

3D Gaussian splatting represents a scene as a huge cloud of tiny translucent blobs — Gaussians — that are sorted and blended to produce a photorealistic image, and it has become the dominant way to render neural scenes quickly. The trouble is scale. A city-scale scene contains an enormous number of Gaussians, far more than a mobile VR headset can store or render in real time, so the rendering has to happen in the cloud on powerful hardware. But virtual reality is unforgiving about latency and fidelity: the image must update within a few milliseconds of the user's head moving (the motion-to-photon budget) or the experience feels sluggish and nauseating, and it must be sharp. Delivering that from the cloud means moving a lot of data over the network very fast, and the obvious way to do it does not fit within VR's bandwidth and latency constraints.
What people did

Render the whole scene in the cloud and stream the finished video

The standard remote-rendering approach, borrowed from cloud gaming, is to run the entire rendering pipeline on the cloud server and stream the finished video frames to the headset. The client becomes a thin display: it receives compressed video, decodes it, and shows it. The heavy scene never has to live on the device, and the client stays simple.
Why it fell short

Video streaming is too fat and too lossy for high-fidelity VR

Streaming finished VR video does not work at the fidelity and smoothness VR needs. Two stereo views at high resolution and high frame rate produce a torrent of pixels, so to fit the network the video must be aggressively, lossily compressed — which blurs detail exactly where VR demands sharpness — and the encode, transmit, decode chain adds latency that eats into the tight motion-to-photon budget. The bandwidth requirement, the paper observes, makes it effectively impossible to deliver high-fidelity and smooth VR content purely as video from the cloud. Meanwhile the alternative of rendering everything locally is off the table because the scene is far too large for the client. Neither pure endpoint works.
This paper's idea

Split the pipeline: cloud does the search, streams Gaussians not pixels, client renders

Nebula is a coherent framework for collaborative rendering that cuts the pipeline in the middle rather than at either end. Large-scale 3DGS rendering has four stages — a level-of-detail search that walks a tree to pick the right set of Gaussians for the current view (the result is called the cut), then preprocessing, sorting, and rasterization that turn that set into an image. Nebula runs the heavy level-of-detail search in the cloud, where the giant scene lives, and streams the intermediate result — the cut of Gaussians — to the client, which finishes the last three stages locally. Streaming that compact set instead of finished video slashes the cloud-to-client data by 19.25 times. Three techniques make it fast: a temporal-aware level-of-detail search that reuses the near-identical cut across frames to tame the cloud's memory traffic, a runtime Gaussian-management scheme that transmits only the handful of Gaussians that changed between frames, and a stereo rasterization on the client that lets the two eyes share almost all their computation via triangulation while staying bit-accurate. Together Nebula reaches a 2.7 times motion-to-photon speedup over lossy video streaming with only minimal hardware additions.

Why this problem is worth solving

Immersive VR at the scale of a whole city is a genuine goal, and the bottleneck is not how prettily you can render a scene but how you deliver it.

3D Gaussian splatting is a technique for rendering a real or synthetic scene captured as a set of Gaussians — small, soft, semi-transparent ellipsoids, each with a position, shape, color, and opacity. To make an image, the renderer figures out which Gaussians are visible from the camera, projects them onto the screen, sorts them by depth, and blends them front to back, producing a smooth photorealistic picture much faster than traditional ray tracing. It has swept through graphics and the computer-architecture community because it is both high quality and fast. But the number of Gaussians grows with the size and detail of the scene, and a city-scale scene needs a staggering number of them — more than a mobile headset’s memory and compute can possibly handle. To keep the count manageable for any given view, large-scale systems build a level-of-detail tree, so distant or unimportant regions are drawn with a few coarse Gaussians and nearby regions with many fine ones; searching that tree for the right cut is itself a heavy, irregular operation. So scale attacks 3DGS on two fronts at once: the scene is too big for the client, and choosing what to draw is expensive — which is why simply moving all the work to the cloud and shipping back pictures does not solve it.

The reason the delivery problem is so hard is that VR sets a latency budget most systems never face.

In an ordinary video call or game stream, a little latency and some compression blur are tolerable. VR is different because the display is strapped to your head: when you turn, the world must move with you almost instantly, or your eyes and inner ear disagree and you feel motion sickness. The interval between a head motion and the corresponding photons reaching your eyes — the motion-to-photon latency — must stay within a few milliseconds, and the image must be crisp because it fills your whole field of view and you can lean in to inspect it. Two consequences follow. First, the pipeline has almost no slack to spend on encoding, transmitting, and decoding video, so any scheme that adds a full compress-send-decode round trip per frame is already in trouble. Second, the fidelity cannot be sacrificed: heavy lossy compression, the usual way to shrink a video stream to fit a network, produces exactly the softness VR cannot afford. So the delivery mechanism must be simultaneously low-latency and low-loss and low-bandwidth — a combination finished-video streaming cannot provide — which is what forces a fundamentally different split of the work between cloud and client, and what makes Nebula’s approach worth the trouble.

What people did before, and where it broke

Render everything on the client. Put the whole scene on the headset and render locally, avoiding the network entirely. Where it broke. A city-scale 3DGS scene has far more Gaussians than a mobile client can store or process in real time, so this is simply impossible at the target scale.
Cloud rendering with video streaming. Render the full pipeline in the cloud and stream compressed video to a thin client, as cloud gaming does. Where it broke. Two high-resolution, high-frame-rate stereo views generate too many pixels for the network, so the video must be lossily compressed — blurring the detail VR needs — and the encode-transmit-decode chain adds latency that violates the motion-to-photon budget.
Ignore scalability in the accelerator. Build a fast 3DGS accelerator tuned for small or medium scenes. Where it broke. The paper notes that current architectural designs often overlook 3DGS scalability, so they are fragile for extremely large scenes; speeding up a pipeline that cannot hold the scene does not deliver city-scale VR.
Render each eye independently on the client. If the client does render, treat the left and right VR views as two separate rendering jobs. Where it broke. That doubles the per-frame rasterization work on already-weak headset hardware, even though the two eyes see almost the same scene from slightly shifted viewpoints, wasting most of the effort.
The gap. No system split the 3DGS pipeline so the cloud handles the heavy, scene-sized level-of-detail search and streams the compact intermediate set of Gaussians (not video) to a client that finishes rendering — while exploiting temporal coherence across frames and stereo coherence across eyes to keep both the network and the client cheap.

How Nebula works

Splitting the 3DGS rendering pipeline between a cloud server and a VR headset

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Cut the pipeline in the middle and stream Gaussians, not pixels. The organizing insight of Nebula is that the choice is not really between rendering on the client and rendering in the cloud — it is about where to split a pipeline that has a natural seam. The level-of-detail search is the stage that genuinely needs the whole giant scene in memory, so it belongs in the cloud; preprocessing, sorting, and rasterization operate only on the chosen cut, which is small, so they can run on the client. By streaming the cut — the intermediate result after the search — instead of the finished video, Nebula sends a fraction of the data (a 19.25 times reduction) and sends it losslessly, sidestepping both the bandwidth blow-up and the compression blur that sink video streaming. Building dedicated support to run the splatting stages efficiently on constrained hardware is the same drive the radiance-field rendering accelerator covered elsewhere on this site pursues, adding architectural support so the sort-and-blend pipeline runs fast without a datacenter GPU; Nebula’s contribution is deciding which stage runs where and shipping the compact thing between them.

Beat the scale limit by never asking the client to hold the scene. City-scale 3DGS fails on the client for one reason: the scene has more Gaussians than the device’s memory can hold. Nebula’s split dissolves that limit, because the client only ever receives and renders the cut for the current view — a tiny slice of the whole scene — while the full, unbounded scene stays in the cloud. This is what lets the paper claim infinite-scale rendering in VR: the client’s work is bounded by what is visible, not by how large the world is. Making a 3D Gaussian scene that is too big for one device’s memory renderable anyway is the same barrier the work on removing the GPU memory barrier for 3D Gaussian splatting covered elsewhere on this site sets out to remove, keeping the bulk of the scene out of scarce fast memory and bringing in only what a step needs. Nebula removes the barrier by geography — the big memory is in the cloud — rather than by clever local paging.

Attack the redundant work that dominates each frame. Once the split is in place, the remaining cost is the repeated work the pipeline does every frame, and Nebula’s three techniques each target a specific redundancy. The cut hardly changes between frames — around 99 percent of it is identical to the previous frame — so re-running the full level-of-detail search wastes almost all its effort; the temporal-aware search reuses prior results to avoid that, and the runtime Gaussian-management scheme sends only the delta of changed Gaussians. And the two VR eyes see nearly the same scene, so rendering them separately doubles the work; the stereo rasterization uses triangulation to let one eye reuse most of the other’s computation. Squeezing real-time neural rendering out of limited hardware by cutting the work that does not need to be redone is the same efficiency drive the real-time neural-rendering-with-pruning work covered elsewhere on this site pursues, pruning away computation that contributes little to the final image. Nebula’s version prunes across frames and across eyes rather than across the scene.

Following one frame through Nebula

From a head movement to a sharp stereo image, split across cloud and client

A trace, drawn from the paper's description of Nebula's pipeline, of how one VR frame is produced collaboratively by the cloud and the headset.

Step 1 of 5

The head moves and the cloud runs the level-of-detail search

The user turns their head, defining a new viewpoint. The cloud, which holds the entire city-scale scene and its level-of-detail tree, searches the tree for the cut — the set of Gaussians at the right detail for this view. This is the one stage that needs the whole scene in memory.

Keeping the search and the giant scene in the cloud is what frees the client from ever holding the full scene.

Why it works

Streaming the cut instead of video removes the bandwidth wall. Two high-resolution stereo video streams are enormous and must be lossily compressed to fit the network → but the cut of Gaussians after the search is far smaller and can be sent losslessly → so the headset gets sharp, high-fidelity content at a small fraction of the data.
Temporal coherence makes the cloud’s search cheap. Re-running the full level-of-detail search every frame repeats nearly identical work → but about 99 percent of the cut is unchanged between consecutive frames → so reusing prior results and streaming the data slashes the redundant, irregular memory traffic that dominates the search.
Sending only the delta keeps the link light. Retransmitting the whole cut each frame would waste the bandwidth the split was meant to save → but a management table lets the cloud send only the newly appeared Gaussians and evict obsolete ones → so steady-state traffic tracks how fast the scene changes, not how large it is.
Sharing the eyes nearly halves the client’s rasterization. Rendering the left and right views independently doubles the per-frame work on a weak headset → but triangulation lets the second eye reuse most of the first eye’s computation → so stereo rendering costs little more than mono while staying bit-accurate.
Minimal hardware augmentation makes it deployable. A rendering trick that demanded a bespoke accelerator would be hard to adopt → but Nebula’s stereo rasterization drops into mainstream 3DGS accelerators with small additions → so the client-side speedup arrives without a custom chip.

What it costs (trade-offs and scope)

It requires a capable client, not a pure thin terminal. Unlike video streaming, where the client only decodes pixels, Nebula’s client must run preprocessing, sorting, and rasterization over the cut. That is far lighter than holding the whole scene, but it is real rendering work and assumes the headset has a 3DGS-capable GPU or accelerator, ideally with the small stereo-rasterization augmentation.
It depends on strong temporal and stereo coherence. The bandwidth and search savings rest on about 99 percent of the cut being stable across frames and the two eyes seeing nearly the same scene. Very fast, erratic head or scene motion that churns the cut, or content with little stereo overlap, would shrink the delta advantage and the eye-sharing advantage.
The split assumes a good network path to the cloud. Streaming the cut is far lighter than video, but it is still a per-frame cloud round trip, so the approach assumes network latency and reliability adequate for the motion-to-photon budget; a poor or highly variable link would still hurt, and the paper’s gains are measured against video streaming rather than against a hypothetical zero-latency local render.
It is specialized to level-of-detail 3D Gaussian splatting. The whole design keys off the four-stage 3DGS pipeline with a level-of-detail tree and a searchable cut. Scenes or renderers without that structure, or non-Gaussian representations, do not directly benefit, and the exact figures are for the paper’s scenes and hardware.
The reported speedups are configuration-specific. The 2.7 times motion-to-photon speedup, 19.25 times data reduction, 52.7 times search speedup, and 21.7 and 5.3 times stereo-rasterization speedups come from the paper’s datasets, baselines, and hardware assumptions; other scenes, headsets, and network conditions would shift them.

The evaluation

Delivering city-scale 3DGS to a VR headset

Property	Local rendering on the client	Cloud rendering, stream video	Nebula (split pipeline)
Can hold a city-scale scene
What crosses the network
Image fidelity
Stereo (two eyes) cost
Reported outcome

A qualitative comparison of ways to deliver large-scale 3DGS to a VR headset. Nebula's reported results are a 2.7 times motion-to-photon speedup and 19.25 times less cloud-to-client data than lossy video streaming, with the cloud search up to 52.7 times faster and the client stereo rasterization up to 21.7 times faster than a mobile GPU and 5.3 times faster than prior accelerators.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s abstract and the arXiv preprint version (arXiv:2512.20495, titled with “Enable City-Scale” rather than “Infinite-Scale”), which are the source for the mechanism described here: the four-stage 3DGS pipeline (level-of-detail search, preprocessing, sorting, rasterization); the split that runs the search in the cloud and streams the cut of Gaussians rather than video; the temporal-aware level-of-detail search that exploits the roughly 99 percent frame-to-frame overlap of the cut; the runtime Gaussian-management scheme that transmits only the changed Gaussians and evicts obsolete ones past a reuse threshold; and the triangulation-based stereo rasterization that lets the two eyes share most computation with bit-accurate results, integrable into mainstream 3DGS accelerators with minimal hardware. The headline figures are the paper’s own: a 2.7 times motion-to-photon speedup and a 19.25 times reduction in cloud-to-client data versus lossy video streaming, an up to 52.7 times faster cloud search, and an up to 21.7 times (versus a mobile GPU) and 5.3 times (versus prior accelerators) faster client stereo rasterization. The paper renders the odd-looking “1925 percent” reduction, which is read here as a 19.25 times reduction. Exact experimental setups, datasets, and hardware parameters should be confirmed in the paper.

Background

Why splitting the pipeline is the recurring move

It helps to see Nebula as one answer to a question that keeps returning in interactive graphics: given a job too big for the endpoint device and a network that cannot carry the finished output cheaply, where do you cut the pipeline? Cutting at the very end (render everything remotely, stream pixels) is simple but pays the full cost of moving the output and, for VR, loses fidelity to compression. Cutting at the very start (send the whole scene, render locally) is impossible when the scene dwarfs the device. The interesting cuts are in the middle, at a stage boundary where the data flowing across is far smaller than either the input scene or the output frames — and for 3DGS that boundary is right after the level-of-detail search, where the cut of Gaussians is compact, lossless, and reusable across frames. Choosing that seam well is what turns an impossible delivery problem into a manageable one, and it recurs because rendering pipelines almost always have a stage where the working set is momentarily small. The related discipline of decoupling the rendering and displaying stages of a graphics pipeline to meet a tight latency target is exactly the concern the decoupled-rendering-and-displaying work for smartphone graphics covered elsewhere on this site is built on, separating stages so each can be scheduled to hit the deadline. Nebula applies the same structural thinking across the cloud-client boundary and the VR frame and eye dimensions at once.

If you remember only three things

1. City-scale 3D Gaussian splatting scenes have far too many Gaussians for a VR headset to store or render, so the rendering must happen in the cloud — but the usual remote-rendering approach of streaming finished video needs enormous bandwidth for two high-resolution stereo views, forces lossy compression that blurs the detail VR needs, and adds latency that breaks VR's strict motion-to-photon budget. Rendering everything locally is impossible and streaming video is too fat and too lossy.
2. Nebula splits the four-stage 3DGS pipeline in the middle: the cloud runs the heavy level-of-detail search over the giant scene and streams only the resulting cut of Gaussians — the intermediate result, not pixels — to the client, which finishes preprocessing, sorting, and rasterization. Three techniques cut the remaining cost: a temporal-aware search that reuses the roughly 99 percent of the cut unchanged between frames, a runtime Gaussian-management scheme that transmits only the changed Gaussians, and a triangulation-based stereo rasterization that lets the two eyes share almost all their computation with bit-accurate results and minimal hardware additions.
3. The payoff is a 2.7 times motion-to-photon speedup and 19.25 times less cloud-to-client data than lossy video streaming, with the cloud search up to 52.7 times faster and the client stereo rasterization up to 21.7 times faster than a mobile GPU and 5.3 times faster than prior accelerators. The approach assumes a 3DGS-capable client, strong temporal and stereo coherence, and an adequate network path, and is specialized to level-of-detail 3D Gaussian splatting.

Check yourself

Check your understanding

1. Why does neither pure local rendering nor cloud video streaming work for city-scale 3DGS in VR?
2. What is Nebula's core idea?
3. How does Nebula keep the cloud search and the network traffic cheap frame to frame?
4. How does Nebula reduce the client's cost of rendering two VR eyes, and what did it achieve overall?

Sources

The paper is “Nebula: Infinite-Scale 3D Gaussian Splatting in VR via Collaborative Rendering and Accelerated Stereo Rasterization” by He Zhu, Zheng Liu, Xingyang Li, Anbang Wu, Jieru Zhao, Fangxin Liu, Yiming Gan, Jingwen Leng, and Yu Feng (Shanghai Jiao Tong University and University of Rochester), published at ASPLOS 2026 (Volume 2), DOI 10.1145/3779212.3790190. It is licensed CC BY.

This explainer is written from the paper’s abstract and its arXiv preprint version (arXiv:2512.20495, “Nebula: Enable City-Scale 3D Gaussian Splatting in Virtual Reality via Collaborative Rendering and Accelerated Stereo Rasterization”), which describe the mechanism in full: the four-stage large-scale 3DGS pipeline (level-of-detail search producing a cut of Gaussians, then preprocessing, sorting, and rasterization); the collaborative split that runs the level-of-detail search in the cloud and streams the intermediate cut rather than finished video; the temporal-aware level-of-detail search that exploits the roughly 99 percent frame-to-frame overlap of the cut to regulate memory access and skip redundant work; the runtime Gaussian-management scheme that transmits only the delta of changed Gaussians and evicts obsolete ones past a reuse threshold; and the triangulation-based stereo rasterization that lets the two eyes share most computation with bit-accurate output and integrates into mainstream 3DGS accelerators with minimal hardware. The reported results are the paper’s own: a 2.7 times motion-to-photon speedup and a 19.25 times reduction in cloud-to-client data over lossy video streaming (the preprint’s “1925 percent” phrasing, read here as 19.25 times), an up to 52.7 times faster cloud-side search, and an up to 21.7 times (versus a mobile Ampere GPU) and 5.3 times (versus prior accelerators) faster client-side stereo rasterization. No numbers beyond the paper’s have been introduced; exact datasets, hardware parameters, and experimental setup should be confirmed in the paper.