GS-Scale: Unlocking Large-Scale 3D Gaussian Splatting Training via Host Offloading

3D Gaussian Splatting builds a photorealistic 3D scene from millions of tiny "Gaussian" blobs, but training a large, high-quality scene needs so many blobs — plus their gradients and optimizer state — that they overflow GPU memory; this paper (Seoul National University and Google — Donghyun Lee, Jae W. Lee, Hongil Yoon, and colleagues) presents GS-Scale, which keeps all the blobs in cheap host (CPU) memory and streams only the ones a given view needs onto the GPU, then uses three tricks to hide the CPU work so it trains about as fast as if everything fit on the GPU — letting a laptop GPU handle four times as many blobs.

The argument in one glance

Problem

High-quality 3D scenes need more Gaussians than a GPU can hold

3D Gaussian Splatting (3DGS) has become a leading way to reconstruct a photorealistic 3D scene from a set of photos: it represents the scene as millions of tiny translucent "Gaussian" blobs, each with a position, shape, orientation, color, and opacity, and trains them to reproduce the input images from any viewpoint. It renders fast and looks great, which is why it has taken off. The problem is scale. Reconstructing a large scene at high quality — a city block, a big building — needs a huge number of Gaussians, and each Gaussian carries not just its parameters but also, during training, its gradients and its optimizer state (the Adam optimizer keeps two extra values per parameter). All of that has to live in GPU memory while training, and it adds up fast: on a large scene, a consumer GPU runs out of memory after only a few million Gaussians, which caps the achievable quality. For example, on one benchmark scene a laptop GPU can only fit about nine million Gaussians, limiting rendering quality. So the people who most want this — someone reconstructing a space on their own laptop, without a datacenter — hit a hard memory wall well before the scene is as detailed as they want.
What people did

Scale up the hardware, or just move the Gaussians to host memory

One answer is more or bigger GPUs — distributed training across multiple high-end cards — but that is expensive and complex, out of reach for the personal-use cases (a hobbyist modeling a room, a designer, a real-estate walkthrough) where 3DGS is most attractive. A cheaper-sounding answer is host offloading: the host machine has far more RAM than the GPU, so keep all the Gaussians' parameters and optimizer states in host (CPU) memory and transfer only the ones a given training image needs onto the GPU for its forward and backward pass, sending gradients back to the CPU afterward. This dramatically cuts GPU memory use, since the GPU only ever holds a working subset.
Why it fell short

Naive offloading shoves slow work onto the CPU and runs about four times slower

Offloading frees GPU memory but relocates two costly operations onto the CPU, where they run badly. The 3DGS training loop has a useful property — for any single training image, only the Gaussians inside that camera's viewing frustum participate in the forward and backward pass, so most Gaussians are idle each step. But two stages are exceptions. First, frustum culling — deciding which Gaussians are visible from this viewpoint — must project every Gaussian, all of them, and that compute-heavy operation is far slower on the CPU's modest FLOPS than on a GPU. Second, the Adam optimizer update must touch every parameter and optimizer state, even Gaussians that received no gradient this step, because Adam's momentum terms stay nonzero; this update is memory-bound, and CPU memory bandwidth is several times lower than GPU memory bandwidth, so it crawls. A third issue: peak GPU memory is still set by the single most demanding image (a far-away viewpoint that sees an unusually large number of Gaussians), which limits how much offloading helps. Together these make a naive host-offloading system about four times slower than GPU-only training. What was missing was a way to keep the memory savings of host offloading while hiding or eliminating the CPU work it creates.
This paper's idea

Offload to host, but keep culling on the GPU and pipeline and defer the CPU work

GS-Scale is a fast, memory-efficient 3DGS training system that keeps all Gaussians in host memory and streams only the needed subset to the GPU per pass, then removes the resulting CPU bottlenecks with three system-level optimizations. First, selective offloading: only the geometric attributes of every Gaussian — its mean, scale, and rotation (about ten small numbers), which are all that frustum culling needs — are kept resident on the GPU, so culling runs fast on the GPU; the much larger appearance attributes (spherical-harmonic color coefficients and opacity) are offloaded to host memory. Second, parameter forwarding: instead of running the CPU's Adam update strictly after the GPU's work, GS-Scale breaks that dependency and pipelines them — it pre-updates just the parameters the next iteration will need and forwards them, so CPU optimizer updates overlap with GPU forward and backward computation rather than serializing behind it. Third, deferred optimizer update: rather than have Adam touch every Gaussian every step, GS-Scale defers the updates for Gaussians with zero gradients, cutting the unnecessary memory-bound CPU work. Together these keep the dramatic GPU-memory savings while restoring speed: GS-Scale reports 3.3 to 5.6 times lower GPU memory demand at training speed comparable to GPU-only training with no offloading. That lets a consumer GPU scale far past its old limit — from four million to eighteen million Gaussians on an RTX 4070 Mobile laptop GPU, improving perceptual quality (LPIPS) by 23 to 35 percent.

Why this problem is worth solving

3D Gaussian Splatting is most valuable exactly where hardware is weakest — on personal, consumer machines — so lifting the memory wall there unlocks the technique for the people who want it most.

The appeal of 3D Gaussian Splatting is that anyone with a set of photos can reconstruct a real place in 3D: a hobbyist capturing a room in virtual reality, an interior designer laying out a virtual space, a real-estate walkthrough. These are personal, single-user scenarios, and the natural hardware is a laptop or a single consumer GPU, not a datacenter. But that is precisely the hardware with the least memory, and large high-quality scenes are exactly what overflow it. So the memory wall does not just slow experts down; it puts the high-quality version of the technique out of reach for its most natural users, forcing them to either accept low quality or rent expensive multi-GPU setups they should not need. Removing that wall — letting a laptop GPU train scenes several times larger than before — directly widens who can use 3DGS and how good their results can be. Because the benefit is measured in achievable scene quality on hardware people already own, it is broad and immediately practical.

The key is a property of the workload: most of the data is idle most of the time, so it does not need to be on the GPU.

The reason host offloading is a natural fit for 3DGS, rather than a desperate hack, is a structural fact about how it trains. Each training step renders one image from one viewpoint, and only the Gaussians inside that camera’s frustum contribute to that image’s forward and backward pass. In a large scene, that is a small fraction of all Gaussians; the rest are simply not involved this step. That means the full set of Gaussians does not need to sit in fast GPU memory at once — only the active subset does. Host memory, which is far larger and cheaper than GPU memory, is the right home for the whole collection, with the GPU holding just the working set. The catch is that two operations break this convenient sparsity: culling has to look at every Gaussian to decide which are active, and Adam has to update every Gaussian because its momentum never fully zeroes. Those two are what a naive offloading design pays dearly for. So the whole design problem reduces to preserving the memory win from the workload’s sparsity while specifically neutralizing the two stages that violate it — which is exactly what GS-Scale’s three optimizations target.

What people did before, and where it broke

Keep all Gaussians in GPU memory (standard 3DGS training). Store every parameter, gradient, and optimizer state on the GPU. Where it broke. A large high-quality scene needs far more Gaussians than fit, so a consumer GPU runs out of memory after a few million, capping quality.
Scale out to multiple high-end GPUs. Distribute training across several large cards. Where it broke. It is expensive and complex to maintain, impractical for the personal, single-user cases where 3DGS is most attractive.
Naively offload all Gaussians to host memory. Keep everything in CPU RAM, transfer only the frustum subset to the GPU. Where it broke. It forces frustum culling and Adam updates onto the CPU, whose lower compute and memory bandwidth make training about four times slower.
Run Adam over every Gaussian each step, as usual. Update all parameters and optimizer states every iteration. Where it broke. On the CPU this memory-bound update touches even zero-gradient Gaussians (Adam’s momentum stays nonzero), wasting scarce CPU bandwidth.
The gap. No system kept the GPU-memory savings of host offloading while hiding or removing the CPU-side culling and optimizer work it creates.

How GS-Scale works

Host offloading with the CPU bottlenecks engineered away

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Keep the whole model in cheap, abundant host memory and move only what the GPU needs. GS-Scale’s foundation is that the GPU should hold only the active working set, while the full collection of Gaussians lives in the host’s much larger memory, streamed across on demand. Storing a model too big for GPU memory in host memory and migrating pieces to the GPU as computation needs them is the same approach the work on tensor migration and prefetching in unified memory (DeepUM) covered elsewhere on this site takes, treating GPU memory as a cache over a larger host-resident whole. GS-Scale applies it to 3DGS, where the natural unit to migrate is the set of Gaussians visible from the current viewpoint.

Hide slow work behind fast work by pipelining instead of serializing. GS-Scale’s parameter forwarding refuses to let the CPU’s Adam update block the GPU: it pre-updates what the next step needs and overlaps the CPU work with GPU computation. Breaking a data dependency so a slow operation runs concurrently with, rather than before, the computation that follows is the same principle the work on overlapping communication with dependent computation covered elsewhere on this site is built on, turning a serial stall into a hidden, pipelined cost. GS-Scale uses it to make the CPU optimizer update effectively free by tucking it under the GPU’s forward and backward pass.

Do work only where it actually matters, and skip the rest. GS-Scale exploits that only a subset of Gaussians participate each step, both to decide what to stream and, in deferred optimizer update, to skip Adam work on Gaussians with zero gradients. Building the system around the fact that only a small, active fraction of the data needs work, and avoiding effort on the inactive remainder, is the same idea the work on stream-ISA processor specialization for sparse computation (SparseCore) covered elsewhere on this site develops, spending compute and bandwidth only on the entries that are live. GS-Scale reads the workload’s sparsity of participation as license to keep most Gaussians untouched most of the time.

Following a training iteration through GS-Scale

Training one image with all Gaussians in host memory

A trace, drawn from the paper, of how GS-Scale runs a single training iteration with its optimizations.

Step 1 of 5

Cull on the GPU using resident geometric attributes

The iteration begins by finding which Gaussians fall in the current image's viewing frustum. Because GS-Scale keeps every Gaussian's geometric attributes (mean, scale, rotation) resident on the GPU, this culling runs fast on the GPU rather than crawling on the CPU.

Only the small geometric attributes need to be GPU-resident for culling.

Why it works

The GPU only holds the active working set. Every Gaussian plus its gradients and optimizer state overflows GPU memory → but keeping the full collection in host memory and streaming only the frustum-visible subset → so GPU memory use drops 3.3 to 5.6 times.
Culling stays fast because only tiny data must be resident. Frustum culling needs every Gaussian, which is slow on the CPU → but geometric attributes are only about ten small numbers, cheap to keep GPU-resident → so culling runs on the GPU while the bulky appearance data stays offloaded.
Pipelining hides the CPU optimizer behind GPU work. A serial CPU Adam update after each GPU pass stalls training → but pre-updating the next iteration’s parameters and forwarding them → so the CPU update overlaps GPU compute and costs almost nothing in wall-clock time.
Deferring zero-gradient updates cuts wasted bandwidth. Adam normally touches every Gaussian because momentum stays nonzero → but deferring updates for Gaussians that got no gradient this step → so the memory-bound CPU update does far less work on the scarce CPU bandwidth.
The workload’s sparsity is what makes it all possible. Only a subset of Gaussians participate per image → but building streaming, culling, and updates around that active subset → so most Gaussians stay untouched in host memory and the memory-for-speed trade nearly disappears.

What it costs (trade-offs and scope)

Data must cross the PCIe interconnect every iteration. Streaming the visible subset’s appearance attributes from host to GPU each step consumes interconnect bandwidth; GS-Scale hides much of this, but the design depends on the transfer being small enough (only the visible subset, only appearance data) and on the interconnect keeping up.
The optimizations exploit 3DGS-specific structure. Selective offloading relies on culling needing only small geometric attributes, and deferred update relies on many Gaussians having zero gradients per step; these are properties of 3D Gaussian Splatting training, so the approach is tailored to it rather than a generic offloading library.
Peak memory is still shaped by the most demanding view. Even with offloading, the working set for a single far-viewpoint image that sees an unusually large number of Gaussians sets a floor on peak GPU memory, so the savings depend on the distribution of viewpoints in the scene.
Deferring updates changes when parameters are applied. Postponing zero-gradient Adam updates is designed to preserve correctness given Adam’s dynamics, but it makes the update schedule more intricate than a plain every-step Adam, adding bookkeeping the baseline does not have.
Reported numbers reflect specific scenes and hardware. The memory and quality figures (3.3 to 5.6 times, four to eighteen million Gaussians, 23 to 35 percent LPIPS) come from particular large-scale datasets and a specific consumer GPU; the exact gains depend on the scene and the hardware.

The evaluation

Training large 3D Gaussian Splatting scenes on a limited GPU

Property	GPU-only training	Naive host offloading	GS-Scale
Where Gaussians live
GPU memory demand
Frustum culling
Optimizer update cost
Training speed and scale

A comparison of ways to train large 3D Gaussian Splatting scenes on a limited GPU. By keeping all Gaussians in host memory and streaming the visible subset, while keeping geometric attributes GPU-resident for fast culling, pipelining CPU optimizer updates with GPU compute, and deferring zero-gradient updates, GS-Scale reports 3.3 to 5.6 times lower GPU memory at speed comparable to GPU-only training, scaling an RTX 4070 Mobile GPU from four million to eighteen million Gaussians for a 23 to 35 percent LPIPS improvement. Exact figures should be read from the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s full text (the authors’ arXiv copy, 2509.15645, of the ASPLOS 2026 paper). The described system is the paper’s: GS-Scale is a fast, memory-efficient training system for 3D Gaussian Splatting that stores all Gaussians (parameters and optimizer states) in host memory and transfers only the subset within each training image’s viewing frustum to the GPU for its forward and backward pass. Naive host offloading forces two operations onto the CPU — frustum culling (which must project all Gaussians) and the Adam optimizer update (which must touch all parameters and states, since Adam’s momentum stays nonzero for zero-gradient Gaussians) — making it about four times slower than GPU-only training. GS-Scale adds three optimizations: selective offloading, which keeps only each Gaussian’s geometric attributes (mean, scale, rotation) GPU-resident so frustum culling runs on the GPU while appearance attributes (spherical harmonics and opacity) are offloaded; parameter forwarding, which pipelines CPU optimizer updates with GPU computation by pre-updating and forwarding needed parameters; and deferred optimizer update, which skips Adam updates for Gaussians with zero gradients. The paper reports 3.3 to 5.6 times lower GPU memory demand at training speed comparable to GPU-only training with no offloading, and shows scaling from four million to eighteen million Gaussians on an RTX 4070 Mobile GPU with a 23 to 35 percent LPIPS improvement (for example, 35.3 percent on the Rubble scene). The step-by-step walkthrough above is an illustration of the described mechanism. Exact algorithms, the pipelining and deferral details, and the full evaluation should be read from the paper.

Background

Why fitting big models onto small GPUs keeps coming back to host memory

It helps to see GS-Scale within a broad pattern: again and again, the way to run a model too big for GPU memory on modest hardware is to stop insisting everything be GPU-resident, keep the bulk in cheaper, larger memory, and move only the active working set across on demand. The GPU’s fast memory is small and expensive; host memory is abundant; and most workloads, at any instant, only touch a fraction of their data. The recurring recipe is therefore to identify that active fraction, stream it in, and then fight the overheads that streaming creates — the transfer latency and any work that got pushed onto the slower processor — by overlapping it with useful computation and by skipping work that does not need doing. GS-Scale is a particularly clean case because 3D Gaussian Splatting has such sharp participation sparsity per view, but the shape of the solution — offload, stream, pipeline, prune — is the same one that recurs whenever demanding models meet memory-limited accelerators. This push to make heavy training and inference fit on the consumer-grade GPUs people actually own, rather than requiring datacenter hardware, is the same terrain the work on streaming weights for modern DNN workloads on mobile GPUs (FlashMem) covered elsewhere on this site occupies, refusing to let limited memory be the ceiling on what a small device can run. GS-Scale’s contribution is to show that, with culling kept cheap and the CPU work hidden and pruned, host offloading can grow a laptop GPU’s 3DGS capacity several-fold at almost no speed cost.

If you remember only three things

1. Training large, high-quality 3D Gaussian Splatting (3DGS) scenes needs millions of Gaussian primitives, and each carries parameters plus, during training, gradients and Adam optimizer states — which overflow GPU memory, so a consumer GPU caps out after a few million Gaussians and cannot reach high quality. Offloading all Gaussians to the host's abundant memory and streaming only the subset a given view needs frees the GPU, but it forces two operations onto the CPU — frustum culling (which must examine every Gaussian) and the Adam update (which must touch every parameter, since its momentum stays nonzero even for zero-gradient Gaussians) — and the CPU's far lower compute and memory bandwidth make this naive offloading about four times slower than GPU-only training.
2. GS-Scale keeps host offloading's memory savings while removing its CPU bottlenecks with three optimizations. Selective offloading keeps only each Gaussian's small geometric attributes (mean, scale, rotation) GPU-resident so frustum culling runs fast on the GPU, offloading the larger appearance attributes (spherical-harmonic color, opacity) to host. Parameter forwarding pipelines the CPU optimizer update with GPU computation by pre-updating and forwarding the parameters the next iteration needs, so the CPU work overlaps rather than stalls. Deferred optimizer update skips Adam updates for Gaussians that received no gradient this step, cutting wasted memory-bound CPU work. The design exploits that only the Gaussians in the current view participate each step.
3. The paper reports GS-Scale lowers GPU memory demand by 3.3 to 5.6 times while training at speed comparable to GPU-only training with no offloading, enabling a consumer RTX 4070 Mobile laptop GPU to scale from four million to eighteen million Gaussians and improve perceptual quality (LPIPS) by 23 to 35 percent (for example, 35.3 percent on the Rubble scene). The trade-offs: data crosses the PCIe interconnect each iteration, the optimizations exploit 3DGS-specific structure, peak memory is still shaped by the most demanding view, and deferring updates adds bookkeeping. It is the first host-offloading-based 3DGS training system, and the reported numbers reflect specific scenes and hardware.

Check yourself

Check your understanding

1. Why does training a large, high-quality 3D Gaussian Splatting scene overflow GPU memory?
2. Why is naively offloading all Gaussians to host (CPU) memory about four times slower than GPU-only training?
3. What is "selective offloading," and how does it fix the frustum-culling bottleneck?
4. How do "parameter forwarding" and "deferred optimizer update" restore speed, and what did GS-Scale achieve?

Sources

The paper is “GS-Scale: Unlocking Large-Scale 3D Gaussian Splatting Training via Host Offloading” by Donghyun Lee, Dawoon Jeong, Jae W. Lee, and Hongil Yoon (Seoul National University and Google), published at ASPLOS 2026, DOI 10.1145/3779212.3790167. It is available on arXiv as 2509.15645.

This explainer is written from the paper’s full text (the authors’ arXiv copy). GS-Scale is a fast, memory-efficient training system for 3D Gaussian Splatting that stores all Gaussians (parameters and optimizer states) in host memory and transfers only the subset within each training image’s viewing frustum to the GPU for forward and backward passes. Because naive host offloading forces frustum culling and Adam optimizer updates onto the CPU — where lower compute and memory bandwidth make training about four times slower — GS-Scale adds three optimizations: selective offloading, keeping only geometric attributes (mean, scale, rotation) GPU-resident for fast culling while offloading appearance attributes (spherical harmonics and opacity); parameter forwarding, pipelining CPU optimizer updates with GPU computation by pre-updating and forwarding needed parameters; and deferred optimizer update, skipping Adam updates for zero-gradient Gaussians. The paper reports 3.3 to 5.6 times lower GPU memory demand at training speed comparable to GPU-only training, and scaling from four million to eighteen million Gaussians on an RTX 4070 Mobile GPU with a 23 to 35 percent LPIPS improvement. Numbers are the paper’s own; exact mechanisms and full results should be confirmed in the paper.