CLM: Removing the GPU Memory Barrier for 3D Gaussian Splatting

Hexu Zhao, Xiwen Min, Xiaoteng Liu, Moonjun Gong, Yiming Li, Ang Li and 3 others

3D Gaussian Splatting renders photorealistic scenes from millions of tiny learned blobs, but a big scene needs so many that its data blows past a GPU's memory, forcing people onto expensive multi-GPU rigs or to throw away detail; this paper (NYU and PNNL) notices that any single camera view actually touches under one percent of the blobs, so it keeps most of them in plentiful CPU memory and streams only the needed ones to the GPU, cleverly overlapping the transfers with computation and reordering the work so nearby views reuse what's already loaded — letting a single consumer RTX 4090 train a scene of 102 million blobs at state-of-the-art quality.

The argument in one glance

Problem

Photorealistic scenes need millions of blobs, and they do not fit in a GPU

3D Gaussian Splatting has become a popular way to turn a set of photos into a 3D scene you can view from new angles, because it renders fast and looks great. It does this by representing the scene as an enormous collection of tiny anisotropic 3D Gaussians — think of them as soft, colored blobs — each carrying dozens of learnable parameters for its position, shape, color, and opacity, and it trains them by gradient descent to reproduce the input images. The trouble is that quality scales with the number of Gaussians: a large area, intricate detail, or high resolution demands more blobs, and the memory footprint balloons. A city-scale scene can need a hundred million Gaussians, and once you count not just the parameters but their gradients and the optimizer state, the memory requirement reaches over a hundred gigabytes — while a consumer GPU like an RTX 4090 has only twenty-four. So GPU memory capacity has become a hard barrier: the scenes people most want to capture are exactly the ones that will not fit.
What people did

Use multiple GPUs, or shrink the scene representation

Prior work scales 3D Gaussian Splatting two ways. One is to spread the Gaussians across several GPUs, pooling their memory so the whole scene fits. The other is to keep to a single GPU but reduce the fidelity of the representation — using fewer Gaussians, or compressing them — so the smaller footprint fits in the available memory.
Why it fell short

More GPUs cost more, and fewer Gaussians cost quality

Both escapes give something up. Spreading the scene across multiple GPUs works, but it multiplies the hardware cost and complexity, putting large scenes out of reach for anyone without a cluster — which defeats much of the appeal of a method prized for running on ordinary hardware. Shrinking the representation keeps it on one GPU but directly attacks the thing that makes 3D Gaussian Splatting good: with fewer or coarser Gaussians, the rendered images lose detail and fidelity, so you trade away exactly the quality you were trying to achieve. Neither path lets a single, affordable GPU render a large, intricate scene at full quality. The obvious alternative — keep the extra Gaussians somewhere cheaper and bring them in as needed — sounds simple, but the traffic between that cheaper storage and the GPU is slow enough that a naive version is far too costly, so nobody had made offloading actually fast for this workload.
This paper's idea

Keep the blobs in CPU memory, and stream in only the tiny subset each view needs

CLM removes the memory barrier on a single GPU by exploiting a fact about how 3D Gaussian Splatting computes: it is inherently sparse. Rendering any one camera view only touches the Gaussians that fall within that camera's frustum — for a large scene, under one percent of them. So CLM keeps the full scene in plentiful CPU memory and loads into the GPU only the small subset a view actually needs. The hard part is making the CPU-to-GPU traffic cheap enough, and CLM does so with a 3DGS-specific offloading strategy built on four observations: the needed Gaussians can be computed ahead of time so their loading overlaps the previous iteration's compute; different views reuse many of the same Gaussians so a GPU cache avoids reloading them; nearby views access similar Gaussians so training iterations can be reordered to maximize that reuse; and the same locality lets the optimizer update overlap with computation. The result trains a hundred-million-Gaussian scene on a single RTX 4090 at state-of-the-art quality.

Why this problem is worth solving

The method’s strength and its memory problem are the same thing: quality comes from sheer numbers of Gaussians.

3D Gaussian Splatting learns a scene from posed photos as a large set of anisotropic 3D Gaussians, each with several dozen learnable parameters — a 3D position, an anisotropic covariance giving its scale and rotation, spherical-harmonic coefficients for view-dependent color, and an opacity, about fifty-nine numbers in all. Training is gradient descent: pick a camera view, select the Gaussians in its frustum, rasterize them into an image, compare to the ground-truth photo, and backpropagate to update the Gaussians, periodically adding more where detail is missing. Because fidelity depends on having enough Gaussians, capturing a large or intricate scene at high resolution needs a great many, and the memory cost is not just the parameters. Gradient-based training also stores a gradient for every parameter and the optimizer’s state, several times the parameters again, so a scene of a hundred million Gaussians needs well over a hundred gigabytes. Consumer GPUs top out around twenty-four gigabytes, so the biggest, most useful scenes simply cannot be trained on one. This memory wall is what stands between 3D Gaussian Splatting’s promise and its application to the large scenes people actually want, which is why removing it matters.

The way out has to avoid both the cost of more hardware and the loss of quality, and the enabling fact is sparsity.

If most of a scene’s Gaussians are not needed at any given moment, they need not occupy scarce GPU memory. And in 3D Gaussian Splatting they are not: rendering a single view only involves the Gaussians inside that camera’s frustum, and for a large scene that is a tiny fraction — measured at under one percent, with an average view of a city-scale scene touching about four tenths of a percent of all Gaussians and the busiest view about one percent. Larger scenes are even sparser, because the frustum’s volume stays fixed while the Gaussian count grows. This makes offloading the obvious idea: store the full scene in the far more plentiful CPU memory, and pull the needed subset into the GPU on demand. The catch is that the link between CPU and GPU is slow relative to GPU computation, so a naive offloading scheme spends most of its time waiting on data transfers and on running the optimizer update on the CPU, without any way to hide that cost behind useful work. Making offloading fast enough to be practical, rather than merely possible, is the real problem, and it requires exploiting the specific structure of how 3D Gaussian Splatting accesses its Gaussians.

What people did before, and where it broke

Multi-GPU training. Distribute the Gaussians across several GPUs to pool their memory. Where it broke. It multiplies hardware cost and complexity, so large scenes need a cluster rather than the ordinary single GPU the method is prized for.
Reduce the representation’s fidelity. Use fewer or compressed Gaussians to shrink the footprint onto one GPU. Where it broke. Fewer or coarser Gaussians directly lower rendered image quality, sacrificing the very thing 3D Gaussian Splatting is good at.
Naive offloading (ZeRO-Offload-style). Keep Gaussians, gradients, optimizer state, and the optimizer computation on the CPU, sending gradients back for the CPU to run the update. Where it broke. It pays heavy CPU-GPU communication and CPU optimizer cost with no way to hide either behind GPU work, and no way to cut the volume of data transferred, so it is far too slow.
The gap. No approach let a single consumer GPU train a large, high-fidelity scene by offloading, because none made the offloading fast — overlapping transfers with computation and shrinking the transfer volume using 3D Gaussian Splatting’s access patterns. Doing that is what CLM contributes.

How CLM works

Sparsity-guided offloading with overlap and locality-aware scheduling

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Store the scene where memory is plentiful, and move only the tiny active subset. CLM’s foundation is that 3D Gaussian Splatting’s computation is sparse — any one view touches under one percent of a large scene’s Gaussians — so the vast majority can sit in CPU memory while only the needed subset occupies the GPU. This turns a GPU-memory-capacity problem into a data-movement problem: keep the full model in the big, cheap tier and migrate the working set into the small, fast tier on demand. Migrating tensors between plentiful host memory and scarce GPU memory, bringing in only what the computation is about to use and prefetching it ahead of need, is exactly the tiering-and-prefetch strategy the work on tensor migration and prefetching in unified memory covered elsewhere on this site is built on, treating the GPU as a fast cache over a larger backing store rather than a hard capacity limit.

Hide the transfers by overlapping them with computation, and reorder the work to help. The reason naive offloading fails is that the CPU-GPU link is slow and its cost is exposed; CLM’s answer is to never let a transfer sit on the critical path. Because each view’s needed Gaussians can be computed ahead of time, CLM pipelines single-image microbatches so that loading one microbatch’s data overlaps the previous one’s backward pass, returning gradients overlaps the next one’s forward pass, and the CPU optimizer update overlaps later GPU passes. It then goes further, reordering microbatches — since their order does not affect the accumulated result — to place views that share Gaussians next to each other, maximizing both cache reuse and update overlap. Restructuring a computation so that communication is overlapped with the dependent computation, rather than paid serially, is the same latency-hiding discipline the work on overlapping communication with dependent computation via decomposition covered elsewhere on this site applies, decomposing and rescheduling work so the expensive transfers happen in the shadow of useful compute.

Let a single commodity GPU do what took a cluster, without losing quality. The payoff is that the memory barrier falls without either buying more GPUs or coarsening the scene: CLM trains a hundred-million-Gaussian scene on one consumer RTX 4090 at state-of-the-art fidelity, several times larger than what the same GPU could hold before. Removing the memory ceiling that forced large-model training onto expensive multi-GPU systems, so that a single commodity GPU server suffices, is the same democratizing goal the work on fine-tuning large-scale models on commodity GPU servers covered elsewhere on this site pursues, using offloading to fit a model far bigger than the GPU’s memory would normally allow. CLM brings that capability to 3D Gaussian Splatting, where the payoff is measured directly in the size and quality of scenes an ordinary machine can reconstruct.

Following a training batch through CLM

Training one batch of a huge scene on a single GPU

A trace, following the paper's design, of how CLM trains a batch whose Gaussians far exceed GPU memory.

Step 1 of 5

The scene is too big for GPU memory

A city-scale scene needs a hundred million Gaussians, over a hundred gigabytes with gradients and optimizer state — far beyond the GPU's twenty-four. CLM stores them all in pinned CPU memory, keeping only their small position, rotation, and scale attributes on the GPU.

Those selection-critical attributes are enough to decide which Gaussians each view sees.

Why it works

Any one view needs almost none of the scene. A large scene has enormous Gaussians in total → but a camera frustum has fixed volume, so a single view touches under one percent of them → so keeping the full scene in CPU memory and loading only a view’s subset fits the GPU with room to spare.
The needed subset is known before it is used. Offloading normally stalls waiting for data → but frustum culling can compute each view’s Gaussians ahead of time using only small on-GPU attributes → so loading them can start early and overlap the previous iteration’s computation instead of blocking.
Consecutive views reuse many of the same Gaussians. Reloading everything every iteration would swamp the link → but nearby views share large fractions of their Gaussians → so caching what a microbatch shares with the next removes redundant transfers, cutting communication volume.
Reordering microbatches is free but increases reuse. The transfer savings depend on how much consecutive microbatches overlap → but microbatch order does not change the accumulated batch result, so it can be chosen freely → so scheduling similar views together, via a traveling-salesman formulation, maximizes caching and update overlap at no cost to correctness.
The optimizer update hides behind later work. Running the optimizer on the CPU is slow and could serialize each step → but Gaussians finalized by an early microbatch can be updated on the CPU while later microbatches compute on the GPU → so the update cost is overlapped rather than added to the critical path.

What it costs (trade-offs and scope)

It trades scarce GPU memory for CPU memory and a fixed GPU tax. CLM needs enough pinned CPU memory to hold the whole scene, and it keeps the small position, rotation, and scale attributes of every Gaussian permanently on the GPU; that per-Gaussian GPU cost is small but grows with the Gaussian count, so at extreme scales even it becomes a consideration.
The speedup depends on having compute to hide transfers behind. CLM overlaps transfers with GPU computation, so its advantage over naive offloading is larger on slower GPUs, where more compute time is available to hide the offloading — measured at about 1.9 times faster on a slower GPU versus about 1.6 times on a faster one; a very fast GPU leaves less to overlap.
The benefit is for large scenes; small ones pay overhead. The approach relies on sparsity, which is strong for large scenes but weaker for small ones where each view sees a larger fraction; for scenes that already fit in GPU memory, CLM adds modest overhead over plain GPU training and is not the point.
Scheduling is a heuristic, and offloading is not free throughput. The microbatch ordering is a greedy traveling-salesman solution rather than optimal, and CPU-side optimizer computation is slower than on the GPU, hidden by overlap but not eliminated; CLM removes the memory ceiling on one GPU rather than matching the raw throughput of many.
It targets 3D Gaussian Splatting’s specific structure. The design leans on frustum-culling sparsity, spatial locality across views, and gradient accumulation over microbatches, so it is tailored to this workload rather than a general offloading system, and the evaluation is on specific scenes and consumer GPUs.

The evaluation

Scaling 3D Gaussian Splatting on a single GPU

Property	GPU-only baseline	Naive offloading
Largest scene on one RTX 4090
Reconstruction quality (BigCity)		n/a
Communication handling
Optimizer update	On GPU
Throughput vs naive offloading (BigCity)	n/a	Baseline

From the paper's evaluation on RTX 4090 and RTX 2080 Ti against a GPU-only baseline (Grendel-GS run single-GPU) and a naive offloading implementation. CLM trains a 102-million-Gaussian city-scale scene on a single RTX 4090 at a state-of-the-art PSNR of 25.15, versus the GPU-only baseline's 15.3-million-Gaussian limit and PSNR of 23.93; it scales to roughly 5.7 to 6.1 times larger models than the baseline and 2.2 to 2.3 times larger than naive offloading, while running 1.58 to 1.92 times faster than naive offloading, with larger gains on slower GPUs. Full results are in the paper.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

The contribution is a practical way to lift the GPU memory ceiling on 3D Gaussian Splatting without a cluster or a quality cut, by turning a capacity problem into a well-hidden data-movement problem. CLM’s insight is that the workload is sparse — any view touches under one percent of the Gaussians — so it keeps the full scene in CPU memory, keeps only the small selection-critical attributes on the GPU to run frustum culling, and streams the needed subsets in. Its four observations make the offloading fast: precomputable access sets enable pipelined overlap of loading, gradient return, and CPU optimizer updates; heavy Gaussian reuse across views enables GPU caching; and spatial locality enables reordering microbatches, as a traveling-salesman problem, to maximize that reuse. The result trains a hundred-million-Gaussian scene on one RTX 4090 at state-of-the-art quality, several times larger than before, at up to nearly twice the speed of naive offloading. Read it within scope. It requires ample CPU memory and a fixed per-Gaussian GPU cost; its speedup depends on having GPU compute to hide transfers behind, so it helps more on slower GPUs; it targets large, sparse scenes and adds overhead on small ones; its scheduling is heuristic and CPU-side updates are slower but overlapped; and it is tailored to 3D Gaussian Splatting’s structure. Within those bounds it convincingly removes the memory barrier for single-GPU high-fidelity scene reconstruction.

Background

Why the GPU memory wall keeps reshaping systems

It helps to see the recurring pressure this work responds to. Across machine learning, models and their working data keep outgrowing the memory of the GPUs meant to run them, and GPU memory is both scarce and expensive to add. The field’s repeated answer is to stop treating GPU memory as a hard wall and instead treat it as the fast tier of a hierarchy, with the plentiful CPU memory (and beyond) as a larger backing store, moving data between them so that only the active working set sits on the GPU. The art is entirely in hiding the cost of that movement, because the link to the GPU is slow relative to how fast the GPU computes, so the technique only pays off when transfers are overlapped with computation, prefetched ahead of need, and minimized by exploiting access locality. This pattern shows up wherever a workload is too big for the GPU it must run on, and 3D Gaussian Splatting is a striking new instance, because its memory demand grows with the size and detail of the scene rather than with a fixed model. The broader project of building efficient hardware and systems support for these rendering workloads, so that photorealistic reconstruction becomes cheap and widespread, is the same trajectory the work on efficient radiance-field rendering covered elsewhere on this site is part of, and CLM contributes the memory-scaling piece that lets a single ordinary GPU take on scenes that used to demand far more.

If you remember only three things

1. 3D Gaussian Splatting represents a scene as millions of learnable 3D Gaussians and its rendered quality scales with their number, so large, intricate, high-resolution scenes need so many — a city scale needs a hundred million, over a hundred gigabytes with gradients and optimizer state — that they exceed a consumer GPU's twenty-four gigabytes; prior scaling either used multiple GPUs (costly) or reduced the number of Gaussians (sacrificing quality), and naive CPU offloading was far too slow because the CPU-GPU link is slow and its cost was fully exposed.
2. CLM removes the barrier on a single GPU by exploiting that 3DGS is inherently sparse — any one view, via frustum culling, touches under one percent of a large scene's Gaussians — so it keeps the full scene in plentiful CPU memory and streams in only each view's needed subset, keeping just the small selection-critical attributes (position, rotation, scale) of all Gaussians permanently on the GPU so frustum culling needs no extra transfer, and making the offloading fast with four ideas: precomputing each view's access set to pipeline loading over the previous iteration's compute, caching Gaussians shared by consecutive microbatches to cut transfer volume, reordering microbatches (safe under gradient accumulation) via a traveling-salesman formulation to maximize that reuse, and overlapping the CPU optimizer update with later GPU work.
3. On a single RTX 4090, CLM trains a 102-million-Gaussian city-scale scene at a state-of-the-art PSNR of 25.15, versus the GPU-only baseline's 15.3-million-Gaussian limit and PSNR of 23.93 — scaling to roughly 5.7 to 6.1 times larger models than the baseline and 2.2 to 2.3 times larger than naive offloading, while running 1.58 to 1.92 times faster than naive offloading with larger gains on slower GPUs; the trade-offs are that it needs ample CPU memory plus a small fixed per-Gaussian GPU cost, its speedup depends on having compute to hide transfers behind, it benefits large sparse scenes while adding modest overhead on small ones, and it is tailored to 3D Gaussian Splatting's access structure.

Check yourself

Check your understanding

1. Why does 3D Gaussian Splatting hit a GPU memory barrier on large scenes?
2. What is the key insight that lets CLM offload without huge slowdown?
3. How does CLM make the CPU-GPU offloading fast?
4. What did CLM achieve, and what are its trade-offs?

Sources

The paper is “CLM: Removing the GPU Memory Barrier for 3D Gaussian Splatting” by Hexu Zhao, Xiwen Min, Xiaoteng Liu, Moonjun Gong, Yiming Li, Ang Li, Saining Xie, Jinyang Li, and Aurojit Panda (New York University, and Pacific Northwest National Laboratory and the University of Washington), published at ASPLOS 2026, DOI 10.1145/3779212.3790140. Its code is open-sourced.

This explainer is written from the full text of the authors’ arXiv version (arXiv:2511.04951). Everything above comes from the paper’s own sections: the introduction and background on novel view synthesis and 3D Gaussian Splatting, its representation as anisotropic 3D Gaussians with about fifty-nine learnable parameters (position, covariance, spherical harmonics, opacity) trained by minibatch gradient descent with frustum culling, rasterization, loss, backpropagation, and adaptive densification; the memory barrier, with the table of scenes and their memory demands (up to a hundred gigabytes for a hundred-million-Gaussian scene against a 24-gigabyte RTX 4090) and the limitations of prior multi-GPU and fidelity-reducing approaches; the four observations that computation is sparse (a single view accessing under one percent of Gaussians, an average of about 0.39 percent and a maximum of 1.06 percent for the largest scene), that views’ accessed Gaussians overlap enabling caching, that spatial locality enables scheduling, and that locality enables overlapping the optimizer update; the design, including keeping selection-critical attributes (position, rotation, scale) of all Gaussians on the GPU for frustum culling while non-critical attributes are offloaded to CPU, microbatch pipelining with gradient accumulation, precise Gaussian caching, overlapped CPU Adam, and pipeline order optimization framed as a traveling-salesman problem solved with a greedy heuristic; and the evaluation on RTX 4090 and RTX 2080 Ti against Grendel-GS as a GPU-only baseline and a naive offloading implementation, reporting training of a 102.2-million-Gaussian scene with a PSNR of 25.15 versus the baseline’s 15.3-million limit and PSNR of 23.93, scalability of roughly 5.7 to 6.1 times over the baseline and 2.2 to 2.3 times over naive offloading, and throughput of 1.58 to 1.92 times naive offloading with larger gains on slower GPUs.