GSCore: Efficient Radiance Field Rendering via Architectural Support for 3D Gaussian Splatting

Photorealistic 3D scenes made of millions of tiny Gaussian blobs render beautifully on desktop GPUs and crawl on headsets and edge devices — GSCore is a 4-square-millimeter chip that renders them 16× faster than a mobile GPU by refusing to sort, test, or shade anything that will not end up in a pixel.

Why this problem is worth solving

Novel-view synthesis — photograph a scene from some positions, then render it from any position — was revolutionized by radiance-field methods. NeRF stores the scene implicitly in a neural network and pays for beauty with millions of network evaluations per frame. 3D Gaussian Splatting (2023) is the breakthrough that made radiance fields fast: it represents the scene explicitly, as a dense set of 3D Gaussians — soft ellipsoid blobs, each with a position, shape (scale plus rotation), opacity, and view-dependent color stored as spherical-harmonic coefficients, 59 learned parameters in all. Rendering is rasterization, not neural inference: project each ellipsoid onto the screen as a 2D ellipse (a “splat”), sort splats by depth, and blend their colors front to back until each pixel is opaque. Unity, Unreal and Blender began adopting it almost immediately.

The catch is where the technique most wants to run. Virtual-reality headsets and edge devices are exactly the platforms hungry for photorealistic 3D — and exactly the platforms whose power budgets cap GPU performance. The paper’s motivating measurement: a scene that renders at over 100 frames per second on a desktop RTX 3090 achieves 6.4 frames per second on a Jetson Xavier NX, with the Oculus Quest 2’s GPU only marginally better. Battery-powered devices will not grow desktop-class GPUs; the computation must shrink instead.

Profiling shows where the time goes: Gaussian sorting and rasterization together consume about 80% of rendering time across scenes and platforms. But the paper’s contribution begins with refusing to accelerate those steps blindly — because much of that 80% is work whose output is never used.

What people did before, and where it broke

The reference pipeline has four stages. Frustum culling discards Gaussians behind or beside the camera. Feature computation projects each survivor to a 2D ellipse with a screen-space covariance, and collapses its 48 spherical-harmonic coefficients into one RGB color for this viewpoint. Sorting orders each tile’s Gaussians by depth, because blending must run front to back. Rasterization computes each Gaussian’s alpha — its opacity at each pixel, an exponential falloff from the ellipse center — and accumulates colors until early termination: once a pixel is effectively opaque, everything behind it is skipped.

Three measured inefficiencies hide inside this sensible design:

False tile assignments. Which tiles does an ellipse overlap? The GPU implementation answers cheaply: bound every ellipse with an axis-aligned square whose half-width is the ellipse’s semi-major axis. For round splats, fine. But trained scenes are full of highly elongated ellipses — and a long thin ellipse in a fat square gets assigned to every tile the square touches, mostly falsely (75% of tiles in the paper’s illustration). Every false assignment inflates both sorting and rasterization downstream.

Sorting for the bin. Early termination means back-of-scene Gaussians are often never blended: on average 18–55% of a tile’s assigned Gaussians — up to 98% in some tiles — are discarded after being fully sorted. The pipeline pays a precise global sort for elements whose order will never be consulted.

Lockstep waste. A GPU thread block processes one tile, all 256 threads marching through the same Gaussian list. When a small splat touches only a corner of the tile, the other threads compute its alpha anyway — multiple floating-point operations including an exponential — just to discover irrelevance. The fraction of computed Gaussians that actually contribute color to a pixel is small across every scene measured.

The idea

Shape-aware intersection: fit the box to the blob

The fix for false assignments is a tighter box: an oriented bounding box (OBB) that rotates with the ellipse instead of hugging the axes. Testing tile-versus-OBB overlap costs more — it needs the Separating Axis Theorem, checking projected overlap on four axes — so GSCore spends it opportunistically: only when the cheap axis-aligned test found multiple candidate tiles and the ellipse is elongated (major-to-minor axis ratio above 2). Round splats keep the cheap test; skewed splats — the ones that generate false positives — get the strict one. In the paper’s example the OBB cuts 16 falsely-intersected tiles to 4. Crucially the test errs only one way: no false negatives, so the rendered image is untouched. On the GPU alone, this one change yields a 1.71× speedup; in GSCore, dedicated OBB intersection units (two per CCU) make it nearly free.

Hierarchical sorting: don’t sort what termination will discard

Sorting is restructured into two stages. Approximate sorting quickly partitions a tile’s Gaussians into depth-ordered chunks (pick a pivot depth, split lower/higher — recursively, like one level of quicksort): every Gaussian in chunk i is nearer than every one in chunk i+1, but chunks are internally unordered. Precise sorting then orders one chunk at a time, just before the renderer needs it. Two savings compound: precise sorting of later chunks overlaps with rasterization of earlier ones (hiding sort latency), and when early termination fires mid-tile, the remaining chunks are simply never sorted. The blending order that matters is preserved exactly — laziness, not approximation. Chunk size tunes the trade (small chunks expose memory latency, huge ones re-import the waste); 256 Gaussians is the sweet spot.

Subtile skipping: know which pixels a Gaussian can touch

Per-pixel Gaussian lists would eliminate lockstep waste but cost too much state. GSCore takes the middle ground: divide each 16×16 tile into subtiles, and during preprocessing store, per Gaussian, an N-bit bitmap — bit i says whether the Gaussian’s ellipse (its exact 3-sigma boundary, computed from the covariance eigenvalues) overlaps subtile i. During rendering, the Volume Rendering Unit processes one subtile at a time across its 4×4 grid of cores (each core owning 2×2 pixels); a zero bit skips the whole subtile’s alpha computation for that Gaussian. Measured bitmap sparsity runs 35–52% — a third to half of all per-subtile alpha work vanishes, exactly the work that produced nothing.

Hardware shaped to the surviving work

With the waste removed, the remaining pipeline maps onto three specialized units — culling/conversion, sorting, and volume rendering — connected by a double-buffered feature store, all in FP16 arithmetic (measured harmless: at most a 0.5% PSNR change, none perceptible). The units pipeline across tiles the way the pure algorithm cannot on lockstep GPU hardware. Total: 3.95 mm² at 28 nm, 1 GHz, 0.87 W — a small corner of a mobile SoC.

AABB test: 16 candidate tiles
OBB (SAT) test: 4 real intersections
false positives eliminated: 75%

Why it works

Because the waste was identified before the hardware was designed, each optimization removes a class of dead work rather than accelerating it: false tile assignments (shape-aware testing), sorted-then-discarded Gaussians (hierarchical laziness), and alpha computations outside a Gaussian’s reach (subtile bitmaps). → Because every optimization is exact — no false negatives, preserved blending order, precise ellipse extents — the speedups compound without touching image quality: PSNR and LPIPS match the reference to FP16 rounding. → Because the surviving work has a fixed, known structure, small specialized units (OBB testers, pivot-based sorters, a 4×4 blending grid with bitmap gating) execute it at far higher utilization than lockstep GPU warps — rendering-core utilization rises roughly 4×. → Because sorting now overlaps rendering and skipping shrinks memory traffic, the pipeline stays busy end to end on 0.87 watts. → Therefore a 3.95 mm² block turns 6.4 frames per second into 91.2 on the same scene at the same quality — real-time radiance-field rendering inside an edge-device power budget.

What it costs

• It is new silicon. GSCore is a synthesized 28 nm design evaluated in a cycle-level simulator (with functional output images) — an IP block a mobile SoC would need to adopt, not software you can ship today. The algorithmic ideas partially transfer (1.71× + 15% on the GPU itself), the full 15.86× does not.
• Rendering only. Training the Gaussians — the cloning, splitting and gradient descent that create the scene — stays on big GPUs; GSCore accelerates inference-time view synthesis.
• Preprocessing does more work up front. OBB tests and bitmap generation add cost to the cheapest pipeline stage to save the expensive ones — a good trade, but one whose balance (axis-ratio threshold 2, chunk size 256, 4×4 cores) was tuned empirically and inherits the evaluated scenes.
• FP16 is a quality judgment call. The measured degradation is negligible on six scenes; the exponential in alpha computation already needed care, and different content could stress precision differently.
• The pipeline is frozen in hardware. Gaussian Splatting is evolving fast (the paper predates many variants); fixed-function units bet that the ellipse-sort-blend skeleton stays stable.

The evaluation

Six scenes — four real-world captures (Train, Truck, Playroom, Drjohnson) and two synthetic (Lego, Palace) — each trained for 30K iterations, at varying resolutions. Baseline: Jetson Xavier NX (mobile Volta GPU) running the reference implementation. GSCore: SystemVerilog, synthesized on a commercial 28 nm library at 1 GHz, evaluated in a cycle-level simulator with a detailed DRAM model that also produces the rendered images.