ASDR: Exploiting Adaptive Sampling and Data Reuse for CIM-based Instant Neural Rendering

Fangxin Liu, Haomin Li, Bowen Zhu, Zongwu Wang, Zhuoran Song, Haibing Guan and 1 other

Neural Radiance Fields render photorealistic 3D scenes from a neural network, but doing it is so slow and power-hungry that it cannot run in real time on a phone-class device, because to color each pixel the renderer shoots a ray, samples many points along it, and runs two neural networks plus big table lookups at every point; this work from Shanghai Jiao Tong co-designs the algorithm and a compute-in-memory chip to spend effort only where it matters — sampling fewer points on easy pixels, skipping the expensive color network for points that barely show up, and laying out the lookup tables to avoid memory conflicts — for up to a 70 times speedup with almost no quality loss.

The argument in one glance

Problem

Neural rendering is too slow and power-hungry for real time

NeRF renders photorealistic 3D scenes from a neural network, but to color each pixel it shoots a ray, samples many 3D points along it, and runs table lookups plus two neural networks at every point. This hammers memory and compute, so it cannot hit real-time, low-power targets on practical devices.
What people did

Accelerate one stage, treating every pixel the same

Existing NeRF accelerators optimize a single stage — the encoding, the neural networks, or the compositing — in isolation, and sample a fixed number of points per pixel and run both networks fully at every point.
Why it fell short

The work done far exceeds what the image needs

Pixels differ hugely in rendering difficulty, neighboring rays share data, the two networks are lopsided in cost (the color one is far heavier), and the hash-table accesses collide. Optimizing one stage and treating everything uniformly leaves all of this on the table, so the renderer does far more work than the final image requires.
This paper's idea

Co-design the algorithm and a compute-in-memory chip (ASDR)

ASDR spends effort only where it matters: adaptive sampling shoots fewer points at easy pixels, an MLP-decoupling scheme skips the expensive color network for points that barely contribute, and a compute-in-memory (CIM) architecture with careful data mapping and reuse avoids hash-table conflicts. Together they deliver up to 9.55 times speedup over prior NeRF accelerators and 69.75 times over a mobile GPU, with only 0.1 quality loss.

Why this problem is worth solving

Neural radiance fields changed how photorealistic 3D content can be created, but their cost keeps them off the devices that would most benefit.

A Neural Radiance Field (NeRF) reconstructs a 3D scene from a limited set of 2D photographs by training a neural network to represent the scene implicitly: given a 3D position and a viewing direction, the network returns how much light-blocking material is there (the density) and what color it emits. To render a new view, NeRF works pixel by pixel. For each pixel it casts a ray from the camera into the scene and samples many 3D points along that ray. Each sampled point’s coordinate is encoded into a feature vector — in the leading fast model, Instant-NGP, by looking it up in large multi-resolution hash tables and interpolating — and that feature is fed through two small multilayer perceptrons (MLPs): a density network and a color network. Finally, volume rendering composites the colors of all the points along the ray, weighting each by how much it actually contributes given the densities in front of it, to produce the pixel’s final color. This implicit, learned representation yields strikingly realistic novel views, which is why NeRF is central to view synthesis, 3D reconstruction, and related applications.

The difficulty is that this pipeline is enormously expensive per pixel, and existing accelerators leave most of the waste in place.

The per-pixel cost compounds: many sample points per ray, a hash-table lookup and interpolation at each point (the tables can be tens of megabytes), and two neural networks run for every point, all before compositing. The hash-table accesses are irregular and memory-heavy, and the total computation is large, so even Instant-NGP renders slowly on high-end GPUs, far short of the real-time, low-power budgets of practical devices, which frequently must operate under about thirty watts. What makes this frustrating is how much of the work is unnecessary, and prior accelerators miss it in four ways. First, they treat every pixel the same, sampling a fixed number of points, even though pixels differ enormously in rendering difficulty — a flat, empty region needs far fewer points than an intricate one. Second, neighboring rays sample points in the same or adjacent regions, so there is spatial locality and data reuse to exploit, which existing methods do not. Third, the two networks are lopsided: the density network is only about eight percent of the neural computation while the color network is about ninety-two percent, yet both are run fully at every point. Fourth, when neighboring rays are processed together, their hash-table lookups collide, and in a compute-in-memory chip the way tables are laid out can force those accesses to serialize. The result is a renderer doing far more work than the final image needs.

What people did before, and where it broke

Run NeRF on high-end GPUs. Use a powerful desktop GPU to render, accepting the power draw. Where it broke. Even fast models like Instant-NGP render slowly on such GPUs, and their power consumption is far beyond the roughly thirty-watt budget of practical, mobile, or embedded devices.
Accelerate a single stage in isolation. Build hardware that speeds up the encoding, or the neural networks, or the compositing, one at a time. Where it broke. The stages interact, and optimizing one leaves the others as bottlenecks, so the end-to-end gain is limited.
Sample a fixed number of points per pixel. Apply the same sampling budget to every ray regardless of the scene. Where it broke. Easy pixels get far more points than they need while the budget ignores that difficulty varies, wasting memory accesses and computation on points that barely affect the image.
Run both networks fully at every point. Execute the density and color MLPs for every sampled point. Where it broke. The color network is about ninety-two percent of the neural cost, and many points contribute almost nothing to the final color, so running it everywhere is hugely wasteful.
The gap. No design spent computation adaptively across pixels, skipped the expensive color work where it did not matter, reused shared data, and laid out the hash tables to avoid conflicts, all together. Providing that co-design is what ASDR does.

How ASDR works

From uniform, single-stage NeRF to adaptive, reuse-aware co-design

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Spend computation only where the image is hard, and skip what barely shows. ASDR’s guiding principle is that NeRF does far more work than the picture requires, so it targets the waste directly: it senses each pixel’s rendering difficulty online and samples fewer points where the scene is simple, and it recognizes that the color network — about ninety-two percent of the neural cost — is run even for points whose contribution to the final pixel is negligible, so by decoupling density from color and approximating the volume rendering, it can skip the expensive color computation where a point hardly matters. Detecting the parts of a neural workload that contribute little to the result and omitting them, concentrating effort on what actually shapes the output, is the same move the weak-attention-pruning work covered elsewhere on this site makes, identifying and discarding the computations whose influence on the final result is small so that the accelerator spends its budget only on the work that changes the answer.

Do the neural math inside the memory that holds it. NeRF’s cost is not only arithmetic but the constant movement of weights and features between memory and compute; ASDR addresses this by running the networks’ matrix-vector multiplications directly inside a ReRAM compute-in-memory array, where each memory cell participates in the multiplication in place, so the weights never have to be hauled to a separate processing unit. This slashes the data movement and power that a conventional accelerator spends feeding the MLPs. Performing a model’s linear algebra within the memory device itself, so computation happens where the data already lives instead of shuttling it back and forth, is the same design the in-memory-accelerator work covered elsewhere on this site is built on, turning the memory array into the compute engine so that the energy and time of moving operands are largely eliminated.

Lay the data out so the parallel hardware does not trip over itself. Compute-in-memory only pays off if the accesses it needs can proceed in parallel, but NeRF’s multi-resolution hash tables, packed into the crossbars, can force lookups at different resolutions to serialize when neighboring rays request them at once, and neighboring rays’ near-camera points collide on the same table entries. ASDR co-designs a data mapping and reuse scheme that arranges the tables to minimize these conflicts and reuses the shared points, so the crossbar’s parallelism is actually usable. Carefully arranging where data sits across the memory-compute fabric to avoid access conflicts and exploit locality, so the hardware’s parallel resources are not stalled fighting over the same banks, is the same discipline the near-data-processing data-access work covered elsewhere on this site follows, co-optimizing data placement and access so that the processing elements stay busy rather than serializing on contended memory.

Following a frame through ASDR

Rendering a NeRF frame with adaptive sampling and CIM

A trace of ASDR rendering an image, following the paper's algorithm-architecture co-design.

Step 1 of 5

Cast rays and gauge pixel difficulty

For the target view, ASDR casts a ray per pixel and senses each pixel's rendering difficulty online, deciding how many 3D points that ray needs — many for intricate regions, few for flat, empty ones.

Adaptive sampling replaces a fixed per-pixel budget.

Why it works

Adaptive sampling matches effort to difficulty. Because pixels vary hugely in how many points they need → sensing difficulty and sampling fewer points on easy pixels → so memory accesses and computation are cut without hurting the image where detail is sparse.
Skipping cheap color removes the dominant cost. Because the color network is about ninety-two percent of the neural work and many points barely contribute → decoupling density from color and approximating volume rendering skips color where it does not matter → so the biggest computational burden is slashed with negligible quality loss.
Compute-in-memory kills weight movement. Because the MLPs’ matrix-vector multiplications run inside the ReRAM array → the weights never move to a separate compute unit → so the data-movement energy and latency that dominate conventional inference are largely eliminated.
Good data mapping unlocks the crossbar’s parallelism. Because multi-resolution tables packed in one crossbar would serialize concurrent lookups → mapping them to minimize conflicts and reusing shared points → so the CIM parallelism is actually usable instead of stalling on contention.
End-to-end co-design compounds the gains. Because the four wastes interact across stages → attacking sampling, color cost, data movement, and access conflicts together → so the speedups multiply rather than being capped by whichever bottleneck was left unaddressed.

What it costs (trade-offs and scope)

It requires a specialized ReRAM CIM accelerator. ASDR is an algorithm-architecture co-design built on a ReRAM compute-in-memory chip with custom data mapping. It is not a software optimization for existing GPUs, so realizing its gains needs the hardware, and ReRAM CIM has its own maturity and precision considerations.
The optimizations are approximations, with a small quality cost. Adaptive sampling and skipping the color network trade a little fidelity for speed; the paper reports only 0.1 loss in the standard image-quality metric (PSNR), which is negligible, but it is not lossless, and aggressive settings on very challenging scenes could show more.
It is specialized to NeRF-style rendering. The design exploits the specific structure of neural radiance fields — ray sampling, hash encoding, density-and-color MLPs, volume rendering. Other neural graphics methods (for example newer point- or splat-based ones) have different structure and would not benefit directly.
Difficulty sensing and mapping add their own logic. Judging pixel difficulty online and arranging conflict-free data mappings require hardware and control that cost some area and complexity; the benefit is net positive but the mechanisms are part of the design, not free.
The headline speedups are against particular baselines. The up to 9.55 times over state-of-the-art NeRF accelerators and 69.75 times over a mobile GPU (and the roughly twenty-times figures the paper reports in some comparisons) depend on those specific baselines, scenes, and configurations; other comparisons would shift the numbers.
Gains depend on scene and view content. The savings come from difficulty variation, spatial locality, and low-contribution points; scenes that are uniformly complex, or views with little reuse, would offer less headroom than the favorable cases.

The evaluation

Ways to run NeRF neural rendering

Property	Mobile / high-end GPU	Single-stage NeRF accelerator
Sampling per pixel
Color network cost		Run fully everywhere
Data movement for weights
Hash-table access conflicts	Not addressed
Rendering speed and quality

Ways to run Neural Radiance Field rendering, as compared in the paper. NeRF colors each pixel by casting a ray, sampling many 3D points, looking each up in large multi-resolution hash tables, running a density and a color MLP per point, and compositing via volume rendering — so it is slow and power-hungry, missing real-time, sub-thirty-watt targets even with a fast model like Instant-NGP. GPUs render it slowly at high power, and prior NeRF accelerators optimize a single stage while sampling every pixel uniformly and running both networks fully at every point, leaving four wastes unaddressed: pixels differ in rendering difficulty, neighboring rays share data, the color network is about ninety-two percent of the neural cost while the density network is about eight, and hash-table lookups collide and can serialize in a compute-in-memory crossbar. ASDR is an algorithm-architecture co-design that attacks all four: adaptive sampling senses each pixel's difficulty online and shoots fewer points at easy pixels while reusing shared points across neighboring rays; an MLP-decoupling scheme approximates the volume rendering to skip the expensive color network for points that barely contribute; a ReRAM compute-in-memory architecture runs the networks' matrix-vector multiplications inside the memory to eliminate weight movement; and a data mapping and reuse microarchitecture arranges the multi-resolution tables to minimize access conflicts. Together these deliver up to 9.55 times speedup over state-of-the-art NeRF accelerators and 69.75 times over a Xavier NX mobile GPU, with only 0.1 loss in PSNR (a negligible drop in image quality).Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

ASDR is a thorough, well-motivated accelerator paper whose strength is that it refuses to treat neural rendering as a monolith and instead finds waste in every stage. Neural radiance fields produce gorgeous novel views but at a cost that has kept them off real-time, low-power devices: to color one pixel, NeRF casts a ray, samples many 3D points, does a large hash-table lookup and two small neural networks at each point, and composites the result. The paper’s insight is that most of this work does not change the final image, and that prior accelerators miss it by optimizing one stage and treating every pixel and point identically. It attacks four specific wastes: pixels vary in difficulty, so it samples adaptively; neighboring rays share points, so it reuses them; the color network is about ninety-two percent of the neural cost while contributing little at most points, so it decouples density from color and skips cheap color via a volume-rendering approximation; and hash-table accesses conflict, so it maps the tables across a ReRAM compute-in-memory fabric to avoid serialization while running the networks inside memory to kill weight movement. The payoff is large — up to 9.55 times over prior NeRF accelerators and 69.75 times over a mobile GPU — at only 0.1 PSNR loss, essentially free quality-wise. Read the scope carefully. This is a specialized ReRAM CIM design, not a GPU software trick, so the gains require the hardware and inherit ReRAM’s maturity and precision constraints; the optimizations are approximations, negligibly lossy here but not lossless; the approach is tailored to NeRF’s specific pipeline and would not transfer directly to differently-structured neural graphics; and the speedups are against particular baselines and favorable scenes with difficulty variation and reuse to exploit. The durable idea is that accelerating a neural pipeline is often less about raw compute than about noticing how much of it is unnecessary — that effort should scale with how hard each output is, that the dominant sub-computation should be skipped where it does not matter, and that on compute-in-memory hardware the data layout is as decisive as the arithmetic.

Background

Why neural 3D representations keep pushing hardware co-design

It helps to see ASDR within the broader push to make neural representations of 3D scenes fast and cheap enough for real use, which has become a rich driver of hardware and algorithm co-design. Neural radiance fields and their relatives replace explicit geometry with a learned function evaluated by sampling, which is wonderfully general but turns rendering into a heavy, sampling-and-inference-bound workload whose naive execution wastes enormous effort. The voxel-caching work for 3D occupancy mapping covered elsewhere on this site reflects the same frontier from the autonomous-systems side, accelerating a neural 3D scene representation by exploiting the structure and locality of the volumetric data so that real-time, resource-constrained operation becomes feasible. The common thread is that neural 3D methods are computationally lavish by default, and making them practical on real devices depends on recognizing where their work is redundant — repeated sampling, low-contribution points, reusable neighbors — and on matching the computation to hardware that minimizes data movement. As neural rendering spreads into interactive graphics, robotics, and augmented reality, the pattern ASDR exemplifies, of co-designing adaptive, reuse-aware algorithms with memory-centric accelerators, is likely to be how these representations are made to run within the tight time and power budgets those settings impose.

If you remember only three things

1. A Neural Radiance Field (NeRF) renders a photorealistic 3D scene from a neural network, but it is slow and power-hungry: to color each pixel it casts a ray, samples many 3D points along the ray, and at each point looks up features in large multi-resolution hash tables and runs two small neural networks (a density MLP and a color MLP), then composites everything via volume rendering. Even the fast leading model, Instant-NGP, renders slowly on high-end GPUs and cannot meet the real-time, low-power (often under about thirty watts) budgets of practical devices. Prior accelerators optimize a single stage and treat every pixel and point the same, missing four wastes: pixels differ enormously in rendering difficulty, neighboring rays share sampled points (spatial locality), the two networks are lopsided (the density MLP is about eight percent of the neural cost and the color MLP about ninety-two percent) yet both run fully at every point, and the hash-table accesses collide and can serialize on a compute-in-memory crossbar. The renderer does far more work than the final image needs.
2. ASDR is an algorithm-architecture co-design that spends effort only where it matters. At the algorithm level: adaptive sampling senses each pixel's rendering difficulty online and shoots fewer points at easy pixels (reducing memory access and computation), and reuses points shared across neighboring rays; and an MLP-decoupling scheme approximates the volume rendering to run density first and skip or approximate the expensive color network for points whose contribution to the pixel is negligible — the key lever, since the color MLP is about ninety-two percent of the neural cost. At the architecture level: a ReRAM compute-in-memory (CIM) design runs the networks' matrix-vector multiplications inside the memory array, eliminating the movement of weights; and a data mapping and reuse microarchitecture arranges the multi-resolution hash tables across the crossbars to minimize the access conflicts that would otherwise serialize lookups, so the CIM parallelism is usable. The four optimizations attack interacting bottlenecks across the whole pipeline rather than one stage.
3. On graphics rendering tasks, ASDR achieves up to 9.55 times speedup over state-of-the-art NeRF accelerators and 69.75 times over a Xavier NX mobile GPU, with only 0.1 loss in PSNR (a negligible drop in image quality, because the skipped work barely affected the image). Scope: it requires a specialized ReRAM CIM accelerator rather than running on existing GPUs, and inherits ReRAM's maturity and precision constraints; its optimizations are approximations (negligibly lossy here but not lossless); it is tailored to NeRF's specific ray-sampling, hash-encoding, density-and-color-MLP, volume-rendering pipeline and would not transfer directly to differently-structured neural graphics; and the speedups are against particular baselines and favorable scenes with difficulty variation and reuse. The durable lesson is that accelerating a neural pipeline is often less about raw compute than about noticing how much of it is unnecessary — scaling effort to each output's difficulty, skipping the dominant sub-computation where it does not matter, and, on compute-in-memory hardware, treating data layout as as decisive as the arithmetic.

Check yourself

Check your understanding

1. Why is NeRF neural rendering so slow and power-hungry?
2. What four sources of waste does ASDR identify that prior accelerators miss?
3. How does ASDR reduce the neural computation, and why does it barely hurt quality?
4. What role does the compute-in-memory (CIM) architecture play, and what did ASDR achieve?

Sources

The paper is “ASDR: Exploiting Adaptive Sampling and Data Reuse for CIM-based Instant Neural Rendering” by Fangxin Liu, Haomin Li, Bowen Zhu, Zongwu Wang, Zhuoran Song, Haibing Guan, and Li Jiang, published at ASPLOS 2025, DOI 10.1145/3676642.3736117. The authors are at Shanghai Jiao Tong University and the Shanghai Qi Zhi Institute.

This explainer is based on the full text of the paper, obtained from the authors’ arXiv version (arXiv:2508.02304). The problem framing (Neural Radiance Fields reconstructing 3D scenes from 2D images and rendering novel views; the per-pixel pipeline of ray casting, sampling many 3D points, multi-resolution hash-table encoding, density and color MLPs, and volume rendering, with Instant-NGP as the leading fast model; and the resulting inference latency and power that miss real-time, sub-thirty-watt device budgets), the four identified challenges (pixels varying in rendering difficulty under uniform sampling, underutilized spatial locality between neighboring rays, the imbalance where the density MLP is about eight percent and the color MLP about ninety-two percent of the neural FLOPs, and hash-table access conflicts that serialize in a compute-in-memory crossbar), the ASDR design (a pixel-difficulty-aware adaptive sampling strategy that reduces sampled points and reuses shared points; an MLP-decoupling scheme based on volume-rendering approximation that skips the expensive color MLP for low-contribution points; a ReRAM compute-in-memory architecture running matrix-vector multiplications in memory; and a data mapping and reuse microarchitecture that reduces hash-table access conflicts), and the results (up to 9.55 times speedup over state-of-the-art NeRF accelerators and 69.75 times over a Xavier NX GPU with only 0.1 PSNR loss, and roughly twenty-times speedups reported in some comparisons) are all drawn directly from the paper. Background explanations of NeRF, multilayer perceptrons, hash encoding, volume rendering, computing-in-memory, ReRAM crossbars, and PSNR are standard and provided to make the paper’s contribution understandable.