Neural Render Proxies for Interactive and Differentiable Lighting
Authors: Sergio Sancho, Alexander Rath, Marco Manzi, Pascal Chang, Amit H. Bermano, Derek Nowrouzezahrai, Markus Gross, Marios Papas
Organizations: ETH Zurich, Switzerland · DisneyResearch|Studios, Switzerland · Tel Aviv University, Israel · McGill University, Canada · Mila – Quebec AI Institute · CIFAR AI Chair
Within the CG animation production pipeline, the challenges artists tackle in Lighting can be immense. Even minor adjustments require re-rendering massive scenes with slow offline renderers; global illumination has to be sampled and complex shaders have to be evaluated, leading to iteration times of minutes to hours per frame. To accelerate this process, we introduce a novel neural render proxy (NRP) that enables differentiable relighting of static scenes with fixed camera and materials at interactive rates. Our main insight is the decoupling of traditional rendering into path sampling and emission computation. From a single, light-agnostic render pass, we collect light transport data in the form of path samples. This enables rapidly sampling lighting conditions on the fly, and training a scene-specific lightweight neural network that learns how light is transported from any scene location to any image pixel. This approach is compatible with non-differentiable production renderers, induces minimal memory requirements during inference, and scales only with resolution and the number of light sources and parameters, but independently of scene and appearance complexity. Our evaluation demonstrates interactive frame rates for relighting (~30-60 Hz) while closely approximating the visual fidelity of ground-truth path tracing. In addition, the differentiable NRP enables fast, gradient-based inverse workflows, allowing artists to efficiently solve for lighting parameters from intuitive image-space edits or generative targets.
Figures & tables
Figure 1: Our Pipeline . Training (left): We collect lighting agnostic paths from path tracing the scene before generating denoised image with random lighting conditions to serve as training data. Our neural render proxy (NRP; middle) is a lightweight MLP with hashgrid positional encoding that predicts light transport to a sensor pixel from an input light. To reduce training time and avoid spectral bias, we condition the NRP on auxiliary features (albedo, depth and normals) at sensor image coordinates. At inference time (top right), we can relight the scene at interactive rates by inserting virtual light sources into the scene. Since the NRP is also differentiable, we can optimize for lighting parameters (bottom right) using gradient descent, driven by any number of image-space guidance approaches.
Figure 2: Classic path tracing vs. our decoupled approach
Figure 4: Qualitative image-based relighting comparison. Our relighting setting matches that of image-based relighting using generative networks. Two leading approaches with similar input settings (image, auxiliary features, and lighting condition) are presented. Generative models do not preserve scene geometry, do not adhere to material appearance behavior, and produce lighting that is plausible, but is not physically reasonable. Our method produces high-quality and high-fidelity appearance in real-time. Note that the aspect ratio is different since the generative models used consider squared images.
Reconstruction
32 spp
64 spp
128 spp
256 spp
512 spp
Den.
Time (ms)
1.05
1.86
3.48
6.90
13.89
2.60
GPU Memory (GB)
1.22
2.42
4.82
9.63
19.25
—
Inference
16/4
32/4
64/4
128/8
256/8
512/8
Time (ms)
0.83
0.98
1.20
3.13
9.13
31.21
Model Size (MB)
5.6
5.6
5.7
6.1
7.4
13.3
Table 1: Reconstruction by spp (top) and inference performance by network size (bottom) for the Kitchen scene ( 885×500 pixels).
Figure 5: Path-based training. PSNR (dB) in tonemapped space, averaged across three scenes. Each bar is a network trained on N images at S spp with NEE (N × S). Blue: increasing dataset size at 1 spp. Orange: fixed budget of 1024 total samples per pixel, redistributed across images. Dashed red line: our method, trained on light-agnostic path data from a 64 spp render. Despite 16 × fewer sampled paths, our approach outperforms all image-based configurations.
Figure 6: Quality metrics vs. samples per pixel. Mean prediction quality for network size 256×8 (blue) and denoised reconstruction from the GatherLight pass (red), averaged across three scenes. Higher spp improves both reconstruction and prediction quality. Notably, at low spp the learned model can outperform the reconstruction it was trained on. Unless stated otherwise, we use 512 spp for training.
Configuration
SMAPE ↓
PSNR ↑
SSIM ↑
FLIP ↓
Mitsuba
None
0.0533
35.2913
0.9488
0.1546
Aux.
0.0290
39.9576
0.9848
0.1017
Aux. + Den.
0.0296
39.9304
0.9851
0.1084
Aux. + Enc.
0.0566
34.7786
0.8570
0.1232
Aux. + Enc. + Den.
0.0263
41.0686
0.9885
0.1014
Table 2: Component ablation study (averaged per scene group)
Figure 7: Effect of network size on prediction quality (SMAPE). Left: average over the three Mitsuba scenes; right: production scene ( Cave ). The dashed line indicates the Monte Carlo reconstruction at the same sample budget. Larger networks consistently improve quality, with diminishing returns beyond 256×8 .
Figure 8: Inverse optimization comparison. A light source is optimized from a random initial position to match a target appearance. Error maps (middle) show the relative squared error of path-traced re-renders against the target. Our NRP approach converges to lower parameter error in less time than traditional (Mitsuba) and neural (ZeroGrads) baselines (right). This comparison serves as validation: Baselines operate at low spp for tractable runtimes, and their accuracy could improve with higher sample counts at proportionally longer optimization times.
1 Light
3 Lights
5 Lights
20 Lights
Pix. frac.
PSNR
Time
PSNR
Time
PSNR
Time
PSNR
Time
Initial
14.3
—
14.9
—
15.8
—
19.2
—
100%
39.5
21s
29.6
63s
32.1
138s
32.4
551s
50%
39.4
11s
29.9
33s
32.0
55s
32.6
286s
25%
39.4
5.7s
29.9
17s
32.2
28s
32.5
149s
5%
39.1
1.7s
29.8
4.8s
32.4
7.9s
32.6
31s
Table 3: Cache optimization ablation: Mini-batch SGD. PSNR (dB, higher is better) is computed on exposure-normalized Reinhard-tonemapped images re-rendered with Mitsuba, averaged across three scenes. Time is for 500 optimization iterations.
Figure 9: Multi-view lighting edits. By training an NRP for each view, our method enables interactive and consistent lighting edits across multiple views simultaneously.
Figure 12: Our formulation extends to dense participating media (left) and alternative light parameterizations, such as quad lights with adjustable size, position, and orientation (right).
Figure 13: Limitations in narrow regions. When queried in under-sampled spaces, our NRP can struggle to accurately model occlusions. As shown here, placing the light source inside the narrow cavity of the lamp results in incorrect light propagation.
High-fidelity 3D assets exhibit intriguing global illumination effects like subsurface scattering, glossy interreflections, and fine-scale fiber scatterings, which often involve long scattering paths that are expensive to simulate. We introduce 8D neural assets (8DNA) to pre-bake these light transport effects into neural representations. Unlike prior methods that assume far-field lighting and precompute light transport into 6D functions, 8DNA learns the full 8D light transport, enabling accurate rendering under near-field illumination. Our training leverages a distribution-learning formulation that learns light transport from forward path-traced samples, which produces less optimization variance with lower training budget than the prior regression-based approaches. Experiments show our 8DNA rendering closely matches path-traced results under various scene configurations, yet it achieves improved variance reduction and fast inference speeds on challenging assets.
Liwen Wu, Haolin Lu, Bing Xu +2
University of California San Diego, USA · NVIDIA, USA
Implicit neural representations (INRs) have gained momentum in scientific visualization due to their compactness and scalability to large datasets, making them well suited for integration with direct volume rendering (DVR). However, real-time volume rendering of INR with advanced illumination effects, such as shadows, remains computationally expensive, as evaluating shadow terms via ray marching is costly. Alternatively, precomputing and storing shadows for many lighting directions is prohibitive in both memory and storage. To address this, we introduce a diffusion-based shadow caching framework that compresses a vast set of pre-calculated shadow INRs into a single diffusion model. Rather than focusing on generalizing to unseen directions, our method effectively memorizes and reconstructs a dense set of pre-trained lighting conditions on the fly. We first encode a collection of shadow coefficient volumes as shadow INRs, and then train a diffusion model conditioned on lighting direction to predict the corresponding shadow INR weights at inference time. This design integrates directly with standard INR renderers without additional runtime sampling. Experiments show that our approach achieves faster rendering than traditional methods while bypassing the massive storage bloat of independent INRs, producing shadows that closely match most of the reference results.
Kai-Chen Tung, Qi Wu, David Bauer +3
University of California, Davis, Davis, CA, United States · NVIDIA Spatial Intelligence Lab, Santa Clara, CA, United States · Argonne National Laboratory, Lemont, IL, United States
Recent advances in neural scene representations enable photorealistic novel-view synthesis, yet most methods remain tightly coupled to a single rendering paradigm, limiting their versatility and integration with conventional graphics workflows. We introduce Floating Radiance Networks (FlaRe), a neural scene representation combining explicit ray-traceable geometry with continuous neural radiance functions. A scene is represented by floating planar generalized Gaussian primitives, each carrying a compact latent descriptor of a local radiance field. A lightweight decoder shared across the scene maps this descriptor, local surface coordinates, and viewing direction to color and opacity. This formulation preserves the expressiveness of neural fields while providing an explicitly addressable structure that can be efficiently queried and manipulated. Hardware-accelerated primitive intersections enable interactive rendering and recursive ray-tracing, including reflections, refractions, transparency, and shadows. The same representation further supports primitive-level deformation, mesh extraction, and appearance stylization directly in its learned descriptor space. Experiments across standard reconstruction benchmarks demonstrate competitive rendering quality while using a compact set of primitives. Together, these results establish FlaRe as a versatile representation that brings high-fidelity neural rendering, ray-tracing, geometric manipulation, and appearance editing into a unified scene model. Source code is available online. Source code can be found at: https://github.com/KByrski/FlaRe
Krzysztof Byrski, Rafał Tobiasz, Grzegorz Wilczyński +5
Jagiellonian University, Faculty of Mathematics and Computer Science · IDEAS Research Institute · Poznań University of Technology, Institute of Robotics and Machine Intelligence