RehearsalNeRF: Decoupling Intrinsic Neural Fields of Dynamic Illuminations for Scene Editing

Changyeon Won; Hyunjun Jung; Jungu Cho; Seonmi Park; Chi-Hoon Lee; Hae-Gon Jeon

cs.CVMar 30, 2026

RehearsalNeRF: Decoupling Intrinsic Neural Fields of Dynamic Illuminations for Scene Editing

Authors: Changyeon Won, Hyunjun Jung, Jungu Cho, Seonmi Park, Chi-Hoon Lee, Hae-Gon Jeon

Organizations: Department of AI Convergence, GIST, Gwangju, Korea. · AI R&D Division, CJ Corporation, Seoul, Korea. · Department of Artificial Intelligence, Yonsei University, Seoul, Korea.

Abstract

Although there has been significant progress in neural radiance fields, an issue on dynamic illumination changes still remains unsolved. Different from relevant works that parameterize time-variant/-invariant components in scenes, subjects' radiance is highly entangled with their own emitted radiance and lighting colors in spatio-temporal domain. In this paper, we present a new effective method to learn disentangled neural fields under the severe illumination changes, named RehearsalNeRF. Our key idea is to leverage scenes captured under stable lighting like rehearsal stages, easily taken before dynamic illumination occurs, to enforce geometric consistency between the different lighting conditions. In particular, RehearsalNeRF employs a learnable vector for lighting effects which represents illumination colors in a temporal dimension and is used to disentangle projected light colors from scene radiance. Furthermore, our RehearsalNeRF is also able to reconstruct the neural fields of dynamic objects by simply adopting off-the-shelf interactive masks. To decouple the dynamic objects, we propose a new regularization leveraging optical flow, which provides coarse supervision for the color disentanglement. We demonstrate the effectiveness of RehearsalNeRF by showing robust performances on novel view synthesis and scene editing under dynamic illumination conditions. Our source code and video datasets will be publicly available.

Figures & tables

Figure 1: RehearsalNeRF jointly optimizes five neural fields for dynamic lighting and static/moving subjects in a training step. Each field cannot only be rendered independently, but can also be composited at once to represent a whole scene. Compared to the baseline ( Wu et al., 2022 ) , the details from RehearsalNeRF are distinguishable and the rendered colors are well decoupled from the scene lights. We provide a variety of applications for video editing, such as controlling lighting while stopping motion and vice versa.

Figure 2: An overview of RehearsalNeRF in training phase. Our RehearsalNeRF consists of five neural fields for rendering static, dynamic objects and illuminations. Each field takes location ( $\mathbf{x}$ ), viewing direction ( $\mathbf{d}$ ) and time-stamp ( $\tau$ ) for the same sample point as input, except the time-stamp for the static fields. They are used to infer the radiance. Particularly, the illumination field predicts the probability $P_{H}$ of the illumination vector $v_{h}$ corresponding to $\tau$ . To decouple static/dynamic objects and illumination components of the main stage video well, two additional regularizations, $L_{reh}$ and $L_{dyn}$ , using the rehearsal prior, are designed.

Figure 3: Visualizations of the procedure how to work the illumination vector. (a) The initialization of the learnable illumination vector is done by sampling the hue channel of the dynamic lighting from the difference between the rehearsal and the main stage video. (b) Back propagation from the rendered hue channel is achieved by computing a weighted sum of the probability $P_{H}$ and the slice of the $v_{h}$ for time $\tau$ , which finally optimize the hue channel of the dynamic lighting effects.

Figure 4: Qualitative comparisons on our real-world dataset. The goal of scene editing is to render the red-colored illumination. Only our method produces the high-quality edited scenes through the successful dynamic illumination decomposition.

Method	Original Dataset				Subsampled Dataset ( $5\times$ Speed)
	Full Image		Dynamic Objects		Full Image		Dynamic Objects
	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$
D 2 NeRF ( Wu et al., 2022 )	29.776	0.857	22.962	0.978	28.274	0.853	21.899	0.976
K-Planes ( Fridovich-Keil et al., 2023 )	28.113	0.867	25.122	0.984	27.849	0.854	25.784	0.986
TensoIR PBR ( Jin et al., 2023 )	18.681	0.707	16.934	0.972	16.955	0.708	15.347	0.971
TensoIR VR ( Jin et al., 2023 )	28.663	0.865	21.918	0.978	28.389	0.848	23.238	0.981

Table 1: Quantitative comparison of novel view synthesis performance on our real-world dataset. We compare results from the original dataset and the temporally subsampled version (taking every

5^{th}

frame) to evaluate robustness against rapid scene dynamics.

Figure 5: Qualitative comparison of novel view synthesis performance for dynamic objects on our real-world dataset. The second row presents error map between rendered image and GT. Our model consistently achieves superior quality in dynamic regions.

Figure 6: Qualitative comparison of NVS performance for dynamic objects on the temporally subsampled real-world dataset. Due to rapid motion, baselines struggle to reconstruct the fine details of dynamic objects.

Method	Novel View Synthesis		Scene Editing
Method	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$
D 2 NeRF ( Wu et al., 2022 )	27.234	0.872	25.201	0.864
K-Planes ( Fridovich-Keil et al., 2023 )	27.495	0.864	25.783	0.853
TensoIR PBR ( Jin et al., 2023 )	15.056	0.668	15.153	0.671
TensoIR VR ( Jin et al., 2023 )	28.769	0.885	27.762	0.883
Ours	30.117	0.883	29.186	0.878

Table 2: Quantitative evaluation for novel view synthesis and scene editing on our synthetic dataset with state-of-the-art methods. To evaluate scene editing performance, we construct a synthetic video dataset with time-shifted lighting effects relative to the original video used as the learning objective.

Figure 7: Qualitative comparison for scene editing on the synthetic dataset. The comparison methods fail to handle the harsh illuminations, mistakenly optimizing dynamic lights as dynamic objects.

Figure 8: Examples from our real-world (a) and synthetic datasets (b,c). The synthetic dataset allows us to test ours and the comparison models in severe illumination conditions, including (b) texture degradation and (c) highly saturated regions.

Method	L2 ( $\times 1000$ ) $\downarrow$	L1 ( $\times 1000$ ) $\downarrow$	PSNR $\uparrow$
K-Planes	14.897	81.276	19.523
D 2 NeRF	3.147	17.811	25.245
Ours	0.398	7.630	34.972

Table 3: Quantitative comparison of dynamic lighting disentanglement on our synthetic dataset. We report L2 loss, L1 loss, and PSNR.

Figure 9: Qualitative comparison of dynamic lighting disentanglement.

Figure 10: Qualitative results for scene editing. (a) We shift the lighting effects to the former frame (see yellow allows). Note that, we do not modify the intensity of illuminations at this point. (b) We tune $v_{h}$ with the red-colored illumination. (c) An impact on the modified intensity value $i$ . We can infer the brighter and darker illumination by applying the lower and higher value of $i$ , respectively. Plus, we can implement the spatially-variant adjustment.

Figure 11: Handling a large misalignment of cameras between the main stage and the rehearsal stage.

Figure 12: Visualizations of the results on ablation study. We can see that each component of our RehearsalNeRF contributes to the visually pleasing looks. The dynamic object disappearance (yellow arrow), incorrect scene editing (green arrow) and inaccurate shadow (blue arrow) are observed if we do not have either one of the proposed components.

	NVS(Real-world)		NVS(Synthetic)		SE(Synthetic)
	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
Ours	28.250	0.851	30.117	0.883	29.186	0.878
w/o $\mathcal{L}_{dyn}$	27.787	0.859	29.769	0.879	25.948	0.859
w/o $\mathcal{L}_{reh}$	27.845	0.827	25.896	0.852	25.190	0.846
w/o $v_{h}$ hue init.	27.378	0.832	29.439	0.878	28.574	0.872
w/o $\psi_{A}$	28.229	0.845	30.040	0.885	29.104	0.880

Table 4: Ablation study for NVS on both real-world and synthetic dataset, and SE.

Ours		+30 frame shift		+60 frame shift		Only use 1 st frame
PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
28.046	0.883	28.039	0.882	28.069	0.883	28.005	0.881

Table 5: Quantitative result for experiments with time mis-sync rehearsal videos.

Figure 13: Experimental results when the subjects are captured with the bulky LED screen. Adjusting the hyper-parameter $k$ makes it possible to represent complex lighting effects, such as those from an LED screen.

Figure 14: Experimental results with non-synchronized rehearsal video input. The time shift between the rehearsal and main stage videos results in a misalignment of dynamic motions. Using only the first frame of the rehearsal video as the rehearsal prior exacerbates this misalignment. Our RehearsalNeRF remains unaffected by such misalignments.

Figure 15: Results with various sizes of mask. The impact of the mask is marginal and our method robustly produce novel view and edited scene.

Figure 16: Results with various rehearsal prior conditions.

Figure 17: Qualitative results for the generative rehearsal prior. We demonstrate our rehearsal-free pipeline using a generative rehearsal prior. The rehearsal images are generated from a single frame of the input video without any lighting effects. Our model operates effectively with the generated rehearsal prior without performance degradation. While the rehearsal prior is essential to our model, its accuracy does not need to be strict, nor does it have to be captured from real data.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

	Confocal fluorescence microscope	RehearsalNeRF
Illumination	various wavelength LEDs/Lasers	dynamic illuminations at a concert hall
Sensor	multiple detectors	multiple cameras
Subject	live cells and tissues	performers
Output	3D volume rendering of live cells and tissues	radiance fields for components of a scene

Appendix

Table 6: Similarity of input/output configurations between confocal fluorescence microscopy and RehearsalNeRF.

Figure 18: Applying RehearsalNeRF framework to confocal fluorescence microscopy. (a) Each sample type exhibits different activations according to the wavelengths of exposed lights. (b) Through multi-view observation such as light field microscopy, we can generate multi view images which can be used to train RehearsalNeRF. (c) We can render the 3D volume of the subject. Please check our supplementary video.

Figure 19: Examples of main stage video in our dataset. The real-world dataset are sampled in the upper row and the synthetic dataset in the lower row.

Dataset	# of scenes	# of dynamic subjects	# of backgrounds
Nerfies Validation Rig ( Park et al., 2021a )	4	3	3
Neural 3D video ( Li et al., 2022b )	6	1	2
Meet Room ( Li et al., 2022a )	3	3	1
Ours (real-world)	10	4	3
Ours (synthetic)	4	4	4

Appendix

Table 7: Statistic of public datasets for NeRFs with dynamic motion and our dataset

Figure 20: Applications of RehearsalNeRF to video manipulation. (a) We can change the light colors after recording the video. (b) Our RehearsalNeRF produces the natural-looking dance video under artificially stopping the light effects. (c) We show a video with the frozen motion, but the light sources still work. Please check our supplementary video.

Figure 21: Examples of synthetic datasets under various light intensities in dynamic illumination. From left to right, we incrementally increase the intensity of lights. The intensity predominantly used in our main paper corresponds to the ’high intensity’ setting, as shown in the third column.

Figure 22: Quantitative results for the synthetic dataset under the lighting conditions with varying intensities. We evaluate the results across dynamic objects, static objects, and the entire image.

Method	VRAM (MB)	Training Time (sec)	Inference Time (sec)
D 2 NeRF	3,739	16,406	72.10
K-Planes	1,081	10,491	7.93
TensoIR VR	2,449	27,110	3.54
TensoIR PBR	15,641	27,110	66.70
Ours	1,639	32,250	14.80

Appendix

Table 8: Comparison of computational efficiency. We report the peak GPU memory usage during inference, training time, and inference time measured on a single RTX 3090 GPU.

Neural Radiance Fields

Neural radiance fields represent 3D scenes as continuous functions of position and viewing direction to synthesize images from new camera views.

See all 80 papers

Aug 6, 2026

Floating Radiance Networks

Recent advances in neural scene representations enable photorealistic novel-view synthesis, yet most methods remain tightly coupled to a single rendering paradigm, limiting their versatility and integration with conventional graphics workflows. We introduce Floating Radiance Networks (FlaRe), a neural scene representation combining explicit ray-traceable geometry with continuous neural radiance functions. A scene is represented by floating planar generalized Gaussian primitives, each carrying a compact latent descriptor of a local radiance field. A lightweight decoder shared across the scene maps this descriptor, local surface coordinates, and viewing direction to color and opacity. This formulation preserves the expressiveness of neural fields while providing an explicitly addressable structure that can be efficiently queried and manipulated. Hardware-accelerated primitive intersections enable interactive rendering and recursive ray-tracing, including reflections, refractions, transparency, and shadows. The same representation further supports primitive-level deformation, mesh extraction, and appearance stylization directly in its learned descriptor space. Experiments across standard reconstruction benchmarks demonstrate competitive rendering quality while using a compact set of primitives. Together, these results establish FlaRe as a versatile representation that brings high-fidelity neural rendering, ray-tracing, geometric manipulation, and appearance editing into a unified scene model. Source code is available online. Source code can be found at: https://github.com/KByrski/FlaRe

Krzysztof Byrski, Rafał Tobiasz, Grzegorz Wilczyński +5

Jagiellonian University, Faculty of Mathematics and Computer Science · IDEAS Research Institute · Poznań University of Technology, Institute of Robotics and Machine Intelligence

Neural Rendering Novel View Synthesis 3D Scene Representation

Ask PDF

Mar 9, 2026

HDR-NSFF: High Dynamic Range Neural Scene Flow Fields

Radiance of real-world scenes typically spans a much wider dynamic range than what standard cameras can capture. While conventional HDR methods merge alternating-exposure frames, these approaches are inherently constrained to 2D pixel-level alignment, often leading to ghosting artifacts and temporal inconsistency in dynamic scenes. To address these limitations, we present HDR-NSFF, a paradigm shift from 2D-based merging to 4D spatio-temporal modeling. Our framework reconstructs dynamic HDR radiance fields from alternating-exposure monocular videos by representing the scene as a continuous function of space and time, and is compatible with both neural radiance field and 4D Gaussian Splatting (4DGS) based dynamic representations. This unified end-to-end pipeline explicitly models HDR radiance, 3D scene flow, geometry, and tone-mapping, ensuring physical plausibility and global coherence. We further enhance robustness by (i) extending semantic-based optical flow with DINO features to achieve exposure-invariant motion estimation, and (ii) incorporating a generative prior as a regularizer to compensate for limited observation in monocular captures and saturation-induced information loss. To evaluate HDR space-time view synthesis, we present the first real-world HDR-GoPro dataset specifically designed for dynamic HDR scenes. Experiments demonstrate that HDR-NSFF recovers fine radiance details and coherent dynamics even under challenging exposure variations, thereby achieving state-of-the-art performance in novel space-time view synthesis. Project page: https://shin-dong-yeon.github.io/HDR-NSFF/

Shin Dong-Yeon, Kim Jun-Seong, Kwon Byung-Ki +1

School of Electrical Engineering, KAIST · Dept. of Electrical Engineering, POSTECH · Grad. School of Artificial Intelligence, POSTECH +1

HDR Video Reconstruction HDR Imaging Dynamic Scene Reconstruction Dynamic Novel View Synthesis Neural Radiance Fields

Ask PDF

Sep 30, 2026

Reconstructing the Dynamic World: A Representation-Centric View of 4D Scene Reconstruction

4D scene reconstruction aims to recover the evolving geometry, appearance, and motion of dynamic environments from visual observations. Despite substantial progress in neural scene representations, reconstructing dynamic scenes remains challenging due to non-rigid motion, occlusions, temporal inconsistencies, and the trade-offs between reconstruction fidelity and computational efficiency. Recent advances in Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have introduced diverse approaches to representing and reconstructing dynamic scenes, yet their relationships, underlying design choices, and evaluation protocols remain fragmented. In this paper, we present a unified perspective on 4D scene reconstruction, organizing existing methods around their scene representations, temporal modeling strategies, reconstruction pipelines, and optimization objectives. Through this framework, we examine how different design choices affect geometric fidelity, appearance consistency, motion representation, and computational efficiency. We further consolidate commonly used datasets and evaluation metrics, identify limitations in current experimental practices, and discuss open challenges in reconstructing complex, dynamic real-world environments. By connecting methodological developments with their underlying assumptions and evaluation evidence, this work provides a structured foundation for understanding existing approaches and identifying future research directions. An evolving collection of relevant papers and resources is available at https://github.com/ZiyangYan/Awesome-4D-Scene-Reconstruction.

Ziren Gong, Guo Chen, Yongjia Li +13

Department of Computer Science and Engineering, University of Bologna, Italy · Wangxuan Institute of Computer Technology, Peking University, China · Department of Computing, The Hong Kong Polytechnic University, Hong Kong +8

3D Gaussian Splatting 4D Reconstruction Dynamic Scene Reconstruction 3D Scene Representation Neural Radiance Fields

Ask PDF

Abstract

Neural Radiance Fields

Explore similar work

Floating Radiance Networks

HDR-NSFF: High Dynamic Range Neural Scene Flow Fields

Reconstructing the Dynamic World: A Representation-Centric View of 4D Scene Reconstruction