Existing 3D relighting methods operate through either explicit material decomposition, diffusion-based view-space generation, or a combination of both, requiring full recomputation for each new lighting condition. We observe that recent latent scene representations, which encode multi-view images into a set of compact tokens with no fixed physical semantics, open up a novel design space for relighting. We present LumiTokens, a framework that formulates 3D relighting as a direct transformation on latent scene tokens, without explicit 3D representations, rendering equations, or physics-based decomposition. Our model introduces a Scene Token Editor that processes scene tokens jointly with light-ray tokens through self-attention, producing updated tokens that can be decoded into multi-view-consistent relit images. To support diverse lighting types through a unified interface, all lighting signals, including environment maps, point lights, and area lights, are parameterized as Plucker ray tokens, enabling native 3D user interaction with a representation that carries no explicit spatial structure. Crucially, this design supports progressive relighting: because the editor's output remains in the same latent space as its input, a user can incrementally build up illumination one light source at a time, with each edit composing in token space. Experiments demonstrate that LumiTokens achieves comparable or superior relighting quality to other methods and supports progressive, composable lighting edits. Project page: https://neu-vi.github.io/LumiTokens/
Figures & tables
Figure 1: Given multi-view input images (left), LumiTokens relights the scene under diverse lighting types. Top: an object relit under an environment map and a point light, rendered from novel views. The red and blue dots indicate the rendering views, yellow dots show the position of the point light sources in 3D, and the cyan rectangle illustrates an area light. Middle: relighting generalizes to multi-object scenes with inter-object shadows and reflections. Bottom: progressive relighting, where the user incrementally adds light sources to edit the lighting condition.
Figure 2: Comparison of relighting paradigms . (a) Inverse rendering recovers explicit geometry and BRDF properties via per-scene optimization, limited by the expressiveness of the material model. (b) Diffusion-based methods use powerful generative priors to synthesize relit views, without a shared scene representation to anchor cross-view consistency or persist lighting edits. (c) Hybrid methods combine feed-forward geometry encoding with generative appearance synthesis, inheriting limitations from both paradigms. (d) LumiTokens (Ours) performs relighting entirely in latent token space: scene tokens are edited via self-attention with light-ray tokens and decoded into relit views, supporting progressive, incremental lighting edits.
Figure 3: Overview of the LumiTokens architecture. An encoder maps posed multi-view images, together with learnable empty tokens z0 , into unstructured latent scene tokens z . The scene token editor transforms these tokens conditioned on target lighting: all light types (environment maps, point lights, area lights) are discretized into Plücker ray tokens and processed jointly with the scene tokens through self-attention, producing relit scene tokens z′ . A decoder with a DPT head then renders z′ into novel views given target camera Plücker rays. Because the editor’s output remains in the same latent space as its input, the architecture naturally supports progressive relighting (dashed arrow): multiple lighting edits chain directly in token space without decoding between steps.
Input
GT
Ours
Neural Gaffer
Diffusion Renderer
LightSwitch
view1
view2
view1
view2
Figure 4: Qualitative comparison of multi-view relighting. Columns correspond to methods and rows show two viewpoints for each of two scenes. Our method preserves appearance and lighting effects consistently across views, while baselines show artifacts such as residual source illumination and inconsistent highlights.
Method
Image-Level Rescaling (ILR)
Scene-Level Rescaling (SLR)
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
LightSwitch [ 23 ]
21.22
0.868
0.105
21.19
0.868
0.131
Neural Gaffer [ 14 ]
28.40
0.947
0.030
28.23
0.955
0.037
DiffusionRenderer [ 20 ]
26.39
0.951
0.038
26.24
0.923
0.086
Ours
30.48
0.954
0.045
30.36
0.917
0.086
Table 1: Multi-view relighting accuracy. ILR: image-level rescaling (per-image scale alignment). SLR: scene-level rescaling (single scale across all views per object).
Method
#Input
Synthetic
Objects-with-Lighting
Stanford-ORB
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
LightSwitch [ 23 ]
16
21.61
0.86
0.13
25.43
0.84
0.30
32.02
0.98
0.03
NVDiffrecMC [ 8 ]
50
22.95
0.86
0.10
20.24
0.73
0.40
31.60
0.97
0.04
TensoIR [ 15 ]
50
26.17
0.92
0.07
26.12
0.77
0.38
25.27
0.94
0.06
Ours
8
27.76
0.91
0.06
26.76
0.92
0.36
31.01
0.97
0.03
Table 2: Novel-view relighting accuracy. #Input denotes how many source views each method uses; our method requires significantly fewer.
GT
Ours
Light Switch
TensoIR
NVDiff recMC
GT
Ours
Light Switch
TensoIR
NVDiff recMC
Figure 5: Visual comparison of novel-view relighting.
Encoder
Regularization
PSNR ↑
SSIM ↑
LPIPS ↓
Frozen
None
25.06
0.910
0.073
Unfrozen
None
26.12
0.927
0.046
Unfrozen
Explicit (predict albedo)
27.09
0.934
0.057
Unfrozen
Implicit (invariance loss)
27.05
0.952
0.045
Table 3: Ablation study on encoder training and lighting invariance regularization for novel-view relighting.
Figure 6: Progressive relighting ablation. Top plots: PSNR and SSIM over multiple editing steps, comparing token-space editing (ours) with pixel-space decode-and-re-encode. Token-space editing preserves fidelity over early steps, while pixel-space editing degrades from the first step onward. Bottom: visual comparison across editing steps showing the same trend.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Light tokenization: different light sources (environment map, point light, area light) are converted into a unified ray-token representation.
Method
End-to-end
Relight Only
Peak GPU Mem.
CPU Memory
Ours
3.79
3.77
6108
3015
NeuralGaffer
16.85
16.85
24064
5622
DiffusionRenderer
22.81
14.36
28755
7460
LightSwitch
532.33
127.10
22696
2882
TensoIR
3952.91
294.40
10979
5038
NVDiffrecMC
1691.75
225.41
6312
2258
Appendix
Table 4: Runtime and memory comparison. End-to-end inference and relight-only times (sec), peak GPU memory (MB), and CPU Memory (MB). Evaluated on a NVIDIA A100 GPU.
Figure 8: Multi-view relighting comparisons. Per-view optimal tone scales (shown as sliders above each image) reveal that LumiToken exhibits more stable scaling across views than baselines, demonstrating better cross-view consistency.
Figure 9: Multi-view relighting comparisons. Per-view optimal tone scales (shown as sliders above each image) reveal that LumiToken exhibits more stable scaling across views than baselines, demonstrating better cross-view consistency.
Figure 10: Qualitative relighting results on synthetic table-top scenes.
Figure 11: Qualitative relighting results on synthetic table-top scenes (continued).
GT
Ours
Light Switch
TensoIR
NVDiff recMC
Appendix
Figure 12: Visual comparison of novel-view relighting. The first two rows are synthetic experiments; the last two rows are real-world examples from Stanford-ORB.
This paper presents a method for image relighting that enables precise and continuous control over multiple illumination attributes in a photograph. We formulate relighting as a conditional image generation task and introduce attribute tokens to encode distinct lighting factors such as intensity, color, ambient illumination, diffuse level, and 3D light positions. The model is trained on a large-scale synthetic dataset with ground-truth lighting annotations, supplemented by a small set of real captures to enhance realism and generalization. We validate our approach across a variety of relighting tasks, including controlling in-scene lighting fixtures and editing environment illumination using virtual light sources, on synthetic and real images. Our method achieves state-of-the-art quantitative and qualitative performance compared to prior work. Remarkably, without explicit inverse rendering supervision, the model exhibits an inherent understanding of how light interacts with scene geometry, occlusion, and materials, yielding convincing lighting effects even in traditionally challenging scenarios such as placing lights within objects or relighting transparent materials plausibly. Project page: vrroom.github.io/tokenlight/
Indoor scene relighting demands photorealism, precise spatial control, and strict multi-view consistency. While diffusion-based image editing models enable semantic lighting manipulation via text prompts, enforcing exact 3D light placement often disrupts their generative priors. We propose Lume-Palette, a progressive framework that leverages semantic lighting priors for spatially controllable multi-view indoor relighting. The approach decouples relighting into two stages: (1) illumination distillation, which extracts canonical illumination palettes from a pretrained diffusion model to preserve realistic material-light interactions, and (2) illumination casting, which explicitly maps target spatial lighting conditions defined from coarse 3D geometry. To efficiently handle dense multi-view and multi-modal inputs, we introduce an asymmetric multi-view conditioning strategy that selectively injects essential spatial context. Experiments on diverse synthetic scenes and real-world scenes demonstrate that Lume-Palette produces photorealistic, spatially controllable, and multi-view consistent relighting results. Project Page: https://cjeen.github.io/lumepalette
Chenjian Gao, Linning Xu, Tianfan Xue
Multimedia Laboratory, The Chinese University of Hong Kong · Shanghai AI Laboratory · CPII under InnoHK
We propose a 3D latent representation that jointly models object geometry and view-dependent appearance. Most prior works focus on either reconstructing 3D geometry or predicting view-independent diffuse appearance, and thus struggle to capture realistic view-dependent effects. Our approach leverages that RGB-depth images provide samples of a surface light field. By encoding random subsamples of this surface light field into a compact set of latent vectors, our model learns to represent both geometry and appearance within a unified 3D latent space. This representation reproduces view-dependent effects such as specular highlights and Fresnel reflections under complex lighting. We further train a latent flow matching model on this representation to learn its distribution conditioned on a single input image, enabling the generation of 3D objects with appearances consistent with the lighting and materials in the input. Experiments show that our approach achieves higher visual quality and better input fidelity than existing methods.