Existing 3D relighting methods operate through either explicit material decomposition, diffusion-based view-space generation, or a combination of both, requiring full recomputation for each new lighting condition. We observe that recent latent scene representations, which encode multi-view images into a set of compact tokens with no fixed physical semantics, open up a novel design space for relighting. We present LumiTokens, a framework that formulates 3D relighting as a direct transformation on latent scene tokens, without explicit 3D representations, rendering equations, or physics-based decomposition. Our model introduces a Scene Token Editor that processes scene tokens jointly with light-ray tokens through self-attention, producing updated tokens that can be decoded into multi-view-consistent relit images. To support diverse lighting types through a unified interface, all lighting signals, including environment maps, point lights, and area lights, are parameterized as Plucker ray tokens, enabling native 3D user interaction with a representation that carries no explicit spatial structure. Crucially, this design supports progressive relighting: because the editor's output remains in the same latent space as its input, a user can incrementally build up illumination one light source at a time, with each edit composing in token space. Experiments demonstrate that LumiTokens achieves comparable or superior relighting quality to other methods and supports progressive, composable lighting edits. Project page: https://neu-vi.github.io/LumiTokens/
Figures & tables
Figure 1: Given multi-view input images (left), LumiTokens relights the scene under diverse lighting types. Top: an object relit under an environment map and a point light, rendered from novel views. The red and blue dots indicate the rendering views, yellow dots show the position of the point light sources in 3D, and the cyan rectangle illustrates an area light. Middle: relighting generalizes to multi-object scenes with inter-object shadows and reflections. Bottom: progressive relighting, where the user incrementally adds light sources to edit the lighting condition.
Figure 2: Comparison of relighting paradigms . (a) Inverse rendering recovers explicit geometry and BRDF properties via per-scene optimization, limited by the expressiveness of the material model. (b) Diffusion-based methods use powerful generative priors to synthesize relit views, without a shared scene representation to anchor cross-view consistency or persist lighting edits. (c) Hybrid methods combine feed-forward geometry encoding with generative appearance synthesis, inheriting limitations from both paradigms. (d) LumiTokens (Ours) performs relighting entirely in latent token space: scene tokens are edited via self-attention with light-ray tokens and decoded into relit views, supporting progressive, incremental lighting edits.
Figure 3: Overview of the LumiTokens architecture. An encoder maps posed multi-view images, together with learnable empty tokens z0 , into unstructured latent scene tokens z . The scene token editor transforms these tokens conditioned on target lighting: all light types (environment maps, point lights, area lights) are discretized into Plücker ray tokens and processed jointly with the scene tokens through self-attention, producing relit scene tokens z′ . A decoder with a DPT head then renders z′ into novel views given target camera Plücker rays. Because the editor’s output remains in the same latent space as its input, the architecture naturally supports progressive relighting (dashed arrow): multiple lighting edits chain directly in token space without decoding between steps.
Input
GT
Ours
Neural Gaffer
Diffusion Renderer
LightSwitch
view1
view2
view1
view2
Figure 4: Qualitative comparison of multi-view relighting. Columns correspond to methods and rows show two viewpoints for each of two scenes. Our method preserves appearance and lighting effects consistently across views, while baselines show artifacts such as residual source illumination and inconsistent highlights.
Method
Image-Level Rescaling (ILR)
Scene-Level Rescaling (SLR)
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
LightSwitch [ 23 ]
21.22
0.868
0.105
21.19
0.868
0.131
Neural Gaffer [ 14 ]
28.40
0.947
0.030
28.23
0.955
0.037
DiffusionRenderer [ 20 ]
26.39
0.951
0.038
26.24
0.923
0.086
Ours
30.48
0.954
0.045
30.36
0.917
0.086
Table 1: Multi-view relighting accuracy. ILR: image-level rescaling (per-image scale alignment). SLR: scene-level rescaling (single scale across all views per object).
Method
#Input
Synthetic
Objects-with-Lighting
Stanford-ORB
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
LightSwitch [ 23 ]
16
21.61
0.86
0.13
25.43
0.84
0.30
32.02
0.98
0.03
NVDiffrecMC [ 8 ]
50
22.95
0.86
0.10
20.24
0.73
0.40
31.60
0.97
0.04
TensoIR [ 15 ]
50
26.17
0.92
0.07
26.12
0.77
0.38
25.27
0.94
0.06
Ours
8
27.76
0.91
0.06
26.76
0.92
0.36
31.01
0.97
0.03
Table 2: Novel-view relighting accuracy. #Input denotes how many source views each method uses; our method requires significantly fewer.
GT
Ours
Light Switch
TensoIR
NVDiff recMC
GT
Ours
Light Switch
TensoIR
NVDiff recMC
Figure 5: Visual comparison of novel-view relighting.
Encoder
Regularization
PSNR ↑
SSIM ↑
LPIPS ↓
Frozen
None
25.06
0.910
0.073
Unfrozen
None
26.12
0.927
0.046
Unfrozen
Explicit (predict albedo)
27.09
0.934
0.057
Unfrozen
Implicit (invariance loss)
27.05
0.952
0.045
Table 3: Ablation study on encoder training and lighting invariance regularization for novel-view relighting.
Figure 6: Progressive relighting ablation. Top plots: PSNR and SSIM over multiple editing steps, comparing token-space editing (ours) with pixel-space decode-and-re-encode. Token-space editing preserves fidelity over early steps, while pixel-space editing degrades from the first step onward. Bottom: visual comparison across editing steps showing the same trend.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Light tokenization: different light sources (environment map, point light, area light) are converted into a unified ray-token representation.
Method
End-to-end
Relight Only
Peak GPU Mem.
CPU Memory
Ours
3.79
3.77
6108
3015
NeuralGaffer
16.85
16.85
24064
5622
DiffusionRenderer
22.81
14.36
28755
7460
LightSwitch
532.33
127.10
22696
2882
TensoIR
3952.91
294.40
10979
5038
NVDiffrecMC
1691.75
225.41
6312
2258
Appendix
Table 4: Runtime and memory comparison. End-to-end inference and relight-only times (sec), peak GPU memory (MB), and CPU Memory (MB). Evaluated on a NVIDIA A100 GPU.
Figure 8: Multi-view relighting comparisons. Per-view optimal tone scales (shown as sliders above each image) reveal that LumiToken exhibits more stable scaling across views than baselines, demonstrating better cross-view consistency.
Figure 9: Multi-view relighting comparisons. Per-view optimal tone scales (shown as sliders above each image) reveal that LumiToken exhibits more stable scaling across views than baselines, demonstrating better cross-view consistency.
Figure 10: Qualitative relighting results on synthetic table-top scenes.
Figure 11: Qualitative relighting results on synthetic table-top scenes (continued).
GT
Ours
Light Switch
TensoIR
NVDiff recMC
Appendix
Figure 12: Visual comparison of novel-view relighting. The first two rows are synthetic experiments; the last two rows are real-world examples from Stanford-ORB.