Learning Latent Proxies for Controllable Single-Image Relighting

Haoze Zheng; Zihao Wang; Xianfeng Wu; Yajing Bai; Yexin Liu; Yun Li; Xiaogang Xu; Harry Yang

cs.CVMar 16, 2026

Learning Latent Proxies for Controllable Single-Image Relighting

Authors: Haoze Zheng, Zihao Wang, Xianfeng Wu, Yajing Bai, Yexin Liu, Yun Li, Xiaogang Xu, Harry Yang

Organizations: HKUST · HKPolyU · CUHK

Abstract

Single-image relighting is highly under-constrained: small illumination changes can produce large, nonlinear variations in shading, shadows, and specularities, while geometry and materials remain unobserved. Existing diffusion-based approaches either rely on intrinsic or G-buffer pipelines that require dense and fragile supervision, or operate purely in latent space without physical grounding, making fine-grained control of direction, intensity, and color unreliable. We observe that a full intrinsic decomposition is unnecessary and redundant for accurate relighting. Instead, sparse but physically meaningful cues, indicating where illumination should change and how materials should respond, are sufficient to guide a diffusion model. Based on this insight, we introduce LightCtrl that integrates physical priors at two levels: a few-shot latent proxy encoder that extracts compact material-geometry cues from limited PBR supervision, and a lighting-aware mask that identifies sensitive illumination regions and steers the denoiser toward shading relevant pixels. To compensate for scarce PBR data, we refine the proxy branch using a DPO-based objective that enforces physical consistency in the predicted cues. We also present ScaLight, a large-scale object-level dataset with systematically varied illumination and complete camera-light metadata, enabling physically consistent and controllable training. Across object and scene level benchmarks, our method achieves photometrically faithful relighting with accurate continuous control, surpassing prior diffusion and intrinsic-based baselines, including gains of up to +2.4 dB PSNR and 35% lower RMSE under controlled lighting shifts.

Figures & tables

Figure 2: Overview of LightCtrl and the object-level rendering pipeline. Our model integrates a few-shot latent proxy, a lighting-aware mask, and a DPO-refined PBR encoder to condition the UNet during denoising. The implicit PBR encoder extracts physically-based latent proxy features $t_{\mathrm{phys}}$ , which are concatenated with image tokens $t_{\mathrm{image}}$ and light condition tokens $t_{\mathrm{light}}$ to guide the diffusion process. A lighting mask predictor takes the lighting change $\Delta\ell$ as input to produce a spatially-aware attention mask for the UNet. The output column shows relit results under three different lighting configurations ( $\Delta\ell_{1}$ , $\Delta\ell_{2}$ , $\Delta\ell_{3}$ ), demonstrating the model’s ability to produce diverse and physically plausible relighting effects from a single input image.

Dataset	#Objects	Lighting Control	Pose / Light
OpenIllumi [ 17 ]	64	$\sim$	$\times$
OWL [ 32 ]	8	$\sim$	$\sim$
Rene [ 31 ]	20	$\sim$	$\times$
LightProp [ 11 ]	80K+	$\checkmark$	$\times$
ScaLight (Ours)	300K+	$\checkmark$	$\checkmark$

Table 1: Comparison with other object-level datasets. ScaLight features large-scale scalability, detailed camera and light metadata (recording exact camera and illumination parameters), and fully controllable lighting setups (supporting diverse and flexible lighting effects). Symbols:

\checkmark

full;

\sim

limited;

\times

none.

Method	RMSE $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
RGB $\leftrightarrow$ X [ 40 ]	0.389	0.425	8.25
IC-Light [ 41 ]	0.413	0.337	7.94
LumiNet [ 35 ]	0.139	0.904	17.20
Ours	0.167	0.655	18.30

Table 2: Scene-level relighting comparison on the MIIW testset. Bold indicates the best performance.

Method	Label	Temperature (CCT shift)			Position (yaw/pitch)			Energy (intensity)
Method	Label	RMSE $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	RMSE $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	RMSE $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
IC-Light [ 41 ]	text	0.397	0.346	8.21	0.375	0.363	8.65	0.380	0.382	8.63
DreamLight [ 19 ]	text	0.349	0.559	9.40	0.221	0.662	13.3	0.221	0.671	13.5
x2rgb [ 40 ]	G-buffer	0.472	0.493	7.81	0.487	0.592	8.32	0.484	0.676	8.89
LBM [ 5 ]	image	0.064	0.957	27.8	0.084	0.936	23.1	0.073	0.956	25.3
Luminet [ 35 ]	image	0.172	0.848	15.8	0.146	0.847	17.8	0.164	0.825	16.2

Table 3: Quantitative comparison on relit color output under three lighting variations. All metrics are computed on the normalized RGB range

[-1,1]

. Temperature : correlated color temperature (CCT) shift; Position : light direction/placement change (yaw/pitch); Energy : light intensity scaling.

Figure 3: Visual comparison of intrinsic decomposition on a chair object. From left to right: ground-truth (GT), our method with DPO, our method without DPO, rgb2x [ 40 ] , and Neural_Lightrig [ 11 ] . The DPO fine-tuning clearly suppresses artifacts and produces more accurate albedo and normal predictions, demonstrating the effectiveness of our DPO strategy.

Figure 4: Qualitative results on in-the-wild images. Our model produces realistic relighting on Internet and product photos (right). Hand-captured examples confirm robust generalization to uncontrolled lighting.

Figure 5: Object-level relighting comparison. Given a single input image (left), we compare our method against IC-Light, Qwen Image Edit, RGB-X, and DiLightNet. Competing methods often introduce strong color shifts, texture distortions, or inconsistent shading, while our approach produces well-relighting results that preserve materials, geometry, and object identity. These examples demonstrate our model’s superior ability to perform physically plausible object-centric relighting.

Method	User Preference Rate ( $\uparrow$ )
Method	Scene-Level	Object-Level
LumiNet [ 35 ]	12.83%	4.3%
IC-Light [ 41 ]	9.37%	11.45%
NanoBanana	22.07%	2.8%
Ours	55.73%	81.45%

Table 4: User preference study (

N=35

). We evaluate human preference separately on complex real-world scenes (MIIW and RWR) and controlled objects (ScaLight). Our method significantly outperforms existing baselines, achieving the highest preference rate in both settings.

Figure 6: Scene-level relighting comparison on indoor environments. From left to right: input, ground-truth relighting, our method, ChatGPT Image, and LumiNet. Our approach produces realistic global illumination and consistent local shading, especially around the reflective light probes highlighted by red boxes, whereas the generative baselines tend to make color shifts or over-smoothed shading.

Figure 7: Targeted real-scene failure analysis on MIIW and RWR [ 39 ] , showing representative errors in shadow recasting, structure preservation, and highlight-heavy regions.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Illustration of our controllable lighting formulation. A lighting edit is first defined in yaw–pitch space (e.g., Light A $\rightarrow$ Light B with $\Delta\mathrm{yaw}=90^{\circ}$ , $\Delta\mathrm{pitch}=0^{\circ}$ ). These angular parameters are mapped to unit vectors on the hemisphere, then projected onto a low-order spherical-harmonic (SH) basis. The resulting SH difference $\Delta\mathbf{s}_{\mathrm{SH}}$ forms the directional component of our relative illumination encoding $\Delta\ell$ .

#PBR Samples	RMSE $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
1K	0.468	0.487	12.8
3K	0.421	0.534	14.7
6K	0.331	0.612	19.3
9K	0.194	0.873	23.5
12K	0.105	0.921	28.6

Appendix

Table 5: Effect of PBR supervision scale on relighting quality. Increasing PBR annotations steadily improves performance, yet our experiments show that even sparse supervision provides meaningful benefits for relighting tasks.

Figure 9: Qualitative relighting results on ScaLight under varying light positions . Left: reference image; middle: ground-truth relit image; right: LightCtrl prediction.

Figure 10: Qualitative relighting results on ScaLight under varying light positions .

Figure 11: Qualitative relighting results on ScaLight under varying light intensity . Left: reference image; middle: ground-truth relit image; right: LightCtrl prediction.

Figure 12: Qualitative relighting results on ScaLight under varying light intensity .

Figure 13: Qualitative relighting results on ScaLight under varying light temperature . Left: reference image; middle: ground-truth relit image; right: LightCtrl prediction.

Figure 14: Qualitative relighting results on ScaLight under varying light temperature

Figure 15: Qualitative relighting results on MIIW . Left: reference image; middle: ground-truth relit image; right: LightCtrl prediction.

Figure 16: Qualitative relighting results on MIIW . Left: reference image; middle: ground-truth relit image; right: LightCtrl prediction.

Physics-Informed Diffusion Models

Physics-informed diffusion models use physical laws and measurements to guide generative sampling, reconstruction, and forecasting.

See all 49 papers

May 18, 2026

PIXLRelight: Controllable Relighting via Intrinsic Conditioning

We present PIXLRelight, a feed-forward approach for physically controllable single-image relighting. Existing methods either provide limited lighting control (e.g. through text or environment maps), accumulate errors when chaining inverse and forward rendering, or require costly per-image optimization. Our key idea is to bridge physically based rendering (PBR) and learned image synthesis through a shared intrinsic conditioning that can be obtained from either real photographs or PBR renders. At training time, paired multi-illumination photographs are decomposed into albedo, diffuse shading, and non-diffuse residuals, which condition the model. At inference time, the same conditioning is computed from a path-traced render of a coarse 3D reconstruction of the input under user-specified PBR lights. A transformer-based neural renderer then applies the target illumination to the source photograph, preserving fine image detail through a per-pixel affine modulation. PIXLRelight enables arbitrary PBR-style lighting control, achieves state-of-the-art relighting quality, and runs in under a tenth of a second per image. Code and models are available at https://mlfarinha.github.io/pixl-relight/.

Miguel Farinha, Ronald Clark

Department of Computer Science University of Oxford

Neural Rendering Image Relighting

Ask PDF

Mar 28, 2026

LightCtrl: Training-free Controllable Video Relighting

Recent diffusion models have achieved remarkable success in image relighting, and this success has quickly been extended to video relighting. However, existing methods offer limited explicit control over illumination in the relighted output. We present LightCtrl, the first controllable video relighting method that enables explicit control of video illumination through a user-supplied light trajectory in a training-free manner. Our approach combines pre-trained diffusion models: an image relighting model processes each frame individually, followed by a video diffusion prior to enhance temporal consistency. To achieve explicit control over dynamically varying lighting, we introduce two key components. First, a Light Map Injection module samples light trajectory-specific noise and injects it into the latent representation of the source video, improving illumination coherence with the conditional light trajectory. Second, a Geometry-Aware Relighting module dynamically combines RGB and normal map latents in the frequency domain to suppress the influence of the original lighting, further enhancing adherence to the input light trajectory. Experiments show that LightCtrl produces high-quality videos with diverse illumination changes that closely follow the specified light trajectory, demonstrating improved controllability over baseline methods. Code is available at: https://github.com/GVCLab/LightCtrl.

Yizuo Peng, Xuelin Chen, Kai Zhang +1

Tsinghua University · GVC Lab, Great Bay University · Adobe Research

Video Relighting Video Editing Controllable Video Generation

Ask PDF

Jul 15, 2026

FreeLit: Paired-Free Indoor Relighting via Physics-Guided Diffusion

Image-based indoor scene relighting remains challenging due to the complex interplay between cluttered geometry and local illumination, requiring precise modeling of light position, color, and intensity. Existing data-driven methods implicitly learn this relationship via paired multi-illumination datasets. Nevertheless, this data is costly and fails to scale, which is essential for accurate light-source-level control. Conversely, inverse-rendering methods reduce the data dependency by incorporating physical priors; however, they lack the robustness of intrinsic estimation in challenging conditions. In this paper, we present FreeLit, a paired-free framework for controllable indoor relighting that explicitly manipulates light-source location, color, and intensity. Instead of relying on paired supervision, we construct a physics-guided illumination prior from intrinsic scene properties, generating a structured lightmap along with a pseudo-relit image to guide diffusion-based synthesis. To address instability in intrinsic estimation, especially in low-light scenes, we introduce a relighting-guided intrinsic stabilization strategy that enforces illumination-invariant reflectance through structure-aware distillation and consistency constraints. Furthermore, we propose controllability-oriented evaluation metrics to quantify alignment with user-specified illumination color and intensity. Experimental results demonstrate that FreeLit achieves stable, physically consistent, and controllable relighting, with improved robustness in low-light indoor scenes, without requiring paired supervision.

Chi-En Yen, Duy-Khanh Ngo, Wen-Wei Tang +3

National Yang Ming Chiao Tung University

Image Relighting

Ask PDF

Abstract

Physics-Informed Diffusion Models

Explore similar work

PIXLRelight: Controllable Relighting via Intrinsic Conditioning

LightCtrl: Training-free Controllable Video Relighting

FreeLit: Paired-Free Indoor Relighting via Physics-Guided Diffusion