Joint Shadow Generation and Relighting via Light-Geometry Interaction Maps

Shan Wang; Peixia Li; Chenchen Xu; Ziang Cheng; Jiayu Yang; Hongdong Li; Pulak Purkait

cs.CVFeb 25, 2026

Joint Shadow Generation and Relighting via Light-Geometry Interaction Maps

Authors: Shan Wang, Peixia Li, Chenchen Xu, Ziang Cheng, Jiayu Yang, Hongdong Li, Pulak Purkait

Organizations: Amazon · Australian National University

Abstract

We propose Light-Geometry Interaction (LGI) maps, a novel representation that encodes light-aware occlusion from monocular depth. Unlike ray tracing, which requires full 3D reconstruction, LGI captures essential light-shadow interactions reliably and accurately, computed from off-the-shelf 2.5D depth map predictions. LGI explicitly ties illumination direction to geometry, providing a physics-inspired prior that constrains generative models. Without such prior, these models often produce floating shadows, inconsistent illumination, and implausible shadow geometry. Building on this representation, we propose a unified pipeline for joint shadow generation and relighting - unlike prior methods that treat them as disjoint tasks - capturing the intrinsic coupling of illumination and shadowing essential for modeling indirect effects. By embedding LGI into a bridge-matching generative backbone, we reduce ambiguity and enforce physically consistent light-shadow reasoning. To enable effective training, we curated the first large-scale benchmark dataset for joint shadow and relighting, covering reflections, transparency, and complex interreflections. Experiments show significant gains in realism and consistency across synthetic and real images. LGI thus bridges geometry-inspired rendering with generative modeling, enabling efficient, physically consistent shadow generation and relighting.

Figures & tables

Panel of Figure 1 — Figure 1: Effectiveness of our joint shadow generation and relighting pipeline. Our method produces realistic, texture-aware shadows consistent with object and scene geometry, while preserving faithful relighting across diverse materials such as wood, leather, metal, and glass. (a-c) Multiple object interactions. (d-e) Generalization to multiple light sources.

Figure 2: Overview of the proposed method. Our approach uses a bridge-matching strategy to transform shadow-free latent codes ( $z_{0}$ ) into shadowed counterparts ( $z_{1}$ ), conditioned on global light cues (e.g., light color, radius) and image-derived light–geometry interaction maps. The key novelty lies in generating these interaction maps from image ( $x_{0}$ ) and light (see 4.1 for details).

Panel of Figure 3 — Figure 3: Elevation difference calculation. Scene objects are shown in blue . Each 2D pixel is lifted to 3D at point ${\bm{p}}$ , from which a ray is cast toward the light source ${\bm{l}}$ . Along this ray, we uniformly sample $n$ points ${\bm{S}}$ within the valid front-facing camera frustum. Each sampled point is reprojected onto the image plane to retrieve its depth from the predicted depth map, yielding a set of reprojected points ${\bm{S}}^{\prime}$ . The elevation angles ${\bm{e}}^{s}$ of these reprojected points are then compared with the elevation angle ${\bm{e}}^{l}$ of the light ray to compute elevation difference ${\bm{e}}^{d}$ . If the light is occluded when viewed from point ${\bm{p}}$ , that point is likely to lie in shadow.

Panel of Figure 4 — Figure 4: Example of LGI maps. They indicate shadows cast in the environment and also reveal self-shadowing and shading effects, which can be useful for relighting.

Method	Overall				Shadow region			Object region
Method	RMSE $\downarrow$	SSIM $\uparrow$	BER $\downarrow$	IoU $\uparrow$	RMSE $\downarrow$	SSIM $\uparrow$	BER $\downarrow$	RMSE $\downarrow$	SSIM $\uparrow$
LBM	0.0417	0.7148	0.0847	0.7166	0.1543	0.5690	0.1549	0.0298	0.6797
Ours	0.0334	0.7227	0.0588	0.8096	0.0898	0.6195	0.1103	0.0282	0.6875

Table 1: Joint shadow synthesis and relighting results on ShadRel dataset.

Figure 5: Qualitative comparison with LBM on synthetic object images. Our method produces shadows and relighting effects consistent with geometry and lighting conditions.

Figure 6: Qualitative comparison with SwitchLight (object relighting without shadow generation) and LBM on real object insertions. Despite being trained solely on synthetic object images, our method outperforms SOTA relighting approaches on real-world object images.

Method	Track 1 (Softness Control)		Track 2 (Horz. D. Control)		Track 3 (Vert. D. Control)
Method	IoU $\uparrow$	RMSE $\downarrow$	IoU $\uparrow$	RMSE $\downarrow$	IoU $\uparrow$	RMSE $\downarrow$
CSG	0.818	0.021	0.780	0.030	0.776	0.028
Ours	0.821	0.021	0.798	0.030	0.785	0.027

Table 2: Comparison with CSG Tasar et al. (2024) on their benchmark.

Figure 7: Visual example of clean-background shadow generation, demonstrating that our light-geometry interaction approach provides accurate control over shadow shapes.

Method	BOS Test Images						BOS-free Test Images
Method	GR $\downarrow$	LR $\downarrow$	GS $\uparrow$	LS $\uparrow$	GB $\downarrow$	LB $\downarrow$	GR $\downarrow$	LR $\downarrow$	GS $\uparrow$	LS $\uparrow$	GB $\downarrow$	LB $\downarrow$
ShadowGAN 45	7.511	67.464	0.961	0.197	0.446	0.890	17.325	76.508	0.901	0.060	0.425	0.842
Mask-SG 12	8.997	79.418	0.951	0.180	0.500	1.000	19.338	94.327	0.906	0.044	0.500	1.000
AR-SG 19	7.335	58.037	0.961	0.241	0.383	0.761	16.067	63.713	0.908	0.104	0.349	0.682
SGRNet 11	7.184	68.255	0.964	0.206	0.301	0.596	15.596	60.350	0.909	0.100	0.271	0.534
DMASNet 32	8.256	59.380	0.961	0.228	0.276	0.547	18.725	86.694	0.913	0.055	0.297	0.574

Table 3: Comparison on Image Harmonization on DESOBAv2 Dataset.

Figure 8: Qualitative comparison with SGDGP Zhao et al. (2025) . Our method achieves higher accuracy in shadow regions by aligning more closely with object geometry and light direction.

Components		Overall				Shadow region			Object region
W-L1	LGI	RMSE $\downarrow$	SSIM $\uparrow$	BER $\downarrow$	IOU $\uparrow$	RMSE $\downarrow$	SSIM $\uparrow$	BER $\downarrow$	RMSE $\downarrow$	SSIM $\uparrow$
		0.0408	0.7109	0.1012	0.7193	0.1236	0.5868	0.1923	0.0330	0.6679
✓		0.0391	0.7148	0.0940	0.7353	0.1154	0.5974	0.1784	0.0317	0.6680
✓	✓	0.0334	0.7227	0.0588	0.8096	0.0898	0.6195	0.1103	0.0282	0.6875
-LGI+Depth		0.0388	0.7148	0.0932	0.7344	0.1166	0.5938	0.1765	0.0315	0.6719
+Depth		0.0339	0.7188	0.0719	0.7921	0.0942	0.6139	0.1364	0.0283	0.6836

Table 4: Analysis of Method Components on ShadRel Dataset

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Overview of the image harmonization pipeline. The light estimation network predicts a full set of lighting parameters, with the estimated lighting direction used to compute LGI maps supervised by ground-truth shadow masks. These estimates are integrated into our light-aware shadow generation and relighting framework to produce consistent and realistic harmonization of the inserted object. Since the LGI module is fully differentiable, it enables end-to-end training of the light estimation network.

Panel of Figure 10 — Figure 10: Example of sunlight LGI maps. They establish a correspondence between light direction and shadow shape when scene geometry is known.

Panel of Figure 11 — Figure 11: Examples of multiple object insertion. (a–c) Inserted objects. (d–f) Scenes with inserted objects after shadow generation and relighting. Our method produces realistic, texture-aware shadows on the table and preserves faithful relighting across wood, leather, metal, and glass materials.

Panel of Figure 12 — Figure 12: Examples of scenes with various light sources (top–middle) rendered using our method, and fused two-light-source scenes (bottom)

N	Overall				Shadow region			Object region
	RMSE $\downarrow$	SSIM $\uparrow$	BER $\downarrow$	IOU $\uparrow$	RMSE $\downarrow$	SSIM $\uparrow$	BER $\downarrow$	RMSE $\downarrow$	SSIM $\uparrow$
8	0.0344	0.7188	0.0612	0.7954	0.0991	0.6125	0.1141	0.0283	0.6875
16	0.0334	0.7227	0.0588	0.8096	0.0898	0.6195	0.1103	0.0282	0.6875
32	0.0334	0.7231	0.0586	0.8099	0.0869	0.6198	0.1096	0.0283	0.6875

Appendix

Table 5: Study on Sample Point Count

	Overall				Shadow region			Object region
	RMSE $\downarrow$	SSIM $\uparrow$	BER $\downarrow$	IOU $\uparrow$	RMSE $\downarrow$	SSIM $\uparrow$	BER $\downarrow$	RMSE $\downarrow$	SSIM $\uparrow$
LBM+depth	0.0395	0.7122	0.1003	0.7284	0.1208	0.5923	0.1901	0.0321	0.6692
Ours	0.0334	0.7227	0.0588	0.8096	0.0898	0.6195	0.1103	0.0282	0.6875

Appendix

Table 6: Comparison to LBM with Depth Input

Figure 13: Examples produced by our method on real-world object images.

Figure 14: Qualitative comparison with and without LGI, illustrating that LGI enables geometry-aligned shadows and relighting effects.

Figure 15: Qualitative results on complex backgrounds. Our method remains robust on both non-planar and textured backgrounds.

Panel of Figure 16 — Figure 16: Complex backgrounds in the DESOBAv2 dataset, including non-planar geometry and rich textures and patterns. Our qualitative results on this dataset show that the proposed method produces reasonable predictions even under these challenging background conditions.

Figure 17: Visual results on challenging in-the-wild backgrounds. Our model maintains lighting consistency and shadow realism across diverse surface geometries and textures.

	Overall				Shadow region			Object region
	RMSE $\downarrow$	SSIM $\uparrow$	BER $\downarrow$	IOU $\uparrow$	RMSE $\downarrow$	SSIM $\uparrow$	BER $\downarrow$	RMSE $\downarrow$	SSIM $\uparrow$
LBM	0.0431	0.7112	0.0954	0.0691	0.1575	0.5609	0.1758	0.0342	0.6743
Ours	0.0341	0.7196	0.0658	0.7940	0.0929	0.6114	0.1241	0.0286	0.6828

Appendix

Table 7: Study on Complex Objects

Figure 18: Qualitative results on complex objects. Our ShadRel dataset includes complex objects, and our method remains robust on these challenging cases. As the dataset provides Ground Truth (GT) for only 5 discrete lighting conditions, the Light 3 shows a direct comparison against the available GT. Light 1 and Light 2 represent unseen illumination directions (sampled from the 24 distinct lights in the supplementary video), demonstrating the model’s generalization capabilities where GT is unavailable.

Figure 19: Qualitative results on complex object-background interaction. ShadRel dataset contains objects with diverse materials, including reflective ones such as glass and porcelain, which introduce challenging shadows from transparent objects, with both primary and secondary reflections. Even in these difficult scenarios, our method produces high-fidelity, physically consistent predictions.

Figure 20: Examples from our dataset. Each sample includes (top) input images of objects from different camera poses, (middle) background images with floor and optionally a wall under point-light illumination, and (bottom) target images combining the objects and backgrounds under the same point light.

Figure 21: Examples illustrating the limitations of our method in reproducing fine-grained details. (Top: Ground Truth; Bottom: Our results.) While our approach can handle certain structural details, tiny elements, such as Logos, may appear inconsistent, distorted, or incomplete.

Image Relighting

Image relighting changes a scene’s illumination while preserving its objects, materials, identity, and visual structure.

See all 34 papers

Embedding Physical Reasoning into Diffusion-Based Shadow Generation Under the Sun and Sky
Dec 5, 2025Shilin Hu, Jingyi Xu, Akshat Dave +2Conditional Image Generation Diffusion-Based Image Generation
Stony Brook University, Stony Brook, NY 11794, USA · UNC Charlotte, Charlotte, NC 28223, USA
JointLight: Human-Centric Video Relighting with Asynchronous RGB-Irradiance Joint Modeling
Oct 21, 2026Henglei Lv, Bailin Deng, Xiaoqiang Liu +4Video Relighting Video Diffusion Models
Institute of Computing Technology, Chinese Academy of Sciences, China and University of Chinese Academy of Sciences, China, China · Cardiff University, United Kingdom · Kuaishou Technology, China
SRUG: Shadow-Guided Relightable Urban Scene with Generation Model
May 23, 2026Yonghao Zhao, Zexin Yin, Jian Yang +2Inverse Rendering 3D Reconstruction
College of Computer Science, Nankai University, China · Nankai University and Nanjing University, China · School of Intelligence Science and Technology, Nanjing University, China

Dec 5, 2025J/K move · Enter open · S save

Embedding Physical Reasoning into Diffusion-Based Shadow Generation Under the Sun and Sky

Shilin Hu, Jingyi Xu, Akshat Dave, Dimitris Samaras+1

Stony Brook University, Stony Brook, NY 11794, USA · UNC Charlotte, Charlotte, NC 28223, USA

Conditional Image Generation Diffusion-Based Image Generation Image Editing

Open paper AskPDF arXiv

Generating realistic cast shadows for inserted foreground objects requires reasoning about scene geometry and illumination. However, most learning-based approaches treat shadow generation as an image translation problem and capture these physical relationships only implicitly. This often results in misaligned or implausible shadows. Motivated by the physics of shadow formation, we introduce explicit geometric guidance for outdoor shadow generation. Given a composite image and a foreground object mask, we recover approximate scene geometry and estimate a dominant light direction to derive a coarse shadow estimate via geometric reasoning. While coarse, this estimate provides a spatial anchor for shadow placement. Because illumination cannot always be uniquely inferred from a single image, we predict confidence scores for both lighting and shadow cues and use them to regulate their influence during generation. These cues (shadow mask, light direction, and their confidence scores) condition a diffusion-based generator that refines the estimate into a realistic shadow. Experiments on DESOBAV2 show substantially improved shadow-region fidelity and localization, with an overall 23% lower shadow-region RMSE and 30% lower shadow-mask BER than the prior state-of-the-art method.

Abstract

Image Relighting

Explore similar work

Embedding Physical Reasoning into Diffusion-Based Shadow Generation Under the Sun and Sky

JointLight: Human-Centric Video Relighting with Asynchronous RGB-Irradiance Joint Modeling

SRUG: Shadow-Guided Relightable Urban Scene with Generation Model