VSDiffusion: Taming Ill-Posed Shadow Generation via Visibility-Constrained Diffusion

Jing Li; Jing Zhang

cs.CVMar 9, 2026

VSDiffusion: Taming Ill-Posed Shadow Generation via Visibility-Constrained Diffusion

Authors: Jing Li, Jing Zhang

Organizations: East China University of Science and Technology, Shanghai, China

Abstract

Generating realistic cast shadows for inserted foreground objects is a crucial yet challenging problem in image composition, where maintaining geometric consistency of shadow and object in complex scenes remains difficult due to the ill-posed nature of shadow formation. To address this issue, we propose VSDiffusion, a visibility-constrained two-stage framework designed to narrow the solution space by incorporating visibility priors. In Stage I, we predict a coarse shadow mask to localize plausible shadow generated regions. And in Stage II, conditional diffusion is performed guided by lighting and depth cues estimated from the composite to generate accurate shadows. In VSDiffusion, we inject visibility priors through two complementary pathways. First, a visibility control branch with shadow-gated cross attention that provides multi-scale structural guidance. Then, a learned soft prior map that reweights training loss in error-prone regions to enhance geometric correction. Additionally, we also introduce high-frequency guided enhancement module to sharpen boundaries and improve texture interaction with the background. Experiments on widely used public DESOBAv2 dataset demonstrated that our proposed VSDiffusion can generate accurate shadow, and establishes new SOTA results across most evaluation metrics.

Figures & tables

Figure 1: (a) Shadow generation is an ill-posed problem: one input may correspond to multiple visually plausible shadows. With visibility analysis, geometric constraints from the light source, caster, and the shadow receiver can significantly narrow the solution space. (b) further illustrates how the solution space is progressively narrowed. The hatched area indicates the intersection of $\Omega_{vis}$ , $\Omega_{data}$ , and $\Omega_{all}$ . S* is the selected solution that balances data fidelity with geometric plausibility.

Figure 2: Framework of our VSDiffusion. It consists of two stages. Stage 1 predicts a coarse foreground shadow mask $M_{fs}^{(1)}$ . Stage 2 includes two sub-modules: (a) the Visibility Control Branch (VCB), which utilizes visibility prior estimator to extract $I_{\text{light}}$ and $I_{\text{depth}}$ from $I_{\text{c}}$ , subsequently encoded by RCE to provide structural guidance. (b) Under this guidance, the U-Net progressively denoises $Z_{t}$ into $Z_{0}$ .

Figure 3: Architecture of the Shadow-Gated Cross-Attention (SGCA) module. SGCA computes cross-attention between U-Net features $X$ (queries) and conditional features $C$ (keys/values). Shadow Gate mechanism, consisting of dropout, convolution, and sigmoid, adaptively modulates the attention output before it is residually added to $X$ , ensuring robust and selective integration of visibility priors.

Figure 4: Architecture of the High-Frequency Guided Enhancement (HFGE) module. HFGE consists of High-Frequency Extraction (blue) to capture structural cues from $F_{e}$ , and High-Frequency Adaptation (green) to align and calibrate these signals via CBAM. The refined guidance is residually added to decoder features $F_{d}$ to enhance shadow boundaries and textures.

Method	BOS Test Images						BOS-free Test Images
Method	GR $\downarrow$	LR $\downarrow$	GS $\uparrow$	LS $\uparrow$	GB $\downarrow$	LB $\downarrow$	GR $\downarrow$	LR $\downarrow$	GS $\uparrow$	LS $\uparrow$	GB $\downarrow$	LB $\downarrow$
ShadowGAN [ 48 ]	7.511	67.464	0.961	0.197	0.446	0.890	17.325	76.508	0.901	0.060	0.425	0.842
Mask-SG [ 16 ]	8.997	79.418	0.951	0.180	0.500	1.000	19.338	94.327	0.906	0.044	0.500	1.000
AR-SG [ 23 ]	7.335	58.037	0.961	0.241	0.383	0.682	16.067	63.713	0.908	0.104	0.349	0.682
SGRNet [ 15 ]	7.184	68.255	0.964	0.206	0.301	0.596	15.596	60.350	0.909	0.100	0.271	0.534
DMASNet [ 40 ]	8.256	59.380	0.961	0.228	0.276	0.547	18.725	86.694	0.913	0.055	0.297	0.574

Table 1: Quantitative comparisons on DESOBAv2. We evaluate shadow generation with RMSE, SSIM, and BER. BOS denotes test images with background shadow references.

\uparrow

\downarrow

indicate higher/lower is better. Best results are highlighted in bold.

Figure 5: Generation of the spatial prior in SWL. Visibility-related inputs $\{I_{\text{light}},I_{\text{depth}},M_{\text{fo}},M_{\text{fs}}^{(1)}\}$ are concatenated and fed into a lightweight predictor $G_{p}$ , followed by sigmoid to obtain $S_{\text{prior}}$ . Mean normalization then produces $\hat{S}_{\text{prior}}$ (with mean 1), which is used for spatial reweighting in SWL.

Figure 6: Qualitative comparison on the DESOBAv2. From left to right: the composite image (without foreground shadows), the foreground object mask, results of ShadowGAN [ 48 ] , AR-SG [ 23 ] , SGRNet [ 15 ] , SGDiffusion [ 26 ] , GPSDiffusion [ 49 ] , our VSDiffusion, and the ground truth. With visibility constraints, the proposed method generates shadows with more consistent projection direction, plausible contact, and sharper boundary alignment, especially under the BOS-free setting.

VCB	HFGE	SWL	GR $\downarrow$	LR $\downarrow$	GS $\uparrow$	LS $\uparrow$	GB $\downarrow$	LB $\downarrow$
			6.655	50.598	0.962	0.364	0.221	0.438
✓			6.592	48.265	0.963	0.366	0.203	0.403
✓	✓		6.571	48.147	0.964	0.374	0.193	0.381
✓	✓	✓	6.422	46.542	0.964	0.377	0.182	0.360

Table 2: Ablation study on DESOBAv2. VCB and SWL are two visibility-inspired ways to inject priors into the diffusion model. SWL is short for the

S_{\text{prior}}

-Weighted Loss. HFGE denotes the High-Frequency Guided Enhancement module.

\uparrow

\downarrow

indicate higher/lower is better, and best results are in bold.

Figure 7: Qualitative ablation of VSDiffusion. Visual comparisons demonstrate that our SWL effectively concentrates supervision on critical shadow boundaries and penumbras. In Effective supervision, the red regions i.e . , high weight $\times$ high error.

Method	GR $\downarrow$	LR $\downarrow$	GS $\uparrow$	LS $\uparrow$	GB $\downarrow$	LB $\downarrow$
Baseline	6.655	50.598	0.962	0.364	0.221	0.438
w/o light	6.545	48.149	0.963	0.373	0.197	0.390
w/o depth	6.456	47.990	0.963	0.371	0.187	0.369
Ours	6.422	46.542	0.964	0.377	0.182	0.360

Table 3: Ablation on light/depth priors in VCB on DESOBAv2. “w/o light” and “w/o depth” remove the corresponding condition while keeping all other components unchanged. The full model achieves the best overall performance.

\downarrow

\uparrow

indicate lower/higher is better, and best results are in bold.

Figure 8: Limitations in BOS-free scenarios: our shadows match geometry but may slightly deviate in intensity from the ground truth.

Compositional Image Generation

Compositional image generation studies how to combine objects, attributes, spatial relations, and visual references into coherent, controllable images.

See all 85 papers

Embedding Physical Reasoning into Diffusion-Based Shadow Generation Under the Sun and Sky
Dec 5, 2025Shilin Hu, Jingyi Xu, Akshat Dave +2Conditional Image Generation Diffusion-Based Image Generation
Stony Brook University, Stony Brook, NY 11794, USA · UNC Charlotte, Charlotte, NC 28223, USA
MultiShadow: Multi-Object Shadow Generation for Image Compositing via Diffusion Model
Mar 3, 2026Waqas Ahmed, Dean Diepeveen, Ferdous SohelImage Editing Diffusion-Based Image Generation
School of Information Technology, Murdoch University, Murdoch, WA 6150, Australia.
ODDR: One-Step Deshadow Diffusion via Reward Guidance
Oct 1, 2026Junseong Shin, Kijun Kim, Minseong Kim +2Synthetic-to-Real Domain Adaptation Diffusion Models for Image Restoration
Department of Artificial Intelligence, Hanyang University · AX Future Technology Institute, KT · Department of Computer Science, Hanyang University

Dec 5, 2025J/K move · Enter open · S save

Embedding Physical Reasoning into Diffusion-Based Shadow Generation Under the Sun and Sky

Shilin Hu, Jingyi Xu, Akshat Dave, Dimitris Samaras+1

Stony Brook University, Stony Brook, NY 11794, USA · UNC Charlotte, Charlotte, NC 28223, USA

Conditional Image Generation Diffusion-Based Image Generation Image Editing

Open paper AskPDF arXiv

Generating realistic cast shadows for inserted foreground objects requires reasoning about scene geometry and illumination. However, most learning-based approaches treat shadow generation as an image translation problem and capture these physical relationships only implicitly. This often results in misaligned or implausible shadows. Motivated by the physics of shadow formation, we introduce explicit geometric guidance for outdoor shadow generation. Given a composite image and a foreground object mask, we recover approximate scene geometry and estimate a dominant light direction to derive a coarse shadow estimate via geometric reasoning. While coarse, this estimate provides a spatial anchor for shadow placement. Because illumination cannot always be uniquely inferred from a single image, we predict confidence scores for both lighting and shadow cues and use them to regulate their influence during generation. These cues (shadow mask, light direction, and their confidence scores) condition a diffusion-based generator that refines the estimate into a realistic shadow. Experiments on DESOBAV2 show substantially improved shadow-region fidelity and localization, with an overall 23% lower shadow-region RMSE and 30% lower shadow-mask BER than the prior state-of-the-art method.

Abstract

Compositional Image Generation

Explore similar work

Embedding Physical Reasoning into Diffusion-Based Shadow Generation Under the Sun and Sky

MultiShadow: Multi-Object Shadow Generation for Image Compositing via Diffusion Model

ODDR: One-Step Deshadow Diffusion via Reward Guidance