Generating realistic cast shadows for inserted foreground objects is a crucial yet challenging problem in image composition, where maintaining geometric consistency of shadow and object in complex scenes remains difficult due to the ill-posed nature of shadow formation. To address this issue, we propose VSDiffusion, a visibility-constrained two-stage framework designed to narrow the solution space by incorporating visibility priors. In Stage I, we predict a coarse shadow mask to localize plausible shadow generated regions. And in Stage II, conditional diffusion is performed guided by lighting and depth cues estimated from the composite to generate accurate shadows. In VSDiffusion, we inject visibility priors through two complementary pathways. First, a visibility control branch with shadow-gated cross attention that provides multi-scale structural guidance. Then, a learned soft prior map that reweights training loss in error-prone regions to enhance geometric correction. Additionally, we also introduce high-frequency guided enhancement module to sharpen boundaries and improve texture interaction with the background. Experiments on widely used public DESOBAv2 dataset demonstrated that our proposed VSDiffusion can generate accurate shadow, and establishes new SOTA results across most evaluation metrics.
Figures & tables
Figure 1: (a) Shadow generation is an ill-posed problem: one input may correspond to multiple visually plausible shadows. With visibility analysis, geometric constraints from the light source, caster, and the shadow receiver can significantly narrow the solution space. (b) further illustrates how the solution space is progressively narrowed. The hatched area indicates the intersection of Ωvis , Ωdata , and Ωall . S* is the selected solution that balances data fidelity with geometric plausibility.
Figure 2: Framework of our VSDiffusion. It consists of two stages. Stage 1 predicts a coarse foreground shadow mask Mfs(1) . Stage 2 includes two sub-modules: (a) the Visibility Control Branch (VCB), which utilizes visibility prior estimator to extract Ilight and Idepth from Ic , subsequently encoded by RCE to provide structural guidance. (b) Under this guidance, the U-Net progressively denoises Zt into Z0 .
Figure 3: Architecture of the Shadow-Gated Cross-Attention (SGCA) module. SGCA computes cross-attention between U-Net features X (queries) and conditional features C (keys/values). Shadow Gate mechanism, consisting of dropout, convolution, and sigmoid, adaptively modulates the attention output before it is residually added to X , ensuring robust and selective integration of visibility priors.
Figure 4: Architecture of the High-Frequency Guided Enhancement (HFGE) module. HFGE consists of High-Frequency Extraction (blue) to capture structural cues from Fe , and High-Frequency Adaptation (green) to align and calibrate these signals via CBAM. The refined guidance is residually added to decoder features Fd to enhance shadow boundaries and textures.
Method
BOS Test Images
BOS-free Test Images
GR ↓
LR ↓
GS ↑
LS ↑
GB ↓
LB ↓
GR ↓
LR ↓
GS ↑
LS ↑
GB ↓
LB ↓
ShadowGAN [ 48 ]
7.511
67.464
0.961
0.197
0.446
0.890
17.325
76.508
0.901
0.060
0.425
0.842
Mask-SG [ 16 ]
8.997
79.418
0.951
0.180
0.500
1.000
19.338
94.327
0.906
0.044
0.500
1.000
AR-SG [ 23 ]
7.335
58.037
0.961
0.241
0.383
0.682
16.067
63.713
0.908
0.104
0.349
0.682
SGRNet [ 15 ]
7.184
68.255
0.964
0.206
0.301
0.596
15.596
60.350
0.909
0.100
0.271
0.534
DMASNet [ 40 ]
8.256
59.380
0.961
0.228
0.276
0.547
18.725
86.694
0.913
0.055
0.297
0.574
Table 1: Quantitative comparisons on DESOBAv2. We evaluate shadow generation with RMSE, SSIM, and BER. BOS denotes test images with background shadow references. ↑ / ↓ indicate higher/lower is better. Best results are highlighted in bold.
Figure 5: Generation of the spatial prior in SWL. Visibility-related inputs {Ilight,Idepth,Mfo,Mfs(1)} are concatenated and fed into a lightweight predictor Gp , followed by sigmoid to obtain Sprior . Mean normalization then produces S^prior (with mean 1), which is used for spatial reweighting in SWL.
Figure 6: Qualitative comparison on the DESOBAv2. From left to right: the composite image (without foreground shadows), the foreground object mask, results of ShadowGAN [ 48 ] , AR-SG [ 23 ] , SGRNet [ 15 ] , SGDiffusion [ 26 ] , GPSDiffusion [ 49 ] , our VSDiffusion, and the ground truth. With visibility constraints, the proposed method generates shadows with more consistent projection direction, plausible contact, and sharper boundary alignment, especially under the BOS-free setting.
VCB
HFGE
SWL
GR ↓
LR ↓
GS ↑
LS ↑
GB ↓
LB ↓
6.655
50.598
0.962
0.364
0.221
0.438
✓
6.592
48.265
0.963
0.366
0.203
0.403
✓
✓
6.571
48.147
0.964
0.374
0.193
0.381
✓
✓
✓
6.422
46.542
0.964
0.377
0.182
0.360
Table 2: Ablation study on DESOBAv2. VCB and SWL are two visibility-inspired ways to inject priors into the diffusion model. SWL is short for the Sprior -Weighted Loss. HFGE denotes the High-Frequency Guided Enhancement module. ↑ / ↓ indicate higher/lower is better, and best results are in bold.
Figure 7: Qualitative ablation of VSDiffusion. Visual comparisons demonstrate that our SWL effectively concentrates supervision on critical shadow boundaries and penumbras. In Effective supervision, the red regions i.e . , high weight × high error.
Method
GR ↓
LR ↓
GS ↑
LS ↑
GB ↓
LB ↓
Baseline
6.655
50.598
0.962
0.364
0.221
0.438
w/o light
6.545
48.149
0.963
0.373
0.197
0.390
w/o depth
6.456
47.990
0.963
0.371
0.187
0.369
Ours
6.422
46.542
0.964
0.377
0.182
0.360
Table 3: Ablation on light/depth priors in VCB on DESOBAv2. “w/o light” and “w/o depth” remove the corresponding condition while keeping all other components unchanged. The full model achieves the best overall performance. ↓ / ↑ indicate lower/higher is better, and best results are in bold.
Figure 8: Limitations in BOS-free scenarios: our shadows match geometry but may slightly deviate in intensity from the ground truth.
Compositional image generation studies how to combine objects, attributes, spatial relations, and visual references into coherent, controllable images.
Generating realistic cast shadows for inserted foreground objects requires reasoning about scene geometry and illumination. However, most learning-based approaches treat shadow generation as an image translation problem and capture these physical relationships only implicitly. This often results in misaligned or implausible shadows. Motivated by the physics of shadow formation, we introduce explicit geometric guidance for outdoor shadow generation. Given a composite image and a foreground object mask, we recover approximate scene geometry and estimate a dominant light direction to derive a coarse shadow estimate via geometric reasoning. While coarse, this estimate provides a spatial anchor for shadow placement. Because illumination cannot always be uniquely inferred from a single image, we predict confidence scores for both lighting and shadow cues and use them to regulate their influence during generation. These cues (shadow mask, light direction, and their confidence scores) condition a diffusion-based generator that refines the estimate into a realistic shadow. Experiments on DESOBAV2 show substantially improved shadow-region fidelity and localization, with an overall 23% lower shadow-region RMSE and 30% lower shadow-mask BER than the prior state-of-the-art method.
Shilin Hu, Jingyi Xu, Akshat Dave +2
Stony Brook University, Stony Brook, NY 11794, USA · UNC Charlotte, Charlotte, NC 28223, USA
Realistic shadow generation is crucial for achieving seamless image compositing, yet existing methods primarily focus on single-object insertion and often fail to generalize when multiple foreground objects are composited into a background scene. In practice, however, modern compositing pipelines and real-world applications often insert multiple objects simultaneously, necessitating shadows that are jointly consistent in terms of geometry, attachment, and location. In this paper, we address the under-explored problem of multi-object shadow generation, aiming to synthesize physically plausible shadows for multiple inserted objects. Our approach exploits the multimodal capabilities of a pre-trained text-to-image diffusion model. An image pathway injects dense, multi-scale features to provide fine-grained spatial guidance, while a text-based pathway encodes per-object shadow bounding boxes as learned positional tokens and fuses them via cross-attention. An attention-alignment loss further grounds these tokens to their corresponding shadow regions. To support this task, we augment the DESOBAv2 dataset by constructing composite scenes with multiple inserted objects and automatically derive prompts combining object category and shadow positioning information. Experimental results demonstrate that our method achieves state-of-the-art performance in both single and multi-object shadow generation settings.
Waqas Ahmed, Dean Diepeveen, Ferdous Sohel
School of Information Technology, Murdoch University, Murdoch, WA 6150, Australia.
Recent advances in deep learning for shadow removal have significantly enhanced image quality and realism. However, most approaches rely on real-world paired datasets, which are costly to collect and often limited in scene diversity, leading to limited generalization. To address these limitations, we propose One-step Deshadow Diffusion via Reward guidance (ODDR), a new framework that achieves efficient and high-fidelity shadow removal without relying on real-world paired supervision. Our method begins with One-step Deshadow Diffusion (ODD), a baseline model trained on synthetic shadow data for efficient one-step shadow-free reconstruction. We further adapt ODD into ODDR using ShadowReward. In contrast to traditional, annotation-heavy approaches, ShadowReward is the first reward model for shadow removal trained entirely without human annotation. It learns to mimic human perceptual judgments by ranking synthetically generated images with controlled degradations, such as texture distortion and boundary artifacts. This reward-guided fine-tuning enables ODDR to close the synthetic-to-real domain gap. Extensive experiments show that ODD achieves strong performance without relying on real-world paired supervision, and ODDR further improves the results, narrowing the gap to fully supervised methods trained on real-world paired data while maintaining higher computational efficiency as a single-step model.
Junseong Shin, Kijun Kim, Minseong Kim +2
Department of Artificial Intelligence, Hanyang University · AX Future Technology Institute, KT · Department of Computer Science, Hanyang University