FreeDeshadow: Shadow Removal for User-Specified Objects With Multimodal Prompts
Authors: Zhiliang Wu, Jihao Gu, Kun Li, Hanyu Xuan, Hehe Fan, Meng Wang, Weisi Lin
Organizations: College of Computing and Data Science, Nanyang Technological University, Singapore 639798, Singapore. · Department of Computer Science, University College London, London WC1E 6BT, U.K. · College of Computing and Artificial Intelligence, United Arab Emirates University, Abu Dhabi 15551, United Arab Emirates. · School of Big Data and Statistics, Anhui University, Hefei 230039, China. · School of Artificial Intelligence, Zhejiang University, Hangzhou 310007, China. · School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601, China.
Existing shadow removal methods either perform global shadow removal or rely on precise shadow masks to guide restoration. Such paradigms limit their ability to selectively remove the shadow cast by a user-specified object in practical applications. To this end, we introduce an object-aware controllable shadow removal task, which allows users to specify a target object through a simple prompt and removes only its cast shadow while preserving other content. However, object-shadow asymmetry makes it challenging to establish accurate correspondence between the specified object and its spatially separated cast shadow and to confine restoration to the target shadow region. To address these challenges, we propose FreeDeshadow, a unified controllable shadow removal framework that supports both visual and textual prompts. It consists of an object-aware generator and a shadow-aware local refiner. The former interprets the user prompt and produces a high-quality shadow-free proposal, while the latter establishes the correspondence between the object and its cast shadow and constrains the generated changes to the target shadow region. To support training and evaluation, we further construct PCSRD+, a large-scale dataset containing 154,571 quadruples at 1024×1024 resolution and covering six types of user prompts. The visual prompts include a dot, a line, and an object mask, while the textual prompts describe the target object using attributes, spatial relations, or states. Extensive experiments demonstrate that FreeDeshadow accurately removes the shadow of the target object under both visual and textual prompts while effectively preserving non-target regions. It also outperforms existing methods in shadow removal performance and controllability. Project Page: https://wzlwlm13.github.io/FreeDeshadow
Figures & tables
Fig. 1 : Overview of object-aware controllable shadow removal. The object-aware shadow removal framework identifies the target object and selectively removes its cast shadow while preserving shadows belonging to other objects. Rather than relying on precise shadow-region masks as control inputs, FreeDeshadow enables flexible control through visual prompts of varying granularity (dots, lines, or object masks) and textual prompts describing attributes, spatial relations, or states.
Fig. 2 : Overview of the FreeDeshadow framework. (a) The object-aware generator employs Qwen2.5-VL to extract multimodal features from the input image and user prompt, and leverages pretrained generative priors to produce a high-quality shadow-free proposal. (b) The shadow-aware local refiner transforms the multimodal features into sparse shadow prompts and employs cross-scale feature interactions to establish the correspondence between the specified object and its cast shadow. Guided by this correspondence, the refiner aggregates the shadow-free proposal with the latent features of the source image, thereby constraining the generated changes to the target shadow region while preserving non-target content.
Annotation
Visual Prompt
Textual Prompt
Dataset
Samples
Resolution
Task
Pair
S-mask
Obj-S
Dot
Line
Obj-mask
V-attr.
Spatial
State
ISTD [ 4 ]
1,870
480×640
SR
✓
✓
✗
✗
✗
✗
✗
✗
✗
ISTD+ [ 3 ]
1,870
480×640
SR
✓
✓
✗
✗
✗
✗
✗
✗
✗
SRD [ 34 ]
3,088
640×480
SR
✓
✗
✗
✗
✗
✗
✗
✗
✗
USR [ 59 ]
4,215
400×400
SR
✗
✗
✗
✗
✗
✗
✗
✗
✗
DESOBA [ 60 ]
3,623
256×256
SG
✓
✓
✓
✗
✗
✓
✗
✗
✗
TABLE I : Comparison with existing datasets. S-mask: shadow mask; Obj-S: object-shadow association; Obj-mask: object mask; Task: SR, SG, and CSR denote shadow removal, shadow generation, and controllable shadow removal, respectively.
Fig. 3 : Pipeline for the construction of PCSRD+.
Fig. 4 : Statistics for the PCSRD+ dataset.
Shadow Regions
Non-Shadow Regions
All Regions
Method
PSNR ↑
SSIM ↑
RMSE ↓
LPIPS ↓
PSNR ↑
SSIM ↑
RMSE ↓
LPIPS ↓
PSNR ↑
SSIM ↑
RMSE ↓
LPIPS ↓
Visual Prompts (PCSRD+, 1024×1024 )
BMNet [ 67 ]
9.885
0.3043
87.0258
0.0281
24.350
0.8205
17.2356
0.2826
21.951
0.8086
22.3503
0.2980
ControlNet-Inpainting [ 54 ]
10.231
0.2802
83.4050
0.0264
25.146
0.7059
16.5155
0.1424
22.639
0.6969
21.3499
0.1593
Inpaint4Shadow [ 68 ]
11.005
0.3376
77.3426
0.0267
27.807
0.8224
11.7897
0.2795
24.485
0.8114
17.2013
0.2967
ShadowFormer [ 14 ]
10.826
0.3340
78.7570
0.0269
28.836
0.8296
10.6695
0.2717
24.907
0.8183
16.5844
0.2893
TABLE II : Quantitative comparison on the PCSRD+ dataset.
Fig. 5 : Qualitative results of FreeDeshadow under six visual and textual prompt types on the PCSRD+ dataset. The predicted target-shadow masks closely align with the corresponding ground-truth masks, demonstrating the model’s ability to accurately localize the shadow cast by the user-specified object while preserving non-target content.
Method
PSNR ↑
SSIM ↑
RMSE ↓
LPIPS ↓
BMNet [ 67 ]
22.224
0.7571
21.416
0.3517
Inpaint4Shadow [ 68 ]
27.602
0.7660
11.178
0.3434
ShadowFormer [ 14 ]
27.618
0.7668
11.172
0.3468
ShadowDiffusion [ 69 ]
27.763
0.7698
11.060
0.3353
HomoFormer [ 16 ]
27.188
0.7633
11.594
0.3499
PACSRNet [ 20 ]
28.900
0.7898
9.1525
0.2564
TABLE III : Quantitative results on the PCSRD dataset [ 20 ] .
Fig. 6 : Qualitative results of features in object-aware reasoning. “CSG-Layer” denotes the layer in the cross-scale shadow grounding module.
Fig. 7 : Qualitative comparison across six prompt types. The scalar denotes the all-region PSNR of each sample.
Fig. 8 : Comparison with existing shadow removal methods.
Fig. 9 : Real-image shadow removal under visual prompts.
Fig. 10 : User Study. (a) Comparison with image editing methods. (b) Comparison with shadow removal methods.
Cue Cond.
Shadow Ground.
Prompt Adapter
Context Agg.
Shadow PSNR ↑
Non-Shadow PSNR ↑
Shadow IoU ↑
✗
✓
✓
✓
18.097
41.067
0.510
✓
✗
✓
✓
18.744
49.496
0.051
✓
✓
✗
✓
20.221
49.208
0.121
✓
✓
✓
✗
20.357
39.333
0.819
✓
✓
✓
✓
22.018
51.078
0.832
TABLE IV : Module ablation on FreeDeshadow. ✓ / ✗ indicate that a component is enabled or ablated, respectively.
Fig. 11 : Effectiveness of the shadow-aware local refiner. “ Δ ” denotes the difference between the result and ground-truth.
Fig. 12 : Visualization of multi-object shadow removal.
Fig. 13 : Zero-shot cross-dataset results. The model is evaluated on images from the WSRD dataset [ 81 ] without fine-tuning.
Fig. 14 : Qualitative results of FreeDeshadow’s controllability.
Recent advances in deep learning for shadow removal have significantly enhanced image quality and realism. However, most approaches rely on real-world paired datasets, which are costly to collect and often limited in scene diversity, leading to limited generalization. To address these limitations, we propose One-step Deshadow Diffusion via Reward guidance (ODDR), a new framework that achieves efficient and high-fidelity shadow removal without relying on real-world paired supervision. Our method begins with One-step Deshadow Diffusion (ODD), a baseline model trained on synthetic shadow data for efficient one-step shadow-free reconstruction. We further adapt ODD into ODDR using ShadowReward. In contrast to traditional, annotation-heavy approaches, ShadowReward is the first reward model for shadow removal trained entirely without human annotation. It learns to mimic human perceptual judgments by ranking synthetically generated images with controlled degradations, such as texture distortion and boundary artifacts. This reward-guided fine-tuning enables ODDR to close the synthetic-to-real domain gap. Extensive experiments show that ODD achieves strong performance without relying on real-world paired supervision, and ODDR further improves the results, narrowing the gap to fully supervised methods trained on real-world paired data while maintaining higher computational efficiency as a single-step model.
Junseong Shin, Kijun Kim, Minseong Kim +2
Department of Artificial Intelligence, Hanyang University · AX Future Technology Institute, KT · Department of Computer Science, Hanyang University
Existing supervised and unsupervised shadow removal methods often suffer from limited generalization due to the insufficient diversity of available training datasets, while zero-shot methods tend to produce artifacts and require time-consuming test-time optimization. To address these issues, we propose FreeShadow, a training-free shadow removal method built upon pretrained diffusion models, which exploits diffusion priors for shadow removal without any training or optimization. For illumination recovery, we propose an illumination transfer attention (ITA), which re-weights the self-attention maps in diffusion model to transfer illumination cues from non-shadow to shadow regions. For content preservation, we analyze the effects of illumination variations on self-attention maps and latent high-frequency features in diffusion model, and selectively preserve illumination-invariant components to maintain content fidelity while suppressing residual shadows. We further propose local texture-preserving relighting (LTPR) to mitigate local texture misalignment caused by VAE compression. Extensive experiments demonstrate that our method achieves strong generalization and produces realistic shadow-free images.
Yinan Wang, Yan Huang, Yong Xu +1
School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China · Pazhou Lab, Guangzhou 510005, China · Polytech Nantes, Universit´e de Nantes, Nantes 44306, France
Shadow removal looks nearly solved on established benchmarks, yet remains brittle in the real world. Models have advanced; the paired training data they rely on have barely changed in nearly a decade. The reason is simple: obtaining a shadow-free target requires removing the occluder while keeping the scene, camera, and illumination otherwise unchanged, making diverse paired data difficult to capture. Meanwhile, large shadow detection datasets already contain diverse real-world images and masks, but no shadow-free targets. To turn this abundant but incomplete data into paired supervision, we propose an offline agentic workflow combining physics-motivated generation, failure detection, feedback-driven retry, candidate selection, and deterministic correction. Using this workflow, we construct AgenticShadow, a dataset of 17,138 image-mask-target triplets spanning general scenes, faces, and remote sensing. Our construction workflow reduces Color Distribution Difference by 50.5% over previous shadow removal work, while training existing shadow removal models on AgenticShadow reduces cross-domain LAB RMSE by 19.7-37.5%.
Shilin Hu, Jingyi Xu, Dimitris Samaras +1
Department of Computer Science Stony Brook University · Department of Computer Science University of North Carolina at Charlotte