FreeDeshadow: Shadow Removal for User-Specified Objects With Multimodal Prompts
Authors: Zhiliang Wu, Jihao Gu, Kun Li, Hanyu Xuan, Hehe Fan, Meng Wang, Weisi Lin
Organizations: College of Computing and Data Science, Nanyang Technological University, Singapore 639798, Singapore. · Department of Computer Science, University College London, London WC1E 6BT, U.K. · College of Computing and Artificial Intelligence, United Arab Emirates University, Abu Dhabi 15551, United Arab Emirates. · School of Big Data and Statistics, Anhui University, Hefei 230039, China. · School of Artificial Intelligence, Zhejiang University, Hangzhou 310007, China. · School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601, China.
Existing shadow removal methods either perform global shadow removal or rely on precise shadow masks to guide restoration. Such paradigms limit their ability to selectively remove the shadow cast by a user-specified object in practical applications. To this end, we introduce an object-aware controllable shadow removal task, which allows users to specify a target object through a simple prompt and removes only its cast shadow while preserving other content. However, object-shadow asymmetry makes it challenging to establish accurate correspondence between the specified object and its spatially separated cast shadow and to confine restoration to the target shadow region. To address these challenges, we propose FreeDeshadow, a unified controllable shadow removal framework that supports both visual and textual prompts. It consists of an object-aware generator and a shadow-aware local refiner. The former interprets the user prompt and produces a high-quality shadow-free proposal, while the latter establishes the correspondence between the object and its cast shadow and constrains the generated changes to the target shadow region. To support training and evaluation, we further construct PCSRD+, a large-scale dataset containing 154,571 quadruples at 1024×1024 resolution and covering six types of user prompts. The visual prompts include a dot, a line, and an object mask, while the textual prompts describe the target object using attributes, spatial relations, or states. Extensive experiments demonstrate that FreeDeshadow accurately removes the shadow of the target object under both visual and textual prompts while effectively preserving non-target regions. It also outperforms existing methods in shadow removal performance and controllability. Project Page: https://wzlwlm13.github.io/FreeDeshadow
Figures & tables
Fig. 1 : Overview of object-aware controllable shadow removal. The object-aware shadow removal framework identifies the target object and selectively removes its cast shadow while preserving shadows belonging to other objects. Rather than relying on precise shadow-region masks as control inputs, FreeDeshadow enables flexible control through visual prompts of varying granularity (dots, lines, or object masks) and textual prompts describing attributes, spatial relations, or states.
Fig. 2 : Overview of the FreeDeshadow framework. (a) The object-aware generator employs Qwen2.5-VL to extract multimodal features from the input image and user prompt, and leverages pretrained generative priors to produce a high-quality shadow-free proposal. (b) The shadow-aware local refiner transforms the multimodal features into sparse shadow prompts and employs cross-scale feature interactions to establish the correspondence between the specified object and its cast shadow. Guided by this correspondence, the refiner aggregates the shadow-free proposal with the latent features of the source image, thereby constraining the generated changes to the target shadow region while preserving non-target content.
Annotation
Visual Prompt
Textual Prompt
Dataset
Samples
Resolution
Task
Pair
S-mask
Obj-S
Dot
Line
Obj-mask
V-attr.
Spatial
State
ISTD [ 4 ]
1,870
480×640
SR
✓
✓
✗
✗
✗
✗
✗
✗
✗
ISTD+ [ 3 ]
1,870
480×640
SR
✓
✓
✗
✗
✗
✗
✗
✗
✗
SRD [ 34 ]
3,088
640×480
SR
✓
✗
✗
✗
✗
✗
✗
✗
✗
USR [ 59 ]
4,215
400×400
SR
✗
✗
✗
✗
✗
✗
✗
✗
✗
DESOBA [ 60 ]
3,623
256×256
SG
✓
✓
✓
✗
✗
✓
✗
✗
✗
TABLE I : Comparison with existing datasets. S-mask: shadow mask; Obj-S: object-shadow association; Obj-mask: object mask; Task: SR, SG, and CSR denote shadow removal, shadow generation, and controllable shadow removal, respectively.
Fig. 3 : Pipeline for the construction of PCSRD+.
Fig. 4 : Statistics for the PCSRD+ dataset.
Shadow Regions
Non-Shadow Regions
All Regions
Method
PSNR ↑
SSIM ↑
RMSE ↓
LPIPS ↓
PSNR ↑
SSIM ↑
RMSE ↓
LPIPS ↓
PSNR ↑
SSIM ↑
RMSE ↓
LPIPS ↓
Visual Prompts (PCSRD+, 1024×1024 )
BMNet [ 67 ]
9.885
0.3043
87.0258
0.0281
24.350
0.8205
17.2356
0.2826
21.951
0.8086
22.3503
0.2980
ControlNet-Inpainting [ 54 ]
10.231
0.2802
83.4050
0.0264
25.146
0.7059
16.5155
0.1424
22.639
0.6969
21.3499
0.1593
Inpaint4Shadow [ 68 ]
11.005
0.3376
77.3426
0.0267
27.807
0.8224
11.7897
0.2795
24.485
0.8114
17.2013
0.2967
ShadowFormer [ 14 ]
10.826
0.3340
78.7570
0.0269
28.836
0.8296
10.6695
0.2717
24.907
0.8183
16.5844
0.2893
TABLE II : Quantitative comparison on the PCSRD+ dataset.
Fig. 5 : Qualitative results of FreeDeshadow under six visual and textual prompt types on the PCSRD+ dataset. The predicted target-shadow masks closely align with the corresponding ground-truth masks, demonstrating the model’s ability to accurately localize the shadow cast by the user-specified object while preserving non-target content.
Method
PSNR ↑
SSIM ↑
RMSE ↓
LPIPS ↓
BMNet [ 67 ]
22.224
0.7571
21.416
0.3517
Inpaint4Shadow [ 68 ]
27.602
0.7660
11.178
0.3434
ShadowFormer [ 14 ]
27.618
0.7668
11.172
0.3468
ShadowDiffusion [ 69 ]
27.763
0.7698
11.060
0.3353
HomoFormer [ 16 ]
27.188
0.7633
11.594
0.3499
PACSRNet [ 20 ]
28.900
0.7898
9.1525
0.2564
TABLE III : Quantitative results on the PCSRD dataset [ 20 ] .
Fig. 6 : Qualitative results of features in object-aware reasoning. “CSG-Layer” denotes the layer in the cross-scale shadow grounding module.
Fig. 7 : Qualitative comparison across six prompt types. The scalar denotes the all-region PSNR of each sample.
Fig. 8 : Comparison with existing shadow removal methods.
Fig. 9 : Real-image shadow removal under visual prompts.
Fig. 10 : User Study. (a) Comparison with image editing methods. (b) Comparison with shadow removal methods.
Cue Cond.
Shadow Ground.
Prompt Adapter
Context Agg.
Shadow PSNR ↑
Non-Shadow PSNR ↑
Shadow IoU ↑
✗
✓
✓
✓
18.097
41.067
0.510
✓
✗
✓
✓
18.744
49.496
0.051
✓
✓
✗
✓
20.221
49.208
0.121
✓
✓
✓
✗
20.357
39.333
0.819
✓
✓
✓
✓
22.018
51.078
0.832
TABLE IV : Module ablation on FreeDeshadow. ✓ / ✗ indicate that a component is enabled or ablated, respectively.
Fig. 11 : Effectiveness of the shadow-aware local refiner. “ Δ ” denotes the difference between the result and ground-truth.
Fig. 12 : Visualization of multi-object shadow removal.
Fig. 13 : Zero-shot cross-dataset results. The model is evaluated on images from the WSRD dataset [ 81 ] without fine-tuning.
Fig. 14 : Qualitative results of FreeDeshadow’s controllability.
School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China · Pazhou Lab, Guangzhou 510005, China · Polytech Nantes, Universit´e de Nantes, Nantes 44306, France