After a Decade: Bringing Shadow Removal into the Real World with Agentic Training Data

Shilin Hu; Jingyi Xu; Dimitris Samaras; Hieu Le

cs.CVSep 29, 2026

After a Decade: Bringing Shadow Removal into the Real World with Agentic Training Data

Authors: Shilin Hu, Jingyi Xu, Dimitris Samaras, Hieu Le

Organizations: Department of Computer Science Stony Brook University · Department of Computer Science University of North Carolina at Charlotte

Abstract

Shadow removal looks nearly solved on established benchmarks, yet remains brittle in the real world. Models have advanced; the paired training data they rely on have barely changed in nearly a decade. The reason is simple: obtaining a shadow-free target requires removing the occluder while keeping the scene, camera, and illumination otherwise unchanged, making diverse paired data difficult to capture. Meanwhile, large shadow detection datasets already contain diverse real-world images and masks, but no shadow-free targets. To turn this abundant but incomplete data into paired supervision, we propose an offline agentic workflow combining physics-motivated generation, failure detection, feedback-driven retry, candidate selection, and deterministic correction. Using this workflow, we construct AgenticShadow, a dataset of 17,138 image-mask-target triplets spanning general scenes, faces, and remote sensing. Our construction workflow reduces Color Distribution Difference by 50.5% over previous shadow removal work, while training existing shadow removal models on AgenticShadow reduces cross-domain LAB RMSE by 19.7-37.5%.

Figures & tables

Figure 1: From constrained paired capture to diverse real-world supervision. ISTD ( Wang et al., 2018 ) and SRD ( Qu et al., 2017 ) largely reflect constrained paired acquisition with limited scene diversity, while WSRD ( Vasluianu et al., 2023 ) remains centered on controlled, close-range object and surface interactions. AgenticShadow draws from SBU ( Vicente et al., 2016 ) , CUHK-Shadow ( Hu et al., 2021 ) , ASFW ( Luo et al., 2026 ) , and S-EO ( Masquil et al., 2025 ) , providing substantially broader scenes, viewpoints, shadow appearances, and application domains.

Figure 2: Agentic dataset-construction pipeline. A VLM evaluates an initial mask-guided probe, triggers additional guided edits or feedback-guided retries, filters major failures, and selects a candidate based on visual quality and Retinex-LoG diagnostics. Segmentation-guided regional correction produces the final shadow-free target. The bottom row illustrates a rejected-probe example.

	SBU	CUHK	ASFW	S-EO	Total
Domain	General	General	Facial	Remote sensing	–
Train	3,974	8,335	928	880	14,117
Test	635	2,087	153	146	3,021
Total	4,609	10,422	1,081	1,026	17,138

Table 1: Composition of AgenticShadow. The four source domains provide complementary scene types and together form a benchmark of 17,138 image-mask-target triplets.

Panel of Figure 3 — Figure 3: Dataset diversity compared with ISTD+. (a) Distribution of shadow-mask coverage. (b) Average spatial distribution of shadow masks. (c) Color contrast between shadow and non-shadow regions. (d) Joint distribution of median log-luminance correction and within-shadow heterogeneity. AgenticShadow contains larger and more spatially distributed shadows, more low-contrast cases, and substantially broader and more heterogeneous correction requirements.

Method	Mean	Std	Min	Max
SID ( Le and Samaras, 2019 )	38.00	45.05	1.66	353.64
SF ( Guo et al., 2023a )	33.63	52.74	0.41	427.34
SD ( Guo et al., 2023b )	89.72	130.54	0.29	650.26
I4S ( Li et al., 2023a )	31.57	48.39	0.43	402.44
SRR ( Hu et al., 2025 )	14.06	26.08	0.24	215.40
Ours-Physics	6.96	18.59	0.01	158.40

Table 2: Agentic target-construction quality on SRR ( Hu et al., 2025 ) . (a) Comparison with previous shadow removal methods. (b) Ablation of the agentic construction pipeline. CDD values are multiplied by

10^{3}

, agentic results are averaged over two independent runs, and lower is better.

Figure 4: Prior-Fusion (PF) baseline. Frozen DINOv2 ( Oquab et al., 2024 ) and Depth Anything V2 ( Yang et al., 2024 ) extract semantic and geometric priors from the shadow image. Learned projections align these priors with intermediate features, and zero-initialized residual fusion incorporates them at the $24\times 24$ bottleneck input and the first $48\times 48$ decoder stage.

	ISTD+		AgenticShadow
Model	PSNR $\uparrow$	LAB $\downarrow$	PSNR $\uparrow$	LAB $\downarrow$
HomoFormer ∗	32.53	1.92	18.85	8.43
PF ∗	32.95	1.87	18.70	8.55
HomoFormer	32.92	1.85	21.95	5.87
PF	32.74	1.86	22.50	5.55

Table 3: Analysis of the Prior-Fusion baseline. (a) Interaction between training data and prior fusion, comparing ISTD+-only training with combined ISTD+ and AgenticShadow training; ∗ denotes the former. (b) Matched E50 prior ablation initialized from HomoFormer E100 ( Xiao et al., 2024 ) . AgenticShadow results are four-domain macro averages.

Official ISTD+ checkpoints
	ISTD+	AgenticShadow: LAB RMSE $\downarrow$				AgenticShadow: Macro
Method	LAB $\downarrow$	SBU	CUHK	ASFW	S-EO	PSNR $\uparrow$	SSIM $\uparrow$	MAE $\downarrow$	LAB $\downarrow$
SID ( Le and Samaras, 2019 )	2.89	7.25	10.86	7.48	10.23	18.28	0.6404	23.77	8.96
ShadowFormer ( Guo et al., 2023a )	1.95	6.63	10.01	7.35	10.74	18.57	0.6592	23.36	8.68
Inpaint4Shadow ( Li et al., 2023a )	2.77	7.12	9.72	7.58	10.08	18.59	0.6119	23.06	8.63
StableSR ( Xu et al., 2025b )	2.03	7.32	8.84	10.01	10.93	17.90	0.6506	25.23	9.28

Table 4: Cross-domain shadow removal evaluation. Official ISTD+ checkpoints are evaluated without adaptation, while † denotes training with both ISTD+ and AgenticShadow. We report per-domain LAB RMSE and equal-weight macro averages across the four AgenticShadow domains. ISTD+ is reported separately. Best and second-best results are bolded and underlined , respectively.

Figure 5: Qualitative cross-domain comparison. We compare HomoFormer ( Xiao et al., 2024 ) and PhaSR ( Lee et al., 2026 ) , their counterparts trained with AgenticShadow ( † ), and PF. Models pretrained on ISTD+ ( Le and Samaras, 2019 ) often leave residual shadows or color shifts on diverse AgenticShadow scenes, while training with AgenticShadow substantially improves removal. PF consistently produces outputs closer to the targets than the AgenticShadow-trained counterparts. Colored insets show mean absolute RGB error using a shared scale.

Figure 6: Generalization to shadow videos. (a) PF is compared with pretrained HomoFormer ( Xiao et al., 2024 ) on ViSha ( Chen et al., 2021 ) using the provided masks. (b) PF-MF is compared with SP+M+I-Net on SBU-TimeLapse ( Le and Samaras, 2022 ) without input masks. In both settings, PF and PF-MF produce cleaner and more complete shadow removal than the compared methods.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Eligible	Selected	Usable	Generated	Train	Test	Total
					Final triplets
SBU	4,723	4,621	4,614	4,609	3,974	635	4,609
CUHK	10,500	10,430	10,430	10,425	8,335	2,087	10,422
ASFW	1,081	1,081	1,081	1,081	928	153	1,081
S-EO	19,162	1,026	1,026	1,026	880	146	1,026
Total	35,466	17,158	17,151	17,141	14,117	3,021	17,138

Appendix

Table 5: AgenticShadow processing and retention. Eligible denotes matched source image-mask pairs, selected denotes examples retained after duplicate review or crop selection, usable denotes inputs ready for target construction after source-specific preprocessing, and generated denotes successful target generation before regional correction.

Source	Generated	Pass	Retry	Multi	Single	Fallback
		Probe routing		Final selection
SBU	4,609	4,500	109	4,588	19	2
SBU	4,609	(97.64%)	(2.36%)	(99.54%)	(0.41%)	(0.04%)
CUHK	10,425	10,193	232	10,364	48	13
CUHK	10,425	(97.77%)	(2.23%)	(99.41%)	(0.46%)	(0.12%)
ASFW	1,081	1,081	0	1,081	0	0

Appendix

Table 6: Dataset-wide target-construction outcomes. Pass and retry describe the initial routing decision. Multi, single, and fallback are mutually exclusive selection outcomes. Percentages are computed within each source and overall.

Figure 7: Semantic scene diversity compared with ISTD+. (a) Joint PCA visualization of DINOv2-L/14 embeddings for all training images. (b) Mean pairwise DINOv2 cosine distance over 100 size-matched resamples of 1,000 images; higher values indicate greater semantic diversity. Distances are computed in the original embedding space rather than the two-dimensional projection.

Figure 8: Generic prompting and physics-motivated additions. Both settings use identical inputs, preservation requirements, evaluator schemas, and agentic control flow. Physics-motivated prompting additionally defines cast and attached shadows as illumination effects caused by light occlusion and provides type-specific guidance for preserving the underlying surface or object.

Figure 9: Construction against captured ISTD+ references. Each group shows the shadow input, our constructed target, the color-corrected ISTD+ reference, and the mean absolute RGB error map using a shared 0–50 scale. The top row presents two low-error cases, while the bottom row presents two higher-error cases that expose remaining texture, detail, and color discrepancies.

Figure 10: Image normalization examples. The first three pairs show low-resolution inputs standardized to $512\times 512$ , while the last pair shows a higher-resolution input normalized to the same resolution. The preprocessing pass improves resolution and image quality while preserving the scene and original shadow appearance.

Prompt setting	Cases	Pass	Retry	Edits	Evaluations	Selectors
Generic	399	391	8	1,205	1,205	394
Physics-motivated	399	389	10	1,207	1,207	397

Appendix

Table 7: API usage for complete SRR runs. Each run processes all 399 evaluation cases. Pass and retry indicate the routing decision for the initial probe. Edits include the probe and subsequent guided or retry candidates, and each edit receives one major-failure evaluation. Selector counts exclude cases with exactly one passing candidate, which are selected directly.

Panel of Figure 11 — Figure 11: Effect of major-failure retry. The evaluator rejects probes with severe residual shadow structure or object-like hallucination in the masked region. Feedback-guided retry produces cleaner candidates before final selection.

Panel of Figure 12 — Figure 12: Effect of shadow-formation grounding and visual candidate selection. Physics-motivated prompting reduces task-level errors and produces stronger candidate edits than generic prompting. All three physics-motivated candidates are shown, with the visually selected output in the final column.

Figure 13: Effect of Retinex-LoG-supported selection. Red frames indicate candidates selected through visual assessment alone, while green frames indicate the final selections obtained with Retinex-LoG structural evidence. The additional evidence helps reject candidates with residual shadows and select candidates that better preserve scene structure.

Figure 14: Effect of segmentation-guided regional correction. Selected proposals remove the dominant shadows but can retain spatially varying brightness or color shifts. Regional correction improves agreement with the surrounding scene while preserving image structure.

Figure 15: Representative construction failures. Each pair shows the mask-overlay input and constructed target. The first example exhibits texture smoothing and loss of valid scene detail. The second illustrates the recurring residual-shadow problem beneath vehicles. The third shows incomplete removal of an attached shadow.

Method	Checkpoint condition	Mask	Inference
SID ( Le and Samaras, 2019 )	Official ISTD+	Yes	Resize to $512\times 512$
ShadowFormer ( Guo et al., 2023a )	Official ISTD+	Yes	Resize to $640\times 480$
Inpaint4Shadow ( Li et al., 2023a )	Official ISTD+	Yes	Resize to $256\times 256$
StableSR ( Xu et al., 2025b )	Official ISTD+ / INS	No	Native resolution; 20 steps
StableSR ( Xu et al., 2025b )	Official ISTD+ / INS	No	Padded to a multiple of 16
ShadowDiffusion ( Guo et al., 2023b )	Official ISTD+	Yes	Full $512\times 512$ inference 25 DDIM steps

Appendix

Table 8: Checkpoint and inference settings. Combined denotes training with ISTD+ and AgenticShadow, while ISTD+ / INS ( Xu et al., 2025a ) denotes separate official checkpoints. Outputs are restored to the target resolution before evaluation.

Method	Online params (M)	Trainable params (M)	Time/image (s)	Peak VRAM (GiB)
ShadowDiffusion	55.52	55.52	2.558	1.51
HomoFormer	17.81	17.81	0.505	0.92
PhaSR	323.37	19.00	0.445	2.80
PF	658.93	19.25	0.999	3.33
PF-MF	658.61	18.92	0.978	3.32

Appendix

Table 9: Model complexity and inference efficiency. We compare ShadowDiffusion ( Guo et al., 2023b ) , HomoFormer ( Xiao et al., 2024 ) , PhaSR ( Lee et al., 2026 ) , PF, and PF-MF. Online parameters include all networks executed during inference. PhaSR uses precomputed geometry, whereas PF and PF-MF execute both frozen prior extractors online. Runtime and peak VRAM are measured on an NVIDIA RTX A6000 during end-to-end processing of

512\times 512

images.

Official ISTD+ checkpoints
	SBU				CUHK-Shadow
Method	PSNR $\uparrow$	SSIM $\uparrow$	MAE $\downarrow$	LAB $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	MAE $\downarrow$	LAB $\downarrow$
SID ( Le and Samaras, 2019 )	20.28	0.6258	18.26	7.25	16.32	0.5643	30.43	10.86
ShadowFormer ( Guo et al., 2023a )	21.02	0.6506	16.99	6.63	17.09	0.6008	28.50	10.01
Inpaint4Shadow ( Li et al., 2023a )	20.51	0.5883	18.30	7.12	17.54	0.5406	26.43	9.72
StableSR ( Xu et al., 2025b )	20.34	0.6345	18.83	7.32	18.38	0.6173	23.94	8.84

Appendix

Table 10: Full evaluation on the SBU and CUHK-Shadow domains. Official ISTD+ checkpoints are evaluated without adaptation, while † denotes training with ISTD+ and AgenticShadow. Best and second-best results are bolded and underlined , respectively.

Official ISTD+ checkpoints
	ASFW				S-EO
Method	PSNR $\uparrow$	SSIM $\uparrow$	MAE $\downarrow$	LAB $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	MAE $\downarrow$	LAB $\downarrow$
SID ( Le and Samaras, 2019 )	20.02	0.7687	19.38	7.48	16.50	0.6027	27.02	10.23
ShadowFormer ( Guo et al., 2023a )	20.18	0.7801	19.36	7.35	16.01	0.6052	28.57	10.74
Inpaint4Shadow ( Li et al., 2023a )	19.90	0.7523	19.88	7.58	16.43	0.5665	27.62	10.08
StableSR ( Xu et al., 2025b )	17.14	0.7367	28.16	10.01	15.75	0.6140	29.98	10.93

Appendix

Table 11: Full evaluation on the ASFW and S-EO domains. Official ISTD+ checkpoints are evaluated without adaptation, while † denotes training with ISTD+ and AgenticShadow. Best and second-best results are bolded and underlined , respectively.

		ISTD+				AgenticShadow
Model	Checkpoint	PSNR $\uparrow$	SSIM $\uparrow$	MAE $\downarrow$	LAB $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	MAE $\downarrow$	LAB $\downarrow$
StableSR	ISTD+	32.23	0.9272	4.84	2.03	17.90	0.6506	25.23	9.28
StableSR	INS	22.76	0.8986	12.14	5.82	16.55	0.6350	28.93	11.12
PhaSR	ISTD+	32.12	0.9330	4.82	2.13	17.90	0.6556	24.75	9.33
PhaSR	INS	26.03	0.9129	8.93	4.32	17.24	0.6415	26.90	10.37

Appendix

Table 12: Synthetic-to-real transfer. We compare official ISTD+ and INS ( Xu et al., 2025a ) checkpoints of StableSR ( Xu et al., 2025b ) and PhaSR ( Lee et al., 2026 ) . AgenticShadow results are four-domain macro averages.

	ISTD+		AgenticShadow
Training data	PSNR $\uparrow$	LAB $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	MAE $\downarrow$	LAB $\downarrow$
ISTD+	32.53	1.92	18.85	0.6597	22.36	8.43
AgenticShadow	27.96	2.74	21.85	0.7152	15.45	5.94
Combined	32.92	1.85	21.95	0.7181	15.25	5.87

Appendix

Table 13: Effect of HomoFormer training data. The AgenticShadow and Combined conditions are initialized from the official ISTD+ checkpoint; Combined denotes training with ISTD+ and AgenticShadow. AgenticShadow results are four-domain macro averages.

Variant	ISTD+	SBU	CUHK	ASFW	S-EO	Macro
HomoFormer	1.860	4.907	6.266	5.434	7.041	5.912
Depth only	1.859	4.835	6.155	5.122	6.964	5.769
Semantic only	1.857	4.731	6.051	4.750	6.807	5.585
PF	1.861	4.723	6.045	4.737	6.804	5.577

Appendix

Table 14: Per-domain PF component analysis. All values are LAB RMSE. Each variant is initialized from HomoFormer E100 and trained for 50 additional epochs; Macro is the unweighted average of the four AgenticShadow domains.

Figure 16: Qualitative comparison with official ISTD+ checkpoints. We compare SID ( Le and Samaras, 2019 ) , ShadowFormer ( Guo et al., 2023a ) , Inpaint4Shadow ( Li et al., 2023a ) , and ShadowDiffusion ( Guo et al., 2023b ) with PF across the four AgenticShadow domains. The official checkpoints frequently leave residual shadows or introduce color shifts, whereas PF produces more complete removal and outputs closer to the constructed targets.

Figure 17: Qualitative comparison of ISTD+ and INS checkpoints. We compare the official ISTD+ and synthetic INS ( Xu et al., 2025a ) checkpoints of StableSR ( Xu et al., 2025b ) and PhaSR ( Lee et al., 2026 ) with PF. The INS-trained models continue to exhibit residual shadows and appearance shifts on real-world AgenticShadow scenes, illustrating that large-scale synthetic supervision does not eliminate the synthetic-to-real gap.

Figure 18: Additional qualitative comparison of AgenticShadow supervision. We compare the official ISTD+ checkpoints of HomoFormer ( Xiao et al., 2024 ) and PhaSR ( Lee et al., 2026 ) , their counterparts trained with ISTD+ and AgenticShadow ( † ), and PF-MF. AgenticShadow supervision produces cleaner and more complete shadow removal, while PF-MF achieves strong results without input masks. Colored insets show mean absolute RGB error using a shared scale.

Explore similar work

ODDR: One-Step Deshadow Diffusion via Reward Guidance
Oct 1, 2026Junseong Shin, Kijun Kim, Minseong Kim +2Video Object Removal Diverse Occlusion-And-Revelation Scenarios
Department of Artificial Intelligence, Hanyang University · AX Future Technology Institute, KT · Department of Computer Science, Hanyang University
Consistency as Regularization for Unsupervised Shadow Removal
Sep 1, 2026Anh-Kiet Duong, Petra Gomez-Krämer, Jean-Michel CarozzaVideo Object Removal Unsupervised
L3i Laboratory, La Rochelle University, 17042 La Rochelle Cedex 1, France · LIENSs Laboratory, La Rochelle University, 17042 La Rochelle Cedex 1, France
Domain-Grounded Candidate Selection for Agentic Image Editing: A Shadow Removal Case
Aug 6, 2026Shilin Hu, Jingyi Xu, Dimitris Samaras +1Image Editing Diverse Occlusion-And-Revelation Scenarios
Stony Brook University, Stony Brook NY 11794, USA · UNC Charlotte, Charlotte NC 28223, USA

Oct 1, 2026, cs.CVJ/K move · Enter open · S save

ODDR: One-Step Deshadow Diffusion via Reward Guidance

Junseong Shin, Kijun Kim, Minseong Kim, Dongjin Kim+1

Department of Artificial Intelligence, Hanyang University · AX Future Technology Institute, KT · Department of Computer Science, Hanyang University

Video Object Removal Diverse Occlusion-And-Revelation Scenarios Synthetic Data

Open paper AskPDF arXiv

Recent advances in deep learning for shadow removal have significantly enhanced image quality and realism. However, most approaches rely on real-world paired datasets, which are costly to collect and often limited in scene diversity, leading to limited generalization. To address these limitations, we propose One-step Deshadow Diffusion via Reward guidance (ODDR), a new framework that achieves efficient and high-fidelity shadow removal without relying on real-world paired supervision. Our method begins with One-step Deshadow Diffusion (ODD), a baseline model trained on synthetic shadow data for efficient one-step shadow-free reconstruction. We further adapt ODD into ODDR using ShadowReward. In contrast to traditional, annotation-heavy approaches, ShadowReward is the first reward model for shadow removal trained entirely without human annotation. It learns to mimic human perceptual judgments by ranking synthetically generated images with controlled degradations, such as texture distortion and boundary artifacts. This reward-guided fine-tuning enables ODDR to close the synthetic-to-real domain gap. Extensive experiments show that ODD achieves strong performance without relying on real-world paired supervision, and ODDR further improves the results, narrowing the gap to fully supervised methods trained on real-world paired data while maintaining higher computational efficiency as a single-step model.

Abstract

Explore similar work

ODDR: One-Step Deshadow Diffusion via Reward Guidance

Consistency as Regularization for Unsupervised Shadow Removal

Domain-Grounded Candidate Selection for Agentic Image Editing: A Shadow Removal Case