cs.CVOct 22, 2026

Where Did the Light Come From? Diagnosing Cast-Shadow Reasoning in VLMs and Borrowing the Missing Shadow Prior from Image Generation Models

Authors: Ryugo Morita, Shunsuke Kitada, Stanislav Frolov, Brian Bernhard Moser, Ko Watanabe, Riku Takahashi, Jinjia Zhou, Andreas Dengel

Organizations: DFKI, Germany · RPTU Kaiserslautern-Landau, Germany · Hosei University, Japan

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Shadows are observable consequences of latent physical causes, arising from the interaction between illumination, object geometry, and scene structure. A model that understands shadows should be able not only to read a shadow's direction, but also to infer where a shadow should fall when it is absent, and to judge whether a shadow is physically consistent with the scene. We introduce ShadowBench, built from real photographs, which poses these three questions to vision-language models (VLMs). From each scene, we build a correct-shadow, a shadow-removed, and a human-annotated wrong-direction state. Evaluating 19 open VLMs (2B to 78B) and frontier proprietary models, we find that VLMs read a visible shadow's direction well above chance and improve with scale, yet infer the absent direction only weakly and fall to chance at judging physical consistency even under a bias-free two-alternative test that removes the yes-bias of single-image naturalness questions. Only Gemini-3.1-pro exceeds chance on validity, and even then only modestly and far below humans. Chain-of-Thought (CoT) does not help. The missing ingredient is not reasoning but knowledge that image generation models (IGMs) already hold. Routing a VLM through an IGM lifts its inference well above its direct attempt and helps it flag the wrong-direction shadows it otherwise accepts. Cast-shadow direction is a single facet of illumination understanding, but the sharp gap it reveals between VLMs and IGMs suggests image-generation or reconstruction objectives as a route toward illumination-aware VLMs.

Figures & tables

Figure 2: Overview of ShadowBench . From each real photograph, we recover the object and its cast shadow as an alpha-matted layer and recompose matched states on a shared background: the shadow-correct IsdwI_{\mathrm{sdw}} , the shadow-removed IobjI_{\mathrm{obj}} , and the physically-wrong composite IflipI_{\mathrm{flip}} , produced by a horizontal flip to change the shadow’s direction. The target object is highlighted in red. These states drive three tasks of increasing demand on the shadow–illumination relation: Direction Reading from a visible shadow at increasing angular resolution (Task 1); Direction Inference , recovering the direction from IobjI_{\mathrm{obj}} once the shadow is removed (Task 2); and Physical Validity , judging whether a shown shadow is consistent with the scene, in single-image and a bias-free two-alternative form on IsdwI_{\mathrm{sdw}} vs. IflipI_{\mathrm{flip}} (Task 3).
StateSourceShadowPhys.Used in
IsdwI_{\mathrm{sdw}}compositecorrectcorrectTask 1,3
IobjI_{\mathrm{obj}}compositeremovedwrongTask 2,3
IflipI_{\mathrm{flip}}compositemirroredwrongTask 3
Table 1: Canonical scene states in ShadowBench. IsdwI_{\mathrm{sdw}} serves as the direction- reading stimulus and the validity positive, IflipI_{\mathrm{flip}} as the wrong-direction negative with the foreground mirrored, and IobjI_{\mathrm{obj}} as the inference input and the negative of the presence 2AFC.
 Reading (T1)Infer (T2)Validity (T3)
Model2w3w5w3wY/N2AFC
≤4\leq 4 B (2–4B)
Qwen2-VL-2B50.033.320.928.550.050.0
Qwen3-VL-2B68.435.718.133.050.050.0
Qwen2.5-VL-3B50.047.227.333.850.350.0
Gemma-4-E4B69.761.833.734.449.750.0
Table 2: Main results on ShadowBench (balanced accuracy, %), all 1919 open and 22 proprietary models. Reading (T1) is at two-/three-/five-way resolution; Infer (T2) is three-way with the shadow removed; Validity (T3) is single-image Yes/No and the bias-free 2AFC. †\dagger : verbose non-parseable answers, scored as incorrect. Reading is well above chance and scales with size, while inference stays near chance and validity is at chance for every model except Gemini-3.1-pro ( 63.0%63.0\% ), far below humans.
 Direction 3-wayValidity Y/NValidity 2AFC
ModelReadInfertopbaltopbaltopboth ✓\checkmark
Gemma-4-12B85.842.0R 7550.0N 10056.0B 5229.2
Gemma-4-26B86.643.8R 8452.5N 9051.9B 6818.8
Qwen3-VL-30B63.137.0L 8151.4Y 9452.5B 939.4
Gemma-4-31B86.842.0R 8751.3N 5050.3B 982.2
InternVL3-38B58.433.6R 9851.6Y 7651.4B 937.5
Table 3: The read-to-infer dissociation, per class. Read / Infer are three-way balanced accuracy with the shadow present/removed; top is the most-predicted class and its share (for validity: Yes/No, or side A/B); both ✓\checkmark is the share of pairs answered correctly under both 2AFC orders, where a fixed-side strategy scores 0%0\% and random guessing 25%25\% . Once the shadow is removed, inference collapses onto a single direction ( top 5757 – 98%98\% ) and balanced accuracy falls toward chance; single-image validity collapses to a fixed Yes/No and the 2AFC to a fixed side. The full panel is in Table 9 .
TaskChanceBaseZS-CoT Δ\DeltaCtx-CoT Δ\Delta
Reading 2-way50.066.5−1.3-1.3−3.3-3.3
Reading 3-way33.350.1−0.6-0.6−2.5-2.5
Reading 5-way20.033.9−0.7-0.7−2.8-2.8
Inference 3-way33.335.6+0.4+0.4+0.4+0.4
Validity Y/N50.050.6−2.2-2.2−0.1-0.1
Validity 2AFC50.051.4−1.0-1.0−0.4-0.4
Table 4: CoT ablation, mean balanced accuracy (%). Base is the non-CoT score, ZS-CoT is zero-shot CoT, and Ctx-CoT is a task-specific scaffold pointing to the relevant lighting cues (comprehension tasks only). Means are over 1919 open models, with InternVL3-8B excluded from the 2AFC for its parser floor, as in Table 10 .
 Direction inferencePhysical validity
ModelDirect++ GenΔ\DeltaDirect++ GenΔ\Delta
≤4\leq 4 B (2–4B)
Qwen2-VL-2B28.533.3+4.8+4.850.050.0+0.0+0.0
Qwen3-VL-2B33.033.6+0.6+0.650.065.9+15.9+15.9
Qwen2.5-VL-3B33.847.6+13.8+13.850.050.0+0.0+0.0
Gemma-4-E4B34.452.1+17.7+17.750.069.7+19.7+19.7
Table 5: Borrowing the IGM’s shadow prior recovers both comprehension tasks (balanced accuracy %; chance 33.333.3 / 5050 ). Direct : the VLM alone (Infer column of Table 2 ; 2AFC on IsdwI_{\mathrm{sdw}} vs. IflipI_{\mathrm{flip}} ). ++ Gen : the IGM places the shadow the scene implies, and the VLM reads it (inference) or matches the candidates against it (validity). Both tasks recover well above chance for every model above the 22 – 33 B tier. † Parser floor on the direct 2AFC; Δ\Delta omitted.
ModelIallI_{\mathrm{all}}IsdwI_{\mathrm{sdw}}Δ\Delta
Qwen2-VL-2B33.333.3+0.0+0.0
Qwen3-VL-2B33.335.7+2.4+2.4
Qwen2.5-VL-3B46.747.2+0.5+0.5
Gemma-4-E4B59.161.8+2.7+2.7
Qwen2-VL-7B73.271.6−1.6-1.6
Qwen2.5-VL-7B49.848.1−1.7-1.7
Table 6: Reading is the same on the capture IallI_{\mathrm{all}} and the composite IsdwI_{\mathrm{sdw}} . Three-way direction reading on each image source and their difference Δ\Delta . The means differ by +1.1+1.1 points, so the reading ceiling is unchanged by the choice of source.
Figure 3: Shadow-detection psychometric curve. The object’s own cast shadow is recomposed at opacities from 00 (no shadow) to 11 (full), and at each level we ask whether the object casts a visible shadow. For the three models shown (InternVL3-8B, Qwen2.5-VL-7B, Qwen3-VL-8B), the detection rate rises smoothly and monotonically with opacity, showing that shadow presence is perceived as a graded quantity that scales with physical strength. Some models instead answer “yes” at every opacity, a response bias rather than a perceptual deficit.
 Shadow length 
Taskshortlongchance
 <1.15<1.15>1.15>1.15 
Reading 2-way73.975.150.0
Reading 3-way61.064.433.3
Inference 3-way36.541.933.3
Table 7: Reading and inference accuracy stratified by shadow length, split at the median shadow reach/object height\text{shadow reach}/\text{object height} ( 1.151.15 ), the mean over the 1919 open models, 229229 scenes per half. Every task improves monotonically with shadow length, and inference on short-shadow scenes sits near the 33%33\% chance level. Bold marks the better half per row.
 PresenceDirection 
Model2AFC2AFCΔ\Delta
Qwen2-VL-2B58.050.0+8.0+8.0
Qwen3-VL-2B51.950.0+1.9+1.9
Qwen2.5-VL-3B54.750.0+4.7+4.7
Gemma-4-E4B85.050.0+35.0+35.0
Qwen2-VL-7B84.651.3+33.3+33.3
Table 8: Presence vs. direction validity 2AFC on the 458458 validity scenes ( 916916 counterbalanced records each). Presence pits the correct-shadow image against the same object casting no shadow. Direction is the main-text validity 2AFC, pitting the correct shadow against a wrong- direction one. Both are counterbalanced, so a constant side cancels to the 50%50\% chance. Every open model above the 22 – 33 B tier is far above chance on presence yet at chance on direction. The gap Δ\Delta isolates shadow direction , not shadow presence, as the locus of the validity failure. †\dagger : InternVL3-8B returns a verbose string rather than a single letter on the direction 2AFC, so that score is a parser floor and its Δ\Delta is omitted. This means we exclude InternVL3-8B, as in Tables 4 and 5 , and the mean Δ\Delta is the difference between the two column means.
 Direction 3-wayValidity Y/NValidity 2AFC
ModelReadInferrec L/D/Rtopbaltopbaltopboth ✓\checkmark
Qwen2-VL-2B33.328.50/67/18D 7350.0Y 10050.0A 1000.0
Qwen3-VL-2B35.733.00/0/99R 10050.0Y 10050.0B 1000.0
Qwen2.5-VL-3B47.233.81/1/100R 10050.3Y 10050.0A 1000.3
Gemma-4-E4B61.834.40/5/98R 9649.7Y 6250.0B 1000.0
Qwen2-VL-7B71.641.79/59/57R 5150.6Y 9851.3A 849.4
Table 9: The read-to-infer dissociation, per class, full panel. Columns as in Table 3 : Read / Infer are three-way direction balanced accuracy with the shadow present/removed, rec L/D/R the per-direction inference recall, and top the single most-predicted class with the percentage of items assigned to it. Open models are listed by size, the two proprietary models below. Reading scales with size while inference stays near the 33.3%33.3\% chance for all but Gemini-3.1-pro, and the gap is largest for the strongest readers; the single-image validity and the 2AFC likewise collapse onto a fixed class. † InternVL3-8B emits a verbose string rather than a single letter on the 2AFC, a parser floor, so its top class is undefined. both ✓\checkmark : share of pairs answered correctly under both presentation orders of the 2AFC; a fixed-side strategy scores 0%0\% and random guessing 25%25\% , so only Gemini-3.1-pro clearly exceeds the random reference.
 Inference 3-wayValidity Y/NValidity 2AFC
ModelBaseZS- Δ\DeltaCtx- Δ\DeltaBaseZS- Δ\DeltaCtx- Δ\DeltaBaseZS- Δ\DeltaCtx- Δ\Delta
Qwen2-VL-2B28.5+2.5+2.5+4.8+4.850.0+1.1+1.1−1.9-1.950.0+0.0+0.0+0.0+0.0
Qwen3-VL-2B33.0−1.1-1.1+0.4+0.450.0−9.1-9.1+0.0+0.050.0+0.2+0.2+1.2+1.2
Qwen2.5-VL-3B33.8−1.0-1.0+3.6+3.650.3+0.5+0.5−0.3-0.350.0−0.9-0.9+0.0+0.0
Gemma-4-E4B34.4+0.3+0.3+2.1+2.149.7−0.6-0.6+1.0+1.050.0+0.0+0.0+0.0+0.0
Qwen2-VL-7B41.7−4.5-4.5−7.6-7.650.6−0.2-0.2+1.7+1.751.3+0.0+0.0+0.5+0.5
Table 10: Per-model CoT ablation on the three comprehension tasks, balanced accuracy (%). Base is the non-CoT score; ZS- Δ\Delta and Ctx- Δ\Delta are the change under generic zero-shot CoT and the task-specific contextual scaffold. Deltas are large in magnitude but sign-inconsistent across models and centered near zero, the signature of added noise rather than added competence; this is the per-model spread summarized in Table 4 . † InternVL3-8B emits no parseable A/B on the 2AFC without CoT (base is a format artifact, not a judgment), so its deltas are omitted and it is excluded from the 2AFC column statistics.
EditorType3-way
FLUX.1-Kontext-devopen79.7
OmniGen2open61.0
gpt-image-2prop.73.4
Gemini-3-pro-imageprop.79.0
Chance 33.3
Table 11: Editor-intrinsic shadow-direction accuracy on the 458458 scenes, scored through the same VLM-driven loop as the main text. Each IGM receives the VLM-authored editing prompt, which never states the shadow direction, and adds a cast shadow to the shadow-removed input; a single fixed VLM reader (Gemma-4-31B) then reports the added shadow’s three-way direction (left/down/right), which we compare to the human GT. This is the same loop, metric, and reader as the ++ Gen inference column of Table 5 (chance 33.333.3 ); the FLUX entry is that same pipeline. prop. is proprietary; Gemini-3-pro-image is nano-banana-pro. Bold is the best. All four IGMs exceed three-way chance by a wide margin; FLUX.1-Kontext-dev is best overall, edging the proprietary Gemini-3-pro-image while remaining open and reproducible, so we route it through the pipeline.
Figure 4: Naturalising the shadow-only rotation with an image editor. (a) the negative a bare rotation produces: the soft shadow matte is rotated about the annotated origin and recomposited, the object pixels left untouched. An image-space rotation cannot yield a silhouette the pictured object could actually cast, and it disturbs the fine compositing statistics along the shadow’s edge – cues a model can exploit without consulting the scene’s lighting at all. We therefore rebuild every negative: (b) the object and its rotated shadow are lifted onto a white canvas, the scene removed so the editor has no lighting context to re-infer the direction from; (c) FLUX.1-Kontext-dev redraws the shadow’s shape under an instruction to keep its direction, length and size, and the result is clamped to the seed’s rotated support so the direction cannot drift; (d) the redrawn shadow is recomposited into the captured background plate, with the object pixels taken unchanged from the positive so that the shadow is the only difference within a pair. Detectability falls from 64.0%64.0\% on (a) to 56.3%56.3\% on (d), while humans answer the same pairs at 94.4%94.4\% (Table 12 ).
Curation levelPooled panelTop-5 models
None (1,374 pairs)64.076.8
Shadow mass ≥ ⁣0.8\geq\!0.8 (200)60.769.2
++ clean compositing (112)57.162.4
Editor-refined (1,374)56.363.8
++ shadow mass ≥ ⁣0.8\geq\!0.8 (358)54.658.3
++ artifact-free (98)52.355.1
Table 12: Shadow-only rotation 2AFC under successively stricter curation (balanced accuracy %, chance 5050 ; open panel of Table 2 , InternVL3-8B excluded from means for its parser floor as there). Top-5 follows the five best models on each construction’s uncurated set through curation. Top block: human curation of the raw rotation (equated shadow mass, then clean compositing). Middle block: the editor-refined construction, in which FLUX.1-Kontext-dev redraws the rotated shadow’s shape on a white canvas (scene hidden) before recompositing; three annotators retain 9898 of 1,3741{,}374 edits as artifact-free. Removing the non-lighting leaks (shadow-mass asymmetry, broken compositing, residual artifacts) drives accuracy monotonically toward chance along both routes, while the mirror construction, which has none of these leaks, sits at chance.
 Reading 3-wayInference 3-way
Modelfixedshuffledfixedshuffled
Qwen2-VL-2B33.333.928.532.0
Qwen3-VL-2B35.734.033.033.1
Qwen2.5-VL-3B47.246.333.832.3
Gemma-4-E4B61.859.134.435.4
Qwen2-VL-7B71.668.441.736.4
Table 13: Shuffling the option-word order changes no conclusion (balanced accuracy %). Reading and inference rerun with the direction words listed in a per-scene random order.
 Random directionGT comp.IGM (FLUX)
Modelvs. GTvs. injectedvs. GTvs. GT
Gemma-4-E4B32.049.961.852.1
Qwen2.5-VL-7B28.841.348.150.4
Qwen3-VL-8B29.750.966.763.0
Gemma-4-12B32.467.285.877.2
Pixtral-12B30.738.445.651.3
Table 14: Only the injected direction carries signal (balanced accuracy %, chance 33.333.3 ). Reading a composited shadow at a random rotation is at chance against the scene’s truth yet far above chance against the direction actually injected; the identical composite in the ground-truth direction reaches 67.9%67.9\% ; the IGM-synthesized shadow ( ++ Gen of Table 5 , same models) is read at 65.0%65.0\% , at ground-truth-composite level.
Edit setHuman 3-way acc. (%)
Synthesis (FLUX)85.9
Repair (FLUX)83.2
Table 15: Human-rated shadow direction of the FLUX edits (chance 33.3%33.3\% ).
Figure 5: Failure cases. Top: direction inference on IobjI_{\mathrm{obj}} (Task 2). Bottom: single-image validity on IflipI_{\mathrm{flip}} (Task 3). The target object is highlighted in red, as in the evaluation prompts.
More in this topic

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

Explore similar work

CardsList
Oct 9, 2026

OccludeBench: Can VLMs Find What's Missing?

How well can vision-language models (VLMs) reason about what is missing from an image? We study this question with OccludeBench, a manually curated benchmark for recovering fully occluded objects from the visual evidence that remains. OccludeBench distinguishes direct cases, where the hidden object must be recovered from a specific physical cue such as a shadow or reflection, from inferential cases, where it can be inferred from broader scene context. Across five VLMs, accuracy averages 42.7% on direct cases versus 62.7% on inferential cases, exhibiting a 20.0-point gap. Humans attain higher accuracies of 63.4% versus 68.5% for direct and inferential images, respectively, with no significant gap. To locate this failure, we introduce OccludeTrace, a four-stage diagnostic that isolates where the visual reasoning process breaks down. The bottleneck emerges when models must recover the relevant object from the projection within the full image: they succeed on only 50% of failed examples, compared with 91% of examples they answer correctly. Extended reasoning, tool use, and few-shot prompting yield inconsistent improvements and do not reliably eliminate the gap. In contrast, an inpaint-then-confront intervention, which prompts models to first generate a candidate hidden object and then evaluate that hypothesis against the visible physical cue, reduces the gap to a statistically non-significant difference for both models with image-generation capabilities. Together, our results establish a tractable, controlled diagnostic for isolating how VLMs reason about visual content that is missing from an image.

Jasin Cekinmez, Addison J. Wu, Ryo Mitsuhashi +3

Princeton University

Occlusion ReasoningVLM ReasoningVLM Evaluation
AskPDF
Dec 5, 2025

Embedding Physical Reasoning into Diffusion-Based Shadow Generation Under the Sun and Sky

Generating realistic cast shadows for inserted foreground objects requires reasoning about scene geometry and illumination. However, most learning-based approaches treat shadow generation as an image translation problem and capture these physical relationships only implicitly. This often results in misaligned or implausible shadows. Motivated by the physics of shadow formation, we introduce explicit geometric guidance for outdoor shadow generation. Given a composite image and a foreground object mask, we recover approximate scene geometry and estimate a dominant light direction to derive a coarse shadow estimate via geometric reasoning. While coarse, this estimate provides a spatial anchor for shadow placement. Because illumination cannot always be uniquely inferred from a single image, we predict confidence scores for both lighting and shadow cues and use them to regulate their influence during generation. These cues (shadow mask, light direction, and their confidence scores) condition a diffusion-based generator that refines the estimate into a realistic shadow. Experiments on DESOBAV2 show substantially improved shadow-region fidelity and localization, with an overall 23% lower shadow-region RMSE and 30% lower shadow-mask BER than the prior state-of-the-art method.

Shilin Hu, Jingyi Xu, Akshat Dave +2

Stony Brook University, Stony Brook, NY 11794, USA · UNC Charlotte, Charlotte, NC 28223, USA

Conditional Image GenerationDiffusion-Based Image GenerationImage Editing
AskPDF
Aug 6, 2026

Domain-Grounded Candidate Selection for Agentic Image Editing: A Shadow Removal Case

Commercial vision-language models are reshaping computer vision, with visual priors broad enough to rival task-specific systems. This raises a natural question: do they reduce the need for classic, physics-informed low-level vision? We study this through shadow removal, a problem shaped by scene geometry, illumination, materials, and occluders, where paired shadow and shadow-free data are hard to collect at scale. We find that a commercial generative editor, used directly, can produce clean shadow-free edits that preserve surface texture and local appearance. However, this comes with a new failure mode: the same editor can regenerate scene content, hallucinate objects, or misread a shadow as material or geometry, producing plausible but physically wrong edits. We address this with an agentic candidate-selection pipeline: the editor generates a guided probe, an evaluator screens for major failures, retries when needed, samples multiple candidates, filters them, and selects a final result balancing shadow removal against scene preservation. Grounding this process in shadow-formation physics makes it more reliable: prompting the generator and evaluator to treat shadows as illumination effects caused by light occlusion, not material or object structure, measurably improves quality and consistency. On the ShadowRemovalRefine benchmark, our physics-oriented pipeline achieves a CDD of 0.0075, reducing CDD by at least 47% over the strongest prior method. These results suggest that commercial vision-language models do not replace classic low-level vision priors; instead, such priors remain useful for constraining and steering physically underconstrained generation.

Shilin Hu, Jingyi Xu, Dimitris Samaras +1

Stony Brook University, Stony Brook NY 11794, USA · UNC Charlotte, Charlotte NC 28223, USA

Shadow RemovalPhysics-Guided Image RestorationImage RestorationImage EditingVLM Hallucination Mitigation
AskPDF