Where Did the Light Come From? Diagnosing Cast-Shadow Reasoning in VLMs and Borrowing the Missing Shadow Prior from Image Generation Models

Ryugo Morita; Shunsuke Kitada; Stanislav Frolov; Brian Bernhard Moser; Ko Watanabe; Riku Takahashi; Jinjia Zhou; Andreas Dengel

cs.CVOct 22, 2026

Where Did the Light Come From? Diagnosing Cast-Shadow Reasoning in VLMs and Borrowing the Missing Shadow Prior from Image Generation Models

Authors: Ryugo Morita, Shunsuke Kitada, Stanislav Frolov, Brian Bernhard Moser, Ko Watanabe, Riku Takahashi, Jinjia Zhou, Andreas Dengel

Organizations: DFKI, Germany · RPTU Kaiserslautern-Landau, Germany · Hosei University, Japan

Abstract

Shadows are observable consequences of latent physical causes, arising from the interaction between illumination, object geometry, and scene structure. A model that understands shadows should be able not only to read a shadow's direction, but also to infer where a shadow should fall when it is absent, and to judge whether a shadow is physically consistent with the scene. We introduce ShadowBench, built from real photographs, which poses these three questions to vision-language models (VLMs). From each scene, we build a correct-shadow, a shadow-removed, and a human-annotated wrong-direction state. Evaluating 19 open VLMs (2B to 78B) and frontier proprietary models, we find that VLMs read a visible shadow's direction well above chance and improve with scale, yet infer the absent direction only weakly and fall to chance at judging physical consistency even under a bias-free two-alternative test that removes the yes-bias of single-image naturalness questions. Only Gemini-3.1-pro exceeds chance on validity, and even then only modestly and far below humans. Chain-of-Thought (CoT) does not help. The missing ingredient is not reasoning but knowledge that image generation models (IGMs) already hold. Routing a VLM through an IGM lifts its inference well above its direct attempt and helps it flag the wrong-direction shadows it otherwise accepts. Cast-shadow direction is a single facet of illumination understanding, but the sharp gap it reveals between VLMs and IGMs suggests image-generation or reconstruction objectives as a route toward illumination-aware VLMs.

Figures & tables

Figure 2: Overview of ShadowBench . From each real photograph, we recover the object and its cast shadow as an alpha-matted layer and recompose matched states on a shared background: the shadow-correct $I_{\mathrm{sdw}}$ , the shadow-removed $I_{\mathrm{obj}}$ , and the physically-wrong composite $I_{\mathrm{flip}}$ , produced by a horizontal flip to change the shadow’s direction. The target object is highlighted in red. These states drive three tasks of increasing demand on the shadow–illumination relation: Direction Reading from a visible shadow at increasing angular resolution (Task 1); Direction Inference , recovering the direction from $I_{\mathrm{obj}}$ once the shadow is removed (Task 2); and Physical Validity , judging whether a shown shadow is consistent with the scene, in single-image and a bias-free two-alternative form on $I_{\mathrm{sdw}}$ vs. $I_{\mathrm{flip}}$ (Task 3).

State	Source	Shadow	Phys.	Used in
$I_{\mathrm{sdw}}$	composite	correct	correct	Task 1,3
$I_{\mathrm{obj}}$	composite	removed	wrong	Task 2,3
$I_{\mathrm{flip}}$	composite	mirrored	wrong	Task 3

Table 1: Canonical scene states in ShadowBench.

I_{\mathrm{sdw}}

serves as the direction- reading stimulus and the validity positive,

I_{\mathrm{flip}}

as the wrong-direction negative with the foreground mirrored, and

I_{\mathrm{obj}}

as the inference input and the negative of the presence 2AFC.

$\leq 4$ B (2–4B)
	Reading (T1)			Infer (T2)	Validity (T3)
Model	2w	3w	5w	3w	Y/N	2AFC
Qwen2-VL-2B	50.0	33.3	20.9	28.5	50.0	50.0
Qwen3-VL-2B	68.4	35.7	18.1	33.0	50.0	50.0
Qwen2.5-VL-3B	50.0	47.2	27.3	33.8	50.3	50.0
Gemma-4-E4B	69.7	61.8	33.7	34.4	49.7	50.0

Table 2: Main results on ShadowBench (balanced accuracy, %), all

19

open and

2

proprietary models. Reading (T1) is at two-/three-/five-way resolution; Infer (T2) is three-way with the shadow removed; Validity (T3) is single-image Yes/No and the bias-free 2AFC.

\dagger

: verbose non-parseable answers, scored as incorrect. Reading is well above chance and scales with size, while inference stays near chance and validity is at chance for every model except Gemini-3.1-pro (

63.0\%

), far below humans.

	Direction 3-way			Validity Y/N		Validity 2AFC
Model	Read	Infer	top	bal	top	bal	top	both $\checkmark$
Gemma-4-12B	85.8	42.0	R 75	50.0	N 100	56.0	B 52	29.2
Gemma-4-26B	86.6	43.8	R 84	52.5	N 90	51.9	B 68	18.8
Qwen3-VL-30B	63.1	37.0	L 81	51.4	Y 94	52.5	B 93	9.4
Gemma-4-31B	86.8	42.0	R 87	51.3	N 50	50.3	B 98	2.2
InternVL3-38B	58.4	33.6	R 98	51.6	Y 76	51.4	B 93	7.5

Table 3: The read-to-infer dissociation, per class. Read / Infer are three-way balanced accuracy with the shadow present/removed; top is the most-predicted class and its share (for validity: Yes/No, or side A/B); both

\checkmark

is the share of pairs answered correctly under both 2AFC orders, where a fixed-side strategy scores

0\%

and random guessing

25\%

. Once the shadow is removed, inference collapses onto a single direction ( top

57

–

98\%

) and balanced accuracy falls toward chance; single-image validity collapses to a fixed Yes/No and the 2AFC to a fixed side. The full panel is in Table 9 .

Task	Chance	Base	ZS-CoT $\Delta$	Ctx-CoT $\Delta$
Reading 2-way	50.0	66.5	$-1.3$	$-3.3$
Reading 3-way	33.3	50.1	$-0.6$	$-2.5$
Reading 5-way	20.0	33.9	$-0.7$	$-2.8$
Inference 3-way	33.3	35.6	$+0.4$	$+0.4$
Validity Y/N	50.0	50.6	$-2.2$	$-0.1$
Validity 2AFC	50.0	51.4	$-1.0$	$-0.4$

Table 4: CoT ablation, mean balanced accuracy (%). Base is the non-CoT score, ZS-CoT is zero-shot CoT, and Ctx-CoT is a task-specific scaffold pointing to the relevant lighting cues (comprehension tasks only). Means are over

19

open models, with InternVL3-8B excluded from the 2AFC for its parser floor, as in Table 10 .

	Direction inference			Physical validity
Model	Direct	$+$ Gen	$\Delta$	Direct	$+$ Gen	$\Delta$
$\leq 4$ B (2–4B)
Qwen2-VL-2B	28.5	33.3	$+4.8$	50.0	50.0	$+0.0$
Qwen3-VL-2B	33.0	33.6	$+0.6$	50.0	65.9	$+15.9$
Qwen2.5-VL-3B	33.8	47.6	$+13.8$	50.0	50.0	$+0.0$
Gemma-4-E4B	34.4	52.1	$+17.7$	50.0	69.7	$+19.7$

Table 5: Borrowing the IGM’s shadow prior recovers both comprehension tasks (balanced accuracy %; chance

33.3

50

). Direct : the VLM alone (Infer column of Table 2 ; 2AFC on

I_{\mathrm{sdw}}

vs.

I_{\mathrm{flip}}

+

Gen : the IGM places the shadow the scene implies, and the VLM reads it (inference) or matches the candidates against it (validity). Both tasks recover well above chance for every model above the

2

–

3

B tier. † Parser floor on the direct 2AFC;

\Delta

omitted.

Model	$I_{\mathrm{all}}$	$I_{\mathrm{sdw}}$	$\Delta$
Qwen2-VL-2B	33.3	33.3	$+0.0$
Qwen3-VL-2B	33.3	35.7	$+2.4$
Qwen2.5-VL-3B	46.7	47.2	$+0.5$
Gemma-4-E4B	59.1	61.8	$+2.7$
Qwen2-VL-7B	73.2	71.6	$-1.6$
Qwen2.5-VL-7B	49.8	48.1	$-1.7$

Table 6: Reading is the same on the capture

I_{\mathrm{all}}

and the composite

I_{\mathrm{sdw}}

. Three-way direction reading on each image source and their difference

\Delta

. The means differ by

+1.1

points, so the reading ceiling is unchanged by the choice of source.

Figure 3: Shadow-detection psychometric curve. The object’s own cast shadow is recomposed at opacities from $0$ (no shadow) to $1$ (full), and at each level we ask whether the object casts a visible shadow. For the three models shown (InternVL3-8B, Qwen2.5-VL-7B, Qwen3-VL-8B), the detection rate rises smoothly and monotonically with opacity, showing that shadow presence is perceived as a graded quantity that scales with physical strength. Some models instead answer “yes” at every opacity, a response bias rather than a perceptual deficit.

Task	short	long	chance
	Shadow length
	$<1.15$	$>1.15$
Reading 2-way	73.9	75.1	50.0
Reading 3-way	61.0	64.4	33.3
Inference 3-way	36.5	41.9	33.3

Table 7: Reading and inference accuracy stratified by shadow length, split at the median

\text{shadow reach}/\text{object height}

(

1.15

), the mean over the

19

open models,

229

scenes per half. Every task improves monotonically with shadow length, and inference on short-shadow scenes sits near the

33\%

chance level. Bold marks the better half per row.

	Presence	Direction
Model	2AFC	2AFC	$\Delta$
Qwen2-VL-2B	58.0	50.0	$+8.0$
Qwen3-VL-2B	51.9	50.0	$+1.9$
Qwen2.5-VL-3B	54.7	50.0	$+4.7$
Gemma-4-E4B	85.0	50.0	$+35.0$
Qwen2-VL-7B	84.6	51.3	$+33.3$

Table 8: Presence vs. direction validity 2AFC on the

458

validity scenes (

916

counterbalanced records each). Presence pits the correct-shadow image against the same object casting no shadow. Direction is the main-text validity 2AFC, pitting the correct shadow against a wrong- direction one. Both are counterbalanced, so a constant side cancels to the

50\%

chance. Every open model above the

2

–

3

B tier is far above chance on presence yet at chance on direction. The gap

\Delta

isolates shadow direction , not shadow presence, as the locus of the validity failure.

\dagger

: InternVL3-8B returns a verbose string rather than a single letter on the direction 2AFC, so that score is a parser floor and its

\Delta

is omitted. This means we exclude InternVL3-8B, as in Tables 4 and 5 , and the mean

\Delta

is the difference between the two column means.

	Direction 3-way				Validity Y/N		Validity 2AFC
Model	Read	Infer	rec L/D/R	top	bal	top	bal	top	both $\checkmark$
Qwen2-VL-2B	33.3	28.5	0/67/18	D 73	50.0	Y 100	50.0	A 100	0.0
Qwen3-VL-2B	35.7	33.0	0/0/99	R 100	50.0	Y 100	50.0	B 100	0.0
Qwen2.5-VL-3B	47.2	33.8	1/1/100	R 100	50.3	Y 100	50.0	A 100	0.3
Gemma-4-E4B	61.8	34.4	0/5/98	R 96	49.7	Y 62	50.0	B 100	0.0
Qwen2-VL-7B	71.6	41.7	9/59/57	R 51	50.6	Y 98	51.3	A 84	9.4

Table 9: The read-to-infer dissociation, per class, full panel. Columns as in Table 3 : Read / Infer are three-way direction balanced accuracy with the shadow present/removed, rec L/D/R the per-direction inference recall, and top the single most-predicted class with the percentage of items assigned to it. Open models are listed by size, the two proprietary models below. Reading scales with size while inference stays near the

33.3\%

chance for all but Gemini-3.1-pro, and the gap is largest for the strongest readers; the single-image validity and the 2AFC likewise collapse onto a fixed class. † InternVL3-8B emits a verbose string rather than a single letter on the 2AFC, a parser floor, so its top class is undefined. both

\checkmark

: share of pairs answered correctly under both presentation orders of the 2AFC; a fixed-side strategy scores

0\%

and random guessing

25\%

, so only Gemini-3.1-pro clearly exceeds the random reference.

Model	Base	ZS- $\Delta$	Ctx- $\Delta$	Base	ZS- $\Delta$	Ctx- $\Delta$	Base	ZS- $\Delta$	Ctx- $\Delta$
	Inference 3-way			Validity Y/N			Validity 2AFC
Qwen2-VL-2B	28.5	$+2.5$	$+4.8$	50.0	$+1.1$	$-1.9$	50.0	$+0.0$	$+0.0$
Qwen3-VL-2B	33.0	$-1.1$	$+0.4$	50.0	$-9.1$	$+0.0$	50.0	$+0.2$	$+1.2$
Qwen2.5-VL-3B	33.8	$-1.0$	$+3.6$	50.3	$+0.5$	$-0.3$	50.0	$-0.9$	$+0.0$
Gemma-4-E4B	34.4	$+0.3$	$+2.1$	49.7	$-0.6$	$+1.0$	50.0	$+0.0$	$+0.0$
Qwen2-VL-7B	41.7	$-4.5$	$-7.6$	50.6	$-0.2$	$+1.7$	51.3	$+0.0$	$+0.5$

Table 10: Per-model CoT ablation on the three comprehension tasks, balanced accuracy (%). Base is the non-CoT score; ZS-

\Delta

and Ctx-

\Delta

are the change under generic zero-shot CoT and the task-specific contextual scaffold. Deltas are large in magnitude but sign-inconsistent across models and centered near zero, the signature of added noise rather than added competence; this is the per-model spread summarized in Table 4 . † InternVL3-8B emits no parseable A/B on the 2AFC without CoT (base is a format artifact, not a judgment), so its deltas are omitted and it is excluded from the 2AFC column statistics.

Editor	Type	3-way
FLUX.1-Kontext-dev	open	79.7
OmniGen2	open	61.0
gpt-image-2	prop.	73.4
Gemini-3-pro-image	prop.	79.0
Chance		33.3

Table 11: Editor-intrinsic shadow-direction accuracy on the

458

scenes, scored through the same VLM-driven loop as the main text. Each IGM receives the VLM-authored editing prompt, which never states the shadow direction, and adds a cast shadow to the shadow-removed input; a single fixed VLM reader (Gemma-4-31B) then reports the added shadow’s three-way direction (left/down/right), which we compare to the human GT. This is the same loop, metric, and reader as the

+

Gen inference column of Table 5 (chance

33.3

); the FLUX entry is that same pipeline. prop. is proprietary; Gemini-3-pro-image is nano-banana-pro. Bold is the best. All four IGMs exceed three-way chance by a wide margin; FLUX.1-Kontext-dev is best overall, edging the proprietary Gemini-3-pro-image while remaining open and reproducible, so we route it through the pipeline.

Figure 4: Naturalising the shadow-only rotation with an image editor. (a) the negative a bare rotation produces: the soft shadow matte is rotated about the annotated origin and recomposited, the object pixels left untouched. An image-space rotation cannot yield a silhouette the pictured object could actually cast, and it disturbs the fine compositing statistics along the shadow’s edge – cues a model can exploit without consulting the scene’s lighting at all. We therefore rebuild every negative: (b) the object and its rotated shadow are lifted onto a white canvas, the scene removed so the editor has no lighting context to re-infer the direction from; (c) FLUX.1-Kontext-dev redraws the shadow’s shape under an instruction to keep its direction, length and size, and the result is clamped to the seed’s rotated support so the direction cannot drift; (d) the redrawn shadow is recomposited into the captured background plate, with the object pixels taken unchanged from the positive so that the shadow is the only difference within a pair. Detectability falls from $64.0\%$ on (a) to $56.3\%$ on (d), while humans answer the same pairs at $94.4\%$ (Table 12 ).

Curation level	Pooled panel	Top-5 models
None (1,374 pairs)	64.0	76.8
Shadow mass $\geq\!0.8$ (200)	60.7	69.2
$+$ clean compositing (112)	57.1	62.4
Editor-refined (1,374)	56.3	63.8
$+$ shadow mass $\geq\!0.8$ (358)	54.6	58.3
$+$ artifact-free (98)	52.3	55.1

Table 12: Shadow-only rotation 2AFC under successively stricter curation (balanced accuracy %, chance

50

; open panel of Table 2 , InternVL3-8B excluded from means for its parser floor as there). Top-5 follows the five best models on each construction’s uncurated set through curation. Top block: human curation of the raw rotation (equated shadow mass, then clean compositing). Middle block: the editor-refined construction, in which FLUX.1-Kontext-dev redraws the rotated shadow’s shape on a white canvas (scene hidden) before recompositing; three annotators retain

98

1{,}374

edits as artifact-free. Removing the non-lighting leaks (shadow-mass asymmetry, broken compositing, residual artifacts) drives accuracy monotonically toward chance along both routes, while the mirror construction, which has none of these leaks, sits at chance.

	Reading 3-way		Inference 3-way
Model	fixed	shuffled	fixed	shuffled
Qwen2-VL-2B	33.3	33.9	28.5	32.0
Qwen3-VL-2B	35.7	34.0	33.0	33.1
Qwen2.5-VL-3B	47.2	46.3	33.8	32.3
Gemma-4-E4B	61.8	59.1	34.4	35.4
Qwen2-VL-7B	71.6	68.4	41.7	36.4

Table 13: Shuffling the option-word order changes no conclusion (balanced accuracy %). Reading and inference rerun with the direction words listed in a per-scene random order.

	Random direction		GT comp.	IGM (FLUX)
Model	vs. GT	vs. injected	vs. GT	vs. GT
Gemma-4-E4B	32.0	49.9	61.8	52.1
Qwen2.5-VL-7B	28.8	41.3	48.1	50.4
Qwen3-VL-8B	29.7	50.9	66.7	63.0
Gemma-4-12B	32.4	67.2	85.8	77.2
Pixtral-12B	30.7	38.4	45.6	51.3

Table 14: Only the injected direction carries signal (balanced accuracy %, chance

33.3

). Reading a composited shadow at a random rotation is at chance against the scene’s truth yet far above chance against the direction actually injected; the identical composite in the ground-truth direction reaches

67.9\%

; the IGM-synthesized shadow (

+

Gen of Table 5 , same models) is read at

65.0\%

, at ground-truth-composite level.

Edit set	Human 3-way acc. (%)
Synthesis (FLUX)	85.9
Repair (FLUX)	83.2

Table 15: Human-rated shadow direction of the FLUX edits (chance

33.3\%

Figure 5: Failure cases. Top: direction inference on $I_{\mathrm{obj}}$ (Task 2). Bottom: single-image validity on $I_{\mathrm{flip}}$ (Task 3). The target object is highlighted in red, as in the evaluation prompts.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

OccludeBench: Can VLMs Find What's Missing?
Oct 9, 2026Jasin Cekinmez, Addison J. Wu, Ryo Mitsuhashi +3Occlusion Reasoning VLM Reasoning
Princeton University
Embedding Physical Reasoning into Diffusion-Based Shadow Generation Under the Sun and Sky
Dec 5, 2025Shilin Hu, Jingyi Xu, Akshat Dave +2Conditional Image Generation Diffusion-Based Image Generation
Stony Brook University, Stony Brook, NY 11794, USA · UNC Charlotte, Charlotte, NC 28223, USA
Domain-Grounded Candidate Selection for Agentic Image Editing: A Shadow Removal Case
Aug 6, 2026Shilin Hu, Jingyi Xu, Dimitris Samaras +1Shadow Removal Physics-Guided Image Restoration
Stony Brook University, Stony Brook NY 11794, USA · UNC Charlotte, Charlotte NC 28223, USA

Oct 9, 2026J/K move · Enter open · S save

OccludeBench: Can VLMs Find What's Missing?

Jasin Cekinmez, Addison J. Wu, Ryo Mitsuhashi, Linrong Cai+2

Princeton University

Occlusion Reasoning VLM Reasoning VLM Evaluation

Open paper AskPDF arXiv

How well can vision-language models (VLMs) reason about what is missing from an image? We study this question with OccludeBench, a manually curated benchmark for recovering fully occluded objects from the visual evidence that remains. OccludeBench distinguishes direct cases, where the hidden object must be recovered from a specific physical cue such as a shadow or reflection, from inferential cases, where it can be inferred from broader scene context. Across five VLMs, accuracy averages 42.7% on direct cases versus 62.7% on inferential cases, exhibiting a 20.0-point gap. Humans attain higher accuracies of 63.4% versus 68.5% for direct and inferential images, respectively, with no significant gap. To locate this failure, we introduce OccludeTrace, a four-stage diagnostic that isolates where the visual reasoning process breaks down. The bottleneck emerges when models must recover the relevant object from the projection within the full image: they succeed on only 50% of failed examples, compared with 91% of examples they answer correctly. Extended reasoning, tool use, and few-shot prompting yield inconsistent improvements and do not reliably eliminate the gap. In contrast, an inpaint-then-confront intervention, which prompts models to first generate a candidate hidden object and then evaluate that hypothesis against the visible physical cue, reduces the gap to a statistically non-significant difference for both models with image-generation capabilities. Together, our results establish a tractable, controlled diagnostic for isolating how VLMs reason about visual content that is missing from an image.

Abstract

VLM Reasoning

Explore similar work

OccludeBench: Can VLMs Find What's Missing?

Embedding Physical Reasoning into Diffusion-Based Shadow Generation Under the Sun and Sky

Domain-Grounded Candidate Selection for Agentic Image Editing: A Shadow Removal Case