How well can vision-language models (VLMs) reason about what is missing from an image? We study this question with OccludeBench, a manually curated benchmark for recovering fully occluded objects from the visual evidence that remains. OccludeBench distinguishes direct cases, where the hidden object must be recovered from a specific physical cue such as a shadow or reflection, from inferential cases, where it can be inferred from broader scene context. Across five VLMs, accuracy averages 42.7% on direct cases versus 62.7% on inferential cases, exhibiting a 20.0-point gap. Humans attain higher accuracies of 63.4% versus 68.5% for direct and inferential images, respectively, with no significant gap. To locate this failure, we introduce OccludeTrace, a four-stage diagnostic that isolates where the visual reasoning process breaks down. The bottleneck emerges when models must recover the relevant object from the projection within the full image: they succeed on only 50% of failed examples, compared with 91% of examples they answer correctly. Extended reasoning, tool use, and few-shot prompting yield inconsistent improvements and do not reliably eliminate the gap. In contrast, an inpaint-then-confront intervention, which prompts models to first generate a candidate hidden object and then evaluate that hypothesis against the visible physical cue, reduces the gap to a statistically non-significant difference for both models with image-generation capabilities. Together, our results establish a tractable, controlled diagnostic for isolating how VLMs reason about visual content that is missing from an image.
Figures & tables
Figure 2: Examples of visual image problems involving direct or inferential evidence, along with their subcategories. We prompt the VLM to answer “ What is the primary object that is completely occluded in this image? Be as specific as you can. ” (see Appendix A.1 ). The responses of each model and the correct answer are shown on the right of each subfigure. Direct cases can be answered with a single visual cue, shadow or reflection, while inferential cases use broader context to arrive at the conclusion. In each question, we group the models by the response they each gave and indicate whether the answer is correct.
Figure 3: Overview of our dataset curation and evaluation pipeline. After we source images from the Internet, existing datasets, and self-collection, human annotators unanimously agree on the correct category for each image (Section 2.1 ). We then evaluate how well the VLM identifies the object hidden in the manually occluded image.
Model
Direct (%)
Inferential (%)
Gap
GPT-5.4
37.6
62.4
24.8
Claude Opus 4.6
51.8
75.1
23.3
Gemini 3.1 Pro
71.6
87.8
16.2
Qwen 3 VL 8B
20.9
46.4
25.5
Llama 4 Maverick
31.6
42.0
10.4
Table 1: VLM accuracy on occluded object identification, split by direct and inferential images. Gap is inferential minus direct accuracy in percentage points. Every model scores lower on direct images.
Figure 4: Model versus human accuracy for occluded object identification. Each dot is a 10-point bin of per-image human accuracy, sized by the number of images. Across all VLMs, questions that are harder for humans are also harder for models ( r=0.44 between per-image human accuracy and mean model accuracy).
Figure 5: Examples of the OccludeTrace stages. (Left) Zoomed-in object recognition: the model sees only the cropped projection and names the object. (Middle) Source mapping: the model chooses which of four candidate regions is the source of the marked projection. (Right) Highlighted-region recovery: the model describes everything in the marked projection region, with no mention of shadows, reflections, or occlusion.
OccludeTrace stage
Accuracy on failed examples
Accuracy on successful examples
1. Cue detection
99%
100%
2. Zoomed-in object recognition
79%
88%
3. Source mapping
85%
82%
4. Highlighted-region recovery
50%
91%
Table 2: Accuracy at each OccludeTrace stage on direct images, computed separately over the examples answered incorrectly and correctly in the base condition. Only the final stage separates the two groups.
Figure 6: Tool-call analysis on direct images. Left: GPT-5.4 accuracy by number of tool calls. GPT-5.4 exhausts its 5-call budget ( n=179 ) at 45% accuracy. There were no traces where GPT-5.4 used exactly one tool. Right: Claude Opus 4.6 accuracy by number of tool calls; accuracy declines monotonically from 62% (2 calls) to 28% (5 calls).
Figure 7: Accuracy on occluded object identification. All VLMs perform worse on direct images than on inferential ones. Tool use and prompt steers yield mixed improvements that vary across models and are insufficient to close the gap.
Figure 8: Accuracies when attempting to improve VLM performance on the occluded object task. Extended reasoning and few-shot prompting improve performance on direct images for most models, but do not reliably close the performance gap between direct and inferential images. Inferential performance sometimes decreases.
Model
Comparison
Rates
Difference
p
GPT-5.4
baseline direct vs inferential
37.6% vs 62.4%
+24.8pp
<0.0001
GPT-5.4
inpaint direct vs inferential
54.3% vs 62.4%
+8.1pp
0.12
Gemini 3.1 Pro
baseline direct vs inferential
71.6% vs 87.8%
+16.2pp
0.0001
Gemini 3.1 Pro
inpaint direct vs inferential
82.2% vs 87.8%
+5.6pp
0.15
Table 3: Accuracy of inpainting multi-step evaluation for GPT-5.4 and Gemini 3.1 Pro. This intervention largely eliminates the gap between identifying the missing object across direct and inferential settings.
Figure 9: Inpaint-then-confront turns identification into hypothesis testing. Reading a hidden object directly off an indirect cue is hard, but checking whether a proposed object is consistent with that cue is easier. (1) Asked to fill in the masked part of a roadside convex mirror, Gemini 3.1 Pro paints in only the empty road. (2) Shown the original next to its infill, the model rejects the infill: the photo is taken from inside a car facing the mirror, so the mirror must reflect that car. (3) It then identifies the hidden object as the car’s reflection, which it missed at baseline. For readability, the figure splits the single confront prompt into its two questions, and the bubbles paraphrase the model’s responses.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Occluded (%)
Unmasked (%)
Model
Direct
Infer.
Direct
Infer.
GPT-5.4
37.6
62.4
91.9
93.4
Claude Opus 4.6
51.8
75.1
86.3
92.8
Gemini 3.1 Pro
71.6
87.8
94.4
92.3
Qwen 3 VL 8B
20.9
46.4
89.3
90.1
Llama 4 Maverick
31.6
42.0
87.8
89.5
Appendix
Table 4: VLM accuracy on occluded and unmasked images. Occluded : original images with the object hidden. Unmasked : bounding box shown but contents visible.
Figure 10: Per-category accuracy across occlusion cue categories for each model, separated by image type. Accuracy differences across subcategories within the direct (left) and inferential (right) image categories are mostly not statistically significant, suggesting that semantic differences within the dataset have little effect.
Model
Direct p
Inferential p
GPT-5.4
0.0037
0.7488
Claude Opus 4.6
0.2020
0.1242
Gemini 3.1 Pro
0.2275
0.7190
Qwen 3 VL 8B
0.0557
0.9393
Llama 4 Maverick
0.0524
0.2715
Appendix
Table 5: Omnibus chi-square p -values testing if accuracy varies by cue category. Bold indicates p<.05 .
Model
Internet
Visual Genome
Author-taken
χ2p
Gemini 3.1 Pro
79%
89%
67%
0.41
Claude Opus 4.6
64%
63%
17%
0.061
GPT-5.4
50%
32%
50%
0.28
Qwen 3 VL 8B
34%
16%
33%
0.26
Llama 4 Maverick
39%
11%
0%
0.008
Appendix
Table 6: Chi-square p -values and accuracy across different image sources. Bold indicates p<.05 .
Model
Category
n
Original Box
New Box
Δ
GPT-5.4
Overall
378
49.5%
50.0%
+0.5
GPT-5.4
Direct
197
37.6%
40.1%
+2.5
GPT-5.4
Inferential
181
62.4%
60.8%
− 1.7
Claude Opus 4.6
Overall
378
63.0%
60.3%
− 2.6
Claude Opus 4.6
Direct
197
51.8%
46.2%
− 5.6
Claude Opus 4.6
Inferential
181
75.1%
75.7%
+0.6
Appendix
Table 7: Accuracy with the original and new bounding boxes (blue interior, yellow border).
Figure 11: Per-image human accuracy distributions for direct and inferential questions. Both distributions span the full difficulty range, with no concentration at the extremes.
Figure 12: Images ranked by human accuracy (easiest to hardest), colored by category. The two categories are interleaved across the difficulty range. Dashed lines show per-category means (direct: 63.4%, inferential: 68.5%).
Metric
Direct
Inferential
Zipf freq (Higher = more common)
4.48
4.22
Concreteness (1 = more abstract, 5 = more concrete)
4.65
4.54
Prevalence (fraction of people who know the word)
1.000
0.996
Out-of-norms (lower = fewer rare terms)
0.035
0.067
Appendix
Table 8: Comparison of lexical metrics between direct and inferential categories.
Figure 13: Tool usage frequency by model on direct images.
Vision Language Models (VLMs) are well known for hallucinating non-existent objects in images. Objects with missing parts present a unique challenge for VLMs, stemming from both real-world knowledge bias and the scarcity of such images in training data. We present MissingBench-Verified, a benchmark designed to evaluate a specific and practically relevant scenario: when vision-language models fail to recognize that an essential component of an object has been removed. Across ten leading models, we observe consistent and significant failure rates that persist even when external tool evidence explicitly contradicts the model's visual perception. We further ask whether granting models access to image processing tools (e.g., cropping, contrast adjustment) enables autonomous inspection to resolve these failures. We find that existing mitigation strategies, including tool-assisted verification, autonomous visual reasoning, longer reasoning durations, and fine-tuning on an easier dataset, provide negligible improvement, indicating that this failure mode cannot be addressed through current prompting or post-hoc correction techniques. Our findings highlight a fundamental limitation of current VLM for inspection and monitoring tasks and underscore the need for architectural or training-level interventions that enable models to override internal expectations when confronted with contradictory evidence.
Wenqi Marshall Guo, Qingyun Qian, Shiyu Zhou +2
Department of CMPS, University of British Columbia, Canada · Weathon Software, Canada · Department of Computer Science, University of British Columbia, Canada
In real-world scenes, target objects may reside in regions that are not visible. While humans can often infer the locations of occluded objects from context and commonsense knowledge, this capability remains a major challenge for vision-language models (VLMs). To address this gap, we introduce SceneFunRI, a benchmark for Reasoning the Invisible. Based on the SceneFun3D dataset, SceneFunRI formulates the task as a 2D spatial reasoning problem via a semi-automatic pipeline and comprises 855 instances. It requires models to infer the locations of invisible functional objects from task instructions and commonsense reasoning. The strongest baseline model (Gemini 3 Flash) only achieves an CAcc@75 of 15.20, an mIoU of 0.74, and a Dist of 28.65. We group our prompting analysis into three categories: Strong Instruction Prompting, Reasoning-based Prompting, and Spatial Process of Elimination (SPoE). These findings indicate that invisible-region reasoning remains an unstable capability in current VLMs, motivating future work on models that more tightly integrate task intent, commonsense priors, spatial grounding, and uncertainty-aware search.
Posheng Chen, Powen Cheng, Gueter Josmy Faure +2
National Taiwan University · Delta Robotics Innovation Center
Benchmark accuracy is often implicitly assumed to reflect grounded visual understanding in vision-language models (VLMs), yet it remains unclear to what extent such scores truly reflect reliance on visual evidence. Motivated by a surprising observation that removing a substantial fraction of image tokens only degrades model performance very slightly on a widely used hallucination benchmark, we systematically investigate this mismatch in a set of open-source VLMs. Our analysis spans multiple levels of granularity, spanning global visual degradation, localized occlusion, question reformulation, answer-space expansion, and decision-level analyses beyond standard accuracy. We further complement these behavioral results with a layer-wise analysis of vision-token geometry. Throughout the experiments, we find that although VLMs do incorporate visual input, their predictions are less sensitive to the loss of fine-grained visual evidence that standard accuracy should have suggested. Even when the final prediction remains unchanged, the model's internal support for the correct answer may already be weakened. We further complement a representation-level analysis, which shows increasing similarity among visual tokens in deeper layers, providing a possible explanation for our findings. Together, these results suggest that current benchmarks are not sufficient to reliably evaluate fine-grained visual grounding in VLMs.
Zixuan Lan, Luzhe Sun, Matthew R. Walter +1
University of Chicago · Toyota Technological Institute at Chicago · Stony Brook University