cs.CVOct 9, 2026
SaveOccludeBench: Can VLMs Find What's Missing?
Organizations: Princeton University
Abstract
How well can vision-language models (VLMs) reason about what is missing from an image? We study this question with OccludeBench, a manually curated benchmark for recovering fully occluded objects from the visual evidence that remains. OccludeBench distinguishes direct cases, where the hidden object must be recovered from a specific physical cue such as a shadow or reflection, from inferential cases, where it can be inferred from broader scene context. Across five VLMs, accuracy averages 42.7% on direct cases versus 62.7% on inferential cases, exhibiting a 20.0-point gap. Humans attain higher accuracies of 63.4% versus 68.5% for direct and inferential images, respectively, with no significant gap. To locate this failure, we introduce OccludeTrace, a four-stage diagnostic that isolates where the visual reasoning process breaks down. The bottleneck emerges when models must recover the relevant object from the projection within the full image: they succeed on only 50% of failed examples, compared with 91% of examples they answer correctly. Extended reasoning, tool use, and few-shot prompting yield inconsistent improvements and do not reliably eliminate the gap. In contrast, an inpaint-then-confront intervention, which prompts models to first generate a candidate hidden object and then evaluate that hypothesis against the visible physical cue, reduces the gap to a statistically non-significant difference for both models with image-generation capabilities. Together, our results establish a tractable, controlled diagnostic for isolating how VLMs reason about visual content that is missing from an image.
Figures & tables
Figure 2: Examples of visual image problems involving direct or inferential evidence, along with their subcategories. We prompt the VLM to answer “ What is the primary object that is completely occluded in this image? Be as specific as you can. ” (see Appendix A.1 ). The responses of each model and the correct answer are shown on the right of each subfigure. Direct cases can be answered with a single visual cue, shadow or reflection, while inferential cases use broader context to arrive at the conclusion. In each question, we group the models by the response they each gave and indicate whether the answer is correct.
Figure 3: Overview of our dataset curation and evaluation pipeline. After we source images from the Internet, existing datasets, and self-collection, human annotators unanimously agree on the correct category for each image (Section 2.1 ). We then evaluate how well the VLM identifies the object hidden in the manually occluded image.
| Model | Direct (%) | Inferential (%) | Gap |
|---|---|---|---|
| GPT-5.4 | 37.6 | 62.4 | 24.8 |
| Claude Opus 4.6 | 51.8 | 75.1 | 23.3 |
| Gemini 3.1 Pro | 71.6 | 87.8 | 16.2 |
| Qwen 3 VL 8B | 20.9 | 46.4 | 25.5 |
| Llama 4 Maverick | 31.6 | 42.0 | 10.4 |
Table 1: VLM accuracy on occluded object identification, split by direct and inferential images. Gap is inferential minus direct accuracy in percentage points. Every model scores lower on direct images.
Figure 4: Model versus human accuracy for occluded object identification. Each dot is a 10-point bin of per-image human accuracy, sized by the number of images. Across all VLMs, questions that are harder for humans are also harder for models ( between per-image human accuracy and mean model accuracy).
Figure 5: Examples of the OccludeTrace stages. (Left) Zoomed-in object recognition: the model sees only the cropped projection and names the object. (Middle) Source mapping: the model chooses which of four candidate regions is the source of the marked projection. (Right) Highlighted-region recovery: the model describes everything in the marked projection region, with no mention of shadows, reflections, or occlusion.
| OccludeTrace stage | Accuracy on failed examples | Accuracy on successful examples |
|---|---|---|
| 1. Cue detection | 99% | 100% |
| 2. Zoomed-in object recognition | 79% | 88% |
| 3. Source mapping | 85% | 82% |
| 4. Highlighted-region recovery | 50% | 91% |
Table 2: Accuracy at each OccludeTrace stage on direct images, computed separately over the examples answered incorrectly and correctly in the base condition. Only the final stage separates the two groups.
Figure 6: Tool-call analysis on direct images. Left: GPT-5.4 accuracy by number of tool calls. GPT-5.4 exhausts its 5-call budget ( ) at 45% accuracy. There were no traces where GPT-5.4 used exactly one tool. Right: Claude Opus 4.6 accuracy by number of tool calls; accuracy declines monotonically from 62% (2 calls) to 28% (5 calls).
Figure 7: Accuracy on occluded object identification. All VLMs perform worse on direct images than on inferential ones. Tool use and prompt steers yield mixed improvements that vary across models and are insufficient to close the gap.
Figure 8: Accuracies when attempting to improve VLM performance on the occluded object task. Extended reasoning and few-shot prompting improve performance on direct images for most models, but do not reliably close the performance gap between direct and inferential images. Inferential performance sometimes decreases.
| Model | Comparison | Rates | Difference | |
|---|---|---|---|---|
| GPT-5.4 | baseline direct vs inferential | 37.6% vs 62.4% | +24.8pp | <0.0001 |
| GPT-5.4 | inpaint direct vs inferential | 54.3% vs 62.4% | +8.1pp | 0.12 |
| Gemini 3.1 Pro | baseline direct vs inferential | 71.6% vs 87.8% | +16.2pp | 0.0001 |
| Gemini 3.1 Pro | inpaint direct vs inferential | 82.2% vs 87.8% | +5.6pp | 0.15 |
Table 3: Accuracy of inpainting multi-step evaluation for GPT-5.4 and Gemini 3.1 Pro. This intervention largely eliminates the gap between identifying the missing object across direct and inferential settings.
Figure 9: Inpaint-then-confront turns identification into hypothesis testing. Reading a hidden object directly off an indirect cue is hard, but checking whether a proposed object is consistent with that cue is easier. (1) Asked to fill in the masked part of a roadside convex mirror, Gemini 3.1 Pro paints in only the empty road. (2) Shown the original next to its infill, the model rejects the infill: the photo is taken from inside a car facing the mirror, so the mirror must reflect that car. (3) It then identifies the hidden object as the car’s reflection, which it missed at baseline. For readability, the figure splits the single confront prompt into its two questions, and the bubbles paraphrase the model’s responses.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Occluded (%) | Unmasked (%) | |||
|---|---|---|---|---|
| Model | Direct | Infer. | Direct | Infer. |
| GPT-5.4 | 37.6 | 62.4 | 91.9 | 93.4 |
| Claude Opus 4.6 | 51.8 | 75.1 | 86.3 | 92.8 |
| Gemini 3.1 Pro | 71.6 | 87.8 | 94.4 | 92.3 |
| Qwen 3 VL 8B | 20.9 | 46.4 | 89.3 | 90.1 |
| Llama 4 Maverick | 31.6 | 42.0 | 87.8 | 89.5 |
Table 4: VLM accuracy on occluded and unmasked images. Occluded : original images with the object hidden. Unmasked : bounding box shown but contents visible.
Figure 10: Per-category accuracy across occlusion cue categories for each model, separated by image type. Accuracy differences across subcategories within the direct (left) and inferential (right) image categories are mostly not statistically significant, suggesting that semantic differences within the dataset have little effect.
| Model | Direct | Inferential |
|---|---|---|
| GPT-5.4 | 0.0037 | 0.7488 |
| Claude Opus 4.6 | 0.2020 | 0.1242 |
| Gemini 3.1 Pro | 0.2275 | 0.7190 |
| Qwen 3 VL 8B | 0.0557 | 0.9393 |
| Llama 4 Maverick | 0.0524 | 0.2715 |
Table 5: Omnibus chi-square -values testing if accuracy varies by cue category. Bold indicates .
| Model | Internet | Visual Genome | Author-taken | |
|---|---|---|---|---|
| Gemini 3.1 Pro | 79% | 89% | 67% | 0.41 |
| Claude Opus 4.6 | 64% | 63% | 17% | 0.061 |
| GPT-5.4 | 50% | 32% | 50% | 0.28 |
| Qwen 3 VL 8B | 34% | 16% | 33% | 0.26 |
| Llama 4 Maverick | 39% | 11% | 0% | 0.008 |
Table 6: Chi-square -values and accuracy across different image sources. Bold indicates .
| Model | Category | n | Original Box | New Box | |
|---|---|---|---|---|---|
| GPT-5.4 | Overall | 378 | 49.5% | 50.0% | +0.5 |
| GPT-5.4 | Direct | 197 | 37.6% | 40.1% | +2.5 |
| GPT-5.4 | Inferential | 181 | 62.4% | 60.8% | 1.7 |
| Claude Opus 4.6 | Overall | 378 | 63.0% | 60.3% | 2.6 |
| Claude Opus 4.6 | Direct | 197 | 51.8% | 46.2% | 5.6 |
| Claude Opus 4.6 | Inferential | 181 | 75.1% | 75.7% | +0.6 |
Table 7: Accuracy with the original and new bounding boxes (blue interior, yellow border).
Figure 11: Per-image human accuracy distributions for direct and inferential questions. Both distributions span the full difficulty range, with no concentration at the extremes.
Figure 12: Images ranked by human accuracy (easiest to hardest), colored by category. The two categories are interleaved across the difficulty range. Dashed lines show per-category means (direct: 63.4%, inferential: 68.5%).
| Metric | Direct | Inferential |
|---|---|---|
| Zipf freq (Higher = more common) | 4.48 | 4.22 |
| Concreteness (1 = more abstract, 5 = more concrete) | 4.65 | 4.54 |
| Prevalence (fraction of people who know the word) | 1.000 | 0.996 |
| Out-of-norms (lower = fewer rare terms) | 0.035 | 0.067 |
Table 8: Comparison of lexical metrics between direct and inferential categories.
Figure 13: Tool usage frequency by model on direct images.
| Question position | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
| Accuracy (%) | 69.9 | 65.3 | 64.2 | 65.1 | 67.3 | 62.0 |
Table 9: Human accuracy by question position.