When Depth Lies: Benchmarking Vision-Language Models on Mirror-Induced RGB-D Ambiguity
Authors: Hao Yin, Tianchen Guo, Heming Du, Yan Ke, Yanbin Liu, Xin Yu
Organizations: The University of Queensland, Australia · Australian Artificial Intelligence Institute University of Technology Sydney, Australia · Australian Institute for Machine Learning Adelaide University, Australia
Vision-language models (VLMs) have made significant progress in spatial reasoning, particularly by integrating RGB images with depth data, which is often assumed to faithfully reflect physical distances. However, mirrors violate this assumption: they distort depth readings while reproducing object appearances, making reflections difficult to distinguish from real objects. As existing benchmarks rarely consider such reflective scenarios, VLM performance in mirror scenes remains largely unexplored. In this work, we introduce a mirror-centric RGB-D dataset from reflective scenes to evaluate the spatial reasoning capabilities of state-of-the-art VLMs in these scenarios. Specifically, we collect 1,011 RGB-D images from 525 real-world scenes, spanning diverse indoor and outdoor environments, six reflective surface types, and both daytime and nighttime conditions. All images are annotated with reflective surface region masks and mask annotations for both directly observed and mirror-reflected objects. Then, we curate a benchmark named MIRA-Bench (Mirror-Induced Reasoning Ambiguity Benchmark), comprising 14,140 QA pairs across eight sub-tasks organized into three cognitive levels: Reflection-aware Perception, Spatial Reasoning, and Scene-level Decision-making. We evaluate state-of-the-art VLMs on MIRA-Bench and find consistent failures across tasks: models misclassify whether objects are directly observed or reflected, fail to match reflected instances to their real-world counterparts, and produce erroneous spatial inferences. Remarkably, providing depth data actually degrades reasoning performance, implying that VLMs cannot identify incorrect depth in mirror scenes. The code and data are available at https://github.com/EvaYin-cloud/MIRA-Bench
Figures & tables
Figure 1: Overview of the MIRA-Bench construction pipeline . Left: data collection from public datasets and newly captured scenes, followed by filtering and selection. Right: annotation pipeline combining reflective surface mask labeling, object instance polygon annotation, and QA pair generation with multi-stage quality control.
Figure 2: Statistics of MIRA-Bench. (a) Distribution of QA pairs across three cognitive levels and eight subtasks. (b) Distribution of images by data source and scenes by lighting condition (daytime versus nighttime).
Refl.-aware Perception
Spatial Reasoning
Scene-level Decision-making
Model
VC
MC
W-Avg
MR
CT
MPAC
W-Avg
RE
AS
CR
W-Avg
Overall
Baselines
Random Guess
12.3
15.0
13.1
3.7
2.5
25.0
8.6
33.3
16.7
33.3
24.1
14.1
Human (RGB)
85.1
82.3
84.2
79.3
78.0
90.8
81.7
92.2
91.5
90.2
91.7
85.0
Human (RGB-D)
92.3
90.5
91.7
85.0
80.2
92.1
85.2
94.4
94.9
91.5
94.5
90.7
Proprietary Models
Table 1: Evaluation results on MIRA-Bench. We report exact accuracy (%) for each sub-task. VC: Visibility Classification (joint); MC: Mirror Correspondence; MR: Mirror To Real Spatial Inference; CT: Counting; MPAC: Mirror Provided Attribute Completion; RE: Reachability; AS: Next Step Action Selection; CR: Collision Risk. W-Avg: weighted average within each cognitive level, weighted by the number of questions per sub-task. Overall: weighted average across all sub-tasks.
Refl.-aware Perception
Spatial Reasoning
Scene-level Decision-making
Input Mode
VC
MC
W-Avg
MR
CT
MPAC
W-Avg
RE
AS
CR
W-Avg
Overall
Qwen3.5-9B
RGB
32.2
39.8
34.6
2.1
20.9
52.7
20.6
57.2
8.1
52.6
29.8
30.3
RGB-D
31.4
37.7
33.4
2.7
20.6
50.4
20.2
61.4
7.9
36.0
30.2
29.6
RGB-M
32.6
42.7
35.8
3.2
19.7
49.8
20.0
62.1
10.3
45.7
32.4
31.3
RGB-DM
32.5
42.4
35.6
2.9
17.2
49.8
19.0
56.0
9.8
38.3
29.3
30.4
Table 2: Impact of input modality on mirror-aware spatial reasoning. We report exact accuracy (%) for each sub-task across four input configurations. RGB: raw image only; RGB-M: image with mirror region mask overlaid; RGB-D: image paired with depth map; RGB-DM: image with both depth map and mirror masks.
Referring Protocol
Overall Acc.
Text-BBox
28.0
Text-BBox+Desc
31.6
Visual-BBox
33.0
Visual-Mask
27.9
Visual-BBox+Desc
33.0
Table 3: Impact of object-referencing protocols on overall accuracy (%). Results shown for Qwen3.5-9B with RGB-M input. Desc denotes textual object descriptions.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Gradio-based QA annotation interface. The interface displays the annotated RGB image and corresponding depth map, together with controls for task selection, instance referencing, question preview, and answer annotation. The example shows a mirror-to-real spatial-inference question with horizontal, vertical, and depth relation labels.
ID
Question template and response specification
Visibility classification
vc_001
Is the object {X} visible in a mirror reflection but not directly in this image? Format: yes_no . Options: yes; no.
vc_002
Is the object {Y} directly visible but not visible in any mirror or other reflective surface in this image? Format: yes_no . Options: yes; no.
vc_003
Is the object {X/Y} visible both directly and in a mirror reflection in this image? Format: yes_no . Options: yes; no.
Mirror correspondence
mc_001
Which real-world instance corresponds to mirror instance {X}? Format: single_choice . Options: candidate real-world instance IDs.
Appendix
Table 4: QA templates for reflection-aware perception. Braced placeholders denote instance identifiers. {X} and {Y} denote objects visible in reflection or directly. Candidate instances for the multiple-choice questions are drawn from the same scene.
ID
Question template and response specification
Mirror-to-real spatial inference
mr_001
Where is the real object corresponding to mirror instance {X} relative to the real object {Y} in real-world space? Format: spatial_3d_axes . Options: horizontal: left, aligned, right; vertical: above, aligned, below; depth: front, same_depth, behind.
mr_003
Where is the real object corresponding to mirror instance {X1} relative to the real object corresponding to mirror instance {X2} in real-world space? Format: spatial_3d_axes . Options: horizontal: left, aligned, right; vertical: above, aligned, below; depth: front, same_depth, behind.
Counting-based spatial reasoning
ct_001
How many objects are there in the real world to the {DIR} of the real object corresponding to instance {X/Y}? (Count only polygon-annotated objects and the mirror itself. Do not count unannotated objects.) Format: count ; integer response. DIR options: left; right; front; behind; above; below.
Mirror-provided attribute completion
Appendix
Table 5: QA templates for spatial reasoning. Braced placeholders denote instance identifiers or spatial directions. {X} and {Y} denote objects visible in reflection or directly. For counting questions, annotators select {DIR} from the six listed directions. Each template specifies the answer format and, where applicable, the response options and text field requirements.
ID
Question template and response specification
Reachability
ac_001
Is real-world instance {X/Y} reachable from the current camera position? Format: yes_no . Options: yes; no; not sure.
Next-step action selection
ac_002
What is the best next action for the camera to move toward real-world instance {X/Y}? Format: single_choice . Options: move left; move right; move forward; move backward; rotate left; rotate right; turn around; stop; not sure; none.
ac_003
How should the camera move to better observe the real object corresponding to mirror instance {X}? Format: single_choice . Options: move left; move right; move forward; move backward; rotate left; rotate right; turn around; stop; not sure; none.
Collision-risk awareness
Appendix
Table 6: QA templates for scene-level decision-making. {X} and {Y} denote objects visible in reflection or directly. Braced placeholders denote instance identifiers or directions. For collision-risk questions, annotators select {DIR} from the six listed directions. The response format and options are provided below each question template.
Figure 4: Examples of the five object-referencing protocols in MIRA-Bench. Each surrounding panel presents the same underlying visibility classification question for the same target object using a different referring protocol. The central panel identifies the target instance. RGB-DM inputs are shown for illustration.
Figure 5: More visualization examples of MIRA-Bench.
Model
RGB
RGB-D
Δ
GPT-5.4
62.01
61.27
−0.74
Gemini-2.5-Flash
62.50
61.98
−0.52
Qwen3.5-9B
66.17
65.62
−0.55
LLaVA-OV-7B
57.30
56.98
−0.32
Appendix
Table 7: Accuracy (%) on 10,000 non-reflective RefSpatial [ 56 ] questions. Δ denotes the accuracy difference (RGB-D minus RGB), measured in percentage points.
Lower-level outcome
GPT-5.4
Gemini-2.5-Flash
Qwen3.5-9B
Perception fails
68.3
65.8
73.4
Reasoning fails
63.0
60.4
64.4
At least one fails
63.9
59.4
65.9
Both correct
45.3
40.8
56.5
Appendix
Table 8: Scene-level failure rate (%) grouped by the outcomes of linked lower-level questions. The groups overlap except for “both correct.”
Model
RGB
RGB-D
Δ
GPT-5.4
54.47
55.04
+0.57
Qwen3.5-9B
49.96
48.38
−1.58
LLaVA-OV-7B
29.77
29.16
−0.61
SpatialRGPT
37.48
39.27
+1.79
Appendix
Table 9: Accuracy (%) on the common single-region question set. Δ denotes the accuracy difference (RGB-D minus RGB), measured in percentage points.
In the era of Vision-Language Models (VLMs), enhancing multimodal reasoning capabilities remains a critical challenge, particularly in handling ambiguous or complex visual inputs, where initial inferences often lead to hallucinations or logic errors. Existing VLMs often produce plausible yet ungrounded answers, and even when prompted to "reflect", their corrections may remain detached from the image evidence. To address this, we propose the MIRROR framework for Multimodal Iterative Reasoning via Reflection On visual Regions. By embedding visual reflection as a core mechanism, MIRROR is formulated as a closed-loop process comprising draft, critique, region-based verification, and revision, which are repeated until the output is visually grounded. To facilitate training of this model, we construct ReflectV, a visual reflective dataset for multi-turn supervision that explicitly contains reflection triggers, region-based verification actions, and answer revision grounded in visual evidence. Experiments on both general vision-language benchmarks and representative vision-language reasoning benchmarks show that MIRROR improves correctness and reduces visual hallucinations, demonstrating the value of training reflection as an evidence-seeking, region-aware verification process rather than a purely textual revision step.
Haoyu Zhang, Yuwei Wu, Pengxiang Li +6
Beijing Institute of Technology · Shenzhen MSU-BIT University
Vision-language models (VLMs) can answer image-based questions confidently, and often correctly, even when no image is provided. This mirage behavior inflates benchmark scores without reflecting visual grounding. Prior work treats this as a single failure mode. We argue it is two. Using Mirage Probes, a contrastive probing framework that pairs paraphrased question variants with matched mirage and non-mirage labels on the same image, we show that mirage behavior is linearly decodable from internal activations across residual stream, MLP, post-attention, and attention-head sites in two open-source VLMs. We demonstrate that a Naive Bayes text baseline cannot recover this signal, ruling out surface lexical confounds. Cross-benchmark separability patterns, together with a novel Prior Harnessing Index (PHI) measuring how much a model can answer from text alone, expose two distinct regimes: textual biases, where the model answers from language priors without engaging visual representations, and spurious images, where it constructs false visual content in latent space and answers as if grounded. The distinction has direct mitigation consequences: text-distribution cleaning can address the first regime but cannot reach the second, since spurious-image mirages live in the model's visual representations rather than its text. Faithful visual grounding will require interventions at the representational level.
Daniel Ben-Levi, Judah Goldfeder, Weiliang Zhao +5
Vision-language models (VLMs) are expected to reason about physical space -- which object is closer, what lies behind what, and how objects are arranged in 3D -- yet they still struggle with such spatial judgments. A natural remedy is to show the model a depth map, but we find that this can make performance worse. We show that depth is not absent: it reaches the language model, but becomes difficult to access for downstream reasoning, while rendered pseudo-depth maps act as noisy auxiliary images that frozen VLMs cannot easily regulate. We propose Depth-Ordinal Prompting (DOP), a training-free method that converts monocular depth into a single question-targeted ordinal text cue at the queried objects, without adding a depth image, training a module, injecting features, or using labels. Our key finding is form dependence: the same depth signal can hurt when shown as an image but help when told as text.Across benchmarks, models, and depth estimators, DOP improves spatial reasoning when pseudo-depth provides reliable object-level ordering and remains largely neutral in strong original-image regimes. It is also competitive with the strongest training-free depth-prompting alternative while being simpler and more targeted.
Quynh Vo, Phuc Dao, Cong-Duy Nguyen +1
National University of Singapore · Center of AI Research, VinUniversity