Seeing Depth but Choosing Cues in Vision-Language Models

Zhengtao Zou; Pekka Marttinen

cs.AIOct 13, 2026

Seeing Depth but Choosing Cues in Vision-Language Models

Authors: Zhengtao Zou, Pekka Marttinen

Organizations: Aalto University

Abstract

Do vision-language models use depth information to judge which object is closer, or rely on image-plane cues? We investigate this question through controlled depth--cue conflicts, representation analysis, and interventions. Among the many image-plane cues relevant to depth perception, we focus on apparent size and vertical position. We compare depth judgments under cue-consistent and cue-conflict conditions in both rendered scenes and real-world photographs. Across model families and scales, depth judgments systematically shift toward misleading cues. Yet the correct depth order remains decodable from the queried objects' visual representations, including when the model answers incorrectly. Image and category controls support interpreting this signal as image-dependent depth information beyond the two target cues. Strengthening the model's own depth readout through object-level write-back can correct conflict decisions without updating model parameters or using ground-truth depth annotations. Interventions show that the effect of this correction depends on the direction and target tokens of the written signal, while its strength varies across architectures and domains. Together, these findings reveal a mismatch between a model's ability to represent depth and use this information when deciding the answer. VLMs can retain useful depth information while allowing competing image-plane cues to exert greater influence on their answers.

Figures & tables

Figure 1: Schematic illustration. A conflict item. The sofa looks larger, so the size cue favors the sofa. Yet the same forward pass contains the correct depth relation in its visual representations, while the final decision follows the conflicting cue.

Figure 2: Examples in which a 2D cue conflicts with true depth. The black dot marks each object’s position projected straight down onto the floor. Bounding boxes highlight the queried objects.

Figure 3: Depth accuracy drops across model families when image-plane cues conflict with true depth.

Panel of Figure 4 — Figure 4: Impact of increasing the object’s size, adjusting its vertical position, or changing its depth on the model’s probability of selecting it as the closest.

Figure 5: Reading depth from object representations. Object tokens are pooled and contrasted to fit the depth direction $v$ . Point clouds and separating planes are schematic.

Figure 6: Depth readouts from answer states and object representations. Points average four consecutive layers; bars indicate within-bin minima and maxima, not confidence intervals.

Figure 7: Intervention to strengthen the depth trace. The sign of the pre-intervention depth trace $z$ determines opposite updates to the two queried objects along $v$ . The intervention uses no ground-truth depth label.

	Rendered						Real
	conflict-size		conflict-vert.		consistent		conflict-size		conflict-vert.		consistent
	van.	int.	van.	int.	van.	int.	van.	int.	van.	int.	van.	int.
Qwen3-VL-8B	50.2	90.8	42.8	71.4	94.4	99.6	77.5	78.9	75.0	80.6	89.2	87.9
Qwen3-VL-4B	54.8	88.8	34.8	75.4	95.4	99.8	74.2	77.5	68.8	75.0	88.2	84.9
Qwen3-VL-32B	34.6	85.0	45.0	84.4	97.2	100.0	75.6	76.1	69.4	77.8	89.2	86.2
LLaVA-OV-1.5-8B	42.0	83.6	7.4	47.0	98.6	96.8	53.1	65.1	46.5	61.8	86.2	85.2

Table 1: Effect of depth-trace intervention across models. Accuracy (%) before and after intervention on rendered and natural images. Each setup is selected on held-out development data and frozen before evaluation. (‘van.’ is vanilla and ‘int.’ is the intervened model)

Figure 8: Example corrections on rendered scenes and natural photographs.

Figure 9: How answers change after the write-back intervention on consistent real-world images. Changes from correct to incorrect are categorized by whether the pre-intervention readout identified the closer object correctly. Numbers show the counts of corresponding images.

	Rendered			Real
	size	vertical	consistent	size	vertical	consistent
Vanilla	50.2	42.8	94.4	77.5	75.0	89.2
Cue-avoidant CoT	45.6	36.0	91.0	74.2	68.8	87.9
Depth input	48.6	51.8	88.8	76.1	72.2	87.9
Random direction	51.6	41.5	94.6	75.5	73.7	87.7
Answer-token write	51.0	47.6	89.8	77.0	77.8	90.2

Table 2: Ablation study on Qwen3-VL-8B. Values are answer accuracy (%). The write-back intervention gives the largest improvement for size and vertical conflicts.

Figure 10: Depth readout versus answer-logit margin before and after write-back. The margin is the logit for the option naming object 0 minus that for naming object 1; positive values favor object 0. Black curves show binned means with 95% confidence bands.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Model	Vanilla	Intervention
SpatialRGPT depth	Qwen3-VL-8B	85.9	85.9
SpatialRGPT conflict	Qwen3-VL-8B	69.2	84.6
SpatialTunnel pooled	Qwen3-VL-8B	89.4	89.8
SpatialTunnel pooled	Qwen3-VL-32B	94.8	94.4
CV-Bench depth	Qwen3-VL-8B	95.7	93.5
CV-Bench depth	Qwen3-VL-32B	95.5	93.6

Appendix

Table 3: External answer accuracy (%). Pooled SpatialTunnel combines 976 phase and 1076 size items that support object masks. SpatialRGPT conflict is directional and contains 13 items.

Figure 11: Vanilla accuracy on three external benchmarks. The upper spokes are cue-consistent and the lower spokes are cue-conflict conditions.

Panel of Figure 12 — Figure 12: Agreement of image-plane cues with depth. Error bars are Wilson 95% intervals; intervals in (b) use approximate reweighted sample sizes.

Panel of Figure 13 — Figure 13: Rendered dataset examples. Depth, apparent size, and vertical position are controlled independently.

Panel of Figure 14 — Figure 14: SUN RGB-D examples. Depth labels come from sensor depth inside the object masks; cue conditions use measured mask area and centroid row.

Model / domain	Conflict	Original	Grey	Mismatched
Qwen3-8B, Render ( $\ell=7$ )	Size	99	52	49
	Vertical	99	63	49
Qwen3-8B, Real ( $\ell=8$ )	Size	84	51	58
	Vertical	82.6	40.3	40
Gemma-12B, Render ( $\ell=20$ )	Size	89	57	52
	Vertical	99	32	52

Appendix

Table 4: Depth sign accuracy (%) under image replacement. The depth direction, question, and object-token positions are fixed. Gemma’s rendered controls use layer 20; the main intervention uses layer 9.

Model	Size conflict	Vertical conflict
Qwen3-VL-4B	$+34.0\ [29.8,\,38.2]$	$+40.6\ [36.2,\,45.0]$
Qwen3-VL-8B	$+40.6\ [36.4,\,44.8]$	$+28.6\ [24.8,\,32.6]$
Qwen3-VL-32B	$+50.4\ [46.0,\,54.6]$	$+39.4\ [35.2,\,43.6]$
LLaVA-OV-1.5-8B	$+41.6\ [37.0,\,46.0]$	$+39.6\ [35.4,\,44.0]$
LLaVA-OV-2-8B	$+33.8\ [29.4,\,38.0]$	$+28.6\ [24.4,\,32.8]$
InternVL3-8B	$+29.0\ [25.2,\,33.0]$	$+25.4\ [21.6,\,29.2]$

Appendix

Table 5: Accuracy changes after write-back on rendered scenes (

n=500

per split), reported as

\Delta

[95% CI] in percentage points.

Model	Size conflict	Vertical conflict
Qwen3-VL-4B	$+3.3\ [2.0,\,7.9]$	$+6.2\ [2.0,\,12.6]$
Qwen3-VL-8B	$+1.4\ [0.3,\,6.7]$	$+5.6\ [2.3,\,12.2]$
Qwen3-VL-32B	$+0.5\ [-0.3,\,6.0]$	$+8.3\ [2.8,\,14.1]$
LLaVA-OV-1.5-8B	$+12.0\ [5.9,\,18.1]$	$+15.3\ [8.3,\,22.3]$
LLaVA-OV-2-8B	$+6.2\ [1.4,\,11.1]$	$+4.9\ [0.6,\,11.0]$
InternVL3-8B	$+1.9\ [0.5,\,3.9]$	$-1.4\ [-2.5,\,0.0]$

Appendix

Table 6: Accuracy changes after write-back on SUN RGB-D, reported as

\Delta

[95% CI] in percentage points.

Method	Size conflict	Vertical conflict	Consistent
Vanilla	50.2	42.8	94.4
CoT	41.8	41.4	93.0
SpatioRoute	42.2	35.2	94.0
VCD	46.6	38.4	93.8
DoLa	49.2	43.4	94.0
Depth input	48.6	51.8	88.8

Appendix

Table 7: Qwen3-VL-8B answer accuracy (%) on rendered scenes.

Figure 15: Matched-readout comparison on real images. Size and vertical subsets contain 157 and 111 pairs; labels give event counts.

VLM Interpretability

Vision-language model interpretability explains how image and language representations shape predictions, evidence use, and multimodal reasoning.

See all 268 papers

Disentangling Pictorial Cue Understanding from Language Bias in VLMs via Depth Ordering Task
Jul 1, 2026Yiqian Liu, Iuliia Kotseruba, John K. TsotsosVLM Evaluation Vision-Language Models
York University, Toronto, Canada · University of Guelph, Guelph, Canada
When Depth Is Better Told Than Shown: Depth-Ordinal Prompting for Vision-Language Spatial Reasoning
Jul 13, 2026Quynh Vo, Phuc Dao, Cong-Duy Nguyen +1Vision-Language Models Visual Spatial Reasoning
National University of Singapore · Center of AI Research, VinUniversity
Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagery
May 26, 2026Yifan Jiang, Ruoxi Ning, Sheng Yao +1VLM Evaluation Vision-Language Models
University of Waterloo · Vector Institute

Jul 1, 2026J/K move · Enter open · S save

Disentangling Pictorial Cue Understanding from Language Bias in VLMs via Depth Ordering Task

Yiqian Liu, Iuliia Kotseruba, John K. Tsotsos

York University, Toronto, Canada · University of Guelph, Guelph, Canada

VLM Evaluation Vision-Language Models 3D VLMs

Open paper AskPDF arXiv

In this paper, we study depth perception of vision-language models (VLMs) to isolate the effects of pictorial depth cues and disentangle vision and language influences on model performance. To this end, we combine depth-ordering and odd-one-out psychophysical tasks: the VLMs are presented with images where one object is at different depth relative to other, otherwise identical, objects, and must determine whether the odd-one-out target is closer or farther to the observer. To create stimuli, we generate 2D views from simulated and real 3D scenes while controlling the presence of individual pictorial depth cues, enabling a fine-grained analysis of cue-level contributions. Language effects are examined by varying referring expression clarity. We also introduce a novel metric to quantify vision-vs-language sensitivities. Applying this methodology, we create the Odd-One-Out Depth (O3-D) dataset with 37K real and synthetic images and 147K image-question pairs. Evaluation of 12 open-source and commercial models on O3-D shows under-utilization of depth cues and depth-ordering accuracies between 47% and 56%, with no model above chance level. At the same time, our metric reveals strong linguistic bias in the answers. Neither chain-of-thought (CoT) nor in-context learning (ICL) significantly improves performance, suggesting that static image data alone may be insufficient for depth understanding. All code, the image generation pipeline, and the O3-D dataset are publicly released at https://github.com/lyiqian/o3-d.

Abstract

VLM Interpretability

Explore similar work

Disentangling Pictorial Cue Understanding from Language Bias in VLMs via Depth Ordering Task

When Depth Is Better Told Than Shown: Depth-Ordinal Prompting for Vision-Language Spatial Reasoning

Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagery