cs.CVNov 17, 2026

CueClash: Frontier Vision-Language Models Judge 3-D Relations Close to the Picture Plane

Authors: Soham Dan

Organizations: Scale AI

Abstract

A vision-language model (VLM) can judge 3-D relations from the picture or from the ground layout. In photographs the two usually agree. We separate them with CueClash, rendered scenes in which the 2-D cue for each of five relations is right on exactly half the statements. The shortcut gap is accuracy when the cue is right minus accuracy when it is wrong. For Claude Opus 5.5, GPT-5.5 and Gemini 3.6 Flash at low reasoning effort it is +18 to +37 percentage points. It is +7 to +29 without two relations whose cue-wrong items are confounded with borderline cases or occlusion. Given ground coordinates and each relation's definition, they answer 99 to 100% correctly. Pixel measurements with camera depths remove most of the gap for objects fitting through openings. The failure lies in the step from the picture to the layout. The maps that the models draw of the scene from above mostly keep the depth order but, like the picture, shorten depth. GPT-5.5's maps stay close to a copy of the picture. Opus 5.5's and Gemini's maps undo about 0.6 of this shortening. Gemini's then shrink all depths by a further uniform factor. Opus 5.5 follows the picture mainly on fits, and its maps correct depth about as much as human observers of real scenes do. The three models also lose accuracy on CV-Bench photographs where a reliable 2-D cue misleads. In two open models the layout information reaches the language model, but their maps lose much of it.

Figures & tables

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Seeing Depth but Choosing Cues in Vision-Language Models

    Oct 13, 2026Zhengtao Zou, Pekka MarttinenVLM Interpretability3D VLMs

  2. Disentangling Pictorial Cue Understanding from Language Bias in VLMs via Depth Ordering Task

    Jul 1, 2026Yiqian Liu, Iuliia Kotseruba, John K. TsotsosVLM EvaluationVision-Language Models