CueClash: Frontier Vision-Language Models Judge 3-D Relations Close to the Picture Plane
Organizations: Scale AI
Abstract
A vision-language model (VLM) can judge 3-D relations from the picture or from the ground layout. In photographs the two usually agree. We separate them with CueClash, rendered scenes in which the 2-D cue for each of five relations is right on exactly half the statements. The shortcut gap is accuracy when the cue is right minus accuracy when it is wrong. For Claude Opus 5.5, GPT-5.5 and Gemini 3.6 Flash at low reasoning effort it is +18 to +37 percentage points. It is +7 to +29 without two relations whose cue-wrong items are confounded with borderline cases or occlusion. Given ground coordinates and each relation's definition, they answer 99 to 100% correctly. Pixel measurements with camera depths remove most of the gap for objects fitting through openings. The failure lies in the step from the picture to the layout. The maps that the models draw of the scene from above mostly keep the depth order but, like the picture, shorten depth. GPT-5.5's maps stay close to a copy of the picture. Opus 5.5's and Gemini's maps undo about 0.6 of this shortening. Gemini's then shrink all depths by a further uniform factor. Opus 5.5 follows the picture mainly on fits, and its maps correct depth about as much as human observers of real scenes do. The three models also lose accuracy on CV-Bench photographs where a reliable 2-D cue misleads. In two open models the layout information reaches the language model, but their maps lose much of it.
Figures & tables
| relation | 3-D truth | 2-D cue (balanced) | reject unless |
|---|---|---|---|
| closer (control) | camera depth of A’s centre B’s | A covers more pixels than B | depth ratio |
| taller | height(A) height(B) | pixel height of A B | height ratio |
| touching | surface distance | visible masks within 3 px | or |
| between | on the ground | same test on visible-mask centroids ( ) | or |
| fits | footprint gap(B,C) width(A) | pixel gap(B,C) pixel width(A) | ratio or |
| easy | medium | hard | |||||||
|---|---|---|---|---|---|---|---|---|---|
| model | acc | gap [95% CI] | w/o between , touching | acc | gap [95% CI] | w/o between , touching | acc | gap [95% CI] | w/o between , touching |
| 2-D cue | 50.0 | +100.0 | – | 50.0 | +100.0 | – | 50.0 | +100.0 | – |
| Random | 52.0 | +3.3 [ 5, +11] | – | 51.0 | 4.7 [ 13, +3] | – | 50.7 | 2.7 [ 10, +5] | – |
| Claude Opus 5.5 | 84.5 | +19.7 [+14, +25] | +7.2 | 85.7 | +21.3 [+16, +27] | +11.7 | 84.2 | +17.7 [+12, +23] | +8.9 |
| GPT-5.5 | 75.8 | +37.0 [+31, +43] | +29.4 | 78.5 | +35.0 [+29, +41] | +27.2 | 76.0 | +28.7 [+22, +35] | +27.2 |
| Gemini 3.6 Flash | 81.2 | +24.3 [+18, +30] | +16.7 | 81.7 | +24.7 [+19, +30] | +20.6 | 78.5 | +23.7 [+18, +30] | +22.2 |
| cue-right | cue-wrong | ||||
|---|---|---|---|---|---|
| tier | reader | fits | does not fit | fits | does not fit |
| easy | size-constancy reader | 97 | 100 | 17 | 97 |
| Claude Opus 5.5 | 100 | 90 | 73 | 80 | |
| GPT-5.5 | 93 | 93 | 27 | 50 | |
| Gemini 3.6 Flash | 97 | 93 | 70 | 63 | |
| medium | size-constancy reader | 97 | 100 | 3 | 93 |
| input (text) | closer | taller | touching | between | fits |
| visible-region centres (text) | – | – | – | 72 † | – |
| ground-contact pixels (text) | – | – | – | 94 † | – |
| pixel measurements (text) | 100 † | 50 † | 62 † | 94 † | 65 † |
| pixel measurements + camera depth (text) | 100 | 69 † | 96 † | 100 | 97 † |
| ground-contact pixels + homography (text) | – | – | – | 100 | – |
| pairwise surface distances (text) | – | – | 100 | – | – |
| Claude Opus 5.5 | GPT-5.5 | Gemini 3.6 Flash | ||||
|---|---|---|---|---|---|---|
| input | gap | vs image [95% CI] | gap | vs image [95% CI] | gap | vs image [95% CI] |
| image | +21.3 | – | +35.0 | – | +24.7 | – |
| stated thresholds | +11.7 | 9.7 [ 14, 5] | +33.0 | 2.0 [ 7, +3] | +22.0 | 2.7 [ 7, +2] |
| verbal definitions (no bands) | +14.7 | 6.7 [ 10, 3] | +33.3 | 1.7 [ 7, +4] | +24.3 | 0.3 [ 5, +5] |
| pixel measurements (text) | +15.0 | 6.3 [ 11, 2] | +30.7 | 4.3 [ 10, +1] | +24.3 | 0.3 [ 6, +5] |
| pixel measurements, no image | +12.0 | 9.3 [ 14, 4] | +19.0 | 16.0 [ 22, 10] | +16.0 | 8.7 [ 16, 1] |
| Claude Opus 5.5 | GPT-5.5 | Gemini 3.6 Flash | ||||
|---|---|---|---|---|---|---|
| input | fits | between | fits | between | fits | between |
| image | +32 | +45 | +65 | +43 | +45 | +43 |
| pixel measurements, with the image | +15 | +33 | +68 | +30 | +42 | +37 |
| pixel measurements, no image | 3 | +23 | +42 | +18 | +18 | +15 |
| pixel measurements + camera depths, with the image | +10 | +33 | +17 | +28 | +20 | +37 |
| ground coordinates, no image, no definitions | +0 | +28 | +0 | +23 | 2 | +10 |
| re-derived from the drawn map | map gap by relation | ||||||||||||
| model | scenes parsed | RMSE (units, median) | depth-to-width ratio [95% CI] | constancy index [95% CI] | share corrected | % com- pressed | cue- right | cue- wrong | gap | closer | touching | between | fits |
| medium tier | |||||||||||||
| Claude Opus 5.5 | 304/304 | 0.24 | 0.83 [0.80, 0.84] | 0.60 [0.58, 0.63] | 0.64 | 88 | 80 | 60 | +20 | 7 | +0 | +47 | +42 |
| GPT-5.5 | 304/304 | 0.46 | 0.61 [0.60, 0.65] | 0.01 [ 0.07, 0.07] | 0.18 | 87 | 75 | 38 | +38 | +0 | +42 | +65 | +43 |
| Gemini 3.6 Flash | 304/304 | 0.44 | 0.63 [0.60, 0.65] | 0.07 [0.01, 0.12] | 0.58 | 96 | 74 | 43 | +30 | +3 | +17 | +55 | +47 |
| Pixtral Large | 303/304 | 0.86 | 0.62 [0.58, 0.65] | – | – | 86 | 48 | 33 | +15 | 8 | +23 | +33 | +15 |
| Depth: gap [95% CI] under each cue | Distance | ||||
| model | acc. | larger box | lower box | acc. | smaller pixel distance |
| Claude Opus 5.5 | 98.2 | +0 [ 2, +3] | +10 [+4, +17] | 95.3 | +19 [+12, +27] |
| GPT-5.5 | 95.0 | +1 [ 2, +5] | +18 [+10, +27] | 90.3 | +48 [+38, +57] |
| Gemini 3.6 Flash | 95.7 | +3 [ 0.3, +7] | +22 [+13, +31] | 91.7 | +28 [+20, +37] |
| Qwen3-VL-4B | 95.8 | +3 [ 0.5, +6] | +6 [+0.1, +13] | 88.7 | +34 [+25, +43] |
| Qwen3-VL-8B | 94.3 | 1 [ 5, +3] | +7 [+0.4, +14] | 90.8 | +44 [+35, +53] |
| reader | depth [95% CI] | lateral | depth-to-width ratio | acc. | gap [95% CI] | fits gap |
|---|---|---|---|---|---|---|
| Hidden states, layer 16 | 0.93 [0.92, 0.93] | 0.94 | 1.01 | 80.2 | +8 [+2, +14] | +27 |
| Reader of the 2-D measurements (ridge) | 0.87 [0.86, 0.88] | 0.96 | 1.00 | 87.8 | +6 [+1, +11] | +28 |
| Visible boxes only, ridge | 0.86 [0.84, 0.87] | 0.96 | 0.97 | 79.5 | +20 [+13, +26] | +37 |
| Gradient-boosted trees on 2-D measurements | 0.95 [0.94, 0.95] | 0.98 | – | – | – | – |
| Qwen3-VL-4B’s drawn map | – | – | 0.25 | 31.5 | +16 | +15 |
| Qwen3-VL-4B’s answers from the image | – | – | – | 61.8 | +14 | +2 |
| image | coordinates only | coordinates + numeric def. | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| model | acc. | gap | closer | taller | touching | between | fits | acc. | gap | acc. / gap |
| Qwen3-VL-4B | 61.8 | +13.7 | +3 | +33 | +18 | +12 | +2 | 53.0 | +3.3 | 53.3 / +0.7 |
| Qwen3-VL-8B | 63.3 | +14.0 | +10 | +22 | +30 | +18 | 10 | 55.8 | +3.7 | 54.3 / +5.3 |
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
| input (all with numeric definitions) | Claude Opus 5.5 | GPT-5.5 | Gemini 3.6 Flash | solver |
|---|---|---|---|---|
| image | 87.5 / +18.3 [+7, +30] | 78.3 / +36.7 [+23, +50] | 82.5 / +28.3 [+16, +41] | – |
| visible-region centres (text) | 61.7 / +50.0 [+35, +65] | 61.7 / +53.3 [+39, +68] | 60.0 / +53.3 [+39, +68] | 72 † |
| ground-contact pixels (text) | 94.2 / +5.0 [ 3, +14] | 93.3 / +6.7 [ 2, +16] | 92.5 / +5.0 [ 4, +15] | 94 † |
| ground-contact pixels + homography (text) | 99.2 / +1.7 [+0, +6] | 100.0 / +0.0 [+0, +0] | 91.7 / 3.3 [ 14, +6] | 100 |
| ground coordinates (text) | 100.0 / +0.0 [+0, +0] | 100.0 / +0.0 [+0, +0] | 100.0 / +0.0 [+0, +0] | 100 |
| in contact | apart | |||||
|---|---|---|---|---|---|---|
| model | input | masks within 3 px | hidden | masks apart | overlapping | acc / gap |
| Claude Opus 5.5 | image | 63 | 10 | 100 | 100 | 68 / +27 |
| Claude Opus 5.5 | image + numeric definitions | 70 | 20 | 100 | 97 | 72 / +27 |
| Claude Opus 5.5 | coordinates only | 90 | 83 | 100 | 100 | 93 / +3 |
| Claude Opus 5.5 | coordinates only + verbal definitions | 90 | 77 | 100 | 100 | 92 / +7 |
| Claude Opus 5.5 | coordinates only + numeric definitions | 100 | 100 | 100 | 97 | 99 / +2 |
| model | prompt prefix | acc / gap [95% CI] | gap vs. default [CI] | touching | between | fits |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | neither (default prompt) | 85.7 / +21.3 [+16, +27] | – | +27 | +45 | +32 |
| Claude Opus 5.5 | numeric definitions only | 89.0 / +14.0 [+9, +19] | 7.3 [ 13, 2] | +27 | +18 | +23 |
| Claude Opus 5.5 | reject bands only | 92.3 / +12.0 [+8, +16] | 9.3 [ 14, 4] | +7 | +27 | +22 |
| Claude Opus 5.5 | numeric definitions + bands | 94.2 / +9.0 [+5, +13] | 12.3 [ 18, 7] | +17 | +8 | +18 |
| Claude Opus 5.5 | stated thresholds (original wording) | 93.2 / +11.7 [+8, +16] | 9.7 [ 14, 5] | +15 | +15 | +20 |
| Claude Opus 5.5 | verbal definitions | 88.0 / +14.7 [+10, +19] | 6.7 [ 10, 3] | +20 | +28 | +20 |
| image tokens | name tokens | ||||||||
| 4B | 8B | 4B | 8B | ||||||
| relation | edit | rise | fall | rise | fall | rise | fall | rise | fall |
| closer | ground depth | +1.23 | 2.00 | +1.48 | 2.09 | 0.39 | 4.95 | 0.32 | 5.59 |
| picture position | 0.13 | 1.40 | +0.08 | 1.09 | 2.12 | 9.30 | 2.00 | 8.68 | |
| lateral position | 0.02 | 0.09 | 0.06 | 0.06 | 0.27 | 0.63 | 0.38 | 0.61 | |
| random (top PCs) | +0.06 | 0.90 | +0.01 | 0.54 | 0.44 | 1.60 | 0.56 | 1.36 | |
| Qwen3-VL-4B | Qwen3-VL-8B | |||
| reader | acc. | minus answers [95% CI] | acc. | minus answers [95% CI] |
| model’s answers | 51.0 | – | 51.7 | – |
| answer-position probe, layer 24 (as in the text) | 65.8 | +14.8 [+9.5, +20.0] | 64.8 | +13.1 [+7.5, +18.6] |
| same probe, chosen within each training fold | 65.8 | +14.8 [+9.5, +20.0] | 65.4 | +13.7 [+8.1, +19.4] |
| same probe, trained on cue-right items only | 54.6 | +3.5 [ 1.6, +8.5] | 54.4 | +2.7 [ 2.6, +7.7] |
| same probe, trained on a random subset of that size (mean SD of 30 draws) | 58.1 | +7.0 2.0 | 57.3 | +5.7 1.3 |
| model | relation | edit | layer 8 | layer 12 | layer 16 | layer 20 | layer 24 | flips at 12 (%) |
| 4B | closer | ground depth | +1.37 / 2.01 | +1.23 / 2.00 | +0.30 / 0.77 | +0.04 / 0.08 | +0.01 / 0.04 | 8.8 / 1.7 |
| picture position | +0.24 / 1.87 | 0.13 / 1.40 | 0.00 / 0.28 | 0.00 / 0.14 | +0.02 / 0.07 | 3.8 / 3.8 | ||
| both | +1.47 / 3.36 | +0.44 / 2.19 | +0.54 / 0.93 | 0.04 / 0.14 | +0.02 / 0.04 | 7.1 / 2.9 | ||
| lateral position | 0.02 / 0.09 | 0.02 / 0.09 | 0.11 / 0.04 | +0.03 / 0.05 | 0.01 / 0.03 | 0.4 / 0.4 | ||
| random (isotropic) | +0.20 / 0.58 | +0.02 / 0.59 | 0.05 / 0.27 | +0.00 / 0.07 | +0.02 / 0.05 | 1.2 / 2.5 | ||
| random (top PCs) | +0.11 / 1.19 | +0.06 / 0.90 | 0.03 / 0.31 | +0.03 / 0.07 | +0.01 / 0.03 | 2.9 / 2.1 |
| depth/width ratio by camera-elevation tertile | |||||||
|---|---|---|---|---|---|---|---|
| tier | map | scenes | constancy index [95% CI] | low | middle | high | median ratio |
| medium | picture-plane copy | – | 0 | 0.51 | 0.60 | 0.69 | 0.61 |
| Claude Opus 5.5 | 304 | 0.60 [0.58, 0.63] | 0.76 | 0.83 | 0.88 | 0.83 | |
| GPT-5.5 | 304 | 0.01 [ 0.07, 0.07] | 0.53 | 0.61 | 0.72 | 0.61 | |
| Gemini 3.6 Flash | 304 | 0.07 [0.01, 0.12] | 0.57 | 0.61 | 0.70 | 0.63 | |
| Grok 4.7 | 224 | 0.37 [0.30, 0.45] | 0.65 | 0.75 | 0.81 | 0.75 | |
| map | scenes | intercept [95% CI] | slope [95% CI] | share corrected |
|---|---|---|---|---|
| Claude Opus 5.5 | 606 | 0.01 [ 0.04, +0.02] | 0.36 [0.32, 0.39] | 0.64 |
| GPT-5.5 | 606 | 0.01 [ 0.07, +0.05] | 0.82 [0.76, 0.89] | 0.18 |
| Gemini 3.6 Flash | 605 | 0.25 [ 0.30, 0.19] | 0.42 [0.36, 0.48] | 0.58 |
| Grok 4.7 | 430 | 0.06 [ 0.14, +0.02] | 0.40 [0.29, 0.49] | 0.60 |
| one source only | leaving one source out | ||||||
|---|---|---|---|---|---|---|---|
| task (cue) | model | SUN RGB-D | nuScenes | Hypersim | without SUN RGB-D | without nuScenes | without Hypersim |
| Depth (lower box) | items (cue-wrong) | 200 (36) | 200 (12) | 200 (40) | |||
| Claude Opus 5.5 | +8 [+0, +19] | +32 [+7, +58] | +5 [+0, +12] | +11 [+3, +21] | +7 [+1, +13] | +14 [+5, +25] | |
| GPT-5.5 | +33 [+17, +50] | +29 [+4, +57] | +3 [ 4, +11] | +8 [+0, +17] | +17 [+9, +26] | +31 [+17, +44] | |
| Gemini 3.6 Flash | +22 [+8, +36] | +57 [+30, +82] | +11 [+1, +22] | +21 [+11, +33] | +16 [+8, +25] | +30 [+18, +44] | |
| Qwen3-VL-4B | +2 [ 1, +8] | +35 [+9, +61] | +3 [ 3, +11] | +9 [+1, +19] | +3 [ 1, +7] | +9 [+0, +19] | |
| tier | objects | elevation ( ∘ ) | shadows | occlusion | items | scenes | scenes sampled |
|---|---|---|---|---|---|---|---|
| easy | 4–5 | 45–60 | on | low | 600 | 343 | 1402 |
| medium | 5–7 | 25–45 | on | medium | 600 | 304 | 315 |
| hard | 6–8 | 8–20 | off | high | 600 | 302 | 313 |
| tier | closer | taller | touching | between | fits | closer: lower in frame |
|---|---|---|---|---|---|---|
| easy | 63 / 63 / 50 (532) | 87 / 87 / 50 (1754) | 96 / 98 / 3 (1008) | 100 / 99 / 3 (2741) | 89 / 91 / 63 (1722) | 100 (532) |
| medium | 58 / 58 / 50 (1534) | 91 / 91 / 50 (3062) | 89 / 94 / 2 (1824) | 99 / 97 / 3 (7830) | 82 / 84 / 59 (4172) | 100 (1534) |
| hard | 52 / 52 / 50 (2314) | 95 / 95 / 50 (3770) | 84 / 89 / 1 (2208) | 94 / 89 / 3 (10181) | 76 / 82 / 66 (5453) | 100 (2314) |
| passage (64 items) | elevated tier (96 items) | |||
|---|---|---|---|---|
| model | acc | gap [95% CI] | acc | gap [95% CI] |
| Claude Opus 5.5 | 89.1 | +15.6 [+1, +31] | 100.0 | +0.0 [+0, +0] |
| GPT-5.5 | 84.4 | +12.5 [ 5, +30] | 96.9 | +6.2 [+0, +14] |
| Gemini 3.6 Flash | 81.2 | +6.2 [ 12, +25] | 96.9 | +6.2 [+0, +14] |
| Pixtral Large | 57.8 | +3.1 [ 22, +28] | 62.5 | +20.8 [+1, +40] |
| Qwen3-VL-235B | 64.1 | +3.1 [ 20, +27] | 89.6 | +12.5 [+0, +25] |
| model | input | acc | cue right | cue wrong | gap [95% CI] | |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | image | 600 | 85.7 | 96.3 | 75.0 | +21.3 [+16, +27] |
| Claude Opus 5.5 | image + CoT | 600 | 86.5 | 95.0 | 78.0 | +17.0 [+12, +22] |
| Claude Opus 5.5 | image + tools (available) | 600 | 86.3 | 94.3 | 78.3 | +16.0 [+11, +21] |
| Claude Opus 5.5 | image + tools (required first) | 600 | 86.0 | 94.7 | 77.3 | +17.3 [+12, +23] |
| Claude Opus 5.5 | image + stated thresholds | 600 | 93.2 | 99.0 | 87.3 | +11.7 [+8, +16] |
| Claude Opus 5.5 | image + numeric definitions only | 600 | 89.0 | 96.0 | 82.0 | +14.0 [+9, +19] |
| taller | touching | between | fits | |||||
|---|---|---|---|---|---|---|---|---|
| model | natural | constructed | natural | constructed | natural | constructed | natural | constructed |
| Claude Opus 5.5 | 3 (35) | +3 (145) | +0 (13) | +31 (167) | +57 (8) | +41 (172) | +23 (12) | +27 (168) |
| GPT-5.5 | +20 (35) | +21 (145) | +69 (13) | +38 (167) | +57 (8) | +43 (172) | +78 (12) | +62 (168) |
| Gemini 3.6 Flash | +11 (35) | +17 (145) | +0 (13) | +22 (167) | +51 (8) | +42 (172) | +23 (12) | +46 (168) |
| model | repeats | accuracies | gaps | gap s.d. | item-level agreement |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 3 | 85.7/86.5/86.2 | +21.3/+19.7/+19.0 | 1.2 | 92.8 |
| GPT-5.5 | 3 | 78.5/77.7/78.8 | +35.0/+36.0/+34.3 | 0.8 | 89.5 |
| Gemini 3.6 Flash | 3 | 81.7/82.0/80.8 | +24.7/+22.7/+21.7 | 1.5 | 88.8 |
| low effort | high effort (16k budget, single run) | high effort (4k budget + retries) | ||||
|---|---|---|---|---|---|---|
| model | acc | gap [95% CI] | acc | gap [95% CI] | acc | gap [95% CI] |
| Claude Opus 5.5 | 85.7 | +21.3 [+16, +27] | 88.0 | +16.7 [+12, +22] | 87.8 | +15.0 [+10, +20] |
| GPT-5.5 | 78.5 | +35.0 [+29, +41] | 83.7 | +25.3 [+20, +31] | 82.3 | +30.0 [+24, +36] |
| Gemini 3.6 Flash | 81.7 | +24.7 [+19, +30] | 83.0 | +17.3 [+12, +23] | 80.3 | +23.3 [+17, +29] |
| plain ground | checkerboard ground | |||
|---|---|---|---|---|
| model | acc | gap | acc | gap [95% CI] |
| Claude Opus 5.5 | 85.7 | +21.3 | 86.8 | +18.3 [+13, +24] |
| GPT-5.5 | 78.5 | +35.0 | 79.2 | +31.0 [+25, +37] |
| Gemini 3.6 Flash | 81.7 | +24.7 | 81.8 | +23.7 [+18, +30] |
| relation | regressor (members) | agree. | VIF | Claude Opus 5.5 | GPT-5.5 | Gemini 3.6 Flash | Grok 4.7 | Qwen3-VL-235B | Pixtral Large | Nova 2 Pro | Llama 4 Maverick | Gemma 3 27B |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| taller | truth | – | 1.4 | +4.0 | +2.6 | +4.1 | +1.8 | +1.0 | +0.6 | +0.8 | +0.5 | +0.1 |
| pixel cues (pixel height, area) | 50/48 | 1.1 | +0.9 | +1.3 | +1.9 | +1.3 | +1.0 | +0.3 | +0.9 | +0.4 | +0.5 | |
| geometry-faithful (depth-corrected height, top y) | 69/71 | 1.5 | +0.3 | +1.1 | +0.6 | +1.0 | +1.2 | +0.6 | +0.9 | 0.5 | +0.4 | |
| touching | truth | – | 2.2 | +2.2 | +0.7 | +0.9 | +0.6 | +0.7 | +0.2 | +0.9 | +0.3 | 0.8 |
| silhouette cues (mask distance, bbox overlap) | 50/50 | 1.3 | +1.3 | +1.4 | +0.5 | +1.5 | +1.0 | +0.6 | +1.0 | +1.1 | +0.2 | |
| geometry-faithful (base points close) | 82 | 2.5 | 0.2 | +0.3 | +0.6 | +0.2 | +0.4 | +0.2 | +0.3 | +0.3 | +1.0 |
| model | tier | raw gap | adjusted gap [95% CI] | raw gap by margin tertile (T1/T2/T3) | adjusted gap by margin tertile |
|---|---|---|---|---|---|
| Claude Opus 5.5 | easy | +19.7 | +10.3 [+5, +15] | +29 / +4 / 4 | +19 / +2 / 2 |
| Claude Opus 5.5 | medium | +21.3 | +14.6 [+11, +18] | +33 / +10 / +0 | +28 / +10 / +0 |
| Claude Opus 5.5 | hard | +17.7 | +12.3 [+7, +17] | +18 / +12 / +12 | +16 / +12 / +3 |
| GPT-5.5 | easy | +37.0 | +28.6 [+21, +35] | +40 / +34 / +3 | +29 / +27 / +14 |
| GPT-5.5 | medium | +35.0 | +27.5 [+22, +33] | +42 / +22 / +9 | +31 / +27 / +14 |
| GPT-5.5 | hard | +28.7 | +23.0 [+17, +29] | +37 / +27 / +22 | +28 / +27 / +17 |
| tool use | acc / gap by episode | gap by relation | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| input | acc | gap [95% CI] | gap vs. low [CI] | calls/item | % used | % python | % zoom | tools used | not used | closer | taller | touching | between | fits |
| image, no tools (low effort) | 85.7 | +21.3 [+16, +27] | – | – | – | – | – | – | – | +0 | +3 | +27 | +45 | +32 |
| image, no tools (high effort, 16k) | 88.0 | +16.7 [+12, +22] | 4.7 [ 9, 1] | – | – | – | – | – | – | +0 | +2 | +30 | +42 | +10 |
| tools available | 86.3 | +16.0 [+11, +21] | 5.3 [ 9, 2] | 0.5 | 50 | 22 | 30 | 77 / +15 | 96 / +11 | +0 | +0 | +22 | +40 | +18 |
| tools required first | 86.0 | +17.3 [+12, +23] | 4.0 [ 8, 1] | 0.6 | 55 | 25 | 32 | 80 / +18 | 93 / +13 | +0 | +2 | +22 | +42 | +22 |
| accuracy | grouped by view-1 cue (231 / 234) | grouped by view-2 cue (397 / 68) | |||||
|---|---|---|---|---|---|---|---|
| model | view 1 | view 2 | [95% CI] | gap, view 1 | gap, view 2 | gap, view 1 | gap, view 2 |
| Claude Opus 5.5 | 81.9 | 86.5 | +4.5 [+1, +8] | +26.4 [+20, +33] | +11.4 [+6, +17] | 7.4 [ 16, +1] | +3.1 [ 6, +12] |
| GPT-5.5 | 72.7 | 83.2 | +10.5 [+6, +15] | +44.0 [+37, +51] | +11.8 [+5, +18] | +2.5 [ 9, +14] | +30.3 [+19, +42] |
| Gemini 3.6 Flash | 76.8 | 82.6 | +5.8 [+2, +10] | +30.7 [+23, +38] | +12.2 [+5, +19] | 8.3 [ 17, +2] | +8.9 [ 2, +20] |
| model | items | image + numeric definitions (acc / gap) | drawn map | drawn map, depth-corrected | gap [95% CI] | answers = solver (%) |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 240 | 87.1 / +20.8 | 77.1 / +35.8 | 79.2 / +28.3 | 7.5 [ 12, 3] | 100 |
| GPT-5.5 | 240 | 72.1 / +52.5 | 67.9 / +50.8 | 70.4 / +44.2 | 6.7 [ 15, +2] | 100 |
| Gemini 3.6 Flash | 240 | 77.9 / +37.5 | 67.5 / +30.0 | 73.3 / +25.0 | 5.0 [ 14, +4] | 100 |
| common support | gap (points) | adjusted gap, within relation | ||||||
|---|---|---|---|---|---|---|---|---|
| relation | model | % off | supported (cue right/wrong) | raw | supported | IPW | binary construct [CI] | construct type [CI] |
| taller | Claude Opus 5.5 | 9 | 102 (43/59) | +3 | +3 | +3 | +3 [ 6, +13] | +5 [ 2, +14] |
| taller | GPT-5.5 | 9 | 102 (43/59) | +17 | +16 | +13 | +10 [ 1, +27] | +15 [+1, +33] |
| taller | Gemini 3.6 Flash | 9 | 102 (43/59) | +17 | +17 | +15 | +14 [+4, +29] | +15 [+4, +33] |
| touching | Claude Opus 5.5 | 20 | 87 (42/45) | +27 | +29 | +31 | +32 [+15, +44] | +33 [+18, +48] |
| touching | GPT-5.5 | 20 | 87 (42/45) | +50 | +46 | +51 | +39 [+20, +57] | +39 [+21, +59] |
| model | tier | closer: raw / shared [CI] / within | taller: raw / shared [CI] / within | touching: raw / shared [CI] / within | between: raw / shared [CI] / within | fits: raw / shared [CI] / within |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | easy | 2 / 2 [ 7, +0] / 3 | +5 / 0 [ 9, +7] / +0 | +35 / +27 [+12, +40] / +14 | +42 / +20 [+8, +32] / +12 | +18 / +7 [ 8, +18] / +7 |
| Claude Opus 5.5 | medium | +0 / +0 [ 1, +3] / 0 | +3 / +1 [ 4, +5] / +3 | +27 / +23 [+10, +35] / +27 | +45 / +25 [+16, +34] / +23 | +32 / +24 [+14, +34] / +27 |
| Claude Opus 5.5 | hard | +0 / 0 [ 1, +0] / +0 | 3 / 6 [ 14, 0] / 5 | +25 / +19 [+9, +31] / +24 | +37 / +22 [+9, +35] / +22 | +30 / +26 [+14, +40] / +38 |
| GPT-5.5 | easy | +0 / 0 [ 1, +0] / +0 | +33 / +25 [+16, +35] / +24 | +50 / +49 [+34, +64] / +49 | +47 / +35 [+19, +49] / +21 | +55 / +34 [+13, +51] / +41 |
| GPT-5.5 | medium | +0 / 0 [ 1, +0] / 0 | +17 / +11 [+1, +21] / +11 | +50 / +41 [+25, +56] / +38 | +43 / +29 [+12, +44] / +10 | +65 / +56 [+42, +69] / +61 |
| GPT-5.5 | hard | +0 / 0 [ 6, +4] / +2 | +13 / +10 [+3, +17] / +15 | +22 / +19 [+6, +34] / +20 | +40 / +30 [+15, +45] / +25 | +68 / +57 [+40, +72] / +68 |
| image | ground coordinates only | image + top-down map | ||||
|---|---|---|---|---|---|---|
| model | seed 7 | seed 11 | seed 7 | seed 11 | seed 7 | seed 11 |
| Claude Opus 5.5 | 85.7 / +21.3 | 83.8 / +19.7 | 95.5 / +6.3 | 95.7 / +2.7 | 93.0 / +4.0 | 93.7 / +4.0 |
| GPT-5.5 | 78.5 / +35.0 | 76.8 / +35.0 | 92.5 / +6.3 | 91.8 / +2.3 | 93.0 / +5.3 | 93.5 / +4.3 |
| Gemini 3.6 Flash | 81.7 / +24.7 | 79.3 / +22.7 | 92.8 / +3.7 | 92.2 / +3.0 | 92.0 / +4.7 | 92.0 / +6.0 |
| direct / map-derived correct | map gap | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| model | scenes parsed | items | both | direct only | map only | neither | agree % | MH odds ratio | % in band | parsed | parse-incl. | |
| Claude Opus 5.5 | 304/304 | 480 | 305 | 91 | 30 | 54 | 75 | 0.34 | 2.4 | 23 | +20 | +20 |
| GPT-5.5 | 304/304 | 480 | 245 | 118 | 27 | 90 | 70 | 0.38 | 2.1 | 31 | +38 | +38 |
| Gemini 3.6 Flash | 304/304 | 480 | 260 | 120 | 21 | 79 | 71 | 0.39 | 3.9 | 28 | +30 | +30 |
| Grok 4.7 | 224/304 | 353 | 188 | 75 | 31 | 59 | 70 | 0.33 | 2.3 | 25 | +18 | +16 |
| model | input | (true) | 30–40 ∘ | 40–60 ∘ | |
|---|---|---|---|---|---|
| Claude Opus 5.5 | image | 100 (60) | 18 (17) | 24 (17) | 100 (26) |
| Claude Opus 5.5 | verbal definitions | 100 (60) | 53 (17) | 47 (17) | 100 (26) |
| Claude Opus 5.5 | stated thresholds | 98 (60) | 71 (17) | 82 (17) | 100 (26) |
| Claude Opus 5.5 | coordinates only | 100 (60) | 29 (17) | 59 (17) | 100 (26) |
| Claude Opus 5.5 | coordinates only + verbal definitions | 100 (60) | 76 (17) | 94 (17) | 100 (26) |
| Claude Opus 5.5 | coordinates + image | 100 (60) | 35 (17) | 88 (17) | 100 (26) |
| model | tier | agreement | both correct | cue flips | agreement on flips | agreement on non-flips | |
|---|---|---|---|---|---|---|---|
| 2-D cue | easy | 476 | 62.6 | 48.3 | 178 | 0.0 | 100.0 |
| 2-D cue | medium | 465 | 56.1 | 45.6 | 204 | 0.0 | 100.0 |
| 2-D cue | hard | 416 | 52.4 | 44.2 | 198 | 0.0 | 100.0 |
| Claude Opus 5.5 | medium | 465 | 86.9 | 77.6 | 204 | 79.9 | 92.3 |
| GPT-5.5 | medium | 465 | 74.8 | 65.4 | 204 | 57.4 | 88.5 |
| Gemini 3.6 Flash | medium | 465 | 80.0 | 69.7 | 204 | 71.1 | 87.0 |
| model | model identifier | reasoning effort |
|---|---|---|
| Claude Opus 5.5 | claude-opus-5-5 | low, high |
| Gemini 3.6 Flash | gemini-3.6-flash | low, high |
| Grok 4.7 | global.xai.grok-4.7 | provider default |
| Gemma 3 27B | google.gemma-3-27b-it | provider default |
| Qwen3-VL-235B | qwen.qwen3-vl-235b-a22b | provider default |
| Nova 2 Pro | us.amazon.nova-2-pro-preview-20251202-v1:0 | provider default |
| model | responses | unparseable (%) | API error (%) | truncated (%) |
|---|---|---|---|---|
| Claude Opus 5.5 | 1800 | 0.0 | 0.0 | 0.0 |
| Claude Opus 5.5 (high effort) | 600 | 0.0 | 0.0 | 0.0 |
| GPT-5.5 | 1800 | 0.0 | 0.0 | 0.0 |
| GPT-5.5 (high effort) | 600 | 0.3 | 0.0 | 0.3 |
| Gemini 3.6 Flash | 1800 | 0.0 | 0.0 | 0.0 |
| Gemini 3.6 Flash (high effort) | 600 | 0.0 | 0.0 | 0.7 |
| model | calls | input tokens (M) | output tokens (M) |
|---|---|---|---|
| Claude Opus 5.5 | 21,491 | 26.8 | 5.66 |
| Claude Opus 5.5 (high effort) | 1,200 | 1.1 | 0.32 |
| GPT-5.5 | 20,191 | 17.5 | 11.17 |
| GPT-5.5 (high effort) | 1,200 | 0.9 | 3.58 |
| Gemini 3.6 Flash | 20,191 | 25.6 | 8.81 |
| Gemini 3.6 Flash (high effort) | 1,200 | 1.4 | 2.45 |
3D VLMs
Three-dimensional vision-language models connect language with spatially grounded visual data for 3D understanding, reasoning, and action.
See all 228 papers