Open-weight Vision-Language Models (VLMs) score 85-94% on spatial benchmarks (BLINK Spatial, What's Up CLEVR) yet collapse to chance on simple geometric questions when object, scene, and texture priors are stripped away. We introduce and open-source Sophon, a procedurally generated diagnostic of paired colored point-cloud stimuli across three coordinate systems (1D linear, 2D Cartesian, 2D polar) and four question types (binary, categorical, comparative, quantitative). Across four open architectures (LLaVA-NeXT, LLaVA-OneVision, Qwen2.5-VL, Idefics3), accuracy clusters at 47-52% while frontier models (Opus 4.6, Gemini 2.5 Pro, GPT-5.4) clear 77-83% and humans reach 84-94%. We use linear probes and low-rank adaptation (LoRA) to trace the open-weight failures. Probes of the vision encoder (ViT) and language decoder (LLM) recover Sophon spatial information at 83-85% and 80-86% accuracy, respectively, and the language tower's reasoning circuit answers text-only controls (point clouds as sets of coordinates) at +40 pp over visual stimuli. Language tower per-layer attention attribution localizes visual integration to mid-stack layers. At readout, models elevate the correct answer-set tokens but select among them randomly. We localize the mechanism by tuning individual and grouped components across every architecture's stack, including the vision tower (ViT workspace and reader), the projector, and the language tower (LLM workspace and reader). These experiments demonstrate that this mid-stack workspace holds circuitry capable of organizing the spatial signal so that the frozen reader can decode it. LoRA fine-tuning adjusts that circuitry such that 7B-class VLMs answer Sophon spatial reasoning questions at frontier aggregate accuracy. This work suggests that these models do not have a capability gap for spatial reasoning but suffer from a routing bottleneck that targeted fine-tuning repairs.
Figures & tables
Figure 1: The diagnostic gap and its mechanism. (a) Despite 85−94% on existing spatial benchmarks, open models show no spatial competence on Sophon while untrained humans solve with high accuracy. (b) Ablation localizes the fine tuning mechanism upstream: the reader-band discriminative contrast at the readout is a downstream effect of upstream workspace change (§ 5.3 ).
Figure 2: Sample Sophon stimuli, one per (question type, coordinate system) cell: 1D linear / categorical (left), Cartesian / binary (center), polar / quantitative (right). Each panel shows its coordinate system and ground truth transform parameter. No objects, scenes, or textures appear in any stimulus (Appendix B ).
Figure 3: The readout-routing bottleneck: spatial content traverses encoder and decoder, then collapses at the readout. (a) Linear probe accuracy on Sophon direction classification at each encoder layer. Five frozen encoders cluster at 83−85% in mid-to-late layers regardless of pretraining objective (Appendix C ). (b) Probe accuracy at the answer emission position across four open architectures, sampled at five depths each: 80−86% at every layer, including the final one. (c) Final layer probe accuracy (solid) vs. end-to-end Sophon categorical accuracy (hatched) across the same four decoders. Every architecture carries the answer in its residual stream and emits at chance, a readout failure.
Figure 4: Quantitative answer distribution taxonomy. Bar height is Pearson r between parsed answer and GT magnitude ( n=200 ). Filled = stimulus engaged: Perceiver ( r≥0.5 , top-1 <50% ), Direction perceiver ( r≥0.5 , top-1 ≥50% ). Hatched/outline = engagement failure: Noisy ( r<0.5 ), Few anchor (top-3 ≥50% ), Constant (top-1 ≥85% ). Labels above bars are answer uniqueness ( nunique/n ). Human scale r reflects rater unit ambiguity. See all models version in Appendix U .
Appendix figures & tables41 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Per-layer GT vs competitor logit gap, five LoRA conditions, four architectures. Each panel projects the answer position hidden state through the unembedding WU at every decoder layer and reports the mean gap between the ground truth direction token and its same axis competitor (e.g. “left” vs “right”). The shaded region marks the reader band. In every architecture Base and Reader sit at or below zero throughout, so reader-band training installs no contrast. The three trained conditions rise late in the stack and largely coincide, and Workspace ’s contrast appears despite the reader band and WU being frozen. The reader band’s contrast emerges when the workspace is reorganized to feed it (§ 5.3 ).
Figure 6: Full Sophon sample stimuli. Top row, left to right : 1D linear/categorical, Cartesian/quantitative, Cartesian/binary. Bottom row : polar/comparative (double wide A-B panel), polar/categorical.
Figure 7: Binary examples across all three coordinate systems.
Figure 8: Categorical examples across all three coordinate systems.
Figure 9: Comparative examples across the two 2D coordinate systems (Cartesian and polar). 1D linear has no comparative cell by design (Figure 13 ).
Figure 10: Quantitative examples across the two 2D coordinate systems (Cartesian and polar). 1D linear has no quantitative cell by design (Figure 13 ).
Figure 11: DINOv2-large probes across Sophon and external benchmarks. Peak accuracies on Sophon: direction 85.4% (L16), coordinate system 97.5% (L1), magnitude R2=0.86 (L20).
Figure 12: SigLIP-large probes. Peaks on Sophon: direction 85.4% (L11), coordinate system 100.0% (L19), magnitude R2=0.89 (L14).
architecture
tower (depth)
workspace
reader
vision tower
LLaVA-OneVision
SigLIP-so400m (26)
L0–22
L23–25
Qwen2.5-VL
windowed ViT (32)
L0–22 c
L23–31 c
LLaVA-NeXT
CLIP-ViT-L (24)
L0–23 (all) b
— b
Idefics3
SigLIP2 (27)
L0–26 (all) b
— b
language tower
Appendix
Table 1: Workspace/reader bands per architecture. Layer indices are inclusive. Projectors (LLaVA-OneVision and LLaVA-NeXT: 2-layer MLP; Qwen2.5-VL: merger MLP; Idefics3: single linear) are never band-split. a The 32-layer decoders cut after L26, the same proportional depth as L22 of 28. b The LLaVA-NeXT and Idefics3 vision towers lie entirely below the cut, so they have no reader band. For these architectures Readers is identical to Reader , and their vision towers are fully contained in the Workspaces condition. c In Qwen2.5-VL’s vision tower the banded conditions ( Workspaces , Readers ) cover the MLP sublayers only. The band configs target the language-model module names, which match the vision SwiGLU MLP but not its fused attention ( qkv / proj ). The two conditions use identical targets with complementary bands, so they remain exact complements. The single-component Vision condition covers attention and MLP.
condition
trainable components
Vision
entire vision tower
Projector
projector
Language
entire language tower
Workspace
language workspace
Reader
language reader
Workspaces
both towers’ workspaces
Appendix
Table 2: Condition names. Conditions are named by what they train, and all other parameters are frozen. d Full trains both towers in their entirety. The projector’s linear layers match none of the LoRA target names, so the projector is trained only in the Projector condition.
Figure 13: Sophon diagnostic heatmap, grouped by tier. Rows are organized into four bands: Human (Prolific n=30 ), Frontier closed weight (Claude Opus 4.6, Gemini 2.5 Pro, GPT-5.4, Grok 4.20), Open 7B base (Qwen2.5-VL-7B, Qwen2.5-VL-32B-AWQ, LLaVA-NeXT-7B, LLaVA-OneVision-7B), and Chance (per-cell random baseline). Columns are coordinate system × question type accuracy cells (1D Linear has no comparative) plus per-coord system rotation MAE and scale MAE columns. Each cell shows mean accuracy or MAE. Quantitative cells are diagonally hatched when parse rate is below 25% (numerically unreliable). Frontier accuracy cells include the 3-seed cross seed standard deviation ±s from the variance subset. Frontier models match or exceed humans on most binary cells but trail on categorical and comparative. Open 7B models cluster near per-cell chance. Grok 4.20 collapses to a near constant token output under greedy decode (Appendix I ).
Model
Sophon Acc
Scale MAE
BLINK Spatial
Gemini 2.5 Pro
82.6%
0.15–0.23
88.8%
GPT-5.4
81.4%
0.06–0.07
86.7%
Claude Opus 4.6
77.2%
0.06–0.09
86.0%
GPT-4o
74.8%
0.75–0.96
81.1%
Grok 4.20
49.8% ‡
9.32–82.83 ‡
82.5%
Qwen2.5-VL-7B
51.1%
0.60–0.70
88.8%
Appendix
Table 3: Per-model Sophon accuracy with external benchmark context. Sophon Acc is the unweighted mean across the 23 underlying (coordinate system, question type) cells of the diagnostic suite (Fig. 13 aggregates these cells into 12 visible columns, 8 accuracy cells plus 4 MAE columns), with 200 samples per cell. ‡ Grok-4.20 emits a single constant token for 17 of the 23 cells under our default greedy decode (per-cell breakdown in Appendix I , temperature ablation in Appendix I ). Its 49.8% mean is the chance hit rate of those constants on balanced GT distributions.
Cell (spec / qtype)
Constant answer
Coverage
all binary specs (×8)
“yes”
97–99%
Cartesian rotation / categorical
“counterclockwise”
100%
polar rotation / categorical
“counterclockwise”
100%
Cartesian rotation / quantitative
“45” (degrees)
88%
polar rotation / quantitative
“45” (degrees)
97%
Cartesian translation x / categ.
“left”
99%
Appendix
Table 4: Grok-4.20 constant token degeneracy on Sophon. “Coverage” is the fraction of samples in that (spec, question type) cell that received the listed top answer. Cells with ≥85% coverage are shown ( n=17 of 23 cells); cells below 85% omitted.
Figure 14: Decoding temperature partially releases Grok-4.20’s constant token attractor on categorical cells but not on binary cells, and does not improve the per-model mean accuracy of any frontier VLM. Top panel: constant token rate (fraction of samples assigned the most common parsed answer) versus decode temperature for the 15 (model, cell) combinations that collapse at temp = 0 (const ≥85% ). Lines staying above the 85% threshold indicate a persistent attractor, and lines dropping below indicate a decode induced artifact. All three GPT-4o (purple) cells release, as does the single Sonnet 4.6 (orange) cell. Of Grok-4.20’s 11 collapsed cells, 4 release (the categorical cells on Cartesian trans x, Cartesian trans y, Cartesian rotation, plus the rotation quantitative cell) while the 7 binary and comparative cells stay ≥88% at every temperature. Bottom panel: per-model mean accuracy across all 11 scoreable cells in the 5 -spec subset, plotted against decode temperature. Frontier models that succeed at greedy decode (Opus 4.6, Sonnet 4.6, Gemini 2.5 Pro, GPT-5.4) remain approximately flat ( Δ≤+5 pp from τ=0 to τ=1.0 ). GPT-4o degrades ∼11 pp. Grok-4.20 stays at chance across the entire sweep. The full per-cell breakdown backing both panels is reported in Table 5 .
Model
Total cells
Collapsed @ τ=0
Released @ τ=1.0
Mean Δ acc
( τ=0→1.0 )
Opus 4.6
11
0
—
+1 pp
Sonnet 4.6
11
1
1 of 1
+1 pp
Gemini 2.5 Pro
11
0
—
+3 pp
GPT-5.4
11
0
—
+5 pp
GPT-4o
11
3
3 of 3
−11 pp
Appendix
Table 5: Temperature ablation summary across the 5 -spec subset for six frontier VLMs. “Total cells” counts the (coordinate system, question type) cells with a defined accuracy ( 11 per model after excluding quantitative cells, where constant rate is not directly interpretable). “Collapsed @ τ=0 ” = constant rate ≥85% on the dominant parsed answer. “Released @ τ=1.0 ” = constant rate drops below 85% at temp = 1.0. Grok-4.20 collapses on every cell at greedy decode; 4 release at temp = 1.0 (predominantly categorical cells), while binary cells stay above 85% at every temperature. GPT-4o is the only frontier model whose mean accuracy degrades substantially with temperature ( −11 pp). Mean Δ acc is over all 11 cells.
Figure 15: Linear probe accuracy on frozen Qwen2.5-VL ViT features across four benchmarks (Sophon + What’s Up + BLINK). Left of each panel : probe accuracy at every encoder layer, with end-to-end VLM accuracy as the dotted reference. Right : end-to-end VLM accuracy bar.
Figure 16: CLIP encoder probes (LLaVA-NeXT). Note that CLIP’s spatial content peaks early (L5–7) and degrades toward the final layer.
Figure 17: Qwen-VL ViT probes after 10k Sophon fine tuning (LoRA 10k). The shift in Sophon direction peak from L14 to L31 relative to the base (Fig. 15 ) is small relative to the decoder side end-to-end gain.
Task
CLIP Probe
LLaVA-NeXT End-to-End
Coordinate system identification
97.5%
44%
Direction classification
81%
48%
Magnitude regression ( R2 )
0.83
−54.8
Appendix
Table 6: Encoder probe vs. end-to-end accuracy on Sophon tasks. The end-to-end magnitude R2 is computed over the responses that parse to a number. LLaVA-NeXT emits out of scale values on a substantial fraction of magnitude items (e.g., predicting 180∘ or 360∘ when the ground truth rotation is under 45∘ ), yielding R2≪0 . The probe R2 is from sklearn ridge regression over the same training/test split. Per cell parse rates and prediction range diagnostics are in Appendix W .
Model
Overall
Rot.
Trans.
Scale
Description to GT
Contradict.
LLaVA base
53%
46%
55%
58%
57%
29%
LLaVA LoRA
52%
48%
52%
57%
41%
23%
Qwen base
52%
45%
51%
58%
63%
27%
Qwen LoRA (1k)
57%
46%
54%
70%
57%
23%
Qwen LoRA (10k)
58%
43%
52%
71%
67%
25%
Qwen-32B-AWQ
52%
44%
56%
52%
54%
41%
Appendix
Table 7: Two turn CoT accuracy by transform type. Description to GT : fraction of categorical samples whose Turn 1 description mentions the correct direction word. Contradict. : subset of those samples where the Turn 2 final answer disagrees with the description’s stated direction.
Figure 18: Qwen2.5-VL-7B training volume curve, plus 1k points for the two other architectures. Error bars on the Qwen 1k point are 1 σ across seeds 42/43/44, and other points are single seed. Sophon saturates at 5k, What’sUp is stable, and BLINK transfer peaks at 1k and regresses at higher volumes.
Volume
Sophon
WU-CLEVR
WU-Real
BLINK
Base ( n=1 )
54.7
93.6
94.2
88.8
500 ( n=3 )
78.7 ± 1.7
94.0 ± 0.4
93.4 ± 0.2
91.1 ± 1.1
1k ( n=3 )
80.2 ± 0.8
94.0 ± 0.8
92.6 ± 0.8
93.2 ± 0.8
5k ( n=1 )
88.7
94.9
94.2
90.9
10k ( n=1 )
89.2
94.1
93.9
91.6
Appendix
Table 8: Qwen2.5-VL-7B volume curve, full LoRA, full vocabulary random training, merged evals. Sophon saturates at 5k and BLINK transfer peaks at 1k. Values percent.
Architecture
Condition
WU-CLEVR
WU-Real
BLINK
OneVision
Base
87.3
91.7
83.2
Workspace
90.2
93.4
81.1
Full
90.7
93.9
81.8
Qwen2.5-VL
Base
93.6
94.2
88.8
Workspace
93.4
93.7
90.2
Full
94.1
92.7
92.3
Appendix
Table 9: External benchmarks across the retrained grid. Same merged checkpoints as the balanced Sophon evaluation, single seed. Values percent.
Eval coord
Base
LoRA all (1k)
LoRA Cartesian
LoRA polar
Cartesian 2D
55.4%
88.0%
93.0 ± 0.5%
77.6 ± 1.1%
Linear 1D
45.0%
79.5%
66.2 ± 1.3%
72.8 ± 3.1%
Polar 2D
50.8%
62.0%
51.7 ± 2.6%
67.3 ± 0.8%
Appendix
Table 10: Cross-coordinate transfer for Qwen2.5-VL-7B ( n=3 seeds). Cartesian only training drives Cartesian specs above mixed-coord training but leaves polar at base. Polar only training drives polar specs up modestly and also transfers substantially to Cartesian. Mean accuracy across question types per coordinate family.
Figure 19: Cross-coordinate transfer asymmetry across three open architectures we test.
Arch / condition
standard
colors
dot count
role rev.
grid off
black bg
OneVision base
52 ± 1
50 ± 0
50 ± 0
50 ± 0
50 ± 0
50 ± 0
OneVision Reader
54 ± 2
54 ± 3
53 ± 1
51 ± 3
54 ± 1
53 ± 5
OneVision Workspace
84 ± 1
74 ± 3
87 ± 1
82 ± 2
93 ± 1
92 ± 2
OneVision Language
87 ± 2
77 ± 4
87 ± 1
84 ± 2
91 ± 3
89 ± 2
OneVision Full
86 ± 2
74 ± 3
86 ± 1
83 ± 1
94 ± 5
89 ± 5
Qwen2.5-VL base
57 ± 1
55 ± 2
62 ± 0
56 ± 3
52 ± 3
57 ± 5
Appendix
Table 11: Suite-family aggregates, mean over three stimulus seeds ± seed range. The standard family mixes the standard rendering with marker and sparsity variants. Table 12 splits it. Workspace rows use each architecture’s workspace-band cell.
Variant
OneVision
Qwen2.5-VL
NeXT-Mistral
Idefics3
standard
95
79
97
98
marker square
92
77
94
94
marker triangle
91
80
86
96
dots sparse
89
87
90
89
color green/orange
65
56
72
84
palette purple/orange
70
68
86
78
Appendix
Table 12: Variant-level accuracy for the Workspace cells, mean over three stimulus seeds, tasks pooled. Values percent.
Arch / task
n=1
n=3
n=5
n=10
n=20
n=30
OneVision trans_x
90
83
87
95
93
99
OneVision trans_y
97
88
87
91
94
98
OneVision scale
57
57
55
71
81
87
Qwen trans_x
87
92
89
95
97
92
Qwen trans_y
95
97
96
85
58
52
Qwen scale
61
73
85
91
93
94
Appendix
Table 13: Point-count sweep, per task, Workspace cells, mean over three stimulus seeds. Values percent.
Arch
256
384
512 (train)
768
1024
OneVision
85
86
95
97
96
Qwen2.5-VL
81
80
79
81
78
NeXT-Mistral
96
97
97
97
98
Idefics3
98
98
98
98
98
Appendix
Table 14: Render-resolution sweep, Workspace cells, mean over three stimulus seeds, tasks pooled. Training render is 512 px. Values percent.
Sophon
What’sUp
VSR
BLINK
Eyes Wide Shut
BlindTest
FUGU
Stimuli
point clouds
object / CLEVR photos
natural images
natural images
CLIP-blind pairs
2D primitives
scatter plots, labeled axes
Obj./scene/lang. priors
none
objects, language
objects, scene, language
objects, scene
object semantics
mostly stripped
mostly stripped
Ground truth
procedural, exact
template
annotation
annotation
curated
procedural
procedural, exact
Coord-system factorial
yes
no
no
no
no
no
no
Parametric transforms
T, R, S
relational only
relational only
mixed
no
no
no
Continuous difficulty knob
yes
no
no
no
no
partial
partial (point count)
Appendix
Table 15: Sophon vs. neighboring spatial benchmarks. What’sUp [ Kamath et al., 2023 ] , VSR [ Liu et al., 2023a ] , BLINK [ Fu et al., 2025b ] , Eyes Wide Shut / MMVP [ Tong et al., 2024b ] , BlindTest [ Rahmanzadehgervi et al., 2024 ] , FUGU [ Tartaglini et al., 2025 ] .
Figure 20: Vocabulary distractor effect across five categorical specs. Each dumbbell connects random restricted accuracy (open marker, 2 transform appropriate direction words offered) to random full accuracy (filled marker, all 6 direction words offered, order randomized in both conditions). The translation (Cartesian Translation X / Translation Y ) and polar Radial (radial translation, answer choices inward / outward ) cells plunge toward the 6-way chance floor under the full vocabulary across both open (Qwen-7B, NeXT-7B) and frontier (GPT-4o, Opus 4.6) tiers. The extra rotation tokens ( clockwise , counterclockwise ) draw probability mass away from the correct translation answer even though the underlying transform is not rotational. The Cartesian Rotation control stays flat because rotation vocabulary is already among the choices in both conditions, so widening the option set adds no novel distractors. The pattern is consistent across capability tiers, supporting the claim that the failure is at the answer set interface rather than at perception.
No hint
With hint
Spec
MAE
Pearson r
MAE
Pearson r
Cartesian rotation (deg)
10.3
0.43
17.2
−0.19
Cartesian scale (factor)
0.09
0.55
0.13
0.55
Polar rotation (deg)
15.2
0.29
15.1
0.10
Polar scale (factor)
0.06
0.86
0.35
0.31
Appendix
Table 16: Opus 4.6 magnitude estimation, no hint vs. range hint. The hint collapses Opus 4.6’s stimulus sensitive estimation toward canonical values ( 15∘ , 0.85, 0.95) and drops correlation sharply on polar scale. Numbers filtered to plausible range. Pixel coordinate leakage in raw responses fixed upstream (§ 4 ).
Figure 21: All models version of the quant response taxonomy (Fig. 4 shows only Human plus the four current/recent closed weight flagships). Adds Grok 4.20 and four open 7 B– 32 B models. The five classes and the color scheme match the main-paper figure: Perceiver , Direction perceiver , Noisy , Few anchor , Constant (neutral grey). Numerical labels are the percentage of responses that are unique values. None of the additional models reaches the perceiver or direction perceiver class on any cell. Qwen2.5-VL-7B Cartesian rotation ( u=1% , top-1 =98% ) and Grok 4.20 polar-rotation ( u=5% , top-1 =97% ) are canonical Constant cells. LLaVA-NeXT polar rotation ( u=11% , top-1 ∼37% , r=−0.11 ) is the noisiest entry, with many distinct guesses and none correlated with GT.
Human
Frontier
Open
Cell
Opus 4.6
Gemini 2.5 Pro
GPT-5.4
Grok 4.20
Qwen-7B
Qwen-32B
LLaVA-NeXT
OneVision
1D Lin Bin
86
76
64
52
45
53
54
48
64
1D Lin Cat
94
92
84
98
57
44
92
40
46
2D Cart Bin
87
88
89
95
48
60
49
49
54
2D Cart Cat
88
78
76
48
48
50
71
51
40
2D Cart Comp
84
68
86
92
53
52
51
48
56
Appendix
Table 17: Per-cell accuracy and quantitative MAE, aggregated across specs to match Fig. 13 . Accuracy in % . Rotation MAE in degrees, scale MAE in factor units. Human values are from n=30 Prolific raters; human scale-MAE is omitted because raters interpreted the factor prompt heterogeneously (factor vs. percent), see Appendix R . OneVision polar-scale MAE is computed over the 30% of items that parsed to a number (constant non-numeric tokens on the rest). Cross-seed standard deviations ±s are shown in the heatmap (this table reports point estimates).
Cell
Opus 4.6
Gemini 2.5 Pro
GPT-5.4
GPT-4o
Grok 4.20
1D-Lin ⋅ Bin
47
71
52
72
45
1D-Lin ⋅ Cat
94
84
98
64
57
2D-Cart ⋅ Cmp Bin
87
90
98
92
53
2D-Cart ⋅ Cmp Comp
69
84
92
78
59
2D-Cart ⋅ Rot Bin
85
93
93
82
55
2D-Cart ⋅ Rot Cat
52
47
46
46
48
Appendix
Table 18: Per-cell accuracy ( % ): frontier models. Cells are coordinate-system × spec × question type. Reparsed comparative scores used where available (§ 5.1 ).
LLaVA-NeXT-7B
Qwen2.5-VL-7B
Cell
base
LoRA-lang
base
LoRA-full
LoRA-lang
Qwen2.5-VL-32B
1D-Lin ⋅ Bin
47
86
57
75
57
50
1D-Lin ⋅ Cat
40
88
44
96
82
92
2D-Cart ⋅ Cmp Bin
49
100
60
100
86
48
2D-Cart ⋅ Cmp Comp
49
90
51
90
94
62
2D-Cart ⋅ Rot Bin
56
100
52
100
85
56
Appendix
Table 19: Per-cell accuracy ( % ): open base and fine-tuned models. LoRA-language variants apply LoRA only to the language tower; LoRA-full updates both vision and language towers (full-vocab fine-tune, 1k samples). Reparsed comparative scores used where available.
Model
Cart Rot ( ∘ )
Polar Rot ( ∘ )
Cart Scale
Polar Scale
Frontier
Opus 4.6
16
26
0.87
0.07
Gemini 2.5 Pro
20
19
—
—
GPT-4o
48
32
0.96
0.75
Grok 4.20
23
28
9.32
82.83
Open
Appendix
Table 20: Per-cell quantitative MAE (rotation in degrees, scale as factor). Lower is better. Dashes indicate the model emitted a constant token (zero variance) on that cell, which makes MAE-vs-truth a misleading summary; see the per-model degeneracy notes in § 5.1 .
Figure 22: Opus 4.6. Quantitative answer distribution vs. ground truth on the four (coordinate system, transform) cells. Each panel overlays the ground truth histogram (gray fill) with no hint predictions (red, solid edge) and range hint predictions (blue, hatched). The two horizontal range bars at the top show the no hint stimulus support (red dashed) and the explicit range supplied by the hint (blue solid), and mode of response annotations point at the dominant text anchor in each condition. The polar scale panel is the only cell where Opus 4.6 tracks the stimulus continuously ( r=0.80 , perceiver class). The other three panels collapse to a single mode near 90∘ or ∼ 1.0 regardless of GT (canonical few anchor ). Range hints shrink the support of the predicted distribution but do not move the mode toward GT, which is the basis for our claim that hints constrain the answer set rather than reveal perception.
Figure 23: GPT-4o. Same layout as Fig. 22 . Polar scale is the lone non degenerate cell ( r=0.62 , direction perceiver, meaning the model tracks scale direction up vs. down but does not produce continuous magnitude). Cartesian scale collapses to a single anchor ( r=0.09 , constant), and both rotation cells are few anchor at ∼45∘ .
Figure 24: GPT-5.4. Same layout as Fig. 22 (no range hint condition was run for GPT-5.4, so the blue series is absent). Both scale cells reach perceiver ( r=0.79 each), the only frontier model in our suite that does so on Cartesian scale. Rotation distributions remain noisy ( r=0.37 Cartesian, r=0.47 polar). Rotation values spread but are not strongly correlated with GT.
Behavior at τ=1.0 (vs. τ=0 )
Regime
constant rate
Accuracy
Example cells (collapsed at τ=0 )
A: Decode induced
releases ( <0.85 )
recovers ( Δ≥+10 pp)
Grok Cartesian trans x cat ( +16 pp), Grok Cartesian trans y cat ( +19 pp), GPT-4o Cartesian rotation cat.
B: Attractor noise
releases
flat ( ±5 pp)
Qwen most categorical cells. The constant token attractor breaks under sampling, but the underlying answer distribution is still uncorrelated with GT, so accuracy doesn’t move.
C: Fixed prior
stays ( ≥0.85 )
flat
Grok binary cells. Answers “yes” on ∼97% of samples at every τ tested, and sampling does not perturb the constant.
Appendix
Table 21: Three regimes for greedy decode collapse, distinguished by behavior under a temperature sweep ( τ∈{0,0.25,0.5,0.75,1.0} ). A cell is “collapsed at τ=0 ” if its top-1 parsed answer covers ≥85% of samples; the constant rate “releases” if it drops below 0.85 at τ=1.0 ; accuracy “recovers” if Δ acc ≥+10 pp from τ=0 to τ=1.0 . Only Regime A is decode induced (a sampling artifact); Regimes B and C are perceptual failures that stochastic decoding cannot fix.