Where Do Vision Language Models Fail at 3D Reasoning?

Khai Le-Duc; Nguyen Kim Hai Bui; An Ngo; Nam D. Hoang; Yueqi Song; Jiarui Liu; Thai Quoc Hoang; Fang Wu

Model	Mode	T1	T2	T3	T4	T5	T6	T7	T8	T9	T10	T11	T12	T13	Avg
Random baseline	–	–	50.0	–	–	–	50.0	–	50.0	–	50.0	33.3	50	33.3	24.4
Best constant	–	34.0	50.0	5.0	24.0	1.0	77.0	9.0	58.0	43.0	51.0	37.3	51.0	37.3	36.7
Human baseline	–	76.7	73.3	23.3	83.3	93.3	63.3	76.7	53.3	70.0	86.7	76.7	86.7	46.7	70.0
Qwen3.5-0.8B	NT	33.4	40.0	0.0	6.0	0.0	47.6	0.0	58.0	45.6	49.0	37.3	51.0	37.3	31.2
	TH	20.8	50.4	0.2	19.8	0.4	66.4	0.0	52.2	27.0	46.4	37.3	50.6	37.0	31.4
Qwen3.5-2B	NT	22.4	50.8	0.0	24.4	0.0	40.8	0.0	59.0	59.2	49.0	37.3	51.0	37.3	33.2

Appendix

Table 2: Main results on BedBench . Accuracy (%) across all 13 tasks in their default settings. Each open-weight cell is the 5-run mean for one task

\times

model

\times

inference mode condition; the frontier rows are single-run TH results; the human row aggregates 30 participants (one question per task per participant; see Appendix C.4 ). The best-constant row gives the accuracy of always outputting the most frequent ground-truth answer (or candidate position) for each task. Bold = best model accuracy per task and highest model average. NT = Non-Thinking, TH = Thinking mode. Takeaway: Humans average 70.0%, while the strongest model reaches 51.5%, 14.8 pp above the best-constant average of 36.7%; the gap is largest on T5 , where humans reach 93.3% and no model exceeds 7.2%.

Task	Mode	Acc (%)	RMSE	MAE
T3: Difference	NT	0.2	4.14	3.70
T3: Difference	TH	0.1	4.19	3.58
T5: Color Distribution	NT	0.7	1.31	0.80
T5: Color Distribution	TH	1.0	1.33	0.74

Appendix

Table 4: Exact match vs. error magnitude on the numeric floor tasks. Averaged over 12 OW models under the default setting; each cell is a 5-run mean. T3 MAE and RMSE average its three components; T5 MAE and RMSE average all 12 palette colors, including colors absent from the object. Takeaway: Both tasks are near zero under exact match (both

\leq

1%), while T3 errors are larger (RMSE

3.2\times

, MAE

4.6\times

).

Figure 9

Prompt	EM $\uparrow$	MAE $\downarrow$	Present $\uparrow$	Over $\downarrow$
Original	3.0	0.50	48.2	40.6
No example	1.4	0.51	45.5	43.9
Benchmark-matched	5.0	0.41	58.9	29.8

Appendix

Table 6: Prompt-example ablation on T5: Color Distribution . Qwen3.5-27B under Non-Thinking mode with color inputs, mean over five runs on the same 100 instances. The original prompt contains an illustrative 28-block answer; the benchmark-matched example has 11 blocks and 8 colors, matching the benchmark distribution. EM is full-distribution exact match, MAE is the per-color absolute error in blocks over the 12-color palette, Present is exact-count accuracy on colors present in the object, and Over is the share of present colors that are over-counted. Takeaway: Removing the example leaves all metrics within their paired 95% CIs, and a benchmark-matched example improves present-color accuracy by

+10.6

pp , while EM stays

\leq 5.0\%

.

Input	EM $\uparrow$	EM (gold = yellow) $\uparrow$	MAE $\downarrow$	Present $\uparrow$	Over $\downarrow$	Total
3D (two views)	3.0	6.0	0.50	48.2	40.6	14.9
2D tiles	52.4	89.8	0.11	91.0	3.6	11.5
2D $-$ 3D	$+$ 49.4	$+$ 83.8	$-$ 0.39	$+$ 42.8	$-$ 37.0	$-$ 3.4
95% CI	$[+39.8,+59.0]$	$[+75.6,+91.0]$	$[-0.43,-0.34]$	$[+37.6,+47.9]$	$[-41.8,-32.2]$

Appendix

Table 7: 2D visibility control on T5: Color Distribution . Qwen3.5-27B, Non-Thinking, color setting, mean over five runs on the same 100 objects. Metrics as in Table 6 ; EM (gold = yellow) merges the two colors in predictions and ground truth as a post-hoc diagnostic, and Total is the mean predicted number of blocks (ground truth 11.4). Takeaway: With every block visible as a separate tile, EM rises by

+49.4

pp , and the gap remains after merging gold and yellow.

	Zero-shot			LoRA, benchmark palette
Variant	Acc	MAE	RMSE	Acc	MAE	RMSE
color (default)	3.0 $\pm$ 0.0	0.502 $\pm$ 0.002	0.937 $\pm$ 0.003	68.6 $\pm$ 0.5	0.033 $\pm$ 0.001	0.182 $\pm$ 0.003
color + axes	3.0 $\pm$ 0.0	0.481 $\pm$ 0.002	0.923 $\pm$ 0.004	66.8 $\pm$ 1.8	0.036 $\pm$ 0.002	0.191 $\pm$ 0.004

Appendix

Table 8: LoRA fine-tuning of Qwen3.5-27B on T5: Color Distribution . The adapter is trained on 800 generated T5 instances rendered with the benchmark palette, the 12-color, luminance-paired coloring of BedBench. Each cell is the 5-run mean

\pm

std on the 100 benchmark instances. Takeaway: Training on the benchmark palette raises exact match from 3.0% to 68.6%.

T7 evaluation condition	Color	Color + axes
Direct visual questioning	35.0 $\pm$ 0.7	36.4 $\pm$ 0.5
Self-predicted T5 counts as text	74.2 $\pm$ 0.8	73.2 $\pm$ 1.3
Ground-truth T5 counts as text	99.0 $\pm$ 0.0	–
Agreement with the argmax of the input counts	99.6 $\pm$ 0.5	99.0 $\pm$ 0.0
Dominant colors implied by self-predicted counts	74.6	74.2
Text-conditioned accuracy, T5 answer exactly correct	99.4	98.5

Appendix

Table 9: T7 answered from block counts given as text. Fine-tuned Qwen3.5-27B, Non-Thinking, accuracy (%) as 5-run mean

\pm

std on the same 100 objects; the last three rows pool 5 runs

\times

100 objects, and the last two use only the 343 and 334 answers with an exactly correct T5 answer. The implied row scores the argmax of the model’s own T5 counts without querying T7 . Ground-truth counts do not depend on the visual setting and are evaluated once (“–”). Paired object-bootstrap 95% CIs (10,000 resamples): self-predicted counts

-

direct

+

39.2 pp

[+26.8,+51.4]

(color) and

+

36.8 pp

[+23.6,+50.0]

(color + axes); ground-truth

-

self-predicted counts

+

24.8 pp

[+16.4,+34.0]

(color). Takeaway: Given its own counts as text, the model derives the dominant colors almost perfectly whenever those counts are correct, unlike when it answers from the images.

Task (variant)	Zero-shot	LoRA	$\Delta$
T4 (color)	45.2 $\pm$ 1.3	69.6 $\pm$ 0.5	$+$ 24.4
T4 (color + axes)	45.8 $\pm$ 0.8	61.8 $\pm$ 0.4	$+$ 16.0
T7 (color)	18.0 $\pm$ 0.0	35.0 $\pm$ 0.7	$+$ 17.0
T7 (color + axes)	18.4 $\pm$ 0.5	36.4 $\pm$ 0.5	$+$ 18.0
T10 (color)	55.2 $\pm$ 0.4	62.4 $\pm$ 1.5	$+$ 7.2

Appendix

Table 10: Transfer of the T5 LoRA adapter (benchmark palette) to related color tasks. Accuracy (%) in Non-Thinking mode, 5-run mean

\pm

std. Table 9 compares direct T7 answers with answers derived from the model’s own T5 counts. Takeaway: Fine-tuning on T5 improves the related color tasks only partially, most on T4 and least on T10 .

Models	T1	T4	T7
OW, Non-Thinking (12)	4.0 (5.4)	32.1 (16.8)	21.6 (2.4)
OW, Thinking (12)	9.6 (7.8)	55.1 (17.7)	21.9 (3.1)
Frontier (4)	–	65.8 (16.7)	39.0 (5.0)
Qwen3.5-27B, zero-shot	8.8 (11.8)	51.8 (17.8)	23.8 (1.7)
Qwen3.5-27B, fine-tuned	–	69.6 (19.9)	36.8 (2.2)

Appendix

Table 11: Per-object agreement between direct answers and the answer implied by the model’s own T5 counts (%, chance in parentheses): T1 vs.

\sum_{c}\hat{c}_{c}

, T4 vs.

|\{c:\hat{c}_{c}>0\}|

, T7 vs.

\arg\max_{c}\hat{c}_{c}

. Same objects, colored renderings, and runs; group rows average the models, and the Qwen3.5-27B rows are Non-Thinking. Chance permutes the direct answers across objects. “–”: no colored T1 run. With the default uncolored T1 renderings, OW agreement is 5.2% (chance 6.0%) in Non-Thinking and 8.7% (7.3%) in Thinking mode. Takeaway: Direct answers rarely agree with the counts the same model reports, before and after fine-tuning, and for T1 no more often than chance.

Task	Non-Thinking	Thinking	$\Delta_{\text{thinking}}$	95% CI	$\uparrow$	$\downarrow$	$\approx$	Verdict
T1: Volume	29.2	20.2	$-8.9$	$[-11.9,-6.0]$	2	8	2	Non-Thinking wins
T2: Packaging	58.8	72.6	$+13.8$	$[+9.3,+18.4]$	11	1	0	Thinking wins
T3: Difference	0.2	0.1	$-0.1$	$[-0.4,+0.1]$	0	0	12	Tie
T4: Color Count	27.1	41.0	$+13.9$	$[+8.9,+19.1]$	10	1	1	Thinking wins
T5: Color Distribution	0.7	1.0	$+0.3$	$[-0.3,+1.0]$	1	0	11	Tie
T6: Local Structure	70.2	72.9	$+2.8$	$[+0.4,+5.1]$	4	5	3	Thinking wins

Appendix

Table 12: Per-task effect of Thinking. Comparison at the default 1024-token budget under the default visual setting. Non-Thinking and Thinking report mean accuracy (%) over the 12 OW models.

\uparrow

/

\downarrow

/

\approx

count how many of the 12 models win, lose, or tie under Thinking by the same 2 pp threshold (not per-model significance). Verdict labels each task by its mean

\Delta_{\text{thinking}}

: Thinking wins above

+

2 pp, Non-Thinking wins below

-

2 pp, and Tie otherwise. The 95% CI is a paired bootstrap interval for the mean

\Delta_{\text{thinking}}

(10,000 resamples of objects, scenes for T13 , and pairs for T3 , with models and runs fixed; no multiplicity correction). † The interval includes zero. Takeaway: Thinking helps most on T2 and T4 (

+

13.8 and

+

13.9 pp ), hurts most on T1 and T9 (

-

8.9 and

-

7.4 pp ), and moves the mean across tasks by only

+

0.9 pp .

Task	Model	0	1024	2048	4096
T1: Volume	Qwen3.5-27B	39.8	24.2	27.6	26.2
	Gemma4-31B	35.0	17.2	13.0	13.4
	Average	37.4	20.7	20.3	19.8
T4: Color Count	Qwen3.5-27B	45.2	57.2	60.8	57.2
	Gemma4-31B	52.8	56.2	57.6	60.0
	Average	49.0	56.7	59.2	58.6

Appendix

Table 13: Effect of the reasoning budget. Default-setting accuracy (%) of Qwen3.5-27B and Gemma4-31B at four reasoning-token budgets, on three tasks spanning the win/loss spectrum of Table 12 (Appendix B.2.1 ): T1 (most hurt by Thinking), T4 (most helped), and T11 (near the middle). Each cell is a 5-run mean; boldface marks the best budget per row. Takeaway: On average, every nonzero budget leaves T1 below budget 0, as do budgets of 2048 tokens or more for T11 , while T4 improves at every nonzero budget and levels off after 2048 tokens (paired bootstrap intervals in Appendix B.2.2 ).

Task	Mode	Acc (%)	Prec	Recall	F1
T8	NT	58.2	0.383	0.510	0.398
T8	TH	55.4	0.543	0.504	0.430
T2	NT	58.8	0.685	0.588	0.506
T2	TH	72.6	0.783	0.726	0.697
T6	NT	70.2	0.692	0.664	0.595
T6	TH	72.9	0.670	0.733	0.675

Appendix

Table 14: Binary tasks under accuracy vs. macro-averaged metrics. Non-Thinking (NT) vs. Thinking (TH), averaged over 12 OW models under the default setting; each cell is a 5-run mean. Thinking lowers accuracy on T8 and raises it on T2 , T6 , and T10 ; the T8 and T10 changes have 95% intervals that include zero (Table 12 ). Takeaway: On T8 , Thinking lowers accuracy but raises precision and F1; on T2 , T6 , and T10 , recall rises more than precision.

#	Name	Description	Models	Representative quote (qualitative sample)
Good behavior
1	Tool Use	Invokes external code execution to inspect pixel values or crop the input image.	GPT	GPT-5, T5 : samples RGB pixel values via numpy to separate look-alike hues (Tab. 113 , Appendix E.5 )
2	Good Perception	An isolated span where the model’s stated read of the scene is correct and matches the ground-truth structure.	All	Gemini-3-Flash, T7 : “we clearly see a diagonal row of 2 teal blocks forming part of the central structure” (matches GT; Tab. 115 , Appendix E.5 )
3	Good Tactic	A valid problem-solving strategy or check applied correctly, e.g. comparing sorted dimensions or framing a task as set difference.	Grok, Gemini	Gemini-3-Flash, T2 : “Comparing the sorted dimensions: $2\leq 2$ (Fits) …the orientation is valid” (Tab. 110 , Appendix E.5 )
4	Good Intuition	A correct-magnitude estimate reached by heuristic guess rather than a completed derivation.	GPT-5, Grok	GPT-5, T1 : “I estimate a total of around 10 cubes” (GT 10; Tab. 109 , Appendix E.5 )
Bad behavior

Appendix

Table 15: Annotation taxonomy for reasoning traces. Applied to the qualitative thinking-trace panels (Appendix E.5 ). Rows 1–11 are tags applied by hand to individual highlighted spans, so a trace can carry several tags and most spans carry none. Rows 12–13 are whole-trace verdicts, assigned in a second pass by comparing a tagged trace with its scored correctness. Takeaway: Wrong Perception is the most frequent tag (222 of 457 spans, all seven models) and appears in 42 of the 44 incorrect traces.

Task	Qwen3.5	Gemma4
T1: Volume	33.1	24.3
T2: Packaging	59.4	62.0
T3: Difference	0.3	0.0
T4: Color Count	25.0	29.2
T5: Color Distribution	0.5	1.2
T6: Local Structure	70.4	79.7

Appendix

Table 16: Family-level accuracy (%, Non-Thinking). Each entry averages the default-setting 5-run mean accuracy over the models in that family. Takeaway: The families differ by up to 12.3 pp on individual tasks ( T1 , T6 , T9 ) but share the same floor on T3 and T5 and nearly identical scores on T8 and T11 – T13 .

Design element	SigLIP 2	Qwen3.5	Gemma4
Vision encoder
ViT type	isotropic, full-attention	same architecture	same architecture
Patch tokenization	2D, non-overlapping	same architecture	same architecture
Position encoding	learned absolute	RoPE	RoPE
Block normalization	LayerNorm	LayerNorm	RMSNorm
QK/V normalization	none	none	yes

Appendix

Table 17: Architecture of the two model families. Component-level comparison against the SigLIP 2 vision reference, read from the released checkpoints. Both vision encoders are isotropic, full-attention ViTs with 2D patch tokenization: Qwen3.5 stays close to stock SigLIP 2 (differing only in RoPE), while Gemma4 keeps the skeleton but updates block internals (RoPE, RMSNorm, QK/V-norm, gated MLP); in both, a compress-and-project adapter replaces the contrastive MAP pooling head. Takeaway: The families share one vision-encoder design and differ mainly in the connector and, most sharply, in the sequence-mixing primitive of the language backbone.

Figure 7: Scaling curves for all 13 tasks. Non-Thinking and Thinking accuracy for both families, plotted against total parameter count (log scale). Takeaway: The split between scaling-responsive tasks ( T2 , T4 , T6 , T7 , T9 , T11 ) and the remaining tasks holds across both families and both modes, with T6 and T1 as the two exceptions discussed in the text.

	Default		Color add-on				Axes add-on
Task	Color	Axes	Without	With	$\Delta_{\text{color}}$	95% CI	Without	With	$\Delta_{\text{axes}}$	95% CI
T1: Volume	$\bm{\times}$	$\bm{\times}$	29.2	31.8	$+2.6$	$[-0.28,+5.65]$	29.1	27.7	$-1.4$	$[-3.98,+1.13]$
T2: Packaging	$\bm{\times}$	$\bm{\times}$	–	–	–	–	58.8	54.7	$-4.1$	$[-5.85,-2.42]$
T3: Difference	$\bm{\times}$	$\checkmark$	0.2	3.7	$+3.5$	$[+1.75,+5.58]$	–	–	–	–
T4: Color Count	$\checkmark$	$\bm{\times}$	–	–	–	–	27.1	26.4	$-0.7$	$[-2.60,+1.30]$
T5: Color Distribution	$\checkmark$	$\bm{\times}$	–	–	–	–	0.7	0.5	$-0.2$	$[-0.72,+0.20]$

Appendix

Table 18: Effect of visual cues on accuracy. Mean accuracy (%) over the 12 OW models in Non-Thinking mode without and with each cue, the difference

\Delta

(pp), and its paired bootstrap 95% interval (pointwise, without multiplicity correction); colored intervals exclude zero, and “–” marks tasks that do not receive the cue. Default marks whether each task’s default rendering is colored and has coordinate axes. Takeaway: Color raises accuracy on all five tasks, most on T12 ; axes lower it on 10 of 11 tasks, most on T2 , and only T11 gains. Only color on T3 , T11 , and T12 and axes on T2 and T11 have intervals that exclude zero.

Task	SigLIP-2	DINOv2	CLIP	Depth Anything V2-L	Qwen3.5 27B	Gemma4 31B
Count estimates: Spearman $\rho$ . Higher is better.
T1: Volume	$0.70^{*}$ (0.09)	$0.53^{*}$ (0.08)	$0.44^{*}$ (0.09)	$0.65^{*}$ (0.08)	$0.62^{*}$ (0.08)	$0.66^{*}$ (0.09)
T4: Color Count	$0.77^{*}$ (0.09)	$0.63^{*}$ (0.09)	$0.61^{*}$ (0.09)	$0.62^{*}$ (0.09)	$0.68^{*}$ (0.09)	$0.78^{*}$ (0.09)
T9: Visible Floors	$0.89^{*}$ (0.09)	$0.92^{*}$ (0.09)	$0.87^{*}$ (0.08)	$0.93^{*}$ (0.09)	$0.92^{*}$ (0.10)	$0.88^{*}$ (0.09)
Decisions: ROC-AUC (mean over colors for T7 ). Higher is better.
T2: Packaging	$0.70^{*}$ (0.59)	$0.76^{*}$ (0.59)	$0.73^{*}$ (0.58)	$0.74^{*}$ (0.58)	$0.80^{*}$ (0.58)	$0.70^{*}$ (0.59)

Appendix

Table 19: Probe performance on frozen visual features. Cells report score (shuffled-label 95th percentile); ∗ marks scores above this empirical reference (1000 permutations); it is descriptive and not corrected for multiple comparisons. Takeaway: Counts and local-structure decisions are well recovered, while complete T3 / T5 answers remain rare.

Figure 8: Probing frozen encoders on ground-truth targets. (A) Null-adjusted probe decodability (SigLIP-2) against the per-task VLM gap above random. (B) Decodability across three encoder classes and the largest deployed tower of each family. (C) Cross-view retrieval from pooled features by cue condition. (D) T11 zero-shot candidate separation (task-rule top-1 minus reversed-rule top-1; positive values favor the task rule, negative values favor its reverse). Values in (C) and (D) depend on the representation (patch-mean features; a CLS-pooled readout of CLIP loses most of its (D) signal), and the Qwen tower is probed before its patch merger. Takeaway: With pooled probes trained on 100 objects, quantity tasks are well recovered, while the per-part and relational tasks (bottom rows of B) have low null-adjusted probe scores.

Model / feature stage	Readout	EM	MAE	Pres.	T1	T4	T7
Qwen3.5-27B / tower	VLM (zero-shot)	3.0	0.50	48.2	–	–	–
	Constant	0.0	0.68	61.5	17.0	1.0	0.0
	Ridge, averaged	21.0	0.27	82.0	40.0	71.0	43.0
	Attention	$61.7\pm 4.8$	0.12	94.2	63.7	88.7	76.0
	Token-sum	$77.0\pm 2.4$	0.12	96.6	77.0	97.0	83.3
Qwen3.5-27B / connector	Ridge, averaged	21.0	0.27	81.1	39.0	65.0	41.0

Appendix

Table 20: T5 probes on frozen tower and connector features (%, except MAE), tested on 100 benchmark objects. Neural probes train on 800 objects with 100 for validation; ridge fits on all 900. EM = exact count-vector match; Pres. = present colors counted exactly; T1 / T4 / T7 = answers derived from predicted counts. Neural rows report three-seed means, with EM standard deviations. Constant predicts the training mean; VLM rows are full-model Non-Thinking default-setting results. Takeaway: Exact counts are recoverable from both towers and remain recoverable at Qwen3.5-27B’s language-model input.

Figure 9: Deployed vision towers across model scale. Left, middle: probe decodability by task family for every Qwen3.5 and Gemma4 tower (x-axis: LLM size, with tower depth/width). Same-architecture towers attached to different LLM sizes give near-identical values: the Gemma4 E2B/E4B pair shares one tower checkpoint, with identical extracted features, and its 26B-A4B/31B pair agrees to within

0.005

on every probe metric. Right: T11 zero-shot candidate separation per tower. Takeaway: Quantity probes are already strong in the smallest towers and change little with size, and the two families order T11 candidates in opposite directions: the task rule wins in Gemma4 and its reverse wins in Qwen3.5.

	Qwen3.5-27B			Gemma4-31B			GLM-4.6V-Flash
Task	Visual	No-image	$\Delta$	Visual	No-image	$\Delta$	Visual	No-image	$\Delta$
T3: Difference	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
T8: Silhouette Symmetry	57.0	57.0	0.0	56.0	56.0	0.0	58.0	58.0	0.0
T13: Mixture-of-Shapes	36.2	38.1	$+$ 1.9	42.3	40.7	$-$ 1.6	37.3	37.3	0.0
T5: Color Distribution	3.0	0.0	$-$ 3.0	4.0	0.0	$-$ 4.0	0.0	0.0	0.0
T12: Reconstruct	57.2	49.0	$-$ 8.2	54.0	45.4	$-$ 8.6	48.0	51.0	$+$ 3.0

Appendix

Table 21: Accuracy with and without images (Non-Thinking). Visual vs. no-image accuracy for the three models evaluated without images (5-run means). Rows are grouped into shortcut-sensitive, visually dependent, and artifact tasks; T1 appears twice because open-ended counting and the MCQ artifact probe are interpreted separately. Takeaway: Scores on T3 , T8 , and T13 barely change without images, visually dependent tasks such as T4 and T6 lose up to 60 pp, and Gemma4-31B reaches 100% on the fixed-layout T1 MCQ without images.

Figure 10: Open-ended vs. multiple-choice accuracy on T1: Volume . Paired bars per model under Non-Thinking mode and the default visual setting; numbers above the bars give the MCQ

-

OE gain (pp); the dashed line marks the 20% random baseline of the five-option MCQ, whose correct option is always in the third position. Takeaway: The fixed-layout MCQ raises accuracy by

+

39.0 pp on average (up to

+

57.8 pp) without adding any visual information.

	Modal Prediction				Modal Prediction
Task	Qwen3.5	Gemma4	GLM	Task	Qwen3.5	Gemma4	GLM
T1: Volume (OE)	0	0	10	T1: Volume (MCQ)	—	MCQ pos. 3	—
T3: Difference	{0,0,0}	{3,1,1}	{4,0,0}	T8: Silhouette Symmetry	0	0	0
T4: Color Count	0	1	7	T6: Local Structure	0	1	1
T5: Color Distribution	{}	6-color dict	12-color dict	T11: Projection	2	2	1
T7: Color Dominance	[]	[ red ]	12-color list	T2: Packaging	—	1	1

Appendix

Table 22: Constant predictions without images (Non-Thinking). Each cell shows the modal prediction; “—” means no collapse observed for that model. Qwen3.5 = Qwen3.5-27B, Gemma4 = Gemma4-31B, GLM = GLM-4.6V-Flash. T3 predictions abbreviated as {i,a,b} (intersection, only-a, only-b). The color abbreviations denote fixed palettes emitted regardless of input: “6-color dict”

=

{ blue , cyan , green , magenta , red , yellow } , each color mapped to count 1; “12-color dict”

=

the full 12-color palette, each count 1; “12-color list”

=

the same palette as a list of names without counts. All modal shares are 100% except Qwen3.5 T8 (99.0%), Gemma4 T8 (98.0%), GLM T5 (89.8%). Takeaway: Without images, all three models give one fixed answer on every T9 and T10 instance ( 3 and 0 ), which accounts for their no-image scores of 43% and 51% (Table 21 , Appendix B.6.1 ).

Task	Non-Thinking	Thinking	$\Delta$	Effect
T1: Volume (OE)	11.3	15.0	$+$ 3.7	small gain, still weak
T1: Volume (MCQ)	80.9	55.3	$-$ 25.7	disrupts answer-position prior
T3: Difference	0.0	0.0	0.0	no recovery
T4: Color Count	7.3	8.7	$+$ 1.4	minimal gain
T5: Color Distribution	0.0	0.0	0.0	no recovery
T6: Local Structure	41.0	51.4	$+$ 10.4	breaks collapse

Appendix

Table 23: Effect of Thinking without images. Non-Thinking and Thinking accuracy (%) averaged over the three models evaluated without images, under matched task and ablation conditions; each model/mode value is a 5-run mean. Takeaway: Without images, Thinking changes most tasks by only a few points, breaks the T6 collapse (

+

10.4 pp), lowers T9 by 11.0 pp, and lowers the T1 MCQ score by 25.7 pp by disrupting the answer-position prior.

Table 25: T1 Volume instance where all eight models count correctly.

Table 26: T1 Volume instance with a mixed split.

Table 27: T1 Volume instance where all eight models miscount.

Table 28: T2 Packaging instance where all eight models answer correctly.

Table 29: T2 Packaging instance with a mixed split.

Table 30: T2 Packaging instance with a mixed split (1/8 correct).

Table 31: T3 Difference instance with a mixed split.

Table 32: T3 Difference instance where all eight models return an incorrect triple with small per-component error.

Table 33: T3 Difference instance where all eight models return an incorrect triple with large per-component error.

Table 34: T4 Color Count instance where all eight models answer correctly.

Table 35: T4 Color Count instance with a mixed split.

Table 36: T4 Color Count instance where all eight models answer incorrectly.

Table 37: T5 Color Distribution instance with a mixed split.

Table 38: T5 Color Distribution instance with a mixed split.

Table 39: T5 Color Distribution instance where all eight models answer incorrectly.

Table 40: T6 Local Structure instance where all eight models answer correctly.

Table 41: T6 Local Structure instance with a mixed split.

Table 42: T6 Local Structure instance with a mixed split.

Table 43: T7 Color Dominance instance with a mixed split.

Table 44: T7 Color Dominance instance with a mixed split.

Table 45: T7 Color Dominance instance where all eight models answer incorrectly.

Table 46: T8 Silhouette Symmetry instance where all eight models answer correctly.

Table 47: T8 Silhouette Symmetry instance with a mixed split.

Table 48: T8 Silhouette Symmetry instance where all eight models answer incorrectly.

Table 49: T9 Visible Floors instance with a mixed split.

Table 50: T9 Visible Floors instance with a mixed split.

Table 51: T9 Visible Floors instance where all eight models answer incorrectly.

Table 52: T10 Color Connected instance where all eight models answer correctly.

Table 53: T10 Color Connected instance with a mixed split.

Table 54: T10 Color Connected instance with a mixed split.

Table 55: T11 Projection instance with a mixed split.

Table 56: T11 Projection instance with a mixed split.

Table 57: T11 Projection instance where all eight models answer incorrectly.

Table 58: T12 Reconstruct instance where all eight models answer correctly.

Table 59: T12 Reconstruct instance with a mixed split.

Table 60: T12 Reconstruct instance with a mixed split.

Table 61: T13 Mixture-of-Shapes instance where all eight models answer correctly.

Table 62: T13 Mixture-of-Shapes instance with a mixed split.

Table 63: T13 Mixture-of-Shapes instance where all eight models answer incorrectly.

Table 64: T1 Volume, color add-on qualitative panel. Each block compares the same sample with and without color. Color present is likely to help.

T3 Difference
“…analyze both objects and determine the difference in their block compositions…”
(a) Color absent.

GT
$(5,6,10)$	$(10,2,2)$ $\bm{\times}$	$(8,4,5)$ $\bm{\times}$	$(8,4,5)$ $\bm{\times}$	$(2,4,5)$ $\bm{\times}$
(b) Color present (color-aware difference).

Appendix

Table 65: T3 Difference, color add-on qualitative panel. Each block compares the same sample with and without color. Color present is likely to help.

Table 66: T11 Projection, color add-on qualitative panel. Each block compares the same sample with and without color. Color present is likely to help.

Table 67: T12 Reconstruct, color add-on qualitative panel. Each block compares the same sample with and without color. Color present is likely to help.

Table 68: T13 Mixture-of-Shapes, color add-on qualitative panel. Each block compares the same sample with and without color. Color present is likely to help.

Table 69: T1 Volume, axes add-on qualitative panel. Each block compares the same sample with and without a coordinate-axis. Axes present is likely to hurt.

T2 Packaging
“…determine whether the object can fit in the boundary in any possible orientation. Boundary: [2,3,2] …”
(a) Axes absent.

GT
False	False $\checkmark$	False $\checkmark$	True $\bm{\times}$	True $\bm{\times}$
(b) Axes present.

Appendix

Table 70: T2 Packaging, axes add-on qualitative panel. Each block compares the same sample with and without a coordinate-axis. Axes present is likely to hurt.

Table 71: T4 Color Count, axes add-on qualitative panel. Each block compares the same sample with and without a coordinate-axis. Axes present is likely to hurt.

Table 72: T5 Color Distribution, axes add-on qualitative panel. Each block compares the same sample with and without a coordinate-axis. Axes present is likely to hurt.

T6 Local Structure
“…determine whether the object has at least one column (vertical stack of blocks) with a height greater than 3…”
(a) Axes absent.

GT
False	False $\checkmark$	False $\checkmark$	False $\checkmark$	True $\bm{\times}$
(b) Axes present.

Appendix

Table 73: T6 Local Structure, axes add-on qualitative panel. Each block compares the same sample with and without a coordinate-axis. Axes present is likely to hurt.

T7 Color Dominance
“…determine the dominant color(s) in the object — the color(s) with the highest block count…”
(a) Axes absent.

GT
	$\checkmark$	$\checkmark$	$\bm{\times}$	$\bm{\times}$
(b) Axes present.

Appendix

Table 74: T7 Color Dominance, axes add-on qualitative panel. Each block compares the same sample with and without a coordinate-axis. Axes present is likely to hurt.

T8 Silhouette Symmetry
“…determine whether the object’s silhouette is symmetric (vertically or horizontally)…”
(a) Axes absent.

GT
False	False $\checkmark$	False $\checkmark$	False $\checkmark$	False $\checkmark$
(b) Axes present.

Appendix

Table 75: T8 Silhouette Symmetry, axes add-on qualitative panel. Each block compares the same sample with and without a coordinate-axis. Axes present is likely to hurt.

Table 76: T9 Visible Floors, axes add-on qualitative panel. Each block compares the same sample with and without a coordinate-axis. Axes present is likely to hurt.

T10 Color Connected
“…determine whether there exists at least one pair of adjacent blocks (sharing a face) that share the same color…”
(a) Axes absent.

GT
False	True $\bm{\times}$	False $\checkmark$	False $\checkmark$	True $\bm{\times}$
(b) Axes present.

Appendix

Table 77: T10 Color Connected, axes add-on qualitative panel. Each block compares the same sample with and without a coordinate-axis. Axes present is likely to hurt.

Table 78: T11 Projection, axes add-on qualitative panel. Each block compares the same sample with and without a coordinate-axis. Axes present is likely to hurt.

Table 79: T12 Reconstruct, axes add-on qualitative panel. Each block compares the same sample with and without a coordinate-axis. Axes present is likely to hurt.

T2 Packaging – scaling-responsive
“…determine whether the object can fit in the boundary in any orientation. Boundary: [2,2,2]…”

Model	GT	0.8B	2B (E2B)	4B (E4B)	9B	27B (26B-A4B)	35B-A3B (31B)
Qwen3.5	False	True $\bm{\times}$	True $\bm{\times}$	True $\bm{\times}$	True $\bm{\times}$	False $\checkmark$	False $\checkmark$
Gemma4		–	True $\bm{\times}$	False $\checkmark$	–	False $\checkmark$	False $\checkmark$

Appendix

Table 80: T2 Packaging, scaling-responsive instance, across the Qwen3.5 (0.8B–35B-A3B) and Gemma4 (E2B–31B, in parentheses) parameter ladders under Non-Thinking mode. Qwen3.5 flips from wrongly answering True at 0.8B–9B to correctly answering False (the object cannot fit the [2,2,2] boundary in any orientation) at 27B/35B-A3B; Gemma4 flips one step earlier, at E4B and up. Both families converge on the correct answer only at their larger checkpoints, mirroring the coarse-cue scaling gains quantified in Appendix B.3.2 .

Table 81: T4 Color Count, scaling-responsive instance, across the Qwen3.5 (0.8B–35B-A3B) and Gemma4 (E2B–31B, in parentheses) parameter ladders under Non-Thinking mode. Both families only reach the ground truth (6) at their two largest checkpoints, illustrating the scaling-responsive trend quantified in Appendix B.3.2 .

T6 Local Structure – scaling-responsive
“…determine whether any column (vertical stack of blocks) has a height greater than 3…”

Model	GT	0.8B	2B (E2B)	4B (E4B)	9B	27B (26B-A4B)	35B-A3B (31B)
Qwen3.5	False	True $\bm{\times}$	True $\bm{\times}$	False $\checkmark$	False $\checkmark$	False $\checkmark$	False $\checkmark$
Gemma4		–	False $\checkmark$	False $\checkmark$	–	False $\checkmark$	False $\checkmark$

Appendix

Table 82: T6 Local Structure, scaling-responsive instance, across the Qwen3.5 (0.8B–35B-A3B) and Gemma4 (E2B–31B, in parentheses) parameter ladders under Non-Thinking mode,. The Qwen3.5 ladder flips from wrongly answering True at 0.8B/2B to correctly answering False from 4B upward, the trend quantified in Appendix B.3.2 ; Gemma4 is already correct at every checkpoint on this instance, so this row demonstrates the Qwen-side scaling gain rather than a matched two-family transition.

T7 Color Dominance – scaling-responsive
“…determine the dominant color(s) in the object — the color(s) with the highest block count…”

Model	GT	0.8B	2B (E2B)	4B (E4B)	9B	27B (26B-A4B)	35B-A3B (31B)
Qwen3.5		$\bm{\times}$	$\bm{\times}$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Gemma4		–	$\bm{\times}$	$\checkmark$	–	$\checkmark$	$\checkmark$

Appendix

Table 83: T7 Color Dominance, scaling-responsive instance, across the Qwen3.5 (0.8B–35B-A3B) and Gemma4 (E2B–31B, in parentheses) parameter ladders under Non-Thinking mode. The two smallest checkpoints in each family over-list several non-dominant colors alongside the correct pair; every checkpoint at 4B/E4B and above converges exactly on the correct two-color set { cyan , yellow }, mirroring the coarse-cue scaling gains quantified in Appendix B.3.2 (Figure 5 ).

Table 84: T9 Visible Floors, scaling-responsive instance, across the Qwen3.5 (0.8B–35B-A3B) and Gemma4 (E2B–31B, in parentheses) parameter ladders under Non-Thinking mode. Both families walk from wrongly answering 3 at their two smallest checkpoints to correctly answering 4 (the ground truth) from their third checkpoint upward, the same shape as the T4 instance above; the aggregate T9 curve is noisier in between than T2/T4/T6 (Appendix B.3.2 ), but this instance isolates the same net small-to-large improvement.

Table 85: T11 Projection, scaling-responsive instance, across the Qwen3.5 (0.8B–35B-A3B) and Gemma4 (E2B–31B, in parentheses) parameter ladders under Non-Thinking mode. Both families flip from the wrong candidate (2) at their two smallest checkpoints to the correct 1 from their third checkpoint upward, the same shape as the T4/T9/T7 instances above, illustrating the responsive trend quantified in Appendix B.3.2 (Figure 5 ).

Table 86: T1 Volume, an instance that isolates the family-level asymmetry underlying T1’s aggregate curve in Appendix B.3.2 (Figure 5 ): across the Qwen3.5 (0.8B–35B-A3B) and Gemma4 (E2B–31B, in parentheses) parameter ladders under Non-Thinking mode. Qwen3.5 is genuinely scaling-resistant on this instance, landing on the same wrong value (

12

) at 5 of its 6 sizes regardless of scale; Gemma4 is scaling-responsive, flipping from wrong (E2B

=14

) to exactly correct (E4B

=11

, 31B

=11

), with a dip back to wrong at the intervening MoE checkpoint (26B-A4B

=13

) consistent with the paper’s broader finding that MoE checkpoints do not reliably help. Aggregated over all 100 rows, Qwen3.5’s 0.8B

\to

27B delta on T1 is only

+6.4

pp while Gemma4’s E2B

\to

31B delta is

+26.8

pp, this instance is representative of the two families’ opposite directions.

Table 87: T3 Difference, scaling-resistant instance, across the Qwen3.5 (0.8B–35B-A3B) and Gemma4 (E2B–31B, in parentheses) parameter ladders under Non-Thinking mode. Every one of the 10 sizes misses the (intersection, only-A, only-B) triple, and there is no monotonic approach to GT

(2,8,8)

as size grows – e.g. Qwen3.5-27B

(10,4,2)

is not closer than Qwen3.5-4B

(10,2,2)

– illustrating that scale does not recover the cross-object voxel-set alignment this task requires (Appendix B.3.2 ).

Table 88: T5 Color Distribution, scaling-resistant instance, across the Qwen3.5 (0.8B–35B-A3B) and Gemma4 (E2B–31B, in parentheses) parameter ladders under Non-Thinking mode. Every model size in both families misassigns the per-color block counts, including the two largest checkpoints per family that solve T4, illustrating that added scale does not recover exact per-color counts (Appendix B.3.2 ).

T8 Silhouette Symmetry – scaling-resistant
“…determine whether the object’s silhouette is symmetric (vertically or horizontally)…”

Model	GT	0.8B	2B (E2B)	4B (E4B)	9B	27B (26B-A4B)	35B-A3B (31B)
Qwen3.5	True	False $\bm{\times}$	False $\bm{\times}$	False $\bm{\times}$	False $\bm{\times}$	False $\bm{\times}$	False $\bm{\times}$
Gemma4		–	False $\bm{\times}$	False $\bm{\times}$	–	False $\bm{\times}$	False $\bm{\times}$

Appendix

Table 89: T8 Silhouette Symmetry, scaling-resistant instance, across the Qwen3.5 (0.8B–35B-A3B) and Gemma4 (E2B–31B, in parentheses) parameter ladders under Non-Thinking mode. All 10 sizes in both families answer False on a silhouette that is in fact symmetric (GT True), never once pairing the mirrored columns needed to detect it; the modal answer does not shift with scale, illustrating a scale-invariant default-to-False response under uncertainty at every point on the ladder (Appendix B.3.2 ).

T10 Color Connected – scaling-resistant
“…determine whether there exists at least one pair of adjacent blocks (sharing a face) that share the same color…”

Model	GT	0.8B	2B (E2B)	4B (E4B)	9B	27B (26B-A4B)	35B-A3B (31B)
Qwen3.5	False	True $\bm{\times}$	True $\bm{\times}$	True $\bm{\times}$	True $\bm{\times}$	True $\bm{\times}$	True $\bm{\times}$
Gemma4		–	False $\checkmark$	False $\checkmark$	–	False $\checkmark$	True $\bm{\times}$

Appendix

Table 90: T10 Color Connected, scaling-resistant instance, across the Qwen3.5 (0.8B–35B-A3B) and Gemma4 (E2B–31B, in parentheses) parameter ladders under Non-Thinking mode. The entire Qwen3.5 ladder answers True (wrong) at every size from 0.8B to 35B-A3B, showing no scaling trend; Gemma4 is correct at E2B/E4B/26B-A4B but regresses back to the wrong answer at its largest checkpoint (31B). Neither family shows the monotonic small-to-large convergence seen for the responsive tasks above, illustrating T10’s flat aggregate curve in Appendix B.3.2 (Figure 5 ).

Table 91: T12 Reconstruct, scaling-resistant instance, across the Qwen3.5 (0.8B–35B-A3B) and Gemma4 (E2B–31B, in parentheses) parameter ladders under Non-Thinking mode. All 10 sizes across both families collapse onto the same wrong candidate (1), never selecting the correct 0: added scale changes nothing about which cross-view silhouette is picked.

Table 92: T13 Mixture-of-Shapes, scaling-resistant instance, across the Qwen3.5 (0.8B–35B-A3B) and Gemma4 (E2B–31B, in parentheses) parameter ladders under Non-Thinking mode. All 10 sizes across both families collapse onto the same correct candidate (2). This example shows agreement across scales; it is not an instance of persistent failure. The aggregate scaling trends are reported in Appendix B.3.2 .

T1 Volume budget verdict: correct at budget $0$ , drifts wrong as the budget grows
“…count the total number of blocks that make up the 3D object…”

Model	GT	NT@ $0$	TH@ $1024$	TH@ $2048$	TH@ $4096$
	$12$	$12$ $\checkmark$	$12$ $\checkmark$	$13$ $\bm{\times}$	$9$ $\bm{\times}$
		$12$ $\checkmark$	$14$ $\bm{\times}$	$14$ $\bm{\times}$	$14$ $\bm{\times}$
Qwen3.5-27B, TH@1024: […] Top block. Let’s re-examine Image 1. In Image 1, the front column has: Top block (part of the main cross). Middle block. Bottom block. So the column is 3 blocks high below the main cross? Or is the top block part of the cross? Let’s look at the junction. In Image 1, the front-most block of the main horizontal layer is connected to a block below it. So, there is [trace cut off at the budget cap]

Appendix

Table 93: T1: Volume , thinking-budget qualitative panel. Both models answer correctly without thinking and neither recovers the count once the budget grows: Qwen3.5-27B moves

12\rightarrow 12\rightarrow 13\rightarrow 9

and Gemma4-31B settles on

14

from

1024

tokens onward. The quoted traces show the budget spent re-orienting the two views and relabeling arms rather than fixing a block lattice. Budgets run left to right:

0

(non-thinking),

1024

,

2048

, and

4096

reasoning tokens; each cell is the 5-run majority vote of that model at that budget on the single default-setting instance shown.

T4 Color Count budget verdict: wrong at budget $0$ , recovers the exact count at $4096$
“…count the number of unique colors in the object…”

Model	GT	NT@ $0$	TH@ $1024$	TH@ $2048$	TH@ $4096$
	$10$	$12$ $\bm{\times}$	$11$ $\bm{\times}$	$11$ $\bm{\times}$	$10$ $\checkmark$
		$11$ $\bm{\times}$	$10$ $\checkmark$	$10$ $\checkmark$	$10$ $\checkmark$
Qwen3.5-27B, TH@1024: […] In front of Slategray: Orchid. In front of Teal: Magenta. To the right of Teal: Red. To the left of Orchid: Orange. This implies a 2x3 grid on top? Or a cross? Let’s assume a central block. Let’s look at the “cross” shape in View 1. Center: Magenta? No. Let’s look at the connections. The block with the top face “Orchid” (light purple) is connected to “Orange” (left), “Magenta [trace cut off at the budget cap]

Appendix

Table 94: T4: Color Count , thinking-budget qualitative panel. Both models start wrong and reach the exact count as the budget grows. The budget-

4096

trace has committed to an explicit coordinate lattice with one color assigned per cell, which is the grounded intermediate the count is read off; at

1024

the same model is still debating whether the top face is a

2\times 3

grid or a cross. Budgets run left to right:

0

(non-thinking),

1024

,

2048

, and

4096

reasoning tokens; each cell is the 5-run majority vote of that model at that budget on the single default-setting instance shown.

T11 Projection budget verdict: Qwen drifts wrong at $4096$ ; Gemma recovers at $2048$
“…identify the invalid projection…”

Model	GT	NT@ $0$	TH@ $1024$	TH@ $2048$	TH@ $4096$
	$2$	$2$ $\checkmark$	$2$ $\checkmark$	$2$ $\checkmark$	$1$ $\bm{\times}$
		$0$ $\bm{\times}$	$0$ $\bm{\times}$	$2$ $\checkmark$	$2$ $\checkmark$
Qwen3.5-27B, TH@1024: […] Fifth layer down: 1 block (col 2). Wait, the stem is clearly visible. Let’s count the vertical stack in the stem. In View 1, the stem is the second column. It has blocks at levels 3, 4, 5 (counting from top). So, column 2 has height 5. Column 1 has height 3. Column 3 has height 3. Column 4 has height 2. Let’s check this against View 1. Col 1: 3 blocks high [trace cut off at the budget cap]

Appendix

Table 95: T11: Projection , thinking-budget qualitative panel. The two models move in opposite directions: Qwen3.5-27B holds the correct projection through

2048

tokens and drifts wrong at

4096

, while Gemma4-31B is wrong at

0

and

1024

and correct from

2048

onward. The instance-level pattern therefore matches the model average for Qwen3.5-27B only. Budgets run left to right:

0

(non-thinking),

1024

,

2048

, and

4096

reasoning tokens; each cell is the 5-run majority vote of that model at that budget on the single default-setting instance shown.

T1 Volume mode verdict: NT wins
“…count the total number of blocks that make up the 3D object…”

Mode	GT
NT	$10$	$10$ $\checkmark$	$10$ $\checkmark$	$12$ $\bm{\times}$	$10$ $\checkmark$
TH		$11$ $\bm{\times}$	$11$ $\bm{\times}$	$11$ $\bm{\times}$	$20$ $\bm{\times}$
Qwen3.5-27B (TH): […] Actually, let’s look for the longest continuous lines. Hypothesis 1: There is a central vertical column. There is a horizontal row extending to the right. There is a horizontal row extending to the left. There is a vertical column extending up. Let’s look at Image 1 again carefully. Leftmost part: A vertical stack of 3 blocks. Let’s call this Column A. Connected to the middle block of Column A is a block to its right. Let’s call this Block B. Above Block [trace cut off at the budget cap]

Appendix

Table 96: T1: Volume , thinking-mode qualitative panel. Non-Thinking answers

3

of

4

correctly and Thinking

0

of

4

;

3

models flip correct to wrong. The task-level verdict is NT wins . Each cell is the 5-run majority vote of that model on the single default-setting instance shown; NT is Non-Thinking and TH is Thinking at the default

1024

-token budget. Beneath the outcome rows we quote the tail of each model’s TH trace. Kimi-VL-A3B surfaces no reasoning tokens, so it has no quote.

T2 Packaging mode verdict: TH wins
“…determine whether the object can fit in the boundary in any possible orientation…”

Mode	GT
NT	False	True $\bm{\times}$	True $\bm{\times}$	True $\bm{\times}$	True $\bm{\times}$
TH		False $\checkmark$	False $\checkmark$	False $\checkmark$	False $\checkmark$
Qwen3.5-27B (TH): […] It shows a vertical stem going down. Let’s count the width again. 1, 2, 3, 4 blocks wide. Let’s count the height. The vertical stem has 3 blocks below the main horizontal section. The main section is 2 blocks high? Let’s look at the grid. Top row: 4 blocks. Second row: 4 blocks. Third row: 2 blocks (under the 3rd and 4th columns). Fourth row: 1 block (under the 3rd column). Fifth row [trace cut off at the budget cap]

Appendix

Table 97: T2: Packaging , thinking-mode qualitative panel. Non-Thinking answers

0

of

4

correctly and Thinking

4

of

4

;

4

models flip wrong to correct. The task-level verdict is TH wins . Each cell is the 5-run majority vote of that model on the single default-setting instance shown; NT is Non-Thinking and TH is Thinking at the default

1024

-token budget. Beneath the outcome rows we quote the tail of each model’s TH trace. Kimi-VL-A3B surfaces no reasoning tokens, so it has no quote.

T3 Difference mode verdict: tie ( $\Delta\approx 0$ , both wrong)
“…analyze both objects and determine the difference in their block compositions…”

Mode	GT
NT	$(5,5,6)$	$(10,2,2)$ $\bm{\times}$	$(6,3,3)$ $\bm{\times}$	$(8,3,3)$ $\bm{\times}$	$(2,4,5)$ $\bm{\times}$
TH		$(6,2,2)$ $\bm{\times}$	$(7,2,1)$ $\bm{\times}$	$(0,8,10)$ $\bm{\times}$	$(4,4,4)$ $\bm{\times}$
Qwen3.5-27B (TH): […] Let’s count top faces. There are 4 top faces visible in a 2x2 arrangement? No. Let’s look at the grid. Top row (back): 2 blocks. Middle row: 2 blocks. Front row: 1 block? Let’s look at the vertical columns. Back-left column: 2 blocks high. Back-right column: 2 blocks high. Front-left column: 2 blocks high. Front-right column: [trace cut off at the budget cap]

Appendix

Table 98: T3: Difference , thinking-mode qualitative panel. Non-Thinking answers

0

of

4

correctly and Thinking

0

of

4

; no model changes its answer. The task-level verdict is tie (

\Delta\approx 0

, both wrong) . Each cell is the 5-run majority vote of that model on the single default-setting instance shown; NT is Non-Thinking and TH is Thinking at the default

1024

-token budget. Beneath the outcome rows we quote the tail of each model’s TH trace. Kimi-VL-A3B surfaces no reasoning tokens, so it has no quote.

T4 Color Count mode verdict: TH wins
“…count the number of unique colors in the object…”

Mode	GT
NT	$8$	$9$ $\bm{\times}$	$9$ $\bm{\times}$	$10$ $\bm{\times}$	$10$ $\bm{\times}$
TH		$8$ $\checkmark$	$8$ $\checkmark$	$8$ $\checkmark$	$10$ $\bm{\times}$
Qwen3.5-27B (TH): […] Let’s look at the main block face in Image 2. Rightmost column: Red (bottom), Cyan (2nd), Orange (3rd), Orchid (top). This matches the leftmost column of Image 1 (Red, Cyan, Orange, Orchid). Middle column: Cyan (bottom), Yellow (2nd), Orchid (3rd), Orange (top). This matches the middle column of Image 1 (Cyan, Yellow, Orchid, Orange). Leftmost column (of the main block part): Yellow [trace cut off at the budget cap]

Appendix

Table 99: T4: Color Count , thinking-mode qualitative panel. Non-Thinking answers

0

of

4

correctly and Thinking

3

of

4

;

3

models flip wrong to correct. The task-level verdict is TH wins . Each cell is the 5-run majority vote of that model on the single default-setting instance shown; NT is Non-Thinking and TH is Thinking at the default

1024

-token budget. Beneath the outcome rows we quote the tail of each model’s TH trace. Kimi-VL-A3B surfaces no reasoning tokens, so it has no quote.

T5 Color Distribution mode verdict: tie (slight TH lead)
“…give the color distribution in the object…”

Mode	GT
NT	1 1 1 1 1 1 1 1 1 1	2 2 1 2 1 2 1 1 1 1 $\bm{\times}$	1 1 1 1 1 1 1 1 1 $\bm{\times}$	2 2 2 1 2 1 1 1 1 1 $\bm{\times}$	2 2 1 2 2 2 2 2 2 1 1 $\bm{\times}$
TH		1 1 1 1 1 1 1 1 1 1 $\checkmark$	1 1 1 1 1 1 1 1 1 1 $\checkmark$	2 2 2 2 2 2 1 2 1 1 $\bm{\times}$	8 2 1 1 2 1 1 1 1 5 1 $\bm{\times}$
Qwen3.5-27B (TH): […] Let’s look at the “Purple” block. In Image 1, Purple is to the right of Blue. In Image 2, Purple is to the left of Orchid. This implies the structure is: Back Row: Yellow/Grey, Cyan/Teal. Middle Row: Orange/Red, Orchid/?. Front Row: Blue/Magenta, Purple/?. Wait, let’s re-examine the adjacency. Image 1: Top layer visible: Yellow, [trace cut off at the budget cap]

Appendix

Table 100: T5: Color Distribution , thinking-mode qualitative panel. Non-Thinking answers

0

of

4

correctly and Thinking

2

of

4

;

2

models flip wrong to correct. The task-level verdict is tie (slight TH lead) . Each cell is the 5-run majority vote of that model on the single default-setting instance shown; NT is Non-Thinking and TH is Thinking at the default

1024

-token budget. Beneath the outcome rows we quote the tail of each model’s TH trace. Kimi-VL-A3B surfaces no reasoning tokens, so it has no quote.

T6 Local Structure mode verdict: TH wins
“…determine whether the object has at least one column (vertical stack of blocks) with a height greater than 3…”

Mode	GT
NT	False	True $\bm{\times}$	False $\checkmark$	True $\bm{\times}$	True $\bm{\times}$
TH		False $\checkmark$	True $\bm{\times}$	False $\checkmark$	True $\bm{\times}$
Qwen3.5-27B (TH): […] Back-left column: Height 3. (Visible: top). Back-right column: Height 3. (Visible: top, mid, bot on the far right edge). Wait, let’s look at the “L” shape again. In Image 1, there is a block sticking out to the left. Let’s look at the top surface. It looks like a 2x2 square of blocks. Let’s assume the top layer is a 2x2 block. Underneath the front-left top block, there are two more blocks. So column height is 3. [trace cut off at the budget cap]

Appendix

Table 101: T6: Local Structure , thinking-mode qualitative panel. Non-Thinking answers

1

of

4

correctly and Thinking

2

of

4

;

2

models flip wrong to correct and

1

model correct to wrong. The task-level verdict is TH wins . Each cell is the 5-run majority vote of that model on the single default-setting instance shown; NT is Non-Thinking and TH is Thinking at the default

1024

-token budget. Beneath the outcome rows we quote the tail of each model’s TH trace. Kimi-VL-A3B surfaces no reasoning tokens, so it has no quote.

T7 Color Dominance mode verdict: tie (slight TH lead on this instance)
“…determine the dominant color(s) in the object…”

Mode	GT
NT		$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$
TH		$\checkmark$	$\checkmark$	$\bm{\times}$	$\bm{\times}$
Qwen3.5-27B (TH): […] This is getting confusing. Let’s count total blocks. Let’s assume the object is a single connected component. Let’s look at Image 2 again. Rightmost: Orange. Left of Orange: Cyan. Under Cyan: Green. Left of Cyan: Yellow. Under Yellow: Yellow (visible underneath). Left of that Yellow: Yellow. Left/Back of that Yellow: Cyan. Front/Left of that Yellow: Cyan. Let’s try to map Image 1 to Image 2. Image 1 shows the object from a [trace cut off at the budget cap]

Appendix

Table 102: T7: Color Dominance , thinking-mode qualitative panel. Non-Thinking answers

0

of

4

correctly and Thinking

2

of

4

;

2

models flip wrong to correct. The task-level verdict is tie (slight TH lead on this instance) . Each cell is the 5-run majority vote of that model on the single default-setting instance shown; NT is Non-Thinking and TH is Thinking at the default

1024

-token budget. Beneath the outcome rows we quote the tail of each model’s TH trace. Kimi-VL-A3B surfaces no reasoning tokens, so it has no quote.

T8 Silhouette Symmetry mode verdict: NT wins
“…determine whether the object’s silhouette is symmetric (vertically or horizontally)…”

Mode	GT
NT	False	False $\checkmark$	False $\checkmark$	False $\checkmark$	False $\checkmark$
TH		False $\checkmark$	True $\bm{\times}$	False $\checkmark$	True $\bm{\times}$
Qwen3.5-27B (TH): […] Let’s look at Image 2 again. It’s a “T” shape but the crossbar is uneven. Left side of crossbar: 1 block. Right side of crossbar: 2 blocks. Back side of crossbar: 2 blocks. Front side: The vertical column. This is definitely asymmetric. Wait, let me look really closely at the images to see if I’m misinterpreting the structure. Image 1: Let’s trace the blocks. There is a central block. To its left: 2 blocks [trace cut off at the budget cap]

Appendix

Table 103: T8: Silhouette Symmetry , thinking-mode qualitative panel. Non-Thinking answers

4

of

4

correctly and Thinking

2

of

4

;

2

models flip correct to wrong. The task-level verdict is NT wins . Each cell is the 5-run majority vote of that model on the single default-setting instance shown; NT is Non-Thinking and TH is Thinking at the default

1024

-token budget. Beneath the outcome rows we quote the tail of each model’s TH trace. Kimi-VL-A3B surfaces no reasoning tokens, so it has no quote.

T9 Visible Floors mode verdict: NT wins
“…count the number of visible layers (floors/horizontal levels) in the object…”

Mode	GT
NT	$2$	$2$ $\checkmark$	$2$ $\checkmark$	$2$ $\checkmark$	$3$ $\bm{\times}$
TH		$3$ $\bm{\times}$	$3$ $\bm{\times}$	$1$ $\bm{\times}$	$6$ $\bm{\times}$
Qwen3.5-27B (TH): […] Let’s try a different hypothesis. Maybe the object is a single layer, just viewed from an angle? If it’s a single layer, all top faces should be on the same plane. In Image 1, look at the top faces. The top face of the leftmost block is at a certain height. The top face of the block next to it is at the same height. The top face of the block in the “middle” row is at the same height. The top face of the block in the “front” row… wait. [trace cut off at the budget cap]

Appendix

Table 104: T9: Visible Floors , thinking-mode qualitative panel. Non-Thinking answers

3

of

4

correctly and Thinking

0

of

4

;

3

models flip correct to wrong. The task-level verdict is NT wins . Each cell is the 5-run majority vote of that model on the single default-setting instance shown; NT is Non-Thinking and TH is Thinking at the default

1024

-token budget. Beneath the outcome rows we quote the tail of each model’s TH trace. Kimi-VL-A3B surfaces no reasoning tokens, so it has no quote.

T10 Color Connected mode verdict: TH wins
“…determine whether there exists at least one pair of adjacent blocks (sharing a face) that share the same color…”

Mode	GT
NT	True	True $\checkmark$	False $\bm{\times}$	False $\bm{\times}$	True $\checkmark$
TH		True $\checkmark$	True $\checkmark$	True $\checkmark$	True $\checkmark$
Qwen3.5-27B (TH): […] Orange block: Below purple. Green block: Next to yellow (left) and red (below). Wait, let’s look at the second image to confirm the structure. 2. Analyze the second image (View 2): This view is from a different angle, seemingly from the “front” or “bottom-right”. Let’s identify blocks. Left stack: Green on top, Red in middle, Red on bottom. This confirms the stack seen in View 1. [trace cut off at the budget cap]

Appendix

Table 105: T10: Color Connected , thinking-mode qualitative panel. Non-Thinking answers

2

of

4

correctly and Thinking

4

of

4

;

2

models flip wrong to correct. The task-level verdict is TH wins . Each cell is the 5-run majority vote of that model on the single default-setting instance shown; NT is Non-Thinking and TH is Thinking at the default

1024

-token budget. Beneath the outcome rows we quote the tail of each model’s TH trace. Kimi-VL-A3B surfaces no reasoning tokens, so it has no quote.

T11 Projection mode verdict: tie (slight NT lead)
“…identify the invalid projection…”

Mode	GT
NT	1	1 $\checkmark$	1 $\checkmark$	1 $\checkmark$	0 $\bm{\times}$
TH		2 $\bm{\times}$	0 $\bm{\times}$	2 $\bm{\times}$	2 $\bm{\times}$
Qwen3.5-27B (TH): […] Fourth row: 1 square (center-left). Fifth row: 1 square (center-left). This seems like a specific arrangement. Let’s assume the object is made of unit cubes. Let’s try to map it to a 3D grid. Let’s assume the “front” face is what we see in View 1. Columns (left to right): Col 1: 3 blocks high. Col 2: 5 blocks high (top 2 + stem 3? No, let [trace cut off at the budget cap]

Appendix

Table 106: T11: Projection , thinking-mode qualitative panel. Non-Thinking answers

3

of

4

correctly and Thinking

0

of

4

;

3

models flip correct to wrong. The task-level verdict is tie (slight NT lead) . Each cell is the 5-run majority vote of that model on the single default-setting instance shown; NT is Non-Thinking and TH is Thinking at the default

1024

-token budget. Beneath the outcome rows we quote the tail of each model’s TH trace. Kimi-VL-A3B surfaces no reasoning tokens, so it has no quote.

T12 Reconstruct mode verdict: tie (slight NT lead)
“…identify the positive view…”

Mode	GT
NT	1	1 $\checkmark$	0 $\bm{\times}$	1 $\checkmark$	1 $\checkmark$
TH		0 $\bm{\times}$	0 $\bm{\times}$	0 $\bm{\times}$	1 $\checkmark$
Qwen3.5-27B (TH): […] Let’s reconsider the standard set: Top, Front, Right Side. If Image 1 is Top View: It shows a 3x3 grid area. Blocks at: (0,0), (0,1), (0,2), (1,1), (1,2), (2,1). (Using x,y coordinates where x is right, y is up). Wait, let’s map Image 1 to a grid. Row 1 (top): Col 1, Col 2. Row 2 (mid): Col 1, Col 2, Col 3. Row 3 (bot): Col 1. If Image 2 is Front View: It [trace cut off at the budget cap]

Appendix

Table 107: T12: Reconstruct , thinking-mode qualitative panel. Non-Thinking answers

3

of

4

correctly and Thinking

1

of

4

;

2

models flip correct to wrong. The task-level verdict is tie (slight NT lead) . Each cell is the 5-run majority vote of that model on the single default-setting instance shown; NT is Non-Thinking and TH is Thinking at the default

1024

-token budget. Beneath the outcome rows we quote the tail of each model’s TH trace. Kimi-VL-A3B surfaces no reasoning tokens, so it has no quote.

T13 Mixture-of-Shapes mode verdict: tie (slight NT lead)
“…identify the invalid silhouette…”

Mode	GT
NT	2	2 $\checkmark$	2 $\checkmark$	2 $\checkmark$	2 $\checkmark$
TH		2 $\checkmark$	1 $\bm{\times}$	1 $\bm{\times}$	2 $\checkmark$
Qwen3.5-27B (TH): […] Image 4: Silhouette. Shows a vertical stack on the left, a cylinder in the middle, and two squares at the bottom right. Image 5: Silhouette. Shows a large L-shape on the left, a cylinder section, and blocks on the right. Let’s look at the “Candidate Silhouettes” more carefully. The prompt says “three candidate 2D orthographic silhouettes”. Candidate 0 (Image 3): This looks like a top [trace cut off at the budget cap]

Appendix

Table 108: T13: Mixture-of-Shapes , thinking-mode qualitative panel. Non-Thinking answers

4

of

4

correctly and Thinking

2

of

4

;

2

models flip correct to wrong. The task-level verdict is tie (slight NT lead) . Each cell is the 5-run majority vote of that model on the single default-setting instance shown; NT is Non-Thinking and TH is Thinking at the default

1024

-token budget. Beneath the outcome rows we quote the tail of each model’s TH trace. Kimi-VL-A3B surfaces no reasoning tokens, so it has no quote.

Table 109: T1 Volume.

“…Your task is to determine whether the object can fit in the boundary in any possible orientation. Boundary: [2,5,6] …”

GT
True	True $\checkmark$	True $\checkmark$	True $\checkmark$	True $\checkmark$	False $\bm{\times}$	True $\checkmark$	True $\checkmark$	True $\checkmark$

Appendix

Table 110: T2 Packaging.

“…Your task is to analyze both objects and determine the difference in their block compositions…”

GT
(8,5,7)	(4,4,8) $\bm{\times}$	(7,6,6) $\bm{\times}$	(0,10,11) $\bm{\times}$	(4,4,4) $\bm{\times}$	(2,4,5) $\bm{\times}$	(4,6,6) $\bm{\times}$	(0,15,13) $\bm{\times}$	(6,4,3) $\bm{\times}$

Appendix

Table 111: T3 Difference.

Table 112: T4 Color Count.

Table 113: T5 Color Distribution.

“…Your task is to determine whether the object has at least one column (vertical stack of blocks) with a height greater than 3…”

GT
True	True $\checkmark$	False $\bm{\times}$	False $\bm{\times}$	True $\checkmark$	True $\checkmark$	True $\checkmark$	False $\bm{\times}$	False $\bm{\times}$

Appendix

Table 114: T6 Local Structure.

“…Your task is to determine the dominant color(s) in the object — the color(s) with the highest block count…”

GT
	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\checkmark$	$\checkmark$	$\bm{\times}$

Appendix

Table 115: T7 Color Dominance.

“…Your task is to determine whether the object’s silhouette is symmetric (vertically or horizontally)…”

GT
True	False $\bm{\times}$	False $\bm{\times}$	False $\bm{\times}$	True $\checkmark$	False $\bm{\times}$	False $\bm{\times}$	False $\bm{\times}$	False $\bm{\times}$

Appendix

Table 116: T8 Silhouette Symmetry.

“…Your task is to count the number of visible layers (floors/horizontal levels) in the object…”

GT
3	4 $\bm{\times}$	5 $\bm{\times}$	4 $\bm{\times}$	6 $\bm{\times}$	4 $\bm{\times}$	6 $\bm{\times}$	4 $\bm{\times}$	3 $\checkmark$

Appendix

Table 117: T9 Visible Floors.

“…Your task is to determine whether there exists at least one pair of adjacent blocks (sharing a face) that share the same color…”

GT
False	True $\bm{\times}$	False $\checkmark$	True $\bm{\times}$	True $\bm{\times}$	False $\checkmark$	True $\bm{\times}$	False $\checkmark$	False $\checkmark$

Appendix

Table 118: T10 Color Connected.

Table 119: T11 Projection.

Table 120: T12 Reconstruct.

Table 121: T13 Mixture-of-Shapes.

Where Do Vision Language Models Fail at 3D Reasoning?

Abstract

Figures & tables

Appendix

Reasoning for VLMs

Do Vision-Language Models Understand 3D Scenes or Just Catalogue Objects?

SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes

Separating perception from reasoning in vision-language models: a model-free render ceiling for crystal structures

Dataset	3D	Pure	Procedural	Infinite	OE	Linked	No-image	Format	Probe
Breadth-first aggregators
SITE ( Wang et al., 2025 )	$\triangle$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\triangle$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$
EASI ( Cai et al., 2025 )	$\triangle$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\triangle$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$
OmniSpatial ( Jia et al., 2026 )	$\triangle$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$
Spatial-DISE ( Huang et al., 2026 )	$\checkmark$	$\triangle$	$\triangle$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$
SRBench ( Stogiannidis et al., 2025 )	$\triangle$	$\triangle$	$\triangle$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$

Abstract

Reasoning for VLMs

Explore similar work

Do Vision-Language Models Understand 3D Scenes or Just Catalogue Objects?

SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes

Separating perception from reasoning in vision-language models: a model-free render ceiling for crystal structures