SpatialChain: A Benchmark for Auditing Spatial Reasoning Faithfulness in VLMs

Rafael Teixeira Sousa; Vinícius Paulo Lopes de Oliveira; Elisa Ayumi Masasi de Oliveira; Luiza Martins de Freitas Cintra; Fernanda Bufon Färber; Igor Dias Aguiar; Julia Yasmim de Almeida Nobre; Arlindo Rodrigues Galvão Filho

cs.CVOct 5, 2026

SpatialChain: A Benchmark for Auditing Spatial Reasoning Faithfulness in VLMs

Authors: Rafael Teixeira Sousa, Vinícius Paulo Lopes de Oliveira, Elisa Ayumi Masasi de Oliveira, Luiza Martins de Freitas Cintra, Fernanda Bufon Färber, Igor Dias Aguiar, Julia Yasmim de Almeida Nobre, Arlindo Rodrigues Galvão Filho

Organizations: Universidade Federal de Mato Grosso (UFMT) · Universidade Federal de Goiás (UFG) · Advanced Knowledge Center for Immersive Technologies (AKCIT)

Abstract

Thinking-enabled vision-language models (VLMs) report ever-higher accuracy on spatial benchmarks, yet final-answer scores cannot reveal whether a correct prediction reflects faithful spatial reasoning or a linguistic shortcut. We introduce SpatialChain, a dataset of 28,350 training and 899 test examples pairing spatially-oriented GQA questions with scene-graph-grounded reasoning chains, retained only when the generated answer matches the symbolic ground truth, and a two-axis evaluation combining objective chain-overlap metrics with a scene-graph-aware LLM judge that scores faithfulness and completeness independently of the final answer. Applied to nine thinking-enabled VLMs, the protocol surfaces three findings invisible to standard accuracy: (i) four of nine models achieve $\geq$ 79% VQA accuracy while exhibiting shortcut rates above 39%, i.e., correct answers whose reasoning the judge marks as unfaithful; (ii) chain quality significantly predicts answer correctness for seven of nine models, but the two exceptions (Claude Sonnet 4.6, InternVL3.5-8B) reveal qualitatively distinct failure modes, terse output vs. verbose-decorative reasoning, that benchmark accuracy alone conflates; (iii) SFT on SpatialChain improves Qwen3-VL-8B by +6.2 pp in-domain and reduces its shortcut rate to 22%, while a stylistic specialization effect on external benchmarks motivates replay-augmented training as mitigation. The faithfulness judge is validated against 198 human-annotated items, where judge-human agreement matches human-human agreement, and against a second judge from a different provider, which preserves the model ranking ( $ρ$ = 0.88). Data, generation scripts, and evaluation code are released at https://github.com/spatialchain/SpatialChainBenchmark.

Figures & tables

Split	Source	Samples	Images	Yes / No / Attr.	Yes-rate	Avg. chain (w)
Train	GQA train	28,350	19,263	16,516 / 9,926 / 1,908	0.58	115
Test	GQA val	899	719	566 / 329 / 4	0.63	145

Table 1 : SpatialChain splits. Yes / No / Attr. partitions answers between binary spatial verification and attribute-typed answers. Avg. chain is the mean reasoning trace length in words.

Model	Acc	F1	Yes	No	R-1	R-L	F1 tok	Len ${}_{\text{ratio}}$
Qwen3-VL-8B FT	85.03	88.68	92.76	71.73	0.654	0.443	0.643	0.90
Qwen3-VL-4B FT	82.23	86.67	91.34	66.57	0.657	0.450	0.646	0.89
GLM-4.1-9B	81.34	84.49	80.39	82.98	0.438	0.301	0.421	1.47
InternVL3.5-4B †	81.34	87.43	87.28	71.12	0.314	0.227	0.305	3.01
Claude Sonnet 4.6 †	79.89	85.15	87.63	66.57	0.359	0.258	0.354	0.36
InternVL3.5-8B †	79.33	85.29	86.57	66.87	0.323	0.228	0.316	7.07

Table 2 : Model comparison on the SpatialChain test split (

n=895

; see § 6 ). Acc and F1 are overall accuracy and macro-F1 on the binary Yes / No task. Yes / No are accuracies conditional on ground-truth polarity. R-1 , R-L , F1 tok are reasoning-chain overlap metrics against the scene-graph reference chain. Len

{}_{\text{ratio}}

is predicted-to-reference chain length (reference average: 145 tokens). † Unparsed predictions excluded from accuracy (InternVL3.5-4B: 25; InternVL3.5-8B: 16; Claude Sonnet 4.6: 7).

Model	R-1 ✓	R-1 ×	$\Delta$ R-1	F1 ✓	F1 ×	$p$
Qwen3-VL-4B	0.418	0.351	+0.067	0.406	0.342	$<\!0.001^{**}$
Qwen3-VL-8B	0.439	0.373	+0.066	0.426	0.362	$<\!0.001^{**}$
GLM-4.1-9B	0.447	0.399	+0.048	0.429	0.385	$<\!0.001^{**}$
Qwen3-VL-4B FT	0.664	0.625	+0.039	0.653	0.612	$<\!0.001^{**}$
Qwen3-VL-8B FT	0.660	0.626	+0.034	0.649	0.612	$<\!0.001^{**}$
InternVL3.5-4B	0.320	0.288	+0.031	0.311	0.282	$0.003^{**}$

Table 3 : Reasoning-chain quality conditioned on answer correctness.

\Delta

R-1 is the mean R-1 difference between correct and incorrect predictions. ∗

p<0.05

; ∗∗

p<0.01

Model	Acc	Faith.	Compl.	Pass	Shortcut
Qwen3-VL-8B FT	85.03	0.630	0.700	0.820	0.222
Qwen3-VL-4B FT	82.23	0.631	0.708	0.802	0.194
GLM-4.1-9B	81.34	0.527	0.583	0.809	0.453
Claude Sonnet 4.6	79.89	0.541	0.582	0.794	0.390
Qwen3-VL-4B	78.44	0.585	0.658	0.776	0.264
Qwen3-VL-8B	78.88	0.571	0.658	0.765	0.260

Table 4 : LLM-as-judge scores, sorted by pass rate. Faith. and Compl. are mean faithfulness and completeness in

[0,1]

. Pass requires

\mathit{AC}\geq 0.5

and

(\mathit{RF}+\mathit{RC})/2\geq 0.5

. Shortcut is the fraction of correct test items with

\mathit{RF}\leq 0.5

Figure 1 : Pass rate vs. shortcut rate. The bottom-right region ( ideal ) marks high passing with low shortcut behaviour; the top-left ( shortcut-heavy ) marks competitive accuracy achieved with frequent unfaithful reasoning. Fine-tuned Qwen3-VL variants are closest to ideal; InternVL3.5-4B is the clearest shortcut case.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Group	GQA types
Spatial relation verification	relVerify , relVerifyCop , relVerifyCr
Existential with relation	existRelS , existRelSC , existRelSRC
Place verification	placeVerify , placeVerifyC
Two-object comparison	twoDifferent , twoDifferentC , twoSame , twoSameC
Category comparison	diffAnimals , diffAnimalsC , diffGender
Attribute / ordinal selection	chooseAttr , compare

Appendix

Table 5 : GQA question types selected for SpatialChain .

Relation	Count	%	Yes-rate
left_of	9,832	34.7	0.57
right_of	7,919	27.9	0.58
in_front_of	1,987	7.0	0.54
behind	1,406	5.0	0.55
near	347	1.2	0.61
above	763	2.7	0.71

Appendix

Table 6 : Per-relation statistics in the SpatialChain training split. Yes-rate is the fraction of questions with a Yes ground-truth answer.

Figure 2 : Distribution of SpatialChain training (left) and test (right) splits by primary spatial relation (inner ring) and answer class (outer ring; green : yes , red : no , orange : attribute). The test split concentrates on projective relations, reflecting the complexity-biased selection.

Figure 3 : Word-count distribution of reasoning chains in the training (blue) and test (orange) splits. Dashed vertical lines indicate medians.

Figure 4 : Distribution of the number of sentences per thinking trace for the training (left) and test (right) splits. Both distributions are unimodal and right-skewed, peaking at 5–6 sentences. The test split exhibits a slightly higher mean (4.87 vs. 4.58), consistent with its bias toward more complex multi-hop chains.

Figure 5 : Thinking trace length (in characters) as a function of question length (in characters) for the training (left, Pearson r = 0.42) and test (right, Pearson r = 0.23) splits. Longer questions elicit longer reasoning traces.

Panel of Figure 6 — Figure 6: Representative examples from SpatialChain , grouped by reasoning complexity. Top row : single-hop questions (one spatial relation). Bottom row : multi-hop questions chaining two relations ( left_of $\to$ above ; left_of $\to$ left_of ). Underlined phrases mark spatial relations; reasoning is truncated for space. Full chains are in the released dataset.

Hyperparameter	Value
Base model	Qwen3-VL-8B-Thinking
Quantization	4-bit NF4 (BitsAndBytes)
Quantization compute dtype	bfloat16
Double quantization	✓
Attention implementation	Flash Attention 2
LoRA rank $r$	16

Appendix

Table 7 : Model and LoRA configuration.

Hyperparameter	Value
Training examples	28,350
Epochs	2
Per-device batch size	6
Gradient accumulation steps	2
Effective batch size	12
Learning rate	$3\times 10^{-5}$

Appendix

Table 8 : Training configuration for SFT on SpatialChain .

Model	Acc	R-1	R-2	R-L	F1 tok	Prec tok	Rec tok	BLEU-1	Entity-ov.	Step-mk.
Qwen3-VL-4B FT	82.23	0.657	0.407	0.450	0.646	0.701	0.614	0.570	0.481	0.26
Qwen3-VL-8B FT	85.03	0.654	0.402	0.443	0.643	0.694	0.615	0.568	0.484	0.45
GLM-4.1-9B	81.34	0.438	0.212	0.301	0.421	0.474	0.470	0.302	0.305	1.34
Qwen3-VL-8B	78.88	0.425	0.218	0.293	0.412	0.353	0.597	0.328	0.368	2.15
Qwen3-VL-4B	78.44	0.403	0.208	0.279	0.392	0.323	0.613	0.303	0.379	2.77
Claude 4.6	79.89	0.359	0.187	0.258	0.354	0.683	0.248	0.127	0.253	0.00

Appendix

Table 9 : Evaluation of reasoning chains on the SpatialChain test set. Extended version of Table 2 with additional metrics (ROUGE-2, BLEU, token precision/recall, entity overlap, step markers per chain). Models sorted by overall accuracy.

Model	left_of	right_of	in_front_of	behind	above	below	next_to	other
Qwen3-VL-8B FT	0.836	0.868	0.800	1.000	0.833	0.500	0.750	0.923
Qwen3-VL-4B FT	0.816	0.825	0.800	0.667	0.833	1.000	1.000	0.923
GLM-4.1-9B	0.812	0.808	0.933	0.667	0.833	0.500	0.750	1.000
InternVL3.5-4B	0.803	0.815	1.000	0.667	0.667	1.000	0.750	1.000
Qwen3-VL-4B	0.796	0.778	0.733	0.667	0.833	0.000	0.750	0.769
Qwen3-VL-8B	0.789	0.788	0.667	1.000	0.833	0.500	0.750	0.923

Appendix

Table 10 : Per-relation accuracy on the SpatialChain test split. Counts of validation examples for each relation: left_of (314), right_of (327), in_front_of (15), behind (3), above (12), below (4), next_to (8), other (13). Note that some relations have very small sample sizes; results for those are reported for completeness only.

Checkpoint	Traj.	Action	Spatial	State	Task	Multi-v.	Point.	Other	Overall
Base	0.409	0.431	0.524	0.473	0.500	0.405	0.382	0.429	0.453
Step 63	0.288	0.375	0.488	0.527	0.368	0.297	0.353	0.143	0.388
Step 313	0.424	0.458	0.464	0.418	0.500	0.324	0.441	0.071	0.425
Step 625	0.167	0.417	0.464	0.455	0.474	0.324	0.412	0.214	0.380
Step 1875	0.303	0.417	0.405	0.436	0.421	0.351	0.324	0.429	0.385
Step 2813	0.303	0.389	0.405	0.436	0.342	0.351	0.441	0.143	0.372

Appendix

Table 11 : Qwen3-VL-4B accuracy on ERQA by category over fine-tuning. Step counts refer to optimisation steps with effective batch size 12.

Checkpoint	Left	Right	Above	Under	Close	Far	Overall
Base	0.943	0.905	0.782	0.772	0.699	0.640	0.791
Step 63	0.937	0.913	0.797	0.698	0.676	0.608	0.773
Step 313	0.929	0.873	0.819	0.701	0.670	0.582	0.763
Step 625	0.904	0.845	0.809	0.724	0.642	0.609	0.757
Step 1875	0.904	0.858	0.809	0.704	0.613	0.567	0.744
Step 2813	0.909	0.873	0.807	0.721	0.624	0.601	0.757

Appendix

Table 12 : Qwen3-VL-4B accuracy on EmbSpatial-Bench by relation category over fine-tuning. Vertical ( above ) and projective ( left , right ) relations are preserved; depth-related relations ( close , far ) are most affected.

Panel of Figure 7 — Figure 7: Both models answer correctly, but through qualitatively different processes. The reference chain (top) was generated from the GQA scene graph by SpatialChain and follows a step-by-step spatial structure. The fine-tuned model (left) closely mirrors this structure and names the bird as direct evidence. InternVL3.5-8B (right) arrives at the correct answer via hallucinated scene description. Full outputs for all nine models are in Appendix K .

Figure 8 : Image 2328879 used as input fot the discussed question "Do you see kittens to the right of the bread that is to the right of the knife".

Model	Full prediction	Correct
Reference (scene graph)	“Looking at the image, let me trace through this step-by-step: (1) Locating the knife: I can see a knife on the left side of the plate, positioned near the bottom left. (2) Finding the bread to the right of the knife: there is a large piece of bread that sits on the plate to the right of where the knife is placed. (3) Examining what is to the right of that bread: to the right of the bread I can see what appears to be gray birds standing on the plate. These are birds with wings and legs visible. (4) Looking for kittens: I do not see any kittens anywhere in the image. The animals present in the scene are birds — specifically gray colored birds that appear to be standing on or near the bread on the plate.” Answer: No, there is a bird to the right of the bread.	—
Qwen3-VL-4B FT (fine-tuned)	“Looking at the image, I need to trace through this step by step: (1) First, I can see a knife on the left side of the image, positioned near the edge. (2) To the right of the knife, there is bread visible in the center-left portion of the image. (3) To the right of that bread, I can see a white plate with a bird perched on it. (4) Looking further to the right of the bread (and the plate with the bird), I see what appears to be a small bird on the table, but examining the entire right side of the bread, I do not see any kittens anywhere in the image. The image shows a table setting with a woman, food items including bread, a plate, a bird, and other dining elements. There are no kittens visible anywhere in the scene, let alone to the right of the bread.” Answer: No, there is a bird to the right of the bread.	✓
Qwen3-VL-8B FT (fine-tuned)	“Looking at the image, I can see a white plate with various food items on it. On the left side, there’s a knife, and to the right of it is a piece of bread. Moving further to the right of the bread, I can see what appears to be a small brown bird standing on the plate. However, I don’t see any kittens anywhere in this image. The only animal visible to the right of the bread is a small brown bird. There are no kittens present in the scene.” Answer: No, there are no kittens visible in the image.	✓
Qwen3-VL-4B (base)	“So, let us break this down. First, the question is asking if there are kittens to the right of the bread that is to the right of the knife. Let us identify each element. First, the knife: there is a knife on the left side of the plate, near the bread. The bread is the sandwich in the center of the plate. Now, ‘the bread that is to the right of the knife’ — so the knife is on the left, the bread is to the right of that knife. Then, we need to look to the right of that bread. What is there? On the plate, to the right of the bread (sandwich), there is a sauce (red sauce, maybe tomato) and some other food. But the question is about kittens. Are there any kittens in the image? Let us check the image. The image shows a plate with a sandwich, some sauce, a knife, and two small birds (sparrows) on the plate or near it. There are no kittens visible. The birds are sparrows, not kittens. So, to the right of the bread (which is the sandwich), there are no kittens. The birds are there, but they are birds, not kittens. So the answer is no.” Answer: No.	✓
Qwen3-VL-8B (base)	“Got it, let’s break this down. First, identify the elements. The question is about kittens to the right of the bread that’s to the right of the knife. Let’s look at the image. First, find the knife. There’s a knife on the left side of the plate, next to the fork. Then, the bread: the hamburger bun is on the plate, to the right of the knife. Now, to the right of that bread (hamburger bun), what’s there? There’s a bird, not a kitten. The image shows birds, not kittens. So no kittens are present. The question asks if there are kittens to the right of the bread (hamburger) which is right of the knife. Since there are birds, not kittens, the answer is no.” Answer: The image shows birds (not kittens) to the right of the hamburger bun, which is to the right of the knife. No kittens are visible. Answer: No, there are no kittens in the image	✓
GLM-4.1-9B	“Got it, let’s analyze the image. First, look for the knife, then the bread to its right, then check if there are kittens to the right of that bread. The image shows a plate with food, a knife on the left, bread (the bun) next to it, and then birds, not kittens. So there are no kittens in that area. Therefore, the answer would be no.” Answer: To determine this, first locate the knife on the left side of the plate, then identify the bread (bun) to the right of the knife. Next, check for kittens to the right of that bread. The area to the right of the bread shows birds, not kittens. Therefore, the answer is No.	✓

Appendix

Table 13 : Complete model outputs for the qualitative example in Figure 7 ( imageId : 2328879). Question: “Do you see kittens to the right of the bread that is to the right of the knife?” Ground-truth answer: No, there is a bird to the right of the bread. Outputs are reproduced verbatim; <think> / </think> delimiters are omitted for readability.

Dimension	human $\leftrightarrow$ human	judge $\leftrightarrow$ individual human	judge $\leftrightarrow$ human consensus
Answer correctness	0.673	0.655	0.679
Reasoning faithfulness	0.443	0.488	0.574
Reasoning completeness	0.416	0.551	0.643

Appendix

Table 14 : Rank correlation (Spearman

\rho

) between raters on

n=198

items. Mean absolute error of the judge against the human consensus is 0.14 (correctness), 0.27 (faithfulness), 0.21 (completeness).

Model	Claude Sonnet 4.6 judge	GPT-5.6 judge
Qwen3-VL-8B FT	0.222	0.466
Qwen3-VL-4B FT	0.194	0.396
GLM-4.1-9B	0.453	0.595
Claude Sonnet 4.6	0.390	0.546
Qwen3-VL-4B	0.264	0.574
Qwen3-VL-8B	0.260	0.588

Appendix

Table 15 : Shortcut rate under two independent judges (fraction of correct test set,

\mathit{RF}\leq 0.5

Model	$\tau{=}0.3$	$\tau{=}0.4$	$\tau{=}0.5$	$\tau{=}0.6$	$\tau{=}0.7$
Qwen3-VL-8B FT	0.026	0.029	0.222	0.227	0.358
Qwen3-VL-4B FT	0.019	0.020	0.194	0.195	0.305
GLM-4.1-9B	0.003	0.003	0.453	0.455	0.627
Claude Sonnet 4.6	0.008	0.009	0.390	0.390	0.473
Qwen3-VL-4B	0.010	0.010	0.264	0.265	0.433
Qwen3-VL-8B	0.018	0.018	0.260	0.265	0.427

Appendix

Table 16 : Shortcut rate vs. faithfulness threshold

\tau

(

\mathit{RF}\leq\tau

; fraction of correct test set).

Figure 9 : Shortcut rate as a function of the faithfulness threshold $\tau$ . The shaded region marks the plateau $\tau\in[0.5,0.6]$ on which all rates are identical.

Subset	$n$	Pearson $r$	Spearman $\rho$	Kendall $\tau$
All models	9	$+0.690$ ( $p=0.040$ )	$+0.644$ ( $p=0.061$ )	$+0.479$ ( $p=0.075$ )
Zero-shot only	7	$+0.445$ ( $p=0.317$ )	$+0.270$ ( $p=0.558$ )	$+0.195$ ( $p=0.543$ )
Qwen3 family only	4	$+0.926$ ( $p=0.074$ )	$+0.800$ ( $p=0.200$ )	$+0.667$ ( $p=0.333$ )

Appendix

Table 17 : Cross-model correlation between VQA accuracy and reasoning-chain quality (ROUGE-1).

Model	Acc single	Acc multi	$\Delta$ Acc	RF single	RF multi	$\Delta$ RF
Qwen3-VL-8B FT	0.847	0.845	$-0.002$	0.630	0.630	$-0.000$
Qwen3-VL-4B FT	0.834	0.809	$-0.024$	0.641	0.623	$-0.017$
GLM-4.1-9B	0.810	0.815	$+0.005$	0.517	0.533	$+0.016$
Claude Sonnet 4.6	0.777	0.823	$+0.047$	0.529	0.550	$+0.022$
Qwen3-VL-4B	0.764	0.805	$+0.042$	0.566	0.597	$+0.031$
Qwen3-VL-8B	0.748	0.815	$+0.067$	0.550	0.586	$+0.036$

Appendix

Table 18 : Accuracy and reasoning faithfulness by question complexity (single-hop:

n=385

; multi-hop:

n=509

\Delta

is multi

-

single.

Explore similar work

ProSR: Process-Shaped Spatial Reasoning for Reliable Chain-of-Thought in VLMs
May 25, 2026Jiangyang Li, Cong Wan, Changjie Wu +8Spatial Reasoning Reasoning Trajectory
1Xi’an Jiaotong University · 2Amap, Alibaba Group · 4Shenzhen University +1
Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs
Aug 5, 2026Yang Yang, Jiawei Chen, Tairan Chen +1Spatial Reasoning Multimodal Large Language Models
Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University · Zhongguancun Academy · Stevens Institute of Technology
Why Far Looks Up: Probing Spatial Representation in Vision-Language Models
May 28, 2026Cheolhong Min, Jaeyun Jung, Daeun Lee +5Entanglement
Seoul National University · The Ohio State University · NVIDIA

May 25, 2026, cs.CVJ/K move · Enter open · S save

ProSR: Process-Shaped Spatial Reasoning for Reliable Chain-of-Thought in VLMs

Jiangyang Li, Cong Wan, Changjie Wu, Songlin Dong+7

1Xi’an Jiaotong University · 2Amap, Alibaba Group · 4Shenzhen University+1

Spatial Reasoning Reasoning Trajectory Chain-of-Thought Reasoning

Open paper AskPDF arXiv

Reliable spatial reasoning remains a core bottleneck for vision-language models (VLMs). Existing mainstream training paradigms for spatial reasoning largely rely on outcome alignment or process imitation, lacking explicit constraints on the reasoning process, and therefore struggle to ensure genuine visual dependence and stable reasoning trajectories. In this paper, we construct a high-quality CoT dataset covering diverse spatial phenomena and diagnose the model's reasoning process, revealing two typical types of process degradation during reinforcement learning optimization: Spurious Grounding, which bypasses visual evidence, and Tail Instability, where uncertainty abnormally rises in the later stage of reasoning. To address these issues, we propose ProSR, a process-shaping optimization framework for spatial reasoning. Through a Counterfactual Invariance Penalty and a Tail Drift Penalty, ProSR extends the optimization objective from single answer correctness to two process-level dimensions: visual dependence and trajectory stability. Experiments on multiple complex and out-of-distribution spatial reasoning benchmarks show that ProSR improves answer accuracy while generating reasoning trajectories that are more stable and more dependent on visual evidence.

Abstract

Explore similar work

ProSR: Process-Shaped Spatial Reasoning for Reliable Chain-of-Thought in VLMs

Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs

Why Far Looks Up: Probing Spatial Representation in Vision-Language Models