We introduce GSU, a text-only grid dataset to evaluate the spatial reasoning capabilities of LLMs over 3 core tasks: navigation, object localization, and structure composition. By forgoing visual inputs, isolating spatial reasoning from perception, we show that while most models grasp basic grid concepts, they struggle with frames of reference relative to an embodied agent and identifying 3D shapes from coordinate lists. We also find that exposure to a visual modality does not provide a generalizable understanding of 3D space that VLMs are able to utilize for these tasks. Finally, we show that while the very latest frontier models can solve the provided tasks (though harder variants may still stump them), fully fine-tuning a small LM or LORA fine-tuning a small LLM show potential to match frontier model performance, suggesting an avenue for specialized embodied agents.
Figures & tables
Figure 1: The GSU Dataset tasks and settings visualized for clarity (note that the models do NOT receive these visuals and instead receive a textual version of the environment shown in Appendix E ). The grey arrows indicate the heading, i.e. direction that the embodied agent is facing. In the Navigation task, they remain aligned with +Y in the Cardinal setting and rotate to reflect the last direction of travel in the Egocentric setting, which affects the direction of the second step and the final coordinates. For the Object Localization task, we show how different headings affect the spatial relations between the target and the reference. For the Structure Composition task, we show the 3 structure categories that models may be asked to describe.
Navigation
Object Localization
Structure Composition
Follower
Instructor
Ego-
Allo-
Os
Oc
Of
On
2D
3D
Card.
Ego.
Card2Ego
centric
centric
Mistral-7b
0.30
0.38
0.59
0.15
0.27
30.32
46.50
49.75
72.60
30.21
20.01
Pixtral
0.33
0.50
0.87
0.26
0.35
40.83
48.18
66.25
87.68
42.16
55.43
Llama-8b
0.33
0.47
0.69
0.06
0.09
46.17
50.18
62.04
86.69
22.70
37.75
LlamaVL-11b
0.32
0.47
0.74
0.14
0.28
48.40
48.08
64.48
86.58
44.44
44.05
Table 1: 3 GSU task results across 3 model categories (prompted LLMs and VLMs , prompted frontier models, and fine-tuned models ) with best scores from each bolded. Navigation results are accuracies: final coordinate accuracies for the Egocentric Follower and instruction sequence accuracy for the 2D Instructor , while OL and SC results are term overlaps: spatial Os , color Oc , shape/form Of , and numerical On . Other settings in Appendix D .
Figure 2: Allocentric front/back flipping
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: GSU Dataset Tasks and Settings
Egocentric
Cardinal
2D
3D
2D
3D
Left
30.4%
17.69%
21.33%
16.67%
Right
29.6%
18.77%
34.67%
14.58%
Front
20.8%
13.72%
25.33%
12.5%
Back
19.2%
13.72%
18.67%
13.54%
Up
0
20.94%
0
29.17%
Appendix
Table 2: Navigation Direction Distribution (reported out of 250 possible steps)
Egocentric
Allocentric
Left
18.88%
18.69%
Right
18.37%
13.64%
Forward
16.84%
17.17%
Backward
14.80%
6.57%
Up
20.41%
18.37%
Down
14.29%
13.64%
Appendix
Table 3: Distribution of Ground Truth Spatial Direction in the Egocentric and Allocentric Object Localization settings
Plain
Dict
Set
Os↑
Oc↑
Of↑
Os↑
Oc↑
Of↑
Os↑
Oc↑
Of↑
0-shot
Mistral-7b
15.92
42.64
12.45
13.72
47.93
10.87
15.88
46.70
13.83
Pixtral
37.67
74.36
14.64
44.67
70.13
13.62
32.08
72.91
7.25
Llama-8b
21.29
63.90
13.00
22.78
63.91
15.56
25.47
68.83
18.65
LlamaVL-11b
35.83
69.07
11.83
24.57
60.48
12.09
20.94
62.14
13.37
Qwen-7b
32.17
68.72
11.48
40.03
68.32
16.73
28.52
62.76
16.65
Appendix
Table 4: Structure Composition results for Plain, Dict, and Set Representations
Percentage
Spatial Relations
Above
19.67%
Below
21.2%
Front
19.67%
Back
10.61%
Left
12.12%
Right
16.67%
Appendix
Table 5: Distribution of Spatial Relations, Colors, and Shapes in the Structure Composition task
Follower
Instructor
Egocentric 2D
Egocentric 3D
Cardinal
Egocentric
Card2Ego
acc ↑
dist. ↓
acc ↑
dist. ↓
acc ↑
dist. ↓
acc ↑
dist. ↓
acc ↑
dist. ↓
Flan-T5 w/o finetune
0
34.64
0
28.28
0
8.79
0
7.82
0
8.23
Flan-T5 w/ finetune
0.3
4.21
0.71
1.68
0.99
0.02
0.9
0.42
1.0
0
BART w/o finetune
0
34.64
0
28.28
0
8.22
0
7.44
0
8.21
BART w/ finetune
0.24
2.37
0.09
7.82
0.26
2.72
0
6.29
0.85
1.37
Appendix
Table 6: BART-base and Flan-T5-large Navigation Ablations
Object Localization
Structure Composition
Egocentric
Allocentric
Os
Oc
Of
On
Flan-T5 w/o finetune
16.5
16.67
67
7.11
16.6
8.9
Flan-T5 w/ finetune
19.5
94.5
99
96.17
88.83
97.07
Appendix
Table 7: BART-base and Flan-T5-Large Ablations over Object Localization and Structure Composition tasks
Follower
Instructor
Egocentric 2D
Egocentric 3D
Cardinal
Egocentric
Card2Ego
Mistral-7b
6.85
6.14
2.84
11.87
11.23
Pixtral
6.31
5.60
0.75
7.65
9.01
Llama-8b
7.03
4.85
1.95
11.87
12.30
LlamaVL-11b
7.36
5.78
1.52
11.96
9.59
Qwen-7b
22.54
20.44
1.57
8.96
7.52
Appendix
Table 8: The distance off scores for each of the Navigation experiments in Table 1 .
Cardinal 3D
Egocentric 3D
acc
dist
acc
dist
1
Mistral-7b
0.48
4.14
0.38
6.14
Llama-8b
0.62
2.65
0.47
4.85
Qwen-7b
0.63
7.79
0.19
20.44
Few
Mistral-7b
0.19
6.23
0.17
8.32
Llama-8b
0.32
4.84
0.20
7.69
Appendix
Table 9: 1-shot vs Few-shot Navigation: Comparison over 1-shot performance with reasoning vs Few-Shot without reasoning examples
Follower
Instructor
Cardinal
Egocentric
Cardinal
Egocentric
2d
3d
2d
3d
2d
3d
2d
3d
acc. ↑
dist ↓
acc ↑
dist ↓
acc ↑
dist ↓
acc ↑
dist ↓
acc ↑
acc ↑
acc ↑
acc ↑
0-shot
Mistral-7b
0.35
6.15
0.24
14.56
0.22
9.92
0.09
17.49
0.01
0.04
0.0
0.01
Pixtral-12b
0.85
2.39
0.54
5.34
0.29
7.34
0.34
7.05
0.21
0.10
0.09
0.07
Llama-8b
0.81
2.34
0.35
8.52
0.33
6.86
0.36
6.55
0.05
0.11
0.02
0.02
Appendix
Table 10: Navigation task results. We compare 0- and 1-shot prompting on base text-only LLMs, similarly sized VLMs (represented with light blue backgrounds), and a larger LLM variant of the overall best performing model, as well as a fully fine-tuned LM. We report the accuracies out of 1 for the final coordinates and their distance from the true coordinates for the Follower setting and the accuracy out of 1 of the generated instruction for the Instructor setting. The best results for each experiment are bolded while the best in each section are underlined .
Model Name
Sequence Length
1
2
3
4
Qwen-7b
1.0
0.24
0.08
0.08
Qwen-7b-v
0.92
0.2
0.04
0
Qwen-72b
1.0
0.56
0.28
0.12
Chat-GPT
1.0
0.72
0.68
0.44
Appendix
Table 11: 2D Egocentric Follower Navigation Performance Deterioration Based on Sequence Length
+Y
+X
−Y
−X
Mistral-7b
36.90
34.98
29.13
32.26
Llama-8b
51.33
40.23
43.80
37.84
Qwen-7b
77.27
52.10
38.26
44.32
Appendix
Table 12: Average Spatial Overlaps Os in the Egocentric Object Localization task broken down by Viewer Heading
+Y Heading
Facing Reference
adj
rand
adj
rand
0-shot
32.90
38.67
29.67
38.12
1-shot
46.50
39.83
32.32
41.75
Appendix
Table 13: Adjacent vs Random Block placements in the Allocentric Object Localization Task for Mistral-7b
Egocentric
Allocentric
Base
Distractors
+Y heading
Ref. heading
adj. ↑
rand. ↑
adj. ↑
rand. ↑
adj. ↑
adj. ↑
0-shot
Mistral-7b
30.87
34.67
30.20
31.90
32.90
29.67
Pixtral-12b
35.22
42.54
31.42
46.73
40.73
34.43
Llama-8b
36.52
38.78
37.08
33.10
37.11
35.52
LlamaVL-11b
35.33
41.50
33.08
36.55
41.08
39.72
Appendix
Table 14: Object Localization spatial overlap, Os , scores across the Egocentric and Allocentric settings, as well various headings and distractor settings. VLM counterparts are in light blue backgrounds, best results overall are bolded and best results in generation setting are underlined .
Set
Text
Os↑
Oc↑
Of↑
Os↑
Oc↑
Of↑
1-shot
Mistral-7b
26.14
54.12
15.18
34.73
57.10
24.22
Llama-8b
59.98
63.58
18.69
60.26
67.05
22.16
Qwen-7b
54.70
74.86
20.85
57.18
71.78
19.14
Few-shot
Mistral-7b
38.44
71.95
11.75
49.75
72.60
26.33
Llama-8b
59.70
85.83
30.19
62.04
86.69
26.10
Appendix
Table 15: 1-shot vs Few-shot Structure Composition: Comparison over 1-shot performance with reasoning vs Few-Shot without
When faced with complex spatial problems, humans naturally sketch layouts to organize their thinking, and the act of drawing further sharpens their understanding. In this work, we ask whether a similar principle holds for Large Language Models (LLMs): can learning to construct explicit visual layouts from spatial descriptions instill genuine spatial understanding? We introduce Text2Space, a dataset that pairs natural language descriptions with ground-truth ASCII grid layouts and spatial QA pairs, enabling us to separate failures in constructing spatial representations from failures in reasoning over them. We adopt ASCII because it is human-readable, operates entirely within the token space of language models, and encodes spatial relations in a structurally verifiable form. Our evaluation reveals a pronounced "Read-Write Asymmetry": LLMs interpret ASCII representations effectively but struggle to produce them from text, and these construction errors propagate to incorrect answers downstream. To address this limitation, we train models on layout construction (Text→ASCII) and find that it significantly improves spatial reasoning from text alone, even without producing any ASCII at inference time. Combining construction with comprehension training further amplifies these gains. Crucially, these improvements transfer to three external spatial reasoning benchmarks, demonstrating that, much as sketching sharpens human spatial thinking, learning to construct explicit layouts instills spatial understanding that generalizes beyond the training format.
Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language tasks, but continue to struggle with spatial reasoning. Existing spatial MLLMs rely on large-scale datasets, explicit 3D inputs, architecture-specific modifications, or sparse Reinforcement Learning (RL) methods that provide insufficient guidance for spatially-grounded reasoning. We introduce SpatialThinker. To our knowledge, it is the first MLLM unifying Scene Graph Generation (SGG) and visual reasoning in a single pass via online RL. The model simulates human-like spatial perception by constructing a mental scene graph of task-relevant objects and relations, and reasoning toward an answer via dense spatial rewards. Our contributions are threefold: (1) SGG-grounded reasoning: integrating SGG directly within the reasoning chain rather than as a disjoint preprocessing step; (2) STVQA-7K: a high-quality spatial VQA training dataset via a scalable synthesis pipeline; and (3) a dense spatial reward design that enforces structured grounding during RL and generalizes to improve broad visual perception. SpatialThinker-7B achieves 3.6× larger gains over SFT and 1.7× better in- and out-of-distribution generalization than sparse RL. Trained on only 7K samples, SpatialThinker-7B matches GPT-5 and outperforms GPT-4o, while SpatialThinker-30B surpasses both GPT-5 and Claude 4 Sonnet on average across 14 spatial and real-world benchmarks, demonstrating that structured spatial grounding with reward-aligned reasoning enables robust spatial understanding with limited data.
Hunar Batra, Haoqin Tu, Hardy Chen +3
University of Oxford · University of California, Santa Cruz
3D spatial reasoning underpins understanding and acting in the physical world, yet it remains unreliable in current multimodal large language models (MLLMs). These models falter at precise geometric measurement, at transforming between egocentric and allocentric viewpoints, and at grounding fine-grained appearance. The most common remedies fine-tune the model on large-scale curated spatial-reasoning datasets or attach dedicated encoders for 3D geometry, which typically couples the solution to costly supervision and a specific backbone. We instead introduce GraFT, a training-free framework that supplies the missing 3D structure through a compact, easily maintained 3D scene graph (3DSG). GraFT supplies the graph's 3D structure to the frozen model in three forms: as values computed by symbolic tools, as a bird's-eye-view (BEV) the model reads as an image, and as egocentric frames the geometry selects. On ScanQA, GraFT improves every metric over the same-backbone baseline, raising CIDEr by 27%. On VSI-Bench, GraFT improves frozen MLLMs by up to 65%, surpassing every proprietary and general-purpose open-source baseline, and several prominent fine-tuned spatial models.