Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structures

Risham Sidhu; Julia Hockenmaier

cs.CLMar 18, 2026

Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structures

Authors: Risham Sidhu, Julia Hockenmaier

Organizations: University of Illinois at Urbana-Champaign

Abstract

We introduce GSU, a text-only grid dataset to evaluate the spatial reasoning capabilities of LLMs over 3 core tasks: navigation, object localization, and structure composition. By forgoing visual inputs, isolating spatial reasoning from perception, we show that while most models grasp basic grid concepts, they struggle with frames of reference relative to an embodied agent and identifying 3D shapes from coordinate lists. We also find that exposure to a visual modality does not provide a generalizable understanding of 3D space that VLMs are able to utilize for these tasks. Finally, we show that while the very latest frontier models can solve the provided tasks (though harder variants may still stump them), fully fine-tuning a small LM or LORA fine-tuning a small LLM show potential to match frontier model performance, suggesting an avenue for specialized embodied agents.

Figures & tables

Figure 1: The GSU Dataset tasks and settings visualized for clarity (note that the models do NOT receive these visuals and instead receive a textual version of the environment shown in Appendix E ). The grey arrows indicate the heading, i.e. direction that the embodied agent is facing. In the Navigation task, they remain aligned with +Y in the Cardinal setting and rotate to reflect the last direction of travel in the Egocentric setting, which affects the direction of the second step and the final coordinates. For the Object Localization task, we show how different headings affect the spatial relations between the target and the reference. For the Structure Composition task, we show the 3 structure categories that models may be asked to describe.

	Navigation					Object Localization		Structure Composition
	Follower		Instructor			Ego-	Allo-	$O_{s}$	$O_{c}$	$O_{f}$	$O_{n}$
	2D	3D	Card.	Ego.	Card2Ego	centric	centric	$O_{s}$	$O_{c}$	$O_{f}$	$O_{n}$
Mistral-7b	0.30	0.38	0.59	0.15	0.27	30.32	46.50	49.75	72.60	30.21	20.01
Pixtral	0.33	0.50	0.87	0.26	0.35	40.83	48.18	66.25	87.68	42.16	55.43
Llama-8b	0.33	0.47	0.69	0.06	0.09	46.17	50.18	62.04	86.69	22.70	37.75
LlamaVL-11b	0.32	0.47	0.74	0.14	0.28	48.40	48.08	64.48	86.58	44.44	44.05

Table 1: 3 GSU task results across 3 model categories (prompted LLMs and VLMs , prompted frontier models, and fine-tuned models ) with best scores from each bolded. Navigation results are accuracies: final coordinate accuracies for the Egocentric Follower and instruction sequence accuracy for the 2D Instructor , while OL and SC results are term overlaps: spatial

O_{s}

, color

O_{c}

, shape/form

O_{f}

, and numerical

O_{n}

. Other settings in Appendix D .

Figure 2: Allocentric front/back flipping

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: GSU Dataset Tasks and Settings

	Egocentric		Cardinal
	2D	3D	2D	3D
Left	30.4%	17.69%	21.33%	16.67%
Right	29.6%	18.77%	34.67%	14.58%
Front	20.8%	13.72%	25.33%	12.5%
Back	19.2%	13.72%	18.67%	13.54%
Up	0	20.94%	0	29.17%

Appendix

Table 2: Navigation Direction Distribution (reported out of 250 possible steps)

	Egocentric	Allocentric
Left	18.88%	18.69%
Right	18.37%	13.64%
Forward	16.84%	17.17%
Backward	14.80%	6.57%
Up	20.41%	18.37%
Down	14.29%	13.64%

Appendix

Table 3: Distribution of Ground Truth Spatial Direction in the Egocentric and Allocentric Object Localization settings

		Plain			Dict			Set
		$O_{s}\uparrow$	$O_{c}\uparrow$	$O_{f}\uparrow$	$O_{s}\uparrow$	$O_{c}\uparrow$	$O_{f}\uparrow$	$O_{s}\uparrow$	$O_{c}\uparrow$	$O_{f}\uparrow$
0-shot	Mistral-7b	15.92	42.64	12.45	13.72	47.93	10.87	15.88	46.70	13.83
	Pixtral	37.67	74.36	14.64	44.67	70.13	13.62	32.08	72.91	7.25
	Llama-8b	21.29	63.90	13.00	22.78	63.91	15.56	25.47	68.83	18.65
	LlamaVL-11b	35.83	69.07	11.83	24.57	60.48	12.09	20.94	62.14	13.37
	Qwen-7b	32.17	68.72	11.48	40.03	68.32	16.73	28.52	62.76	16.65

Appendix

Table 4: Structure Composition results for Plain, Dict, and Set Representations

		Percentage
Spatial Relations	Above	19.67%
	Below	21.2%
	Front	19.67%
	Back	10.61%
	Left	12.12%
	Right	16.67%

Appendix

Table 5: Distribution of Spatial Relations, Colors, and Shapes in the Structure Composition task

	Follower				Instructor
	Egocentric 2D		Egocentric 3D		Cardinal		Egocentric		Card2Ego
	acc $\uparrow$	dist. $\downarrow$	acc $\uparrow$	dist. $\downarrow$	acc $\uparrow$	dist. $\downarrow$	acc $\uparrow$	dist. $\downarrow$	acc $\uparrow$	dist. $\downarrow$
Flan-T5 w/o finetune	0	34.64	0	28.28	0	8.79	0	7.82	0	8.23
Flan-T5 w/ finetune	0.3	4.21	0.71	1.68	0.99	0.02	0.9	0.42	1.0	0
BART w/o finetune	0	34.64	0	28.28	0	8.22	0	7.44	0	8.21
BART w/ finetune	0.24	2.37	0.09	7.82	0.26	2.72	0	6.29	0.85	1.37

Appendix

Table 6: BART-base and Flan-T5-large Navigation Ablations

	Object Localization		Structure Composition
	Egocentric	Allocentric	$O_{s}$	$O_{c}$	$O_{f}$	$O_{n}$
Flan-T5 w/o finetune	16.5	16.67	67	7.11	16.6	8.9
Flan-T5 w/ finetune	19.5	94.5	99	96.17	88.83	97.07

Appendix

Table 7: BART-base and Flan-T5-Large Ablations over Object Localization and Structure Composition tasks

	Follower		Instructor
	Egocentric 2D	Egocentric 3D	Cardinal	Egocentric	Card2Ego
Mistral-7b	6.85	6.14	2.84	11.87	11.23
Pixtral	6.31	5.60	0.75	7.65	9.01
Llama-8b	7.03	4.85	1.95	11.87	12.30
LlamaVL-11b	7.36	5.78	1.52	11.96	9.59
Qwen-7b	22.54	20.44	1.57	8.96	7.52

Appendix

Table 8: The distance off scores for each of the Navigation experiments in Table 1 .

		Cardinal 3D		Egocentric 3D
		acc	dist	acc	dist
1	Mistral-7b	0.48	4.14	0.38	6.14
	Llama-8b	0.62	2.65	0.47	4.85
	Qwen-7b	0.63	7.79	0.19	20.44
Few	Mistral-7b	0.19	6.23	0.17	8.32
	Llama-8b	0.32	4.84	0.20	7.69

Appendix

Table 9: 1-shot vs Few-shot Navigation: Comparison over 1-shot performance with reasoning vs Few-Shot without reasoning examples

		Follower								Instructor
		Cardinal				Egocentric				Cardinal		Egocentric
		2d		3d		2d		3d		2d	3d	2d	3d
		acc. $\uparrow$	dist $\downarrow$	acc $\uparrow$	dist $\downarrow$	acc $\uparrow$	dist $\downarrow$	acc $\uparrow$	dist $\downarrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$
0-shot	Mistral-7b	0.35	6.15	0.24	14.56	0.22	9.92	0.09	17.49	0.01	0.04	0.0	0.01
	Pixtral-12b	0.85	2.39	0.54	5.34	0.29	7.34	0.34	7.05	0.21	0.10	0.09	0.07
	Llama-8b	0.81	2.34	0.35	8.52	0.33	6.86	0.36	6.55	0.05	0.11	0.02	0.02

Appendix

Table 10: Navigation task results. We compare 0- and 1-shot prompting on base text-only LLMs, similarly sized VLMs (represented with light blue backgrounds), and a larger LLM variant of the overall best performing model, as well as a fully fine-tuned LM. We report the accuracies out of 1 for the final coordinates and their distance from the true coordinates for the Follower setting and the accuracy out of 1 of the generated instruction for the Instructor setting. The best results for each experiment are bolded while the best in each section are underlined .

Model Name	Sequence Length
	1	2	3	4
Qwen-7b	1.0	0.24	0.08	0.08
Qwen-7b-v	0.92	0.2	0.04	0
Qwen-72b	1.0	0.56	0.28	0.12
Chat-GPT	1.0	0.72	0.68	0.44

Appendix

Table 11: 2D Egocentric Follower Navigation Performance Deterioration Based on Sequence Length

	$+Y$	$+X$	$-Y$	$-X$
Mistral-7b	36.90	34.98	29.13	32.26
Llama-8b	51.33	40.23	43.80	37.84
Qwen-7b	77.27	52.10	38.26	44.32

Appendix

Table 12: Average Spatial Overlaps

O_{s}

in the Egocentric Object Localization task broken down by Viewer Heading

	$+Y$ Heading		Facing Reference
	adj	rand	adj	rand
0-shot	32.90	38.67	29.67	38.12
1-shot	46.50	39.83	32.32	41.75

Appendix

Table 13: Adjacent vs Random Block placements in the Allocentric Object Localization Task for Mistral-7b

		Egocentric				Allocentric
		Base		Distractors		+Y heading	Ref. heading
		adj. $\uparrow$	rand. $\uparrow$	adj. $\uparrow$	rand. $\uparrow$	adj. $\uparrow$	adj. $\uparrow$
0-shot	Mistral-7b	30.87	34.67	30.20	31.90	32.90	29.67
	Pixtral-12b	35.22	42.54	31.42	46.73	40.73	34.43
	Llama-8b	36.52	38.78	37.08	33.10	37.11	35.52
	LlamaVL-11b	35.33	41.50	33.08	36.55	41.08	39.72

Appendix

Table 14: Object Localization spatial overlap,

O_{s}

, scores across the Egocentric and Allocentric settings, as well various headings and distractor settings. VLM counterparts are in light blue backgrounds, best results overall are bolded and best results in generation setting are underlined .

		Set			Text
		$O_{s}\uparrow$	$O_{c}\uparrow$	$O_{f}\uparrow$	$O_{s}\uparrow$	$O_{c}\uparrow$	$O_{f}\uparrow$
1-shot	Mistral-7b	26.14	54.12	15.18	34.73	57.10	24.22
	Llama-8b	59.98	63.58	18.69	60.26	67.05	22.16
	Qwen-7b	54.70	74.86	20.85	57.18	71.78	19.14
Few-shot	Mistral-7b	38.44	71.95	11.75	49.75	72.60	26.33
	Llama-8b	59.70	85.83	30.19	62.04	86.69	26.10

Appendix

Table 15: 1-shot vs Few-shot Structure Composition: Comparison over 1-shot performance with reasoning vs Few-Shot without

Spatial Reasoning in Language Models

See all 50 papers

Learning to Draw ASCII Improves Spatial Reasoning in Language Models
Apr 16, 2026Shiyuan Huang, Li Liu, Jincheng He +1Representation Learning Spatial Reasoning
University of California, Santa Cruz
SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards
Nov 10, 2025Hunar Batra, Haoqin Tu, Hardy Chen +3Multimodal Understanding Scene Graph Generation
University of Oxford · University of California, Santa Cruz
S-Agent 2.0: Spatial Reasoning with a General Coding Agent
Oct 20, 2026Rui Gong, Ruoyu Wang, Yalun Dai +9Spatial Reasoning in Language Models 3D Spatial Reasoning
Nanyang Technological University · University of Hong Kong

Apr 16, 2026J/K move · Enter open · S save

Learning to Draw ASCII Improves Spatial Reasoning in Language Models

Shiyuan Huang, Li Liu, Jincheng He, Leilani H. Gilpin

University of California, Santa Cruz

Representation Learning Spatial Reasoning Spatial Reasoning in Language Models

Open paper AskPDF arXiv

When faced with complex spatial problems, humans naturally sketch layouts to organize their thinking, and the act of drawing further sharpens their understanding. In this work, we ask whether a similar principle holds for Large Language Models (LLMs): can learning to construct explicit visual layouts from spatial descriptions instill genuine spatial understanding? We introduce Text2Space, a dataset that pairs natural language descriptions with ground-truth ASCII grid layouts and spatial QA pairs, enabling us to separate failures in constructing spatial representations from failures in reasoning over them. We adopt ASCII because it is human-readable, operates entirely within the token space of language models, and encodes spatial relations in a structurally verifiable form. Our evaluation reveals a pronounced "Read-Write Asymmetry": LLMs interpret ASCII representations effectively but struggle to produce them from text, and these construction errors propagate to incorrect answers downstream. To address this limitation, we train models on layout construction (Text

\rightarrow

ASCII) and find that it significantly improves spatial reasoning from text alone, even without producing any ASCII at inference time. Combining construction with comprehension training further amplifies these gains. Crucially, these improvements transfer to three external spatial reasoning benchmarks, demonstrating that, much as sketching sharpens human spatial thinking, learning to construct explicit layouts instills spatial understanding that generalizes beyond the training format.

Abstract

Spatial Reasoning in Language Models

Explore similar work

Learning to Draw ASCII Improves Spatial Reasoning in Language Models

SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards

S-Agent 2.0: Spatial Reasoning with a General Coding Agent