ESPIRE: A Diagnostic Benchmark for Embodied Spatial Reasoning of Vision-Language Models

Yanpeng Zhao; Wentao Ding; Hongtao Li; Baoxiong Jia; Zilong Zheng

cs.CVMar 13, 2026

ESPIRE: A Diagnostic Benchmark for Embodied Spatial Reasoning of Vision-Language Models

Authors: Yanpeng Zhao, Wentao Ding, Hongtao Li, Baoxiong Jia, Zilong Zheng

Organizations: State Key Laboratory of General Artificial Intelligence, BIGAI

Abstract

A recent trend in vision-language models (VLMs) has been to enhance their spatial cognition for embodied domains. Despite progress, existing evaluations have been limited both in paradigm and in coverage, hindering rapid, iterative model development. To address these limitations, we propose ESPIRE, a diagnostic benchmark for embodied spatial reasoning. ESPIRE offers a simulated world that physically grounds VLMs and evaluates them on spatial-reasoning-centric robotic tasks, thus narrowing the gap between evaluation and real-world deployment. To adapt VLMs to robotic tasks, we decompose each task into localization and execution, and frame both as generative problems, in stark contrast to predominant discriminative evaluations (e.g., via visual-question answering) that rely on distractors and discard execution. This decomposition further enables a fine-grained analysis beyond passive spatial reasoning toward reasoning to act. We systematically design ESPIRE both at the instruction level and at the environment level, ensuring broad coverage of spatial reasoning scenarios. We use ESPIRE to diagnose a range of frontier VLMs and provide in-depth analysis of their spatial reasoning behaviors.

Figures & tables

Benchmark	Localization & Execution	Evaluation		Systematicity	Physically- Grounded	Diagnostic	Clutter Level
Benchmark	Localization & Execution	Paradigm	Tool-Free	Systematicity	Physically- Grounded	Diagnostic	Clutter Level
image- and video-based
Blink ( Fu et al., 2024 )	✗	VQA	✓	✗	✗	✗	high
CV-Bench ( Tong et al., 2024 )	✗	VQA	✓	✗	✗	✗	high
VSI-Bench ( Yang et al., 2024 )	✗	VQA	✓	✗	✗	✗	high
Where2Place ( Yuan et al., 2024 )	✗	Point Gen.	✓	✗	✗	✗	high

Table 1: Comparisons of spatial-reasoning benchmarks. ‘Text Gen.’ and ‘Point Gen.’ indicate that models produce answers in natural language and 2D points, respectively. ‘Fully Gen.’ denotes that models generate positions and rotations in 3D. ‘Tool-Free’ means no external tools are used, thus assessing the intrinsic spatial reasoning of VLMs.

Figure 1: Espire : a simulated physical world. Top: the spatial world of Espire covers key factors of spatial reasoning like spatial aspects (e.g., relationship and distance), reference frames, reference objects (§ 4.1 ). It features a tabletop scene for pick tasks and a shelf scene for place tasks (§ 4.2 ) and supports reasoning at varying granularities (see Table 10 in Appendix A.2 ). Bottom: example Espire tasks that all inherently rely on spatial reasoning.

Models	Pick			Place
Models	accuracy	acceptance	success	accuracy	acceptance	success
w/o reflection
Gemini2.5-Pro	57.72	63.93	34.06	50.61	28.36	5.68
InternVL3-78B	28.31	63.01	17.26	23.66	40.94	9.67
RoboBrain2.0-7B	57.72	18.81	10.87	50.70	15.68	8.64
Qwen3-VL-30B-A3B	54.43	62.56	32.15	45.54	43.47	20.00

Table 2: The localization accuracy (%), acceptance rate (%) in execution, and overall task success rate (%) across different VLMs.

Figure 2: Localization performance across spatial aspects and granularities on pick tasks.

Models	Accuracy
Models	attribute	distance	orientation	relationship
Pick
Gemini2.5-Pro	56.00	53.89	63.81	56.52
InternVL3-78B	29.33	21.67	33.65	30.14
RoboBrain2.0-7B	61.33	56.11	60.95	55.65
Qwen3-VL-30B-A3B	57.33	50.83	57.14	55.07

Table 3: Localization accuracy (%) across four primary spatial aspects.

Models	Accuracy (%)			Acceptance (%)
Models	easy	medium	hard	easy	medium	hard
Pick
Gemini2.5-Pro	60.78	60.98	52.04	70.96	60.98	60.71
InternVL3-78B	24.85	29.00	30.61	60.78	65.85	62.24
RoboBrain2.0-7B	62.57	56.10	55.10	21.56	15.72	19.39
Qwen3-VL-30B-A3B	62.57	53.39	48.87	65.87	66.67	55.87

Table 4: Performance across difficulty levels.

Models	#Localization	#Move	Dist. at success	Dist. before success	Rank
Pick
Gemini2.5-Pro	1.20	2.54	0.07	0.47	1
InternVL3-78B	1.05	2.56	0.05	0.48	3
RoboBrain2.0-7B	1.36	2.54	0.05	0.50	6
Qwen3-VL-30B-A3B	1.16	2.49	0.06	0.38	4
Qwen3-VL-8B	1.17	2.41	0.05	0.42	2

Table 5: The average number of attempts to succeed in localization and execution, and average distance (meter) between the target and end-effector upon execution success and before execution success. ‘Rank’ indicates model ranking in execution.

Panel of Table 6 — Table 6: Qualitative analysis. We categorize intermediate executions into following six types: T1 denotes a grasp-favorable viewpoint; T2 denotes a grasp-infeasible viewpoint; T3 denotes manipulator occlusion; T4 denotes object occlusion; T5 denotes unrecognizable target; and T6 denotes physically-achievable execution.

Pitch	Yaw	Roll	Pick (%)	Place (%)
Pitch	Yaw	Roll	Pick (%)	w/o C	w/ C
			52.73	37.74	43.33
✓			20.91	26.42	26.67
	✓		28.18	29.25	25.00
		✓	30.91	35.85	23.33
✓	✓		4.55	23.58	25.00

Table 7: Acceptance rate with rotation axes generated by Qwen3-VL-235B-A22B. Unchecked axes indicate that ground-truth rotations are used. For place tasks, results are reported on tasks with and without explicit pose constraints (C).

Model	Near Obj.	Distant Obj.	Table
Gemini2.5-Pro	-0.573 $\pm$ 0.634	0.8 $\pm$ 0.274	1.0 $\pm$ 0.0
RoboBrain2.0-7B	-0.674 $\pm$ 0.242	0.8 $\pm$ 0.274	1.0 $\pm$ 0.0
Qwen3-VL-30B-A3B	-0.100 $\pm$ 0.652	0.8 $\pm$ 0.274	1.0 $\pm$ 0.0
Qwen3-VL-8B	-0.674 $\pm$ 0.242	0.8 $\pm$ 0.274	1.0 $\pm$ 0.0
Qwen3-VL-235B-A22B	-0.573 $\pm$ 0.634	0.8 $\pm$ 0.274	1.0 $\pm$ 0.0

Table 8: Agreement of humans and models on reference frames. References are categorized into table, near, and distant objects.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Layouts of the tabletop and shelf scenes within Espire (best viewed in color). The light red region denotes the camera viewpoint sampling area, the light green region indicates where the robot end effector may appear, and the light blue region denotes where distant reference objects are placed. All labeled dimensions are in meters.

Models	Pick				Place
Models	localization (s)	execution (s)	# update	success (%)	localization (s)	execution (s)	# update	success (%)
w/o reflection
Gemini2.5-Pro	22.37	40.53	6.86	34.06	33.00	41.15	9.01	15.70
InternVL3-78B	14.44	32.21	6.92	17.26	19.06	34.04	7.59	9.67
RoboBrain2.0-7B	14.17	27.56	9.07	10.87	19.08	30.83	9.36	8.64
Qwen3-VL-8B	15.44	34.87	6.73	29.32	19.48	35.86	7.83	12.41

Appendix

Table 9: Breakdowns of evaluation time in seconds (s).

Panel of Figure 4 — Figure 4: Example prompts with Qwen3-VL (continued).

Panel of Figure 5 — Figure 5: Example prompts with Qwen3-VL.

Panel of Figure 6 — Figure 6: Pick localization example with reflection, where the prompt simplified for better demonstration.

Panel of Figure 7 — Figure 7: Pick localization example (subsequent attempts and successful execution).

Panel of Figure 8 — Figure 8: Place localization example without reflection, where the prompt simplified for better demonstration.

Panel of Figure 9 — Figure 9: Pick execution example without reflection, where the prompt simplified for better demonstration.

Panel of Figure 10 — Figure 10: Place execution example with reflection, where the prompt simplified for better demonstration.

Spatial Aspect	Gran- ularity	Type	Input	Example Instruction
Attribute	Coarse	Small	nil	take a small book from the table
		Medium	nil	take a medium-sized book
		Large	nil	take a large book
		Empty	nil	place the book in an empty slot
		NonEmpty	nil	place the book in a partly occupied slot
		Emptiest	nil	place the book in the emptiest slot

Appendix

Table 10: Spatial aspects of varying granularities. nil indicates no input parameters are required. Note that left , right , front , behind , above , and below are reloaded as directional relations in ‘Orientation’ ( cf. positional relations in ‘Relationship’).

$S$	$F$	$O$	$R$	$I$	Example Program
Attribute			Small		filterAttr $R (filterBook(TABLE))
			Large
			Medium
			Height	float	filterAttr $R ($ I$ , filterBook(TABLE))
			Width	float	filterAttr $R ($ I$ , filterBook(TABLE))
Distance		viewer distant obj. near obj.	RankClosest	int	filterDist $R ($ I $, filterBook(TABLE),$ O$ )

Appendix

Table 11: Example instruction families of pick tasks. The outermost pick (

\cdot

) is discarded for simplicity. unique (

\cdot

) ensures a unique item from the input set. TABLE returns all items in the tabletop scene.

$S$	$F$	$O$	$R$	$I$	Example Program
Attribute			Small		filterAttr $R (filterSlot(SHELF))
			Large
			Medium
			Height	float	filterAttr $R ($ I$ , filterSlot(SHELF))
			Width	float	filterAttr $R ($ I$ , filterSlot(SHELF))
Distance		viewer distant obj. near obj.	RankClosest	int	filterDist $R ($ I $, filterSlot(SHELF),$ O$ )

Appendix

Table 12: Example instruction families of place tasks. The outermost place (

\cdot

) is discarded for simplicity. unique (

\cdot

) ensures a unique item from the input set. SHELF returns all shelf-scene items.

Panel of Table 13 — Table 13: Tables with different colors and textures.

Panel of Table 14 — Table 14: Shelf with different layouts.

Panel of Table 15 — Table 15: Shelf with different textures.

Panel of Table 16 — Table 16: Near reference objects (w/ and w/o an intrinsic frame of reference) appear on the table or shelf.

Panel of Table 17 — Table 17: Distant reference objects (w/ and w/o an intrinsic frame of reference) appear behind the table or besides the shelf.

Panel of Table 18 — Table 18: Manipulable books. They will be auto-scaled to match three pre-defined sizes: small, medium, and large.

Panel of Table 19 — Table 19: Different bookends are used to help create tilting poses of books.

Name	Type	Oriented	L (cm)	W (cm)	H (cm)
Alarm clock	near	✓	7	13	17
Armento Rider	near	✓	24	7	24
Bicycle sculpture	near	✓	21	8	18
Picture frame	near	✓	13	22	18
Teddy bear	near	✓	20	23	25
Newton’s cradle	near	✗	10	15	14

Appendix

Table 20: Attributes of Espire assets. Oriented assets have an intrinsic frame while non-oriented assets do not.

Embodied Spatial Reasoning

See all 20 papers

SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models
Feb 24, 2026Yuechen Xie, Xiaoyan Zhang, Yicheng Shan +6VLM Reasoning Spatial Reasoning Benchmarks
Zhejiang University · The University of Sydney · ManyCore +2
SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models
Aug 3, 2026Jing Wu, Jianhua Wu, Jiayi Guan +5Vision-Language Models Visual Spatial Reasoning
Xiaomi EV, Beijing, China · College of Automotive and Energy Engineering, Tongji University, Shanghai, China · Independent Researcher
MultihopSpatial: Multi-hop Compositional Spatial Reasoning Benchmark for Vision-Language Model
Mar 19, 2026Youngwan Lee, Soojin Jang, Yoorhim Cho +3Spatial Reasoning Benchmarks Spatial Reasoning
Electronics and Telecommunications Research Institute, South Korea · Korea Advanced Institute of Science and Technology, South Korea · Sungkyunkwan University, South Korea +1

Feb 24, 2026J/K move · Enter open · S save

SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models

Yuechen Xie, Xiaoyan Zhang, Yicheng Shan, Hao Zhu+5

Zhejiang University · The University of Sydney · ManyCore+2

VLM Reasoning Spatial Reasoning Benchmarks Visual Spatial Reasoning

Open paper AskPDF arXiv

Vision-Language Models (VLMs) have been increasingly applied in real-world scenarios due to their outstanding understanding and reasoning capabilities. Although VLMs have already demonstrated impressive capabilities in common visual question answering and logical reasoning, they still lack the ability to make reasonable decisions in complex real-world environments. We define this ability as spatial logical reasoning, which not only requires understanding the spatial relationships among objects in complex scenes, but also the logical dependencies between steps in multi-step tasks. To bridge this gap, we introduce Spatial Logical Question Answering (SpatiaLQA), a benchmark designed to evaluate the spatial logical reasoning capabilities of VLMs. SpatiaLQA consists of 9,605 question answer pairs derived from 241 real-world indoor scenes. We conduct extensive experiments on 41 mainstream VLMs, and the results show that even the most advanced models still struggle with spatial logical reasoning. To address this issue, we propose a method called recursive scene graph assisted reasoning, which leverages visual foundation models to progressively decompose complex scenes into task-relevant scene graphs, thereby enhancing the spatial logical reasoning ability of VLMs, outperforming all previous methods. Code and dataset are available at https://github.com/xieyc99/SpatiaLQA.

Abstract

Embodied Spatial Reasoning

Explore similar work

SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models

SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models

MultihopSpatial: Multi-hop Compositional Spatial Reasoning Benchmark for Vision-Language Model