Humans in Space: Can MLLMs Construct Human-Centric Spatial Understanding via Active Exocentric Exploration?

Tianchen Guo; Tengyue Zhang; Chen Liu; Ling Chen; Xin Yu

cs.CVOct 25, 2026

Humans in Space: Can MLLMs Construct Human-Centric Spatial Understanding via Active Exocentric Exploration?

Authors: Tianchen Guo, Tengyue Zhang, Chen Liu, Ling Chen, Xin Yu

Organizations: The University of Queensland · Follow Me AI Pty LTD · Australian Institute for Machine Learning, Adelaide University · University of Technology Sydney

Abstract

While Multimodal Large Language Models (MLLMs) excel at zero-shot reasoning, their spatial capabilities are primarily evaluated through static, passive observation. In complex, human-centric scenes, occlusion often hides crucial geometric evidence, and passive benchmarks force models to guess rather than recognize their own visual uncertainty. To bridge this gap, we introduce SATE-Bench, a novel interactive benchmark comprising 316 real-world scenes and 2,525 multi-turn QA pairs to evaluate visual self-awareness and active exocentric view-exploration. It combines a Passive multi-view control with an Active exploration setting, where models start from a severely occluded view and must iteratively query adjacent viewpoints until they gather enough evidence to fully ``sate'' the question, scored by a step-penalized efficiency reward. Evaluating state-of-the-art MLLMs reveals that they terminate exploration prematurely while reporting high confidence, and struggle with fine-grained 3D geometric tasks; matched-view controls further attribute the Active--Passive gap primarily to insufficient evidence acquisition. Moreover, fine-tuning with Group Relative Policy Optimization (GRPO) shows that rewarding view-seeking increases active exploration but lowers overall accuracy, as models over-explore and fail to fuse expanding multi-view contexts across turns. These findings indicate that scaling passive multi-view perception does not intrinsically yield active spatial reasoning, highlighting the need for models that know where to look, how to integrate what they see, and when to stop. Code and data are available at https://gtc-gh.github.io/SATE-Bench/

Figures & tables

Benchmark	Act.	Pas.	Real	MT	DEMV	SA	SO	HC
BLINK [ 11 ]	✗	✓	✓	✗	✗	✗	✗	✗
MuirBench [ 48 ]	✗	✓	✓	✗	✗	✗	✗	✗
MMSI-Bench [ 57 ]	✗	✓	✓	✗	✗	✗	✗	✗
AllAnglesBench [ 59 ]	✗	✓	✓	✗	✗	✗	✗	✗
MindCube [ 60 ]	✗	✓	✓	✗	✗	✗	✓	✗
CVBench [ 18 ]	✗	✓	✓	✗	✗	✗	✗	✓

Table 1: Comprehensive comparison of spatial reasoning and visual perception benchmarks. Abbreviations: Act. (Active), Pas. (Passive), Real (Real-world scenes), MT (Multi-Turn, supporting interaction trajectories of

>3

consecutive turns), DEMV (Dense Exocentric Multi-View), SA (Self-Awareness), SO (Severe Occlusion), HC (Human-Centric).

Figure 2: Overview of the SATE-Bench Curation and Evaluation Pipeline. Our framework is constructed and deployed through four distinct stages. Steps 1 & 2 (Data Collection and Quality Control): We source dense, 360-degree multi-view captures across five cognitive spatial categories. Candidate question-answer pairs undergo a rigorous human annotation and modification loop to guarantee accuracy, clarity, and appropriate difficulty. Step 3 (Active Exploration): The multi-turn interactive evaluation setting. The MLLM is initialized with a single view and must dynamically query additional exocentric perspectives until it confidently resolves its visual uncertainty. Step 4 (Passive Observation): The single-turn control setting, where the model is provided a static, comprehensive bag-of-frames to yield a direct prediction without the need for spatial view-exploration.

Question Name	# QA Pairs	Avail. Views	Passive Views
Fine-grained Action and Pose	301	5.23	2.22
Human Counting	308	6.41	3.27
Human Existence	203	7.57	3.33
Human-Object Interaction	165	6.79	2.75
Object Counting	232	5.17	3.46
Object Existence	218	4.06	3.71

Table 2: Dataset Statistics and View Distributions by Task. “Avail. Views” and “Passive Views” denote the average number of available views and selected passive views, respectively.

Text Length Statistics
Metric	Value
Average Question Length	102.89
Max Question Length	253.00
Average Option Length	49.33
Max Option Length	148.00
Average ‘Purely Text’ Option Length	60.67

Table 3: Dataset Text Length Statistics and Answer Distribution.

Model	HC	OC	FAP	HOI	HE	OE	RD-H	RD-O	VDO	VSD	Avg.	Avg.	Avg.	Active	Inf. Time
											Acc (%)	Steps	Reward	Rate (%)	(hours)
Baselines
Random Guess	18.2	22.8	19.5	15.6	50.5	54.0	22.1	16.7	21.3	21.9	24.8	1.0	22.5	0.0	0.0
Blind (Text-only)	0.4	6.3	20.5	3.3	46.8	53.2	21.5	21.7	20.0	31.2	20.9	2.1	19.1	21.5	1.9
Human	88.2	76.8	79.6	83.1	81.4	87.6	78.9	87.1	80.7	84.3	82.7	4.7	55.6	88.1	$>$ 10.0
Oracle-View ∗	42.2	26.3	67.9	17.1	60.4	57.4	73.5	59.7	25.8	33.3	54.7	-	-	-	$<$ 1.0

Table 4: Comprehensive Evaluation Results on SATE-Bench across both Active and Passive settings. Task abbreviations: HC (Human Counting), OC (Object Counting), FAP (Fine-grained Action & Pose), HOI (Human-Object Interaction), HE (Human Existence), OE (Object Existence), RD-H (Relative Distance to Human), RD-O (Relative Distance to Object), VDO (View-Specific Depth Ordering), VSD (View-Specific Direction). In the Passive setting, metrics requiring multi-turn exploration (Rewards, Steps, Active Rate) are marked with “

-

”. For enhanced readability, the reported Average Reward is scaled by

100

. ∗ Evaluated based on Gemini-3.1-flash-lite. Best results among models are bolded. More open-source model results can be found in the Appendix C.8 .

Model	Avg. Acc (%)	Model	Avg. Acc (%)
Gemini-3.5-flash	37.9	GPT-5.2-2025-12-11	37.3
Gemini-3.1-flash-lite	37.9	GPT-5-mini-2025-08-07	39.6
Gemini-2.5-flash	37.3	GPT-5-nano-2025-08-07	35.0

Table 5: Single View Evaluation. In this setting, models are provided strictly with the initial view from the Active set and are evaluated passively (i.e., no spatial exploration is permitted).

Figure 3: Impact of the action reward weight (

r_{3}

) during GRPO fine-tuning on Qwen2.5-VL-3B-Instruct. Increasing the action reward successfully incentivizes proactive behavior, evidenced by a sharp rise in (b) average steps and (d) active exploration rate. However, this aggressive exploration exacerbates contextual interference, leading to a steady decline in (a) active accuracy and (c) average reward. In contrast, the model’s passive accuracy remains highly stable across all reward configurations.

Standard Active ( $\pm 1$ moves)
Model	Avg.	Avg.	Avg.	Active	Inf. Time
	Acc (%)	Steps	Reward	Rate (%)	(hours)
Gemini-3.1-flash-lite	38.0	1.5	31.5	24.8	1.5
Gemini-2.5-flash	45.4	1.9	35.3	41.2	8.3
Expanded Active ( $\pm 2$ moves)
Gemini-3.1-flash-lite	37.9	1.4	33.1	21.0	1.1

Table 6: Evaluation on Expanded Viewpoint Selection. We compare the standard Active setting against an expanded setting where models are provided with broader selection options at each step (allowing shifts of either

\pm 1

\pm 2

views). The total available views in the 360-degree environment remain consistent (8 views).

Model	Setting	Avg. Acc (%)	Avg. Steps	Avg. Reward	Active Rate (%)
Gemini-3.1-lite	Active	38.0	1.5	31.5	24.8
Gemini-3.1-lite	Passive	40.3	-	-	-
GPT-5-mini	Active	41.4	1.4	34.0	29.4
GPT-5-mini	Passive	41.9	-	-	-

Table 7: Performance comparison between Active and Passive settings. Note that “Passive” refers to the Matched-View Passive setting. For brevity, model names are abbreviated: “Gemini-3.1-lite” refers to Gemini-3.1-flash-lite, and “GPT-5-mini” refers to GPT-5-mini-2025-08-07.

Model	Avg.	Avg.	Avg.	Active	Last 3rd	Penult.	Final
	Acc (%)	Steps	Reward	Rate (%)	Conf (%)	Conf (%)	Conf (%)
Gemini-3.1-lite	37.3	1.4	33.0	16.2	82.1	85.2	95.1
GPT-5-mini	41.4	1.4	36.4	26.3	73.2	78.5	84.7

Table 8: Analysis of exploration behavior and terminal confidence, from a separate run with the confidence-elicitation prompt. Model names are abbreviated for brevity: “Gemini-3.1-lite” refers to Gemini-3.1-flash-lite, and “GPT-5-mini” refers to GPT-5-mini-2025-08-07.

Figure 4: Taxonomy and qualitative examples of SATE-Bench tasks. Our benchmark evaluates multi-view spatial reasoning across five core cognitive dimensions: Counting, Existence, Pose and Physical Interaction, Global Relative Proximity, and View-Specific Spatial Geometry. For each sub-task, we illustrate a representative spatial query alongside a subset of the available exocentric observations required to resolve the scene’s geometric ambiguity.

Figure 5: The interactive evaluation interface for the human performance baseline.

Dataset	License Type	Publicly Available	Access Requirement	Released Artifacts	Re-identification
M3GYM	Non-Commercial Academic	Yes	Requires Signed Agreement	Extraction Scripts	No
Ego-Exo4D	Ego4D License Agreement	Yes	Requires Signed Agreement	Extraction Scripts	No
4D-OR	MIT	Yes	Requires Signed Agreement	Extraction Scripts	No
MM-OR	Apache-2.0	Yes	Requires Signed Agreement	Extraction Scripts	No
Harmony4D	MIT	Yes	Direct Download	Multi-View Frames	No
HOI-M3	Apache-2.0	Yes	Direct Download	Multi-View Frames	No

Table 9: Overview of source datasets categorized by licensing and access requirements. To ensure compliance with original distribution terms, raw imagery is withheld for datasets requiring signed agreements.

		No move		After moving
Model	Acc.	T1-Ans.	T1-Abs.	Part.	All	None
Gemini-3.5-flash	56.8	19.0	2.0	65.1	13.5	0.4
Gemini-2.5-flash	45.4	45.9	15.8	28.4	9.2	0.7
Gemini-3.1-flash-lite	38.0	50.6	25.1	17.0	6.9	0.3
Gemini-2.5-flash-lite	28.1	26.2	57.2	9.9	0.5	6.3
GPT-5.2	46.4	34.5	9.5	54.9	0.9	0.2

Table 10: Distribution of active failure cases (%). Acc.: Active accuracy; T1: answered (Ans.) or abstained (Abs.) at the start view; Part.: answered after a partial exploration; All: answered after observing all views; None: no valid answer.

Panel of Figure 6 — Figure 6: Active failure cases A1–A3: the model stops at the start view.

Panel of Figure 7 — Figure 7: Active failure cases A4–A5: errors after exploring.

Panel of Figure 8 — Figure 8: Passive failure cases P1–P2.

Panel of Figure 9 — Figure 9: Passive failure cases P3–P4.

Model	HC	OC	FAP	HOI	HE	OE	RD-H	RD-O	VDO	VSD	Avg.
											Acc (%)
Gemini-3.5-flash	25.8	16.5	39.0	15.6	65.8	50.0	47.4	42.8	29.3	31.2	37.9
Gemini-3.1-flash-lite	25.3	15.0	49.7	15.6	70.3	37.9	47.1	39.4	18.7	38.5	37.9
Gemini-2.5-flash	24.9	15.7	40.0	12.2	77.5	50.0	51.0	34.4	17.3	27.1	37.3
Gemini-2.5-flash-lite	19.1	15.0	17.4	13.3	69.4	37.1	31.7	31.7	26.7	37.5	28.9
GPT-5.2-2025-12-11	26.7	14.2	36.9	21.1	61.3	42.7	54.5	38.3	17.3	32.3	37.3

Table 11: Full Table for Single View Evaluation. In this setting, models are provided strictly with the initial view from the Active set and are evaluated passively (i.e., no spatial exploration is permitted).

Model	HC	OC	FAP	HOI	HE	OE	RD-H	RD-O	VDO	VSD	Avg.	Avg.	Avg.	Active	Inf. Time
											Acc (%)	Steps	Reward	Rate (%)	(hours)
Original Active Setting (Unrestricted 8-View Exploration)
Gemini-3.1-flash-lite	29.8	21.3	47.2	14.4	68.5	40.3	44.6	38.9	18.7	37.5	38.0	1.5	31.5	24.8	1.5
Gemini-2.5-flash	41.3	31.5	48.2	12.2	69.4	57.3	52.9	51.1	28.0	34.4	45.4	1.9	35.3	41.2	8.3
Ablated Active Setting (Restricted to Maximum $\pm 2$ Views)
Gemini-3.1-flash-lite	30.7	18.9	45.6	16.7	68.5	41.1	45.2	36.1	20.0	37.5	37.9	1.4	33.1	21.0	1.1

Table 12: Full Table for Expanded Viewpoint Selection. We compare the standard Active setting against an expanded setting where models are provided with broader selection options at each step (allowing shifts of either

\pm 1

\pm 2

views). The total available views in the 360-degree environment remain consistent (8 views). Task abbreviations follow the main table. Metrics are presented for Gemini-3.1-flash-lite and Gemini-2.5-flash to analyze the impact of increased spatial querying flexibility on exploration efficiency.

Model	Modality	HC	OC	FAP	HOI	HE	OE	RD-H	RD-O	VDO	VSD	Avg. Acc (%)
Gemini-3.1-flash-lite	Multi-Image	59.0	31.6	68.9	26.8	58.5	70.2	82.7	70.8	32.3	27.8	62.6
Gemini-3.1-flash-lite	Video	51.8	30.3	68.9	17.1	52.8	55.3	78.6	66.9	25.8	27.8	57.6
GPT-5-mini-2025-08-07	Multi-Image	61.4	42.1	65.1	34.1	47.2	67.0	85.7	77.3	25.8	36.1	64.6
GPT-5-mini-2025-08-07	Video	67.5	36.8	65.1	24.4	47.2	66.0	84.2	76.0	29.0	36.1	63.7

Table 13: Full Table for Comparison of model performance across Multi-Image and Video modalities.

Model	HC	OC	FAP	HOI	HE	OE	RD-H	RD-O	VDO	VSD	Avg. Acc (%)	Avg.	Avg.	Active
											(Final Step)	Steps	Reward	Rate (%)
Gemini-3.1-flash-lite	28.9	19.7	45.6	16.7	71.2	40.3	42.6	35.6	21.3	38.5	37.3	1.4	33.0	16.2
GPT-5-mini-2025-08-07	32.4	20.5	50.8	10.0	59.5	49.2	51.6	46.1	32.0	34.4	41.4	1.4	36.4	26.3

Table 14: Direct Measurement of Visual Self-Awareness Full Statistics.

Model	History	HC	OC	FAP	HOI	HE	OE	RD-H	RD-O	VDO	VSD	Avg.	Avg.	Avg.	Active
												Acc (%)	Steps	Reward	Rate (%)
Gemini-3.1-flash-lite	Yes	29.8	21.3	47.2	14.4	68.5	40.3	44.6	38.9	18.7	37.5	38.0	1.5	31.5	24.8
Gemini-3.1-flash-lite	No	21.3	7.9	48.2	12.2	69.4	40.3	47.1	38.9	20.0	34.4	36.2	2.0	32.1	23.3
GPT-5-mini-2025-08-07	Yes	34.2	23.6	50.3	10.0	59.5	50.8	49.7	46.7	28.0	34.4	41.4	1.4	34.0	29.4
GPT-5-mini-2025-08-07	No	26.7	29.9	53.8	12.2	64.0	47.6	49.7	43.9	26.7	33.3	41.0	1.7	35.7	25.7

Table 15: Impact of interaction history visibility on active exploration performance.

	Active Setting			Passive Setting
Model	Accuracy (%)	Margin (%)	95% CI	Accuracy (%)	Margin (%)	95% CI
Gemini-3.5-flash	56.76	$\pm$ 2.48	[54.28, 59.24]	69.20	$\pm$ 3.07	[66.13, 72.26]
Gemini-3.1-flash-lite	38.05	$\pm$ 2.43	[35.62, 40.47]	62.64	$\pm$ 3.21	[59.43, 65.86]
Gemini-2.5-flash	45.41	$\pm$ 2.49	[42.92, 47.90]	58.51	$\pm$ 3.27	[55.23, 61.78]
Gemini-2.5-flash-lite	28.08	$\pm$ 2.25	[25.83, 30.33]	45.06	$\pm$ 3.31	[41.75, 48.36]
GPT-5.2-2025-12-11	46.45	$\pm$ 2.50	[43.95, 48.94]	62.99	$\pm$ 3.21	[59.78, 66.20]

Table 16: 95% Confidence Intervals (Wald method) for overall accuracy across Active and Passive settings.

Model	HC	OC	FAP	HOI	HE	OE	RD-H	RD-O	VDO	VSD	Avg.	Avg.	Avg.	Active
											Acc (%)	Steps	Reward	Rate (%)
Active Setting (Multi-turn Spatial Exploration)
Qwen2.5-VL-3B-Instruct	17.8	7.5	34.9	14.3	61.3	41.9	36.2	36.7	23.0	21.9	30.4	1.0	27.5	0.2
Qwen2.5-VL-7B-Instruct	20.0	11.8	23.6	11.1	66.7	37.9	34.6	36.7	18.7	33.3	29.8	1.0	26.9	0.4
Qwen3-VL-8B-Instruct	9.3	9.4	27.2	11.1	58.6	37.1	44.6	44.4	20.0	31.2	30.7	1.6	27.1	15.6
Qwen3-VL-8B-Thinking	12.0	11.8	19.5	8.9	72.1	44.4	42.3	30.6	18.7	34.4	29.8	1.1	26.7	5.8

Table 17: Comprehensive Evaluation Results of Open-Source Vision-Language Models on SATE-Bench across Active (Multi-turn Spatial Exploration) and Passive (Single-turn Static Observation) settings. Task abbreviations: HC (Human Counting), OC (Object Counting), FAP (Fine-grained Action & Pose), HOI (Human-Object Interaction), HE (Human Existence), OE (Object Existence), RD-H (Relative Distance to Human), RD-O (Relative Distance to Object), VDO (View-Specific Depth Ordering), VSD (View-Specific Direction). In the Passive setting, multi-turn exploration metrics are marked with “

-

”. Best results among open-source models are bolded.

Model	Abstention Selection Rate (Active)	Accuracy on Abstention (Active)	Abstention Selection Rate (Passive)	Accuracy on Abstention (Passive)
Gemini-2.5-flash	14.57%	47.37%	2.24%	18.75%
Gemini-2.5-flash-lite	54.38%	92.11%	16.36%	75.00%
Gemini-3.1-flash-lite	22.25%	71.05%	2.10%	18.75%
Gemini-3.5-flash	2.50%	0.00%	0.98%	12.50%
GPT-5-mini-2025-08-07	5.62%	18.42%	0.56%	6.25%
GPT-5-nano-2025-08-07	12.41%	36.84%	0.70%	18.75%

Table 18: Selection rate and accuracy for the Abstention Option across Active and Passive settings.

Model	$\lambda=0.05$	$\lambda=0.1$	$\lambda=0.2$	$\lambda=0.5$
Gemini-3.5-flash	45.85	40.60	32.25	17.48
Gemini-3.1-flash-lite	33.67	31.45	27.56	19.04
Gemini-2.5-flash	38.69	35.31	29.75	18.97
Gemini-2.5-flash-lite	26.15	24.49	21.51	14.83
GPT-5.2-2025-12-11	39.56	36.50	31.15	19.80
GPT-5-mini-2025-08-07	36.24	33.95	29.88	20.69

Table 19: Sensitivity analysis of the step penalty parameter

\lambda

on average reward. The reported Average Reward is scaled by 100.

Metric	Value
Average Question Length	119.63
Max Question Length	344.00
Average Option Length	37.48
Max Option Length	185.00
Average ‘Purely Text’ Option Length	45.03
Average Correct Answer Length	33.74

Table 20: Text Length Statistics for Combined Training and Validation Sets (

N=19,370

Training & Optimization
Hyperparameter	Value
Base Model	Qwen2.5-VL-3B-Instruct
Learning Rate	$1\times 10^{-5}$
Train Batch Size (per device)	2
Gradient Accumulation Steps	16
Max Training Steps	300

Table 21: Key hyperparameters for GRPO fine-tuning.

Model	HC	OC	FAP	HOI	HE	OE	RD-H	RD-O	VDO	VSD	Avg.	Avg.	Avg.	Active
											Acc (%)	Steps	Reward	Rate (%)
Active Setting: Base Model vs. GRPO Tuned Models
Qwen2.5-VL-3B (Base)	17.8	7.5	34.9	14.3	61.3	41.9	36.2	36.7	23.0	21.9	30.4	1.0	27.5	0.2
Qwen2.5-VL-3B-Instruct-005	20.9	8.2	12.8	5.4	73.0	44.4	17.3	14.4	11.5	30.2	22.5	4.3	15.0	49.5
Qwen2.5-VL-3B-Instruct-02	18.7	7.5	11.8	7.1	73.0	46.0	17.6	15.6	13.8	24.0	22.2	4.7	14.6	55.1
Qwen2.5-VL-3B-Instruct-04	24.9	7.5	11.8	7.1	69.3	40.3	14.7	13.3	12.6	28.1	21.7	4.7	14.0	56.3

Table 22: Evaluation Results of GRPO Fine-Tuned Models on SATE-Bench across Active (Multi-turn Spatial Exploration) and Passive (Single-turn Static Observation) settings. Task abbreviations: HC (Human Counting), OC (Object Counting), FAP (Fine-grained Action & Pose), HOI (Human-Object Interaction), HE (Human Existence), OE (Object Existence), RD-H (Relative Distance to Human), RD-O (Relative Distance to Object), VDO (View-Specific Depth Ordering), VSD (View-Specific Direction). In the Passive setting, multi-turn metrics are marked with “

-

”. Best results among tuned models are bolded.

Panel of Figure 10 — Figure 10: Prompts for Passive Evaluation

Panel of Figure 11 — Figure 11: Prompts for Active Evaluation

Panel of Figure 12 — Figure 12: Prompts for Expanded View Selection Evaluation

Active
Model	Fallback Used	Usage (%)
Gemini-2.5-flash	0	0.00
Gemini-2.5-flash-lite	0	0.00
Gemini-3.1-flash-lite	0	0.00
Gemini-3.5-flash	0	0.00
GPT-5-mini-2025-08-07	0	0.00

Table 23: Usage of the LLM fallback for answer extraction.

Panel of Figure 13 — Figure 13: Prompts for Active Confidence Score Evaluation

Panel of Figure 14 — Figure 14: System Prompt for Active Exhaustive Search Evaluation

Spatial Reasoning Benchmarks

Spatial reasoning benchmarks test whether AI systems can perceive, represent, and act on geometric, relational, and viewpoint-dependent information.

See all 163 papers

SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks
Jun 8, 2026Hongcheng Gao, Hailong Qu, Jingyi Tang +18Spatial Reasoning Benchmarks Visual Spatial Reasoning
Tsinghua University · Chongqing University · Peking University +6
SpaceAct: Benchmarking 3D Spatial and Temporal Reasoning via Closed-Loop Action and Active Perception
Oct 28, 2026Chenglong Shi, Zekai Yuan, Yue Song +2Spatial Reasoning Benchmarks Spatiotemporal Reasoning
Tencent HY
OmniView-Space: Reinforcing Spatial Reasoning via Multi-Perspective Spatial Mapping
Jul 1, 2026Xudong Li, Mengdan Zhang, Peixian Chen +7Visual Spatial Reasoning 3D Spatial Reasoning
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Tencent Youtu Lab · Beijing Institute of Technology

Jun 8, 2026J/K move · Enter open · S save

SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks

Hongcheng Gao, Hailong Qu, Jingyi Tang, Jiahao Wang+17

Tsinghua University · Chongqing University · Peking University+6

Spatial Reasoning Benchmarks Visual Spatial Reasoning Long-Horizon Agent Evaluation AI Agent Benchmarks Multimodal Agents Spatial Reasoning

Open paper AskPDF arXiv

Spatial reasoning is a foundational capability for multimodal large language models (MLLMs) to perceive and operate within the physical world. However, existing benchmarks predominantly rely on passive evaluation (e.g., static VQA) or simulator-specific pipelines, failing to assess general interactive spatial understanding. We introduce SpatialWorld, a unified benchmark designed specifically for evaluating the interactive spatial understanding of multimodal agents in complex real-world tasks. Integrating eight heterogeneous simulation backends under a shared, simulator-agnostic protocol, SpatialWorld features 760 human-annotated tasks across diverse domains (e.g., household routines, travel, social collaboration). Agents must solve tasks under vision-only partial observability, actively gathering egocentric visual evidence and expressing decisions via a unified, text-based action interface native to MLLMs. For reliable evaluation, each task includes a human-validated initial state, a reference trajectory, and a terminal-state verifier. Evaluating 15 advanced agents reveals that robust spatial task solving remains challenging: the strongest model, GPT-5, achieves an average task success rate (TSR) of only 17.4%, while the leading open-source model, Qwen-3.5, reaches 14.1%. Further analysis exposes a clear mismatch between task success and execution efficiency, alongside substantial domain-specific performance variations. These bottlenecks in active exploration and long-horizon planning position SpatialWorld as a rigorous testbed for future spatial agents.

Abstract

Spatial Reasoning Benchmarks

Explore similar work

SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks

SpaceAct: Benchmarking 3D Spatial and Temporal Reasoning via Closed-Loop Action and Active Perception

OmniView-Space: Reinforcing Spatial Reasoning via Multi-Perspective Spatial Mapping