Unleashing Spatial Reasoning in Multimodal Large Language Models via Textual Representation Guided Reasoning

Jiacheng Hua; Yishu Yin; Yuhang Wu; Tai Wang; Yifei Huang; Miao Liu

cs.CVMar 24, 2026

Unleashing Spatial Reasoning in Multimodal Large Language Models via Textual Representation Guided Reasoning

Authors: Jiacheng Hua, Yishu Yin, Yuhang Wu, Tai Wang, Yifei Huang, Miao Liu

Organizations: College of AI, Tsinghua University, Beijing, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · The University of Tokyo, Tokyo, Japan

Abstract

Existing Multimodal Large Language Models (MLLMs) struggle with 3D spatial reasoning, as they fail to construct structured abstractions of the 3D environment depicted in video inputs. To bridge this gap, drawing inspiration from cognitive theories of allocentric spatial reasoning, we investigate how to enable MLLMs to model and reason over text-based spatial representations of video. Specifically, we introduce Textual Representation of Allocentric Context from Egocentric Video (TRACE), a prompting method that induces MLLMs to generate text-based representations of 3D environments as intermediate reasoning traces for more accurate spatial question answering. TRACE encodes meta-context, camera trajectories, and detailed object entities to support structured spatial reasoning over egocentric videos. Extensive experiments on VSI-Bench and OST-Bench demonstrate that TRACE yields notable and consistent improvements over prior prompting strategies across a diverse range of MLLM backbones, spanning different parameter scales and training schemas. We further present ablation studies to validate our design choices, along with detailed analyses that probe the bottlenecks of 3D spatial reasoning in MLLMs.

Figures & tables

Figure 1: Motivation for Textual Representation of Allocentric Context from Egocentric Video (TRACE) in video-based spatial reasoning. (a) An egocentric video paired with a query that requires holistic spatial reasoning. (b) A textual description that vividly captures the room layout needed to solve the example spatial question answering (QA). (c) TRACE encodes meta-context, camera trajectory, and entities, serving as an intermediate reasoning trace for spatial QA with MLLMs.

Figure 2: Illustration of our Textual Representation of Allocentric Context from Egocentric Video (TRACE). We construct TRACE by aligning a global coordinate system with the room layout and geometry, logging the camera trajectory across temporal steps, and registering visible objects with key attributes, estimated positions, and spatial relations. Here, we also show the key prompts used to guide MLLMs to generate this intermediate reasoning trace.

Methods	Avg.	Numerical Answer				Multiple-Choice Answer
		Obj. Cnt.	Abs. Dist.	Obj. Size	Room Size	Rel. Dist.	Rel. Dir.	Route	Order
Gemini 3 Pro as base model
Direct	52.61	33.77	32.57	67.09	42.99	62.54	50.52	51.03	70.71
CoT	53.65	30.35	34.54	64.05	40.76	61.78	58.09	61.34	71.96
ToT	58.88	44.55	42.12	72.20	45.55	65.35	57.83	55.62	73.73
LtM	59.52	45.19	40.72	73.36	44.15	65.82	60.40	53.59	73.64

Table 1: Evaluation results on the VSI benchmark. We report average performance and detailed breakdowns across numerical-answer and multiple-choice tasks, under proprietary and open-sourced base models. Best results are in bold , and second-best are underlined .

Methods	Avg.	Agent Visible Info					Agent-object Spatial Relationship						Agent State
		Exist.		Quant.	Divers.	Order	Direct.			Dist.			Pos.		Orient.
		JUD.	TEMP.	CNT.	JUD.	JUD.	JUD.	TEMP.	EST.	JUD.	TEMP.	EST.	JUD.	EST.	JUD.	EST.
Gemini 3 Pro as base model
Direct	59.22	96.72	84.87	68.75	89.66	82.54	54.27	48.15	28.63	60.61	54.55	30.00	71.43	22.78	72.60	22.68
CoT	59.26	82.24	93.99	65.96	96.55	77.78	52.24	62.96	27.84	52.76	50.00	31.64	71.43	24.26	72.60	26.67
ToT	59.20	94.54	83.55	66.67	93.10	80.65	54.27	54.39	31.00	55.61	53.85	32.36	61.76	29.07	72.60	24.58

Table 2: Evaluation results on the OST benchmark. Results are reported across agent state understanding, visible information reasoning, and agent–object spatial relationship tasks, under proprietary and open-sourced base models. Best results are in bold , and second-best are underlined .

Figure 3: Performance gains across models on VSI-Bench. TRACE yields consistent, state-of-the-art performance gains compared to Direct prompting baselines, across various model architectures and parameter scales.

Input Setting	Avg.	Numerical Answer				Multiple-Choice Answer
		Obj. Cnt.	Abs. Dist.	Obj. Size	Room Size	Rel. Dist.	Rel. Dir.	Route	Order
Proprietary model as base
Video Direct	52.61	33.77	32.57	67.09	42.99	62.54	50.52	51.03	70.71
One-Stage	60.15	47.55	38.82	73.90	45.62	63.85	61.70	58.01	72.97
Two-Stage	58.52	42.25	36.73	72.10	52.17	58.75	63.50	51.35	74.01
Text-Only	52.27	28.52	32.66	67.28	48.02	49.58	62.66	52.43	64.93

Table 3: Systematic studies of different prediction settings for utilizing our proposed text-based spatial representation. Among all settings, one-stage prompting yields the best performance on both Gemini-3 Pro and Qwen2.5-VL-72B.

Input Setting	Avg.	Numerical Answer				Multiple-Choice Answer
		Obj. Cnt.	Abs. Dist.	Obj. Size	Room Size	Rel. Dist.	Rel. Dir.	Route	Order
Cognitive Map	21.41	7.82	9.86	8.17	21.22	33.94	35.23	32.99	30.26
Spatial Caption	27.58	14.90	14.20	24.86	30.59	36.06	34.40	36.60	36.73
Ours	31.11	12.83	21.74	39.71	23.51	37.04	33.16	32.99	40.13
Ours w/o Trajectory	29.19	10.16	18.19	33.93	29.41	35.92	37.19	31.96	32.85
Ours w/o Entity Registry	25.87	6.11	28.84	7.41	19.69	41.69	37.50	30.41	33.50

Table 4: Comparison with existing text-based spatial representations and ablation studies of our method. We use Qwen2.5-VL-72B as base and adopt text-only inference for more direct comparison.

Figure 4: A visual illustration demonstrates that TRACE is more effective than the cognitive map (CM) approach. Notably, the CM lacks the 3D granularity required for many spatial reasoning tasks.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Descriptor $\backslash$ Parser	Gemini 3 Pro Avg. (MCA/NA)	Qwen2.5-72B-Instruct Avg. (MCA/NA)	Qwen2.5-32B-Instruct Avg. (MCA/NA)	Qwen2.5-7B-Instruct Avg. (MCA/NA)
Gemini 3 Pro	52.27 (44.12/57.40)	40.86 (42.33/39.47)	36.23 (38.03/41.35)	29.35 (30.00/28.73)
Qwen2.5-VL-72B-Instruct	36.11 (41.58/28.94)	31.11 (35.98/26.51)	26.12 (32.01/20.56)	24.06 (29.60/18.84)
Qwen2.5-VL-32B-Instruct	36.70 (40.74/32.18)	14.66 (29.20/0.94)	23.29 (29.60/17.34)	24.45 (30.24/18.99)
Qwen2.5-VL-7B-Instruct	32.72 (36.38/28.39)	29.08 (32.85/25.53)	24.33 (29.00/19.92)	25.19 (29.28/21.34)

Appendix

Table 5: Decomposition analysis of 3D spatial QA performance (matrix). We adopt the text-only prediction setting and report results for different combinations of visual descriptors and spatial knowledge parsers. Cell shows Avg. with (Multiple-Choice Answer / Numerical Answer).

Descriptor	Parser	Avg.	Numerical Answer				Multiple-Choice Answer
			Obj. Cnt.	Abs. Dist.	Obj. Size	Room Size	Rel. Dist.	Rel. Dir.	Route	Order
Gemini 3 Pro	Gemini 3 Pro	52.27	28.52	32.66	67.28	48.02	49.58	62.66	52.43	64.93
	Qwen2.5-72B	40.86	28.12	28.91	55.38	39.69	39.30	40.60	34.54	50.97
	Qwen2.5-32B	36.23	27.45	21.55	49.81	44.10	40.00	30.99	30.41	45.15
	Qwen2.5-7B	29.35	24.62	22.95	36.13	29.03	28.87	28.31	26.29	35.11
Qwen2.5-VL-72B	Gemini 3 Pro	36.11	10.57	16.57	51.83	31.72	40.25	39.93	34.81	47.39

Appendix

Table 6: Decomposition analysis of 3D spatial QA performance (grouped breakdown). We adopt the text-only prediction setting and report results for different combinations of visual descriptors and spatial knowledge parsers.

Descriptor	Parser	Avg.	Numerical Answer				Multiple-Choice Answer
			Obj. Cnt.	Abs. Dist.	Obj. Size	Room Size	Rel. Dist.	Rel. Dir.	Route	Order
Gemini 3 Pro	Qwen2.5-72B	40.86	28.12	28.91	55.38	39.69	39.30	40.60	34.54	50.97
Gemini 3 Pro	Qwen2.5-VL-72B	39.48	28.21	20.56	54.23	42.15	44.37	34.50	39.18	53.56
Qwen2.5-VL-72B	Qwen2.5-72B	31.11	12.83	21.74	39.71	23.51	37.04	33.16	32.99	40.13
Qwen2.5-VL-72B	Qwen2.5-VL-72B	26.65	9.08	11.08	36.21	20.59	28.31	30.58	36.08	40.78

Appendix

Table 7: Effect of visual tokens and multimodal training on spatial context reasoning.

Method	Avg.	ARKitScenes	ScanNet	ScanNetPP
o3 as base model
Direct	51.15	49.28	53.55	49.78
CoT	52.36	51.76	53.53	51.36
ToT	52.09	51.27	53.46	51.06
LtM	52.50	51.70	54.31	50.80
CM	53.93	52.69	56.24	52.00

Appendix

Table 8: Stratified VSI-Bench results across underlying scene datasets. Performance is broken down by ARKitScenes, ScanNet, and ScanNetPP for both proprietary and open-weight models. TRACE consistently improves over the Direct baseline across all environment distributions, indicating strong cross-environment generalization.

Methods	Avg.	Numerical Answer				Multiple-Choice Answer
		Obj. Cnt.	Abs. Dist.	Obj. Size	Room Size	Rel. Dist.	Rel. Dir.	Route	Order
Gemini 3 Pro as base model
Direct	52.61	33.77	32.57	67.09	42.99	62.54	50.52	51.03	70.71
CoT	53.65	30.35	34.54	64.05	40.76	61.78	58.09	61.34	71.96
ToT	58.88	44.55	42.12	72.20	45.55	65.35	57.83	55.62	73.73
LtM	59.52	45.19	40.72	73.36	44.15	65.82	60.40	53.59	73.64

Appendix

Table 9: Evaluation results on the VSI benchmark. We report average performance and detailed breakdowns across numerical-answer and multiple-choice tasks, under proprietary and open-sourced base models. Best results are in bold , and second-best are underlined .

	GLM-4.5V		MiMo-VL-7B		Qwen2.5-VL-72B		o3		Gemini 3 Pro
Method	Tok	Avg	Tok	Avg	Tok	Avg	Tok	Avg	Tok	Avg
Direct	405.17	37.33	337.36	39.79	3.48	36.28	3.61	51.15	334.35	52.61
CoT	568.36	38.48	579.21	37.49	129.62	28.44	76.20	52.36	479.64	53.65
ToT	1079.97	40.66	1132.86	39.14	308.99	37.68	352.30	52.09	450.82	58.88
LtM	989.55	40.99	1097.05	38.34	229.84	38.01	220.28	52.50	571.88	59.52
CM	722.16	38.93	723.68	36.85	224.72	35.47	81.07	53.93	403.23	59.72

Appendix

Table 10: Token consumption and performance trade-offs of prompting methods across different models. TRACE generally maintains a favorable performance-to-cost ratio relative to branching reasoning baselines, although its token usage varies by backbone. Best results for each model aredo in bold .

3D Spatial Reasoning

3D spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and motion to understand and act in three-dimensional scenes.

See all 202 papers

ViSRA: A Video-based Spatial Reasoning Agent for Multi-modal Large Language Models
May 11, 2026Tingshu Mou, Jiabo He, Renying Wang +5Visual Spatial Reasoning 3D Spatial Reasoning
Fudan University · BCAI · Bosch Center for Artificial Intelligence (BCAI) +1
EgoMind: Activating Spatial Cognition through Linguistic Reasoning in MLLMs
Apr 1, 2026Zhenghao Chen, Huiqun Wang, Di HuangSpatial Reasoning in Language Models Multimodal Spatial Reasoning
State Key Laboratory of Complex and Critical Software Environment, Beihang University · School of Computer Science and Engineering, Beihang University
Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representations
Mar 9, 2026Jiangye Yuan, Gowri Kumar, Baoyuan Wang3D Spatial Reasoning Multimodal Large Language Models
Zillow Group

May 11, 2026J/K move · Enter open · S save

ViSRA: A Video-based Spatial Reasoning Agent for Multi-modal Large Language Models

Tingshu Mou, Jiabo He, Renying Wang, Ce Liu+4

Fudan University · BCAI · Bosch Center for Artificial Intelligence (BCAI)+1

Visual Spatial Reasoning 3D Spatial Reasoning Multimodal Large Language Models VLM Reasoning Vision-Language Agents

Open paper AskPDF arXiv

Recent advances in Multi-modal Large Language Models (MLLMs) target 3D spatial intelligence, yet the progress has been largely driven by post-training on curated benchmarks, leaving the inference-time approach relatively underexplored. In this paper, we take a training-free perspective and introduce ViSRA, a human-aligned Video-based Spatial Reasoning Agent, as a framework to probe the spatial reasoning mechanism of MLLMs. ViSRA elicits spatial reasoning in a modular and extensible manner by leveraging explicit spatial information from expert models, enabling a plug-and-play flexible paradigm. ViSRA offers two key advantages: (1) human-aligned and transferable 3D understanding rather than task-specific overfitting; and (2) no post-training computational cost along with heavy manual curation of spatial reasoning datasets. Experimental results demonstrate consistent improvement across a set of MLLMs on both existing benchmarks and unseen 3D spatial reasoning tasks, with ViSRA outperforming baselines by up to a 15.6% and 28.9% absolute margin respectively.

Abstract

3D Spatial Reasoning

Explore similar work

ViSRA: A Video-based Spatial Reasoning Agent for Multi-modal Large Language Models

EgoMind: Activating Spatial Cognition through Linguistic Reasoning in MLLMs

Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representations