HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models

Huizhi Liang; Yichao Shen; Yu Deng; Sicheng Xu; Zhiyuan Feng; Tong Zhang; Yaobo Liang; Jiaolong Yang

cs.CVMar 26, 2026

HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models

Authors: Huizhi Liang, Yichao Shen, Yu Deng, Sicheng Xu, Zhiyuan Feng, Tong Zhang, Yaobo Liang, Jiaolong Yang

Organizations: Tsinghua University · Microsoft Research Asia · Xi’an Jiaotong University · University of the Chinese Academy of Sciences

Abstract

Achieving human-like spatial intelligence for vision-language models (VLMs) requires inferring 3D structures from 2D observations, recognizing object properties and relations in 3D space, and performing high-level spatial reasoning. In this paper, we propose a principled hierarchical framework that decomposes the learning of 3D spatial understanding in VLMs into four progressively complex levels, from geometric perception to abstract spatial reasoning. Guided by this framework, we construct an automated pipeline that processes approximately 5M images with over 45M objects to generate 3D spatial VQA pairs across diverse tasks and scenes for VLM supervised fine-tuning. We also develop an RGB-D VLM incorporating metric-scale point maps as auxiliary inputs to further enhance spatial understanding. Extensive experiments demonstrate that our approach achieves state-of-the-art performance on multiple spatial understanding and reasoning benchmarks, surpassing specialized spatial models and large proprietary systems such as Gemini-2.5-pro and GPT-5. Moreover, our analysis reveals clear dependencies among hierarchical task levels, offering new insights into how multi-level task design facilitates the emergence of 3D spatial intelligence.

Figures & tables

Figure 2: Overview of our approach. Left: Data construction pipeline which generates spatial-related VQA pairs from either in-the-wild images or existing data with 3D annotations. Right: Hierarchical spatial understanding task taxonomy with representative QA pairs.

Figure 3: Model architecture of our VLM, which integrates metric-scale 3D point map as auxiliary input.

Model	Input	SpatialRGPT-Quantitative $\uparrow$						QSpatial-Bench $\uparrow$
		Width	Height	Direct Distance	Horizontal Distance	Vertical Distance	Avg.	Plus	ScanNet	Avg.
		$\underbrace{\rule{56.9055pt}{0.0pt}}_{\textit{\footnotesize{(Level 1)}}}$		$\underbrace{\rule{125.19194pt}{0.0pt}}_{\textit{\footnotesize{(Level 2)}}}$				$\underbrace{\rule{56.9055pt}{0.0pt}}_{\textit{\footnotesize{(Level 1\&2)}}}$
Proprietary Models
GPT-4o	RGB	51.10	68.40	29.70	25.40	33.00	41.50	61.06	69.41	66.30
GPT-5	RGB	59.09	67.66	29.72	25.21	13.72	40.47	63.37	73.53	68.45
Gemini-2.5-Pro	RGB	37.20	43.51	16.78	17.39	15.53	26.57	40.59	55.46	49.92

Table 1: Accuracy (%) on quantitative VQA benchmarks for level-1 and level-2 spatial understanding.

*

denotes using GT point map.

Proprietary Models
Method	Input	EmbSpatial $\uparrow$	RoboSpatial $\uparrow$	CV-Bench-3D $\uparrow$	CV-Bench-2D Relation $\uparrow$	3DSRBench $\uparrow$
		$\underbrace{\rule{307.28984pt}{0.0pt}}_{\textit{\footnotesize{(Level 2)}}}$				$\underbrace{\rule{56.9055pt}{0.0pt}}_{\textit{\footnotesize{(Level 1-3)}}}$
GPT-4o	RGB	63.38	77.20	84.90	84.62	44.20
Gemini-2.5-Pro	RGB	76.67	77.24	90.80	93.54	48.47
Claude-3.7-Sonnet	RGB	33.33	60.73	85.00	74.15	48.20
Open-Source Vision-Language Models

Table 2: Accuracy (%) on qualitative VQA benchmarks evaluating spatial understanding and reasoning across levels 1–3.

Model	Object-to-Camera Distance (L1) $\uparrow$	Object Direction (L2) $\uparrow$	Spatial Problem Solving (L3) $\uparrow$
GPT-5	47.19	59.27	33.33
Qwen3VL-8B	12.70	22.52	25.64
RoboRefer-8B-SFT	58.63	N/A	26.92
HiSpatial-3B (ours)	92.18	67.21	47.44

Table 3: Accuracy (%) on our custom spatial VQA benchmark.

Benchmark	PaliGemma2-3B	Ours
MMBench $\uparrow$	49.86	69.67
POPE $\uparrow$	87.00	87.97
SEED $\uparrow$	48.32	63.51
RealWorldQA $\uparrow$	47.76	58.95

Table 4: Accuracy (%) on general VQA benchmarks compared to our base model PaliGemma2.

L0	L1	L2	L3	Level 2 Tasks					Level 3 Tasks
				CV-Bench $\uparrow$	RoboSpatial $\uparrow$	3DSR-L2 $\uparrow$	EmbSpatial $\uparrow$	Avg. $\uparrow$	3DSR-L3 $\uparrow$	Problem Solving $\uparrow$	Avg. $\uparrow$
✓	✓	✓	✓	96.64	86.18	61.32	80.71	81.21	65.14	47.44	56.29
		✓	✓	96.55 (-0.09)	82.93 (-3.25)	60.03 (-1.29)	79.25 (-1.46)	79.69 (-1.52)	51.43 (-13.71)	44.87 (-2.57)	48.15 (-8.14)
✓			✓	79.00 (-17.64)	77.24 (-8.94)	37.36 (-23.96)	37.53 (-43.18)	56.21 (-25.00)	43.81 (-21.33)	39.74 (-7.70)	41.78 (-14.51)

Table 5: Inter-level task dependency analysis. Removing lower-level tasks in training reduces higher-level performance; see text for details.

Input	Source	Qualitative $\uparrow$	Quantitative $\uparrow$
RGB	-	83.70	74.16
RGB+Relative Depth	MoGe2	84.29 (+0.59)	75.26 (+0.90)
RGB+XYZ (Ours)	MoGe2	84.79 (+0.50)	82.02 (+6.76)
RGB+XYZ (Ours)	GT	-	82.79 (+0.77)

Table 6: Effect of auxiliary 3D input on model accuracy (%).

Task	Level	# of QAs	Sampling Ratio in Training
Pixel-Wise 3D Point Querying	0	Online Generated	10.51%
Pairwise Depth Ordering	0	Online Generated	2.69%
Object Orientation	1	238,909,307	3.51%
Object Size	1	55,187,863	11.56%
Object Localization	1	36,469,102	8.31%
Relative Direction	2	510,297,825	26.09%

Table 7: Task statistics of our training dataset.

Figure 4: Examples of spatial VQA data constructed using our method, covering different task levels.

Figure 5: Examples of spatial VQA data constructed using our method, covering different task levels.

Figure 6: Examples of our model’s responses on unseen images.

Figure 7: Prompt used to generate Level-3 spatial problem-solving QA pairs.(test set)

Figure 8: Prompt used to evaluate the correctness of answers in our custom benchmark for spatial problem-solving tasks.

Figure 9: Prompt used to generate the object reference in kosmos dataset

Figure 10: Prompt used to vlm grounding for evaluation in kosmos dataset

Figure 11: Prompt used to generate the object reference in objects365 dataset

Figure 12: Prompt used to vlm grounding for evaluation in objects365 dataset

Figure 13: Prompt used to generate the basic problem solving question

Figure 14: Prompt used to generate the quantative problem solving question

Figure 15: Prompt used to generate the problem solving question guided by few shot demonstrations

Figure 16: Examples of few-shot demonstration

Figure 17: Prompt used to generate the answer of the given problem solving question (stage1, part1)

Figure 18: Prompt used to generate the answer of the given problem solving question (stage1, part2)

Figure 19: Prompt used to generate the answer of the given problem solving question (stage1, part3)

Figure 20: Prompt used to generate the answer of the given problem solving question (stage2)

3D Spatial Reasoning

3D spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and motion to understand and act in three-dimensional scenes.

See all 202 papers

Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning
May 28, 2026Chun-Hsiao Yeh, Shengyi Qian, Manchen Wang +3Vision-Language Models 3D Spatial Reasoning
FAIR at Meta · UC Berkeley · HKU
Grounded 3D-Aware Spatial Vision-Language Modeling
May 28, 2026An-Chieh Cheng, Yang Fu, Yatai Ji +12Visually Grounded Reasoning 3D Spatial Reasoning
UCSD · NVIDIA · MIT
SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models
Aug 3, 2026Jing Wu, Jianhua Wu, Jiayi Guan +5Vision-Language Models Visual Spatial Reasoning
Xiaomi EV, Beijing, China · College of Automotive and Energy Engineering, Tongji University, Shanghai, China · Independent Researcher

May 28, 2026J/K move · Enter open · S save

Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning

Chun-Hsiao Yeh, Shengyi Qian, Manchen Wang, Yi Ma+2

FAIR at Meta · UC Berkeley · HKU

Vision-Language Models 3D Spatial Reasoning Geometric Representation Learning 3D VLMs

Open paper AskPDF arXiv

Vision-Language Models (VLMs) often struggle with robust 3D spatial reasoning. Prevailing methods that rely on fine-tuning with 3D visual question-answering (VQA) datasets may overfit dataset-specific biases, while integrating specialized 3D visual encoders is often inflexible and cumbersome. In this paper, we argue that genuine spatial understanding should emerge from learning fundamental geometric priors, not only from high-level VQA supervision. We propose GASP (Geometric-Aware Spatial Priors), a framework that injects these priors directly into the LLM's transformer layers. GASP employs a small correspondence head, applied as a deep supervision signal across all layers, and is trained with a dual objective leveraging ground-truth geometry from large-scale video scenes: a contrastive loss on ground-truth point correspondences enforces 2D view-invariance, while a depth consistency supervision resolves 3D geometric ambiguities. Our analysis first provides a diagnostic showing that standard VLMs' internal correspondence matching accuracy is very low (often below 5%). We then demonstrate that our training substantially improves this behavior, boosting peak layer-wise correspondence to over 70% and maintaining over 85% temporal robustness while baselines remain below 5%. These internal improvements translate to significant gains on downstream spatial benchmarks including +18.2% on All-Angles Bench and +29.0% on VSI-Bench, all without training on any 3D VQA data. Our findings indicate that learning from fundamental geometric priors is a promising and generalizable pathway towards VLMs with more reliable 3D spatial reasoning.

Abstract

3D Spatial Reasoning

Explore similar work

Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning

Grounded 3D-Aware Spatial Vision-Language Modeling

SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models