Think, Then Look: Active Spatial Reasoning for House-Scale 3D Scene Understanding

Hongpei Zheng; Shijie Li; Lin Qian; Zhenghao Li; Qijun Yang; Yanran Li; Hujun Yin

cs.CVDec 2, 2025

Think, Then Look: Active Spatial Reasoning for House-Scale 3D Scene Understanding

Authors: Hongpei Zheng, Shijie Li, Lin Qian, Zhenghao Li, Qijun Yang, Yanran Li, Hujun Yin

Organizations: University of Manchester · Institute for Infocomm Research (I2R), A*STAR, Singapore · University of Bedfordshire

Abstract

Spatial reasoning in large-scale 3D environments remains challenging for current vision--language models, which are typically constrained to room-scale scenarios. We formalize Active House-Scale Spatial Reasoning (AHSR), a new paradigm in which a model reasons over a pre-built house-scale 3D map via virtual spatial tool invocations to answer spatial questions, without exhaustive scene-wide processing. To support AHSR research, we introduce H $^2$ U3D (Holistic House Understanding in 3D), the first benchmark targeting house-scale 3D scene understanding, featuring environments with an average aggregate floor area of 250.8 m $^2$ and up to three floors, together with hierarchical coarse-to-fine visual representations. Building on H $^2$ U3D, we propose SpatialReasoner, an AHSR framework trained via supervised fine-tuning with self-correction, followed by reinforcement learning with a task-aware adaptive exploration reward. SpatialReasoner achieves state-of-the-art performance on H $^2$ U3D with 64.9% overall accuracy, outperforming strong baselines including GPT-5.4 and Gemini-3.5-Flash, and generalizes effectively to MT-HM3D and HM-EQA. These results demonstrate the clear advantage of active map-directed exploration over passive scene-wide processing in house-scale 3D understanding.

Figures & tables

Figure 1: Overview of SpatialReasoner inference. Given a spatial question and a pre-built house-scale 3D map, the model progressively localizes the query-relevant region, invokes zooming and viewpoint-rendering tools, recovers from an uninformative observation, and produces an answer grounded in the final visual evidence.

Figure 2: Two-stage H 2 U3D dataset construction pipeline. Stage 1 (QA Collection): tool calls are invoked to acquire interest region, then multi-floor BEVs and rendered views are processed by VLM to generate question-answer pairs. Stage 2 (CoT Generation): The same visual inputs and tool sequences are used to produce detailed chain-of-thought reasoning annotations.

Figure 3: Question-category distribution of H 2 U3D.

#	Model	Params.	Position	Attribute	Counting	Material	State	Shape	Overall
Tool-Free Models
1	Gemini-3.5-Flash	—	0.604	0.483	0.510	0.608	0.557	0.604	0.553
2	GPT-5.4	—	0.588	0.540	0.566	0.584	0.543	0.648	0.571
3	Qwen3-VL-8B	8B	0.474	0.424	0.336	0.502	0.550	0.576	0.458
4	Qwen3-VL-30B-A3B	30B	0.398	0.329	0.391	0.452	0.550	0.580	0.412
5	Qwen3-VL-235B	235B	0.462	0.453	0.377	0.478	0.595	0.596	0.471

Table 1: Accuracy on the H 2 U3D test set. The best and second-best results are highlighted in bold and underlined, respectively.

Model	Base VLM	MT-HM3D	HM-EQA
ExploreEQA	GPT-4o	0.362	0.474
GraphEQA	GPT-4o	—	0.635
MemoryEQA	GPT-4o	0.431	0.614
SpatialReasoner	Qwen3-VL-8B	0.479	0.668

Table 2: Accuracy on MT-HM3D and HM-EQA benchmarks.

Model	Office $\uparrow$	Laboratory $\uparrow$	Overall $\uparrow$
GPT-5.4	0.567	0.433	0.500
GPT-5.4 w/ Tool	0.667	0.567	0.617
SpatialReasoner	0.733	0.600	0.667

Table 3: Accuracy on two real-world environments reconstructed using 3D Gaussian Splatting, with 30 questions per environment.

Variant	Overall $\uparrow$	Avg. Tool Calls
w/o Error Injection	0.571	2.01
w/o SFT	0.584	2.44
w/o Exploration Reward	0.597	3.12
SFT Only	0.565	1.97
SpatialReasoner	0.649	3.79

Table 4: Ablation study on the H 2 U3D test set. “w/o SFT” applies reinforcement learning directly to the base model, whereas “SFT Only” denotes the supervised checkpoint without reinforcement post-training.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Overview of our task pipeline.

Figure 5: Illustration of the spatial operations in our SpatialReasoner.

Dataset	Scene Source	QA Pairs	Training Split	Multi-floor	Annotation
EXPRESS-Bench ( Jiang et al. 2025 )	HM3D	2,044	✗	✗	VLM-generated
HM-EQA ( Ren et al. 2024 )	HM3D	500	✗	✗	VLM-generated
Open-EQA ( Majumdar et al. 2024 )	ScanNet/HM3D	1,636	✗	✗	Manual
MT-HM3D ( Zhai et al. 2025 )	HM3D	1,587	✗	✗	VLM-generated
H 2 U3D (Ours)	HM3D	34,651	✓	✓	VLM + Human

Appendix

Table 5: Comparison of H 2 U3D with existing house-scale 3D spatial question-answering datasets. H 2 U3D is the only dataset in this comparison that provides both a training split and multi-floor scenes, while containing substantially more question–answer pairs.

Category	Definition	Representative Example
Position	Reasoning about object locations or spatial relations.	“What object is mounted on the wall directly above the toilet?”
Attribute	Identifying visual properties such as color, pattern, or style.	“What color is the handle on the white paneled door next to the window?”
Counting	Determining the number of objects or components.	“How many framed pictures are hanging in a vertical line on the wall?”
Material	Identifying an object’s material, surface, or finish.	“What material is the cabinet located next to the toilet made of?”
State	Identifying the current condition or configuration of an object.	“What is the state of the doorway at the far end of the passage?”
Shape	Identifying the geometric form of an object or component.	“What is the shape of the rug on the floor below the window?”

Appendix

Table 6: Definitions and representative examples of the six H 2 U3D question categories.

Figure 6: Qualitative vocabulary overview of H 2 U3D questions. Word size reflects question-level occurrence frequency after removing common function and template words. Word color indicates the question category in which the term appears most frequently.

Figure 7: Overview of the data pipeline for constructing the H 2 U3D benchmark. The pipeline covers scene selection, multi-floor context processing, expert demonstration collection, annotation, and QA generation, supporting both passive and active spatial reasoning tasks.

Figure 8: Examples of error cases in different operations.

Hyperparameter	Supervised Fine-Tuning	Reinforcement Learning
Initialization	Qwen3-VL-8B-Instruct	SFT checkpoint
Training samples	5,626	22,048
Error-induced self-correction	25%	—
Optimizer / algorithm	AdamW	GRPO
Training epochs	3	1
Learning rate	$1\times 10^{-6}$	$1\times 10^{-6}$

Appendix

Table 7: Training hyperparameters for SpatialReasoner. Batch size denotes the global SFT batch size and the RL training batch size, respectively; rollout batch size denotes the number of questions collected before an RL update.

Reward component	Hyperparameters	Values
Overall reward weights	$(w_{\mathrm{c}},w_{\mathrm{e}},w_{\mathrm{g}},w_{\mathrm{r}})$	$(1.0,1.0,0.25,1.0)$
Exploration bonus and penalty	$(\alpha_{\mathrm{e}},\alpha_{\mathrm{p}})$	$(0.025,0.02)$
Exploration cap and budget	$(N_{\max},N_{\mathrm{budget}})$	$(6,5)$
Group-success thresholds	$(\tau_{\mathrm{low}},\tau_{\mathrm{high}})$	$(0.5,0.8)$
Goal-reward Gaussian widths	$(\sigma_{\mathrm{b}},\sigma_{\mathrm{p}},\sigma_{\theta})$	$(0.2,0.2,0.25)$
Goal-reward term weights	$(\omega_{\mathrm{z}},\omega_{\mathrm{v}})$	$(0.5,0.5)$

Appendix

Table 8: Spatial-reward hyperparameters. The position threshold is expressed on the model-facing

[0,1000]

coordinate scale, and the angular threshold is measured in degrees.

Explore similar work

SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes
Sep 14, 2026Anubhav Khanal, Prabigya Acharya, Roshni Poudel +53D Spatial Reasoning 3D Generation
Nepal Applied Mathematics and Informatics Institute for research (NAAMII), Nepal · INSAIT, Sofia University, Bulgaria · State University of New York at Stony Brook, Korea
SceneScaffold: Active Scene-State Construction for Unified 3D Scene Understanding
Sep 27, 2026Xiangqi Li, Libo Huang, Jiarui Zhao +43D Scene Understanding State-Tracking
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
SpatialForge: Bootstrapping 3D-Aware Spatial Reasoning from Open-World 2D Images
May 12, 2026Zishan Liu, Ruoxi Zang, Yanglin Zhang +53D Spatial Reasoning Spatial Reasoning
1Lingnan University · 2XPENG Robotics

Sep 14, 2026, cs.CVJ/K move · Enter open · S save

SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes

Anubhav Khanal, Prabigya Acharya, Roshni Poudel, Sujan Kapali+4

Nepal Applied Mathematics and Informatics Institute for research (NAAMII), Nepal · INSAIT, Sofia University, Bulgaria · State University of New York at Stony Brook, Korea

3D Spatial Reasoning 3D Generation Scene Understanding Vision-Language Foundation Models

Open paper AskPDF arXiv

Vision-language models excel at 2D image understanding but remain limited in 3D spatial reasoning. Progress is hindered by limitations in current benchmarks. First, 3D datasets often rely on point clouds that capture geometry but discard rich visual features like texture, text, and materials. Second, annotations treat objects in isolation while ignoring real-world hierarchical organization (scenes, rooms, functional areas, object groups). Third, evaluation tasks focus narrowly on basic recognition rather than multi-step spatial reasoning. In this context, we introduce SceneBench, a benchmark of 966 photorealistic 3D scenes reconstructed with Gaussian Splatting and densely annotated with hierarchical semantics spanning scenes, rooms, functional areas, object groups, and individual objects. These annotations are produced through a human-in-the-loop pipeline combining vision-language models with roughly 1,500 human-hours of iterative refinement and verification, producing over 183K annotated nodes with textual descriptions and 3D bounding boxes. Building on this representation, we define three evaluation tasks: Existence-Based Questions probing object attributes, Spatial Intelligence Questions covering counting, size comparison, distance, and directional relations, and Grounded Question-Reasoning-Answer (QRA) triplets requiring multi-step reasoning across semantic levels. Experiments with state-of-the-art vision-language models show that while models perform well on basic recognition tasks (e.g., up to 85% accuracy for detection), performance drops substantially on hierarchical and compositional reasoning (e.g., down to 60% for counting), revealing limitations not captured by existing benchmarks. SceneBench provides a realistic testbed for developing and evaluating models capable of fine-grained spatial reasoning in photorealistic 3D environments.

Abstract

Explore similar work

SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes

SceneScaffold: Active Scene-State Construction for Unified 3D Scene Understanding

SpatialForge: Bootstrapping 3D-Aware Spatial Reasoning from Open-World 2D Images