Organizations: NLPR, MAIS, Institute of Automation, Chinese Academy of Sciences · Alibaba Token Hub, Alibaba Group · The University of Hong Kong · The Chinese University of Hong Kong · Nanjing University
The spatial intelligence of large vision-language models (LVLMs) requires the ability to combine local visual observations into a geometrically consistent representation of the physical world. Many static benchmarks assess isolated spatial judgments, such as relative direction or distance, while interactive embodied evaluations combine perception with exploration, planning, control, and simulator-specific execution. To bridge this gap, we introduce RECONSPATIAL, a benchmark centered on multi-view 3D scene reconstruction across indoor, tabletop, and outdoor environments. Given real multi-view images and target-object queries, LVLMs are required to jointly predict oriented 3D bounding boxes in a shared, camera-anchored metric coordinate frame. Reconstructions are evaluated along two complementary axes: Structure for relative layout and Metric for physical scale and camera-anchored placement. Nine atomic spatial QA tasks, grouped into Geometric Perception and Spatial Relations, probe spatial capabilities involved in joint reconstruction. Offline navigation extends evaluation to path planning. Evaluation of 18 frontier LVLMs reveals a consistent gap between relative scene structure and metric reconstruction; even GPT-6-Astra scores 81.4% on Structure but only 58.7% on Metric. Tracing this gap, diagnostic analyses link atomic spatial capabilities to reconstruction quality and show that relational consistency alone does not ensure metric accuracy. Experiments with 3D map injection further separate scene construction from path planning, revealing how these geometric errors limit navigation success. Finally, spatial fine-tuning yields broad transfer across both spatial and general vision-language benchmarks.
Figures & tables
Figure 1: From spatial QA to joint 3D box prediction for evaluating LVLM spatial reasoning. (a) Static and interactive embodied QA compared with reconstruction from multi-view object queries via structured 3D boxes, visualized as a joint scene; reference geometry illustrates the target output. (b) Task distribution across four evaluation levels. (c) Eighteen models ranked by Overall, alongside their reconstruction score.
Figure 2: Examples across the four levels of ReconSpatial , with reconstruction scoring schematics and navigation annotations on the shown scene. Bold green options mark reference answers. Prompts are abbreviated; complete examples appear in Appendix .
Score
Component
Quantity tested
Structure
Adjacency
Jaccard agreement of local neighbor graphs
Shape
Residual after planar similarity alignment
Axis order
Chance-corrected Kendall order on three axes
Metric
Position
AP over normalized localization thresholds
View IoU
Mean top-down and front-view box overlap
Scale
Penalty for the fitted global scale correction
Table 1: Anatomy of the reconstruction scores. Metric separates scale-corrected geometry from the size of the required scale correction.
Table 2: Main results for eighteen frontier LVLMs. Atomic-task abbreviations are defined in Figure 2 ; Reconstruction reports three-domain Structure (S), Metric (M), and their mean. Scores are percentages; Navigation uses scale-normalized paths and Overall equally averages the four levels. Best and second-best values are bold and underlined, respectively.
Figure 3: Structure–Metric gaps by domain across eighteen models, ordered as in Table 2 . Bars show Structure minus Metric in percentage points. All 54 gaps are positive, with tabletop scenes showing the largest gap for every model.
Figure 4: Geometric information injection for reconstruction. Changes are relative to the same prompts without added cues.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Task
Domain
Instances
Views
Chance (%)
Appendix
Table 5: Data composition of ReconSpatial across L1–L4.
Current evaluations of spatial intelligence can be systematically invalid under modern vision-language model (VLM) settings. First, many benchmarks derive question-answer (QA) pairs from point-cloud-based 3D annotations originally curated for traditional 3D perception. When such annotations are treated as ground truth for video-based evaluation, reconstruction and annotation artifacts can miss objects that are clearly visible in the video, mislabel object identities, or corrupt geometry-dependent answers (e.g., size), yielding incorrect or ambiguous QA pairs. Second, evaluations often assume full-scene access, while many VLMs operate on sparsely sampled frames (e.g., 16-64), making many questions effectively unanswerable under the actual model inputs. We improve evaluation validity by introducing ReVSI, a benchmark and protocol that ensures each QA pair is answerable and correct under the model's actual inputs. To this end, we re-annotate objects and geometry across 381 scenes from 5 datasets to improve data quality, and regenerate all QA pairs with rigorous bias mitigation and human verification using professional 3D annotation tools. We further enhance evaluation controllability by providing variants across multiple frame budgets (16/32/64/all) and fine-grained object visibility metadata, enabling controlled diagnostic analyses. Evaluations of general and domain-specific VLMs on ReVSI reveal systematic failure modes that are obscured by prior benchmarks, yielding a more reliable and diagnostic assessment of spatial intelligence.
Yiming Zhang, Jiacheng Chen, Jiaqi Tan +3
1Simon Fraser University · 2Hong Kong University of Science and Technology · University of Waterloo +1
Vision-language models (VLMs) can benefit from geometric priors for multi-view spatial reasoning, yet answer-only training does not directly supervise the intermediate geometric estimates and their use in deriving quantitative spatial answers. We hypothesize that spatial chain-of-thought (CoT) supervision becomes more effective when the VLM first jointly learns complementary local geometry and global scene context through multi-view reconstruction. We introduce SpatialSpeak, a two-stage framework that connects QA-native reconstruction pretraining with spatial CoT learning. In Stage I, QA-Native Reconstruction Pretraining (QA-RP) combines marked-point 3D queries for fine-grained local geometry with object-center queries for global scene context across views. Both tasks are formulated as text-based question answering, allowing geometric estimation and subsequent reasoning to share the same autoregressive output interface. In Stage II, spatial CoT with Visual Compensation (CoT-VC) trains the model to express question-relevant geometric estimates and use them to derive answers, with reliability assessment and visual compensation supporting answer refinement when needed. On ReVSI, QA-RP increases the gain from CoT-VC from 2.6 to 6.9 points, and ablations show that both local and global reconstruction supervision are beneficial. SpatialSpeak achieves state-of-the-art results on ReVSI, VSI-Bench, and SPAR-Bench, with a ReVSI score of 62.8 that exceeds the strongest compared baseline by 8.7 points.
Yang Cao, Jiaxin Zhang, Dave Zhenyu Chen +4
Hong Kong University of Science and Technology · Harbin Institute of Technology · Huawei Noah’s Ark Lab
Spatial question answering is the dominant paradigm for evaluating spatial intelligence in Vision-Language Models (VLMs), but it leaves a complementary axis of spatial competence under-evaluated: holistic 3D layout inference, which predicts every visible object's pose and extent from a single image in a structured form. To this end, we introduce IDEAL-Bench, an evaluation suite that requires VLMs to predict structured 3D layouts on photorealistic indoor scenes across 10 room types, scored along five numerical dimensions and a perceptual render-and-compare protocol. By operating on semantically realistic scenes with full asset substitution under controlled lighting and viewpoint, IDEAL-Bench moves beyond CLEVR-style simple geometric primitives so that any image-space discrepancy reflects spatial reasoning alone. The benchmark is built on IDEAL-Scenes, a procedurally generated dataset of 1,000 re-renderable Blender environments with ground-truth layouts. Evaluating 15 prominent VLMs reveals three findings: the task remains substantially unsolved, with the strongest model reaching only 62.1/100 overall; all models exhibit a sharp asymmetry between object recognition and geometric regression, indicating that current VLMs are trained to describe scenes rather than to measure them; model rankings partially diverge from those on QA-based and primitive-reconstruction benchmarks: top-tier consensus holds, but mid-tier rankings shift substantially. Collectively, these findings establish IDEAL-Bench as a diagnostic suite, targeting the geometric and structural competencies that QA-based evaluation cannot surface, and paving the way towards more rigorous evaluation of spatial intelligence in next-generation VLMs. Together, these findings position IDEAL-Bench as a principled diagnostic for whether future VLMs achieve genuine spatial understanding rather than linguistic approximations of it.