Despite the remarkable success of large-scale pre-trained image representation models (i.e., vision encoders) across various vision tasks, they are predominantly trained on 2D image data and therefore often fail to capture 3D spatial relationships between objects and backgrounds in the real world, constraining their effectiveness in many downstream applications. To address this, we propose SpatialBoost, a scalable framework that enhances the spatial awareness of existing pre-trained vision encoders by injecting 3D spatial knowledge expressed in linguistic descriptions. The core idea involves converting dense 3D spatial information from 2D images into linguistic expressions, which is then used to inject such spatial knowledge into vision encoders through a Large Language Model (LLM). To this end, we adopt a multi-turn Chain-of-Thought (CoT) reasoning process that progressively incorporates dense spatial knowledge and builds hierarchical spatial understanding. To validate effectiveness, we adapt SpatialBoost to state-of-the-art vision encoders such as DINOv3, and evaluate its performance gains on a wide range of benchmarks requiring both 3D perception and general vision abilities. For instance, SpatialBoost improves DINOv3 performance from 55.9 to 59.7 mIoU on ADE20K, achieving state-of-the-art performance with 3.8% gain over the pre-trained DINOv3.
Figures & tables
Figure 1: Overview of SpatialBoost. We enhance spatial and geometric understanding of pre-trained vision encoders by leveraging language-guided spatial reasoning. SpatialBoost consists of (a) spatial knowledge extraction through depth estimation, 3D reconstruction, segmentation, and region captioning, (b) converting spatial knowledge into multi-turn spatial reasoning from pixel to scene levels, and (c) building a spatial-aware vision encoder with LLM using generated data in (b).
Figure 2: Illustration of multi-turn visual spatial reasoning dataset , exhibiting pixel-level, object-level, and scene-level reasoning QAs. At the pixel-level, the QA task queries the 3D positions of points ( e.g . , via depth estimation). At the object-level, it extracts spatial properties of objects ( e.g . , by predicting bounding cubes or relative positions). At the scene-level, it determines the exact distances between multiple objects that require the rationales of the previous steps. At last, we add 2-turn for general scene caption. These are listed in order and constitute 12 multi-turn visual spatial reasoning conservation.
Figure 3: Illustration of the dual-channel attention layer [ 40 ] , where an additional attention block is introduced alongside the original attention block and merged via a learnable mixture factor α .
Table 3: Results on 3D-centric tasks. We evaluate unified probing on diverse 3D-related tasks from ScanNet [ 23 ] scenes. We report BLEU-1 score for Vision-Language Reasoning (VLR) on ScanQA [ 6 ] and SQA3D [ 58 ] . For Visual Grounding (VG), we report accuracy (%) on overall category of ScanRefer [ 14 ] dataset. For Geometric Understanding (GU), we report Registration Recall (RR) at 0.05m RMSE threshold and Relative Translation Error (RTE). For 3D Semantic Understanding (3D SU), we report accuracy and mIoU. Lower is better for RTE and higher is better for all other metrics.
Figure 4: Examples of visual observations from CortexBench. We train imitation learning agents to learn a mapping from these visual observations to expert actions.
Image classification
Image retrieval
Method
ImageNet (kNN)
ImageNet (lin.)
Oxford-H
Paris-H
Met (GAP)
AmsterTime
Vision-only trained encoder
V-JEPAv2 [ 4 ]
82.1
84.0
40.6
76.1
38.2
31.9
Vision-Language trained encoder
AIMv2 [ 29 ]
85.7
88.3
60.8
86.4
52.0
50.3
dino.txt [ 46 ]
80.0
81.4
49.7
78.5
41.8
33.1
Table 5: Results on image classification and retrieval tasks. We report Top-1 accuracy of kNN performance and linear probing (lin.) for image classification on validation set of ImageNet-1K [ 71 ] . For image retrieval, we report global average precision (GAP) on Met [ 95 ] and mean average precision (mAP) on Oxford-Hard (Oxford-H) [ 65 ] , Paris-Hard (Paris-H) [ 65 ] , and AmsterTime dataset [ 94 ] . For all results, we freeze the encoder backbone.
Figure 5: Effect of dataset scalability. We investigate the effect of the size of analysis of data scalability effects on (a) depth estimation results (AbsRel, RMSE) on NYUd benchmark for SigLIPv2, (b) depth estimation results (AbsRel, RMSE) on NYUd benchmark for DINOv3, and (c) semantic segmentation results (mIoU) on ADE20K benchmark for SigLIPv2 and DINOv3. The results show scalable performance improvements with increased data size.
Figure 6: Effect of dual-channel attention layer. We report the linear evaluation performance of DINOv2-ViT-L/14 across different fine-tuning strategies.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
system _ prompt = [
“ You are a helpful multimodal assistant.
Generate question-answer pairs for given two images.
Both images are came from same scene.
When referring to the image, please call it the first image or the second image. "
]
general _ vqa _ prompt = [
Appendix
Table 10: Prompt examples for generating multi-view VQA data.
Label type
Labels
Positive labels
“ an iPhone photo of an indoor scene "
“ an iphone photo of an outdoor scene "
“ a DSLR photo of an indoor scene "
“ a DSLR of an outdoor scene "
Negative labels
“ a close up shot of a single object "
“ a product displayed in front of a white background "
Appendix
Table 11: CLIP labels for filtering images.
single _ point _ questions = [
“ What is the depth value at pixel point [A]? "
“ How far away is point [A]? "
“ Tell me the depth of point [A]. "
]
single _ point _ answers =[
“ [X] away. "
Appendix
Table 12: Template examples for pixel-level VQA.
bounding _ cube _ questions = [
“ Identify [A] and [B] "
“ What is the center of the 3d bounding box coordinate for [A]? "
]
bounding _ cube _ answers = [
“ [X] "
“ Center: [X] "
Appendix
Table 13: Template examples for object-level VQA.
distance _ questions = [
“ What is the distance between the [A] and the [B]? "
“ How far is the [A] from the [B]? "
“ How distant is the [A] from the [B]? "
“ Measure the distance from the [A] to the [B]. "
]
distance _ answers = [
Appendix
Table 14: Template examples for scene-level VQA.
Depth ↓
Segmentation ↑
Classification ↑
Method
OpenCLIP
DINOv2
OpenCLIP
DINOv2
OpenCLIP
DINOv2
lin.
DPT
lin.
DPT
lin.
+ms
lin.
+ms
Pre-trained
0.56
0.41
0.38
0.29
39.1
45.7
47.7
53.1
83.9
86.3
Pix
0.52
0.40
0.34
0.29
39.6
46.3
48.2
53.4
84.0
86.6
Obj
0.53
0.41
0.37
0.30
39.4
46.3
48.0
53.3
84.7
87.2
Scene
0.53
0.41
0.38
0.32
39.2
45.9
47.7
53.3
84.5
87.1
Appendix
Table 15: Effect of reasoning hierarchy.
Method
Classification ↑
Segmentation ↑
Depth estimation ↓
DINOv2 (Pre-trained)
86.3
47.7
0.38
Full Fine-tuning
79.5
49.4
0.31
LoRA
81.7
49.0
0.32
Dual-Channel Attn.
87.6
49.2
0.32
Appendix
Table 17: Quantitative results of dual-channel attention.
Figure 7: Qualitative results for dual-channel attention. We visualize attention heatmap from (a) DINOv3 and (b) SpatialBoost DINOv3. We rollout attention layers for cosine similarity between patches. Red cross denotes a query patch. We visualize pure attention heatmap (top) and RGB overlayed version (bottom).
Model
Sample size
Classification ↑
Segmentation ↑
Depth estimation ↓
SigLIPv2
Pre-trained
89.1
42.8
0.51
+SpatialBoost
50K
89.5
43.2
0.44
100K
89.7
44.5
0.42
300K
90.0
45.1
0.39
DINOv3
Pre-trained
88.4
55.9
0.31
+SpatialBoost
50K
88.6
56.8
0.29
Appendix
Table 18: Data scalability on classification, segmentation, and depth estimation.
Visual representation learning develops image and video features that capture useful structure for recognition, retrieval, spatial reasoning, and action.
Existing Vision-Language Models (VLMs) exhibits a critical bottleneck in robust spatial reasoning. Recent reinforcement learning (RL) methods aim to close this gap with verifiable outcomes, yet they suffer from poor credit assignment across intermediate reasoning steps. Concurrently, structured reasoning approaches overlook the critical depth perception necessary for comprehensive 3D understanding. To address these challenges, we propose SCOUT (Structured Chain-Of-Thought Utilizing Process-Supervised RL Training). Specifically, we design a structured Chain-of-Thought (CoT) framework that explicitly models 3D environmental perception to ensure robust spatial understanding and reasoning. Furthermore, we introduce a novel RL algorithm featuring multi-objective process rewards and a tailored advantage estimation method, facilitating fine-grained credit assignment across distinct segments of the reasoning trajectory. To support our framework, we develop SCOUT-24k, a structured spatial reasoning CoT dataset synthesized through a customized pipeline. Extensive evaluations demonstrate that SCOUT-3B improves upon baseline models by 16.85% and 6.3% on general spatial benchmarks and complex spatial reasoning tasks respectively. Notably, our larger SCOUT-7B even outperforms GPT-4o by a margin of 4.28%. Moreover, despite being trained exclusively on single image, SCOUT-7B exhibits robust out-of-domain generalization to multi-image and video scenarios. These empirical results render SCOUT as a critical step towards next generation of spatially-aware VLMs.
Zile Zhou, Huining Yuan, Weichen Zhang +2
Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
Achieving human-like spatial intelligence for vision-language models (VLMs) requires inferring 3D structures from 2D observations, recognizing object properties and relations in 3D space, and performing high-level spatial reasoning. In this paper, we propose a principled hierarchical framework that decomposes the learning of 3D spatial understanding in VLMs into four progressively complex levels, from geometric perception to abstract spatial reasoning. Guided by this framework, we construct an automated pipeline that processes approximately 5M images with over 45M objects to generate 3D spatial VQA pairs across diverse tasks and scenes for VLM supervised fine-tuning. We also develop an RGB-D VLM incorporating metric-scale point maps as auxiliary inputs to further enhance spatial understanding. Extensive experiments demonstrate that our approach achieves state-of-the-art performance on multiple spatial understanding and reasoning benchmarks, surpassing specialized spatial models and large proprietary systems such as Gemini-2.5-pro and GPT-5. Moreover, our analysis reveals clear dependencies among hierarchical task levels, offering new insights into how multi-level task design facilitates the emergence of 3D spatial intelligence.
Huizhi Liang, Yichao Shen, Yu Deng +5
Tsinghua University · Microsoft Research Asia · Xi’an Jiaotong University +1
Vision-language models excel at 2D visual understanding but remain limited in 3D spatial reasoning. Existing approaches either depend on explicit 3D modalities, which limits scalability, or inject partial, view-conditioned geometric priors and leave the language model to recover global scene structure from sparse cues. We introduce Spa3R, a self-supervised framework that learns a unified, view-invariant spatial representation from unposed multi-view RGB images. Its Predictive Spatial Field Modeling objective compresses context views into a compact latent representation and predicts aligned geometric and semantic feature fields at novel viewpoints, thereby encouraging coherent encoding of scene geometry and layout. We integrate the pre-trained Spa3R Encoder into a vision-language model through a lightweight residual cross-attention adapter, yielding Spa3-VLM and grounding language reasoning in global spatial context. Spa3-VLM achieves an average score of 58.6% on VSI-Bench and delivers leading or competitive performance across three additional spatial reasoning benchmarks. These results demonstrate that predictive spatial representation learning provides an effective visual foundation for 3D reasoning.
Haoyi Jiang, Liu Liu, Xinjie Wang +5
School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China · Horizon Robotics, Beijing 100094, China · D-Robotics, Shenzhen 518055, China