SpatialBoost: Enhancing Visual Representation through Language-Guided Reasoning

Byungwoo Jeon; Dongyoung Kim; Huiwon Jang; Insoo Kim; Jinwoo Shin

cs.CVMar 23, 2026

SpatialBoost: Enhancing Visual Representation through Language-Guided Reasoning

Authors: Byungwoo Jeon, Dongyoung Kim, Huiwon Jang, Insoo Kim, Jinwoo Shin

Organizations: KAIST · RLWRLD · NAVER Cloud

Abstract

Despite the remarkable success of large-scale pre-trained image representation models (i.e., vision encoders) across various vision tasks, they are predominantly trained on 2D image data and therefore often fail to capture 3D spatial relationships between objects and backgrounds in the real world, constraining their effectiveness in many downstream applications. To address this, we propose SpatialBoost, a scalable framework that enhances the spatial awareness of existing pre-trained vision encoders by injecting 3D spatial knowledge expressed in linguistic descriptions. The core idea involves converting dense 3D spatial information from 2D images into linguistic expressions, which is then used to inject such spatial knowledge into vision encoders through a Large Language Model (LLM). To this end, we adopt a multi-turn Chain-of-Thought (CoT) reasoning process that progressively incorporates dense spatial knowledge and builds hierarchical spatial understanding. To validate effectiveness, we adapt SpatialBoost to state-of-the-art vision encoders such as DINOv3, and evaluate its performance gains on a wide range of benchmarks requiring both 3D perception and general vision abilities. For instance, SpatialBoost improves DINOv3 performance from 55.9 to 59.7 mIoU on ADE20K, achieving state-of-the-art performance with 3.8% gain over the pre-trained DINOv3.

Figures & tables

Figure 1: Overview of SpatialBoost. We enhance spatial and geometric understanding of pre-trained vision encoders by leveraging language-guided spatial reasoning. SpatialBoost consists of (a) spatial knowledge extraction through depth estimation, 3D reconstruction, segmentation, and region captioning, (b) converting spatial knowledge into multi-turn spatial reasoning from pixel to scene levels, and (c) building a spatial-aware vision encoder with LLM using generated data in (b).

Figure 2: Illustration of multi-turn visual spatial reasoning dataset , exhibiting pixel-level, object-level, and scene-level reasoning QAs. At the pixel-level, the QA task queries the 3D positions of points ( e.g . , via depth estimation). At the object-level, it extracts spatial properties of objects ( e.g . , by predicting bounding cubes or relative positions). At the scene-level, it determines the exact distances between multiple objects that require the rationales of the previous steps. At last, we add 2-turn for general scene caption. These are listed in order and constitute 12 multi-turn visual spatial reasoning conservation.

Figure 3: Illustration of the dual-channel attention layer [ 40 ] , where an additional attention block is introduced alongside the original attention block and merged via a learnable mixture factor

\alpha

Table 4

Vision-only trained encoder
	VLR		VG	GU		3D SU
Method	ScanQA $\uparrow$	SQA3D $\uparrow$	ScanRefer (%) $\uparrow$	[email protected] (%) $\uparrow$	RTE (m) $\downarrow$	Acc $\uparrow$	mIoU $\uparrow$
V-JEPAv2 [ 4 ]	37.7	49.0	55.5	95.4	0.08	69.5	33.2
Vision-Language trained encoder
AIMv2 [ 29 ]	37.4	48.1	50.9	47.8	0.31	53.9	11.8
dino.txt [ 46 ]	39.8	50.4	52.5	81.7	0.16	85.7	59.4

Table 3: Results on 3D-centric tasks. We evaluate unified probing on diverse 3D-related tasks from ScanNet [ 23 ] scenes. We report BLEU-1 score for Vision-Language Reasoning (VLR) on ScanQA [ 6 ] and SQA3D [ 58 ] . For Visual Grounding (VG), we report accuracy (%) on overall category of ScanRefer [ 14 ] dataset. For Geometric Understanding (GU), we report Registration Recall (RR) at 0.05m RMSE threshold and Relative Translation Error (RTE). For 3D Semantic Understanding (3D SU), we report accuracy and mIoU. Lower is better for RTE and higher is better for all other metrics.

Panel of Figure 4 — Figure 4: Examples of visual observations from CortexBench. We train imitation learning agents to learn a mapping from these visual observations to expert actions.

Vision-only trained encoder
	Image classification		Image retrieval
Method	ImageNet (kNN)	ImageNet (lin.)	Oxford-H	Paris-H	Met (GAP)	AmsterTime
V-JEPAv2 [ 4 ]	82.1	84.0	40.6	76.1	38.2	31.9
Vision-Language trained encoder
AIMv2 [ 29 ]	85.7	88.3	60.8	86.4	52.0	50.3
dino.txt [ 46 ]	80.0	81.4	49.7	78.5	41.8	33.1

Table 5: Results on image classification and retrieval tasks. We report Top-1 accuracy of kNN performance and linear probing (lin.) for image classification on validation set of ImageNet-1K [ 71 ] . For image retrieval, we report global average precision (GAP) on Met [ 95 ] and mean average precision (mAP) on Oxford-Hard (Oxford-H) [ 65 ] , Paris-Hard (Paris-H) [ 65 ] , and AmsterTime dataset [ 94 ] . For all results, we freeze the encoder backbone.

Panel of Figure 5 — Figure 5: Effect of dataset scalability. We investigate the effect of the size of analysis of data scalability effects on (a) depth estimation results (AbsRel, RMSE) on NYUd benchmark for SigLIPv2, (b) depth estimation results (AbsRel, RMSE) on NYUd benchmark for DINOv3, and (c) semantic segmentation results (mIoU) on ADE20K benchmark for SigLIPv2 and DINOv3. The results show scalable performance improvements with increased data size.

Figure 6: Effect of dual-channel attention layer. We report the linear evaluation performance of DINOv2-ViT-L/14 across different fine-tuning strategies.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

system

\_

prompt

=

[

“ You are a helpful multimodal assistant.

Generate question-answer pairs for given two images.

Both images are came from same scene.

When referring to the image, please call it the first image or the second image. "

]

general

\_

vqa

\_

prompt

=

[

Appendix

Table 10: Prompt examples for generating multi-view VQA data.

Label type	Labels
Positive labels	“ an iPhone photo of an indoor scene "
	“ an iphone photo of an outdoor scene "
	“ a DSLR photo of an indoor scene "
	“ a DSLR of an outdoor scene "
Negative labels	“ a close up shot of a single object "
	“ a product displayed in front of a white background "

Appendix

Table 11: CLIP labels for filtering images.

single

\_

point

\_

questions

=

[

“ What is the depth value at pixel point [A]? "

“ How far away is point [A]? "

“ Tell me the depth of point [A]. "

]

single

\_

point

\_

answers

=[

“ [X] away. "

Appendix

Table 12: Template examples for pixel-level VQA.

bounding

\_

cube

\_

questions

=

[

“ Identify [A] and [B] "

“ What is the center of the 3d bounding box coordinate for [A]? "

]

bounding

\_

cube

\_

answers

=

[

“ [X] "

“ Center: [X] "

Appendix

Table 13: Template examples for object-level VQA.

distance

\_

questions

=

[

“ What is the distance between the [A] and the [B]? "

“ How far is the [A] from the [B]? "

“ How distant is the [A] from the [B]? "

“ Measure the distance from the [A] to the [B]. "

]

distance

\_

answers

=

[

Appendix

Table 14: Template examples for scene-level VQA.

	Depth $\downarrow$				Segmentation $\uparrow$				Classification $\uparrow$
Method	OpenCLIP		DINOv2		OpenCLIP		DINOv2		OpenCLIP	DINOv2
	lin.	DPT	lin.	DPT	lin.	+ms	lin.	+ms
Pre-trained	0.56	0.41	0.38	0.29	39.1	45.7	47.7	53.1	83.9	86.3
Pix	0.52	0.40	0.34	0.29	39.6	46.3	48.2	53.4	84.0	86.6
Obj	0.53	0.41	0.37	0.30	39.4	46.3	48.0	53.3	84.7	87.2
Scene	0.53	0.41	0.38	0.32	39.2	45.9	47.7	53.3	84.5	87.1

Appendix

Table 15: Effect of reasoning hierarchy.

Method	Classification $\uparrow$	Segmentation $\uparrow$	Depth estimation $\downarrow$
DINOv2 (Pre-trained)	86.3	47.7	0.38
Full Fine-tuning	79.5	49.4	0.31
LoRA	81.7	49.0	0.32
Dual-Channel Attn.	87.6	49.2	0.32

Appendix

Table 17: Quantitative results of dual-channel attention.

Figure 7: Qualitative results for dual-channel attention. We visualize attention heatmap from (a) DINOv3 and (b) SpatialBoost DINOv3. We rollout attention layers for cosine similarity between patches. Red cross denotes a query patch. We visualize pure attention heatmap (top) and RGB overlayed version (bottom).

Model	Sample size	Classification $\uparrow$	Segmentation $\uparrow$	Depth estimation $\downarrow$
SigLIPv2	Pre-trained	89.1	42.8	0.51
+SpatialBoost	50K	89.5	43.2	0.44
	100K	89.7	44.5	0.42
	300K	90.0	45.1	0.39
DINOv3	Pre-trained	88.4	55.9	0.31
+SpatialBoost	50K	88.6	56.8	0.29

Appendix

Table 18: Data scalability on classification, segmentation, and depth estimation.

Visual Representation Learning

Visual representation learning develops image and video features that capture useful structure for recognition, retrieval, spatial reasoning, and action.

See all 224 papers

SCOUT: Unlocking Enhanced Spatial Reasoning via Structured Chain-of-Thought and Multi-Objective Process Reward
Aug 12, 2026Zile Zhou, Huining Yuan, Weichen Zhang +2Visual Spatial Reasoning Process Reward Models
Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models
Mar 26, 2026Huizhi Liang, Yichao Shen, Yu Deng +53D Spatial Reasoning 3D VLMs
Tsinghua University · Microsoft Research Asia · Xi’an Jiaotong University +1
Spa3R: Predictive Spatial Field Modeling for 3D Visual Reasoning
Feb 24, 2026Haoyi Jiang, Liu Liu, Xinjie Wang +53D Spatial Reasoning 3D VLMs
School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China · Horizon Robotics, Beijing 100094, China · D-Robotics, Shenzhen 518055, China

Aug 12, 2026J/K move · Enter open · S save

SCOUT: Unlocking Enhanced Spatial Reasoning via Structured Chain-of-Thought and Multi-Objective Process Reward

Zile Zhou, Huining Yuan, Weichen Zhang, Xinlei Chen+1

Shenzhen International Graduate School, Tsinghua University, Shenzhen, China

Visual Spatial Reasoning Process Reward Models 3D Spatial Reasoning CoT Reasoning VLM Reasoning RL for VLMs

Open paper AskPDF arXiv

Existing Vision-Language Models (VLMs) exhibits a critical bottleneck in robust spatial reasoning. Recent reinforcement learning (RL) methods aim to close this gap with verifiable outcomes, yet they suffer from poor credit assignment across intermediate reasoning steps. Concurrently, structured reasoning approaches overlook the critical depth perception necessary for comprehensive 3D understanding. To address these challenges, we propose SCOUT (Structured Chain-Of-Thought Utilizing Process-Supervised RL Training). Specifically, we design a structured Chain-of-Thought (CoT) framework that explicitly models 3D environmental perception to ensure robust spatial understanding and reasoning. Furthermore, we introduce a novel RL algorithm featuring multi-objective process rewards and a tailored advantage estimation method, facilitating fine-grained credit assignment across distinct segments of the reasoning trajectory. To support our framework, we develop SCOUT-24k, a structured spatial reasoning CoT dataset synthesized through a customized pipeline. Extensive evaluations demonstrate that SCOUT-3B improves upon baseline models by 16.85% and 6.3% on general spatial benchmarks and complex spatial reasoning tasks respectively. Notably, our larger SCOUT-7B even outperforms GPT-4o by a margin of 4.28%. Moreover, despite being trained exclusively on single image, SCOUT-7B exhibits robust out-of-domain generalization to multi-image and video scenarios. These empirical results render SCOUT as a critical step towards next generation of spatially-aware VLMs.

Abstract

Visual Representation Learning

Explore similar work

SCOUT: Unlocking Enhanced Spatial Reasoning via Structured Chain-of-Thought and Multi-Objective Process Reward

HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models

Spa3R: Predictive Spatial Field Modeling for 3D Visual Reasoning