DK-VLM: Geometry as Position for Spatial Reasoning in Vision-Language Models

Han Su; Meng Tian; Tianyu Huang; Mingyu Gao; Lennox Wei; Dongxu Li; Jianhua Han; Hang Xu

cs.CVOct 15, 2026

DK-VLM: Geometry as Position for Spatial Reasoning in Vision-Language Models

Authors: Han Su, Meng Tian, Tianyu Huang, Mingyu Gao, Lennox Wei, Dongxu Li, Jianhua Han, Hang Xu,

Organizations: Harbin Institute of Technology · Yinwang Intelligent Technology · Huawei

Abstract

Vision-language models still struggle to understand 3D spatial relations across views and update spatial judgments as the observer moves. A common approach fuses 3D reconstruction features into visual tokens, but aligning these features with pretrained semantic representations remains challenging under limited spatial supervision. We propose DK-VLM with Dual-Kernel Attention, which computes attention from both a visual token's input position and its geometric position in the scene. A semantic kernel retains the pretrained model's native attention and positional encoding, while a separate geometric kernel uses estimated 3D points and camera poses to directly modulate attention through point displacements and relative view transformations. Then, a learned gate fuses their outputs. Geometry thus enters attention as position rather than as fused reconstruction features. We further introduce Reference-Frame Composition (RC) supervision to train the model to update its reference frame across successive judgments. It provides stepwise turn and heading targets derived from verified single-step annotations. Among the compared methods, DK-VLM-8B achieves the highest average score across six spatial benchmarks and the highest overall scores on VSI-Bench (74.33) and ReVSI-32 (58.28). RC supervision further improves multi-step spatial reasoning, while the full model preserves general video understanding. Project page: https://sumuru789.github.io/DK-VLM-website/.

Figures & tables

Figure 1: Geometry as position in DK-VLM. Left: 2D input positions do not fully capture 3D relations. Right: DK-VLM retains the native positional encoding and introduces scene geometry into attention as position.

Figure 2: Geometry as Position via Dual-Kernel Attention. Selected decoder layers retain the backbone-native semantic kernel and introduce a geometric kernel over geometry-valid visual tokens. A learned gate fuses both outputs into the shared residual stream, while $\mathcal{L}_{\mathrm{pair}}$ directly supervises the pre-fusion geometric representation during training.

Figure 3: Reference-frame composition. Verified single-step relations sharing endpoints form a training chain with stepwise turn targets.

(a) VSI-Bench
Model	Overall	Obj. Count	Abs. Dist.	Obj. Size	Room Size	Rel. Dist.	Rel. Dir.	Route Plan	Appr. Order
Qwen2.5-VL-7B	29.30	25.20	10.50	36.40	29.60	38.40	38.00	29.80	26.80
Qwen3-VL-8B	57.90	67.60	47.00	76.30	61.90	58.00	51.00	35.00	66.30
VG-LLM-8B	62.20	71.40	56.80	69.00	69.10	67.90	83.20	47.40	32.50
VLM-3R-7B	60.90	70.20	49.40	69.20	67.10	65.40	80.50	45.40	40.10
Cambrian-S-7B	67.50	73.20	50.50	74.90	72.20	71.10	76.20	41.80	80.10

Table 1: Spatial reasoning on VSI-Bench and ReVSI-32. Both panels report overall scores and task results. Higher is better. Darker and lighter shading mark the best and second best results in each column. ∗ denotes results obtained using public checkpoints and official evaluation code.

Model	Avg.	VSI- Bench	MMSI- Bench	MindCube	ViewSpatial	SITE	CV- Bench
Qwen2.5-VL-7B	40.31	29.30	26.80	36.00	36.80	37.60	75.40
Qwen3-VL-8B	47.73	57.90	28.80	29.80	39.00	45.80	85.10
Cambrian-S-7B	47.28	67.50	25.80	39.60	40.90	33.00	76.90
VLM-3R-7B	45.40	60.90	27.90	40.00	40.50	31.30	71.80
VG-LLM-8B	51.05	62.20	30.00	36.10	45.80	50.50	81.70
GeoThinker-7B	60.43	68.50	31.70	83.60	41.90	54.80	82.10

Table 2: Results across six spatial benchmarks. Avg. is the mean of the six overall scores. Higher is better.

Model	VSI Route	ReVSI Route	MMSI Multi-Step	ViewSpatial Person
DK-VLM (Full)	60.31	47.67	34.34	61.72
DK-VLM w/o RC	51.21	43.48	29.19	56.32

Table 3: Effect of RC supervision on public tasks. Both models use Qwen3-VL-8B. Person denotes the average score across three person-perspective tasks in ViewSpatial: relative direction, object view orientation, and scene simulation relative direction.

Chain Length	w/o RC	w/ RC
2	10.6 / 2.2	52.2 / 45.6
3	9.0 / 0.3	50.0 / 37.5
4 (unseen)	8.2 / 0.0	35.4 / 20.6

Table 4: Performance across reasoning chain lengths. Each entry reports Progress / Chain Acc. (%). See Appendix A.8 for metric definitions.

Model	VideoMME	MVBench
Qwen3-VL-8B + SFT	64.59	68.42
DK-VLM-8B	66.22 (+1.63)	67.93 ( $-0.49$ )

Table 5: General video understanding. Parentheses show changes from the corresponding Qwen SFT baseline in percentage points.

Configuration	Overall
Full	74.33
w/o Point GTA	71.81
w/o Camera GTA	72.46
w/o Point & Camera GTA	70.02
w/o Pairwise Supervision	73.20

Table 6: Component ablations.

Model	w/o RC	w/ RC	$\bm{\Delta}$
Backbone	68.80	69.20	+0.40
DK-VLM	73.28	74.33	+1.05

Table 7: Effect of RC supervision.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Component	DK-VLM-7B	DK-VLM-8B
Backbone	8,292.17	8,767.12
LoRA	40.37	61.34
Geometric branch (including gates)	146.82	209.72
Pairwise prediction head	51.43	67.16
Total trainable parameters	238.62	338.22
Relative to backbone	2.88%	3.86%

Appendix

Table 8: Parameter counts of DK-VLM. Parameter counts are reported in millions (M). Trainable parameters include LoRA, the geometric branch, and the pairwise prediction head. The prediction head is used only during training and removed at inference. All counts exclude the frozen VGGT-

\Omega

frontend.

Chain Length	SPAR Delta	SPAR Heading	VSI Angle	VSI Action	Total
2	80	80	80	80	320
3	80	80	80	80	320
4	80	80	80	8	248

Appendix

Table 9: Composition of the RC evaluation set. Counts denote evaluation examples for each target type and chain length.

Injection Layers	Layer Indices	VSI-Bench
3	6, 18, 33	62.28
5	6, 12, 18, 27, 33	63.05
7	6, 9, 12, 18, 24, 27, 33	63.11

Appendix

Table 10: Effect of the number of injection layers on VSI-Bench.

Placement	Layer Indices	VSI-Bench
Early	6, 9, 12, 15, 18	62.43
Distributed	6, 12, 18, 27, 33	63.05
Late	18, 21, 24, 27, 30	61.29

Appendix

Table 11: Effect of injection layer placement on VSI-Bench. All configurations use five injection layers.

Geometry Frontend	Overall
VGGT- $\Omega$	74.33
VGGT	73.91
MASt3R-SLAM	73.64
Pi3X	73.62
DA3-Large	72.99

Appendix

Table 12: Geometry frontend comparison on VSI-Bench. All configurations use the same DK-VLM-8B checkpoint and 32 input frames, without retraining.

Input Frames	Overall
8	65.29
16	69.94
32	74.33
64	75.13
128	75.32

Appendix

Table 13: Effect of input frame count on VSI-Bench. All configurations use the same DK-VLM-8B checkpoint and VGGT-

\Omega

frontend, without additional training.

Configuration	VSI-Bench Overall $\uparrow$
Qwen3-VL-8B + SFT	59.48
3D positional embeddings at input	59.21
3D positional embeddings at five layers	59.74
Single-kernel GTA (three layers)	45.98
Single-kernel GTA (five layers)	39.14
DK-VLM-8B (five layers)	62.78

Appendix

Table 14: Comparison of geometry injection strategies on VSI-Bench. All configurations omit the pairwise geometric loss.

Visual Spatial Reasoning

Visual spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and physical change from images and video.

See all 251 papers

Reinforcing Dual-Path Reasoning in Spatial Vision Language Models
Jun 16, 2026Yatai Ji, An-Chieh Cheng, Yang Fu +13Visual Spatial Reasoning 3D Spatial Reasoning
The University of Hong Kong · NVIDIA · University of California, San Diego
Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models
Mar 18, 2026Kevin Qu, Haozhe Qi, Mihai Dusmanu +33D VLMs 3D Spatial Reasoning
Microsoft Spatial AI Lab · ETH Zurich · EPFL
Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning
May 28, 2026Chun-Hsiao Yeh, Shengyi Qian, Manchen Wang +3Vision-Language Models 3D Spatial Reasoning
FAIR at Meta · UC Berkeley · HKU

Jun 16, 2026J/K move · Enter open · S save

Reinforcing Dual-Path Reasoning in Spatial Vision Language Models

Yatai Ji, An-Chieh Cheng, Yang Fu, Yukang Chen+12

The University of Hong Kong · NVIDIA · University of California, San Diego

Visual Spatial Reasoning 3D Spatial Reasoning VLM Reasoning Visually Grounded Reasoning RL for VLMs

Open paper AskPDF arXiv

Spatial VLMs have made substantial progress in geometric perception, yet complex spatial reasoning requiring multi-step inference over depth, distance, and scene relations remains challenging. Moreover, different spatial queries call for fundamentally different strategies: some are best addressed through purely linguistic, step-by-step deduction, while others require explicit 3D grounding before quantitative inference. We present Dual-Path Spatial Reasoning via Reinforcement Learning for Spatial VLMs (SR-REAL), a unified framework that equips a spatial VLM with two complementary reasoning paths: Language-Only Reasoning (LOR), which performs step-by-step linguistic deduction, and Detect-Then-Reason (DTR), which detects 3D geometric cues (e.g., centers or bounding boxes) via region tokens before explicit geometric inference. SR-REAL begins with a cold-start supervised fine-tuning stage that constructs LOR and DTR chain-of-thought supervision and exposes a region-to-3D interface, followed by RL that optimizes the policy model with accuracy and format rewards; for DTR, a discrete center-based detection reward further refines geometric alignment. Across diverse spatial benchmarks, SR-REAL significantly outperforms spatial VLM baselines: (i) a single RL-trained model supports both reasoning paths, with DTR excelling in region-aware tasks through precise 3D localization and LOR enhancing general spatial reasoning; (ii) jointly training both paths fosters mutual reinforcement; (iii) high-quality, blended cold-start data is crucial for stable RL optimization; and (iv) the model generalizes across datasets and domains without per-task tuning, demonstrating positive transfer between LOR and DTR.

Abstract

Visual Spatial Reasoning

Explore similar work

Reinforcing Dual-Path Reasoning in Spatial Vision Language Models

Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models

Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning