Organizations: University of Science and Technology of China, Anhui, China · Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · University of Chinese Academy of Sciences, Beijing, China · Stanford University, CA, USA
Advancing towards artificial superintelligence requires rich and intelligent perceptual capabilities. A critical frontier in this pursuit is overcoming the limited spatial understanding of Multimodal Large Language Models (MLLMs), where geometry information is essential. Existing methods often address this by rigidly injecting geometric signals into every input, while ignoring their necessity and adding computation overhead. Contrary to this paradigm, our framework endows the model with an awareness of perceptual insufficiency, empowering it to autonomously engage geometric features in reasoning when 2D cues are deemed insufficient. To achieve this, we first introduce an independent geometry input channel to the model architecture and conduct alignment training, enabling the effective utilization of geometric features. Subsequently, to endow the model with perceptual awareness, we curate a dedicated spatial-aware supervised fine-tuning dataset. This serves to activate the model's latent internal cues, empowering it to autonomously determine the necessity of geometric information. Experiments across multiple spatial reasoning benchmarks validate this approach, demonstrating significant spatial gains without compromising 2D visual reasoning capabilities, offering a path toward more robust, efficient and self-aware multi-modal intelligence.
Figures & tables
Figure 1: Adaptive Geometric Reasoning with GeoSense. (Top) Existing MLLMs typically adopt a static approach to 3D geometry, either ignoring it or rigidly fusing it, which leads to confusion in general tasks or failures in spatial reasoning. (Bottom) GeoSense introduces an adaptive mechanism ( Use with Sense ) that requests geometric features only when necessary. As shown in the radar chart, this flexibility allows GeoSense to achieve SOTA performance across both general visual benchmarks ( e.g. , MMBench ( Liu et al., 2024b ) , WeMath ( Qiao et al., 2024 ) ) and spatial reasoning tasks ( e.g. , VSI-Bench ( Yang et al., 2025a ) and MindCube ( Yin et al., 2025 ) ).
Figure 2: Architectural Overview of GeoSense. We integrate a 3D visual geometry encoder alongside a standard 2D visual encoder, both of which are kept frozen to preserve pretrained representations. Dedicated projection layers map these features into a unified embedding space for the LLM backbone. During inference, the model dynamically makes an “Internal Sense Decision” based on the 2D visual and textual prompt. If the latent state triggers a geometry request ( e.g. , via the <vggt> token), 3D embeddings are concatenated to the sequence for a second re-inference pass. During training, only the projection layers and the LLM backbone are optimized.
Figure 3: Sample distribution of the perception dataset by training objective. Consistent refers to samples with prediction invariance to 3D features, kept as is. Strategy A and Strategy B represent data where 3D geometric cues are essential and where they should be taken as noise, respectively.
Dataset Source
Subset / Type
Data Size
VSI-590K
Sampled Subset
55K
SophiaVL-R1
Chart
5K
General
15K
Knowledge
3K
Mantis-Instruct
LRV-Multi
13K
NLVR2
16K
Table 1: Composition of the Perception Tuning Data.
Spatial Reasoning Benchmarks
General Benchmarks
Overall
Model
FT-Data
Avg.
VSI-Bench
MMSI
MC-Tiny
BLINK
EmbSpatial
CVB(3D)
Avg.
MMBench
MME (P)
POPE
WeMath
Avg.
Rank
Qwen2.5-VL-3B ( Bai et al., 2025 )
-
43.4
27.0
28.6
37.6
33.1
62.3
71.8
53.3
76.6
1526.1
87.5
25.8
48.3
11
Qwen2.5-VL-7B ( Bai et al., 2025 )
-
50.5
32.3
26.8
36.0
55.9
71.8
80.1
57.8
82.6
1693.9
87.8
33.8
54.1
3
InternVL3-2B ( Chen et al., 2024b )
-
43.6
33.0
26.5
37.5
30.0
60.1
74.3
54.1
78.6
1610.2
88.9
22.4
48.8
9
ViLASR-7B ( Wu et al., 2025b )
73K
51.0
44.6
30.2
35.1
51.4
67.3
77.2
54.5
80.8
1634.4
84.8
25.3
52.8
4
SpaceR-sft-7B ( Ouyang et al., 2025 )
151K
49.9
41.6
27.4
38.0
49.6
66.9
75.7
58.9
82.8
1688.1
88.0
39.0
54.4
2
Table 2: Performance comparison on Spatial and General benchmarks. We report average scores for each benchmark collection, seperate benchmarks, and a total average for ranking. For spatial reasoning benchmarks, we include VSI-Bench ( Yang et al., 2025a ) , MMSI ( Yang et al., 2025d ) , MindCube (abbr. as MC) ( Yin et al., 2025 ) , BLINK ( Fu et al., 2024 ) , EmbSpatial ( Du et al., 2024 ) and CV-Bench(abbr. as CVB) ( Tong et al., 2024 ) for comprehensive spatial reasoning evaluation. For general benchmarks, we include MMBench ( Liu et al., 2024b ) , MME ( Fu et al., 2025 ) , POPE ( Li et al., 2023b ) , and WeMath ( Qiao et al., 2024 ) to evaluate the general multimodal reasoning capability. Note that we evaluate the 3D subset of CV-Bench and the MME perception score. * denotes fine-tuning on the same data as our model used.
VSI-Bench
MindCube-Tiny
Model
Injection
Avg.
Obj. Count
Obj. Size
Room Size
Rel. Dir.
Route Plan
Appr. Order
Rel. Dist.
Avg.
Rotation
Among
Around
Qwen2.5-vl-3B
0.00%
48.07
63.45
58.45
41.94
46.02
31.95
59.87
49.43
35.67
34.50
41.50
27.50
+ fusion VGGT (VG-LLM)
100.00%
52.34
64.67
63.96
39.06
61.61
35.56
61.00
54.78
30.58
34.00
30.00
29.75
+ alternative VGGT (GeoSense)
35.68%
54.86
69.43
66.64
58.26
52.85
34.02
71.52
53.66
45.08
31.50
49.17
45.75
Table 3: Ablation study on different geometry injection strategy. We compare the baseline against the additive fusion manner (VG-LLM) and our adaptive independent token insertion approach (GeoSense). All variants are trained with the same data to ensure fair comparison.
Table 7
Figure 5: Case study of internal sense decision. We present representative examples demonstrating how our model adaptively determines whether to trigger 3D geometric features according to the input and task. (a, b) : Rare cases where general tasks explicitly demand geometric embedding. (c) : Typical activation for spatial reasoning. (d, e) : Standard suppression for general visual inputs. (f) : Spatial queries solved effectively without geometric triggers.
Model
Overall
S2
D2
S3
D3
InternVL3-2B
29.25
21.88
27.30
25.58
39.15
Qwen2.5-VL-3B-Instruct
36.79
37.81
44.32
27.91
38.30
Qwen3-VL-2B-Instruct
30.44
19.69
31.62
23.02
43.62
Qwen3-VL-4B-Instruct
36.60
26.56
32.70
29.77
52.77
SpatialLadder-3B
36.86
34.06
34.59
33.72
43.40
VG-LLM-4B
31.13
34.06
36.76
17.67
37.02
Table 5: Comparison of different models on embodied task. Overall denotes the aggregate score, while S2/D2 represent static/dynamic 2D sub-task and S3/D3 represent static/dynamic 3D sub-task (reported in %). The best results are highlighted in bold , and the second-best results are underlined .
College of Computer Science and Artificial Intelligence, Fudan University · School of Computing and Information Systems, Singapore Management University · School of Computer Science and Technology, East China Normal University +2