Organizations: National Engineering Research Center of Visual Technology, National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University · Meituan Inc. · National Biomedical Imaging Center, Peking University
Multimodal large language models (MLLMs) have exhibited remarkable performance in various visual tasks, yet still struggle with spatial reasoning. Recent efforts mitigate this by injecting geometric features from 3D foundation models, but rely on static single-layer extractions. We identify that such an approach induces a task misalignment bias: the geometric features naturally evolve towards 3D pretraining objectives, which may contradict the heterogeneous spatial demands of MLLMs, rendering any single layer fundamentally insufficient. To resolve this, we propose GeoAlign, a novel framework that dynamically aggregates multi-layer geometric features to realign with the actual demands. GeoAlign constructs a hierarchical geometric feature bank and leverages the MLLM's original visual tokens as content-aware queries to perform layer-wise sparse routing, adaptively fetching the suitable geometric features for each patch. Extensive experiments on VSI-Bench, ScanQA, and SQA3D demonstrate that our compact 4B model effectively achieves state-of-the-art performance, even outperforming larger existing MLLMs.
Figures & tables
Figure 1: Task misalignment bias. Feature evolution progressively aligns with the pretraining tasks, thus many geometric features valuable for spatial reasoning tasks are distributed in the preceding layers.
Feature Source
Route Plan.
Appr. Order
Room Size
Obj. Size
Obj. Count
Rel. Dist.
Abs. Dist.
Rel. Dir.
Layer-12
47.9
83.5
74.0
74.4
69.5
66.3
55.1
83.2
Layer-20
43.2
79.9
72.0
73.9
69.9
67.5
59.0
89.0
Δ
-4.7
-3.6
-2.0
-0.5
+0.4
+1.2
+3.9
+5.8
Table 1: Impact of feature layer selection. We evaluate Qwen2.5-VL finetuned with geometric features from two distinct layers (Layer-12 and Layer-20) of the VGGT encoder on VSI-Bench. The Δ row indicates the performance difference (Layer-20 minus Layer-12). The better results for each task are marked in bold .
Figure 2: Overview of the GeoAlign framework. We augment the 2D visual features with aggregated geometric features, which are adaptively selected and fused from a hierarchical feature bank built upon the 3D geometry encoder. In this dynamic routing mechanism, the original visual tokens act as content-aware queries, ensuring that the injected geometric features properly align with diverse spatial reasoning demands.
Models
Avg.
Numerical
Multiple-Choice
Obj. Cnt.
Abs. Dist.
Obj. Size
Room Size
Rel. Dist.
Rel. Dir.
Route Plan
Appr. Order
Proprietary Models
GPT-4o
34.0
46.2
5.3
43.8
38.2
37.0
41.3
31.5
28.5
Gemini-1.5-Pro
45.4
56.2
30.9
64.1
43.6
51.3
46.3
36.0
34.6
Gemini-2.5-Pro
53.6
46.0
37.4
68.7
54.4
62.0
43.9
47.4
68.8
Open-Sourced Models
Table 2: Evaluations on VSI-Bench for spatial reasoning. We compare our proposed GeoAlign model against representative proprietary models, open-sourced models, and spatial-enhanced models. Following the benchmark guidelines, we report the accuracy for multiple-choice tasks, and the mean relative accuracy for numerical tasks. The best performance of each column is marked in bold , and the second best is underlined .
Models
ScanQA
SQA3D
B-4
M
R
C
EM-1
Task-Specific Models
ScanQA
10.1
13.1
33.3
64.9
47.2
SQA3D
11.2
13.5
34.5
-
46.6
3D-VisTA
10.4
13.9
35.7
69.6
48.5
3D/2.5D-Input Models
Table 3: Evaluation on ScanQA and SQA3D for 3D scene understanding. In this table, “B-4”, “M”, “R”, “C”, and “EM-1” denote BLEU-4, METEOR, ROUGE-L, CIDEr, and exact match accuracy, respectively. Among video-input models, the best performance is in bold , and the second best is underlined .
Figure 3: Qualitative visualization of the dynamic routing mechanism. We visualize the mean routing weights ( α , presented as percentages) assigned to each layer of the geometric feature bank across the entire visual sequence. For distinct visual inputs, the routing distributions exhibit significant variations.
Setting
Avg.
Obj. Cnt.
Abs. Dist.
Obj. Size
Room Size
Rel. Dist.
Rel. Dir.
Route Plan
Appr. Order
Geometric Feature Usage
2D-Only
66.8
70.3
49.6
73.5
68.3
64.2
78.1
47.4
82.7
Single
69.3
71.0
58.2
73.7
72.7
69.4
87.2
40.7
81.7
Mean
70.5
71.5
58.3
73.7
73.4
70.7
87.1
47.4
81.9
Dynamic
71.4
71.2
59.8
74.1
75.0
72.0
87.1
50.5
81.7
Geometric Feature Selection
Table 4: Ablation studies. We evaluate key architectural settings: (1) Geometric Feature Usage, comparing our dynamic routing against baselines using no geometric features, single-layer geometric features, or mean geometric features; (2) Geometric Feature Selection, assessing the construction of geometric feature bank using different VGGT layers (uniformly sampled, former half, latter half); (3) Injection Position, investigating injection stages from before the LLM to various internal layers; (4) Sparsity Hyperparameter, determining the number of selected layers ( K ) in our dynamic routing; (5) Projection and Fusion Mechanism, comparing our shared projector and direct residual addition against split, modulated, or gated variants. The best is in bold , and the second best is underlined .
3D spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and motion to understand and act in three-dimensional scenes.