Multimodal Large Language Models (MLLMs) that directly process RGB inputs for tasks like 3D localization and navigation have shown remarkable potential. However, we argue that these RGB-only approaches are fundamentally flawed in their ability to generalize across cameras. By ignoring camera parameters, they entangle an object's physical properties with the camera's perspective, creating an irresolvable ambiguity. We show this leads MLLMs to overfit to the training camera distribution, rather than learning true and generalizable 3D geometric principles. To address this, we propose Camera-Aware MLLM framework for spatial MLLMs. It learns generalizable spatial reasoning by: (i) injecting camera intrinsics via a dense embedding that conditions each visual token; (ii) introducing a camera-aware data augmentation strategy that synthetically varies camera parameters, forcing the model to disentangle camera properties from scene content; and (iii) distilling geometric priors from a 3D vision foundation model. Extensive experiments demonstrate that camera-aware MLLMs substantially outperform their naive counterparts, particularly in cross-camera generalization tests on spatially-grounded tasks, indicating that camera-awareness is not only beneficial but also a prerequisite for robust and generalizable spatial intelligence in MLLMs.
Figures & tables
Figure 1: Illustration of the inherent geometric ambiguity in RGB-only spatial reasoning. The same 2D image can result from different 3D scenes: a nearby object with a wide-angle lens can appear identical to a distant object with a telephoto lens. This ambiguity makes generalizable 3D localization from a single RGB image an ill-posed problem, especially when camera intrinsics are unknown.
Model
Train Data
Evaluate Data
31 common classes
P0.25
R0.25
F10.25
Qwen2.5 - VL 3B
ScanNet
ScanNet-val
47.5
44.2
45.7
ScanNet ARKitScenes 3RScan Matterport3D SUN - RGBD Objectron
ScanNet-val
38.4
33.3
35.4
ScanNet
ScanNet-val x0.8
26.4
22.9
24.3
ScanNet
ScanNet-val x1.2
33.0
30.5
31.6
VG - LLM 4B
ScanNet
ScanNet-val
48.3
45.0
46.5
Table 1: Generalization failure of camera-agnostic MLLMs. 3D object detection performance drops when trained on mixed data sources or evaluated on resized images, exposing a fundamental lack of robustness and generalization.
Figure 2: Visualization of generalization failure in a camera-agnostic MLLM (finetuned Qwen2.5-VL). Simply resizing the input image during inference induces a systematic shift in 3D localization.
Figure 3: Multi-modal distribution of camera intrinsics in mixed datasets.
Figure 4: The proposed Camera-Aware MLLM Framework. (a) The overview of the architecture, where (b) Geometry-Aware Visual Encoder (GAVE) injects camera-awareness and 3D geometric priors into the MLLM.
Figure 5: Illustration of camera-aware geometric augmentation.
Figure 6: Cross-camera generalization on spatially-grounded tasks. While camera-agnostic MLLMs (Qwen2.5-VL, VG-LLM) fail catastrophically on altered camera geometries by rescaling, our method maintains robust performance, proving its ability to generalize across cameras.
Methods
Avg.
Low
Depth-OC
Depth-OC-MV
Depth-OO
Depth-OO-MV
Dist-OC
Dist-OC-MV
Dist-OO
Dist-OO-MV
Medium
PosMatch
CamMotion
ViewChgI
High
DistI-OO
DistI-OO-MV
ObjRel-OC-MV
ObjRel-OO
ObjRel-OO-MV
SpImag-OC
SpImag-OC-MV
SpImag-OO
SpImag-OO-MV
SPAR-Bench (tiny)
Human Level
67.27
55.31
72.75
74.25
28.75
36.25
78.25
52.25
66.5
33.50
72.32
92
64
60.97
76.22
80
94
70
92
80
78
82
50
60
GPT-4o
36.39
29.25
53.80
45.00
15.00
13.60
37.40
34.40
23.40
24.40
24.93
30
16
28.80
45.11
64
64
58
46
46
32
44
30
22
Claude-3.7-Sonnet
21.77
25.43
41.00
45.40
11.20
12.20
42.60
19.60
26.00
5.40
7.33
16
6
0.00
23.33
40
48
22
36
14
12
20
6
12
Qwen2-VL-72B
35.62
35.28
45.40
49.80
13.80
10.00
54.60
49.40
36.80
22.40
23.39
42
18
10.16
40.00
60
68
50
38
44
18
28
18
36
Qwen2.5-VL-72B
39.40
35.35
53.20
46.80
17.80
29.00
49.60
57.40
14.40
14.60
23.05
40
16
13.16
48.44
74
74
60
56
50
20
34
24
44
Table 2: Comparison of MLLMs’ spatial reasoning performance on SPAR-Bench.
Obj. Count
Abs. Dist.
Obj. Size
Room Size
Rel. Dist.
Rel. Dir.
Route Plan
Appr. Order
Avg.
Numerical Answer
Multiple-Choice Answer
Proprietary Generalist MLLMs:
GPT-4o
34.0
46.2
5.3
43.8
38.2
37.0
41.3
31.5
28.5
GPT-5-Chat-0807-global
49.1
52.4
36.6
68.0
55.5
49.9
47.4
33.0
49.8
Gemini-1.5-Flash
42.1
49.8
30.8
53.5
54.4
37.7
41.0
31.5
37.8
Gemini-1.5-Pro
45.4
56.2
30.9
64.1
43.6
51.3
46.3
36.0
34.6
Table 3: Comparison of MLLMs’ spatial reasoning performance on VSI-Bench.
Model
CV-Bench-3D
BLINK-Spatial
Avg.
Depth
Dist.
Avg.
Depth
Spatial Rel.
Multi. View
GPT-4V
69.1
-
-
62.8
60.0
72.7
55.6
GPT-4o
83.0
-
-
67.6
74.2
69.2
59.4
Gemini-2.5-Pro
90.8
91.0
90.7
71.6
87.9
91.6
35.3
Qwen2.5-VL-3B
69.5
74.5
66.2
64.7
68.5
81.1
44.4
Qwen2.5-VL-7B
77.3
84.5
76.5
73.8
79.0
86.7
55.6
Table 4: Comparison of model performance on various spatial understanding datasets.
Components
20 Common Classes
Ray Emb.
Geom. Aug.
Prior Dist.
P0.25
R0.25
F10.25
-
-
-
40.8
37.7
39.1
✓
-
-
42.9
39.7
41.2
-
✓
-
43.8
40.4
42.0
-
-
✓
45.7
41.1
43.1
✓
-
✓
46.2
42.8
44.3
Table 5: Ablation study on the components of our Camera-Aware MLLM framework. Performance measured on ScanNet-val x1.2 to test cross-camera generalization.
Figure 7: Qualitative comparison of 3D object detection results. Our camera-aware MLLM can robustly generalize to new camera setups.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Table 6: The prompt for 3D video object detection.