3D spatial reasoning from video remains challenging for VLMs because videos provide sequential, view-local observations rather than persistent metric scene records. Many questions must be answered in a query-specified perspective rather than the observed camera viewpoint, and prompting with a fixed set of RGB views may miss sparse or occluded visual cues. To address these gaps, we introduce BEV3SR, a training-free and backbone-agnostic framework for RGB-D videos with camera parameters. BEV3SR turns video observations into queryable metric bird's-eye-view (BEV) scene memory and formalizes each query into an evidence-aware spatial task. The Spatial Task Formalizer specifies the query's reference frame, spatial constraints, objective, and evidence requirement. The evidence requirement determines the appropriate reasoning pathway. A query treated as requiring geometric evidence is handled by the Geometric Reasoner, which deterministically computes the answer using the metric BEV records in the query-specified reference frame. Otherwise, a query treated as requiring visual evidence is passed to the Visual Evidence Aligner, which uses BEV-indexed object records and their associated supporting views to retrieve object-consistent and scene-context RGB frames for VLM grounding. Within a standard 16-frame budget, BEV3SR consistently improves different VLM backbones across multiple 3D scene QA benchmarks, delivering large gains on spatially intensive questions and significant improvements on open-ended scene QA. These results demonstrate that BEV3SR offers an effective, efficient, and interpretable framework for explicit 3D spatial reasoning from video.
Figures & tables
Figure 1: Three limitations of the VLM baseline and how the proposed BEV3SR addresses them. The baseline uses Qwen3-VL-8B [ 4 ] prompted with 16 uniformly sampled RGB frames.
Figure 2: Overview of the BEV3SR framework.
VLM
Method
SQA3D (test)
ScanQA (val)
EM
EM-R
EM
B-4
M
R
C
Qwen3-VL-8B [ 4 ]
Baseline
51.6
55.0
27.8
15.9
15.8
41.9
82.6
BEV3SR
62.6
64.8
30.6
16.0
18.5
46.4
95.6
InternVL3.5-8B [ 33 ]
Baseline
52.9
55.2
28.2
14.2
17.4
44.5
90.4
BEV3SR
61.2
62.5
30.2
14.7
16.9
45.4
90.7
LLaVA-OneVision-1.5-8B [ 2 ]
Baseline
49.2
50.1
22.1
4.2
13.3
34.8
66.0
Table 1: Baseline vs. BEV3SR across different VLMs. We report EM (exact match), EM-R (refined exact match), BLEU-4 (B-4), METEOR (M), ROUGE (R), and CIDEr (C).
Model
Avg.
Num.
MC
Obj. Cnt.
Abs. Dist.
Obj. Size
Room Size
Rel. Dist.
Rel. Dir.
2D MLLM
SpatialMLLM-4B [ 34 ]
49.3
65.3
34.8
63.1
45.1
41.3
46.2
VG-LLM-8B [ 48 ]
52.3
67.9
37.7
58.6
62.0
46.6
40.7
Open source
InternVL3.5-8B [ 33 ]
57.3
70.2
42.7
67.4
60.9
54.6
48.1
Table 3: Comparison on the static spatial subset of VSI-Bench.
Method
Backbone
ScanQA
SQA3D
SpatialMind [ 39 ]
Qwen2.5-VL-72B [ 5 ]
B-1 38.0
EM 39.2
CoV [ 46 ]
GPT-4.1 [ 1 ] / Gemini [ 11 ]
EM 31.9
EM 51.1
BEV3SR
Qwen3-VL-8B [ 4 ]
EM 30.6 / B-1 41.0
EM 62.6
Table 4: Contextual comparison with training-free methods.
Method
EM
Δ
EM-R
Δ
Baseline
43.2
–
44.0
–
+ BEV
41.7
−1.5
42.3
−1.7
+ GR
65.5
+22.3
65.9
+21.9
Table 5: Ablation studies on the SQA3D validation set: (a) effectiveness of GR on the geometry-solvable subset; (b) end-to-end contribution of GR and VEA on the full set.
Method
EM
B-1
B-2
B-3
B-4
M
R
C
Baseline
27.8
35.4
23.2
18.2
15.9
15.8
41.9
82.6
VEA
30.6
41.0
27.0
20.7
16.0
18.5
46.4
95.6
Δ
+2.8
+5.6
+3.8
+2.5
+0.1
+2.7
+4.5
+13.0
Table 6: Ablation study for effectiveness of Visual Evidence Aligner (VEA) on ScanQA validation set.
Table 8
Reference-frame Regime
#Q
EM
Good position, good heading ( ep≤1m , eθ≤30∘ )
547
76.97
Good position, bad heading ( ep≤1m , eθ>30∘ )
169
58.58
Bad position, good heading ( ep>1m , eθ≤30∘ )
142
65.49
Bad position, bad heading ( ep>1m , eθ>30∘ )
262
51.15
Table 9: QA accuracy under different reference-frame error regimes on SQA3D val.
3D spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and motion to understand and act in three-dimensional scenes.
We introduce Thinking with Spatial Code, a framework that transforms RGB video into explicit, temporally coherent 3D representations for physical-world visual question answering. We highlight the empirical finding that our proposed spatial encoder can parse videos into structured spatial code with explicit 3D oriented bounding boxes and semantic labels, enabling large language models (LLMs) to reason directly over explicit spatial variables. Specifically, we propose the spatial encoder that encodes image and geometric features by unifying 6D object parsing and tracking backbones with geometric prediction, and we further finetuning LLMs with reinforcement learning using a spatial rubric reward that encourages perspective-aware, geometrically grounded inference. As a result, our model outperforms proprietary vision-language models on VSI-Bench, setting a new state-of-the-art. Code is available at https://github.com/Beckschen/spatialcode.
Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning. To bridge this gap, we present VLM-IE3D, a unified framework that enhances the 3D spatial awareness of VLMs by equipping them with both implicit and explicit 3D geometries learned from RGB videos. Our VLM-IE3D introduces Implicit Geometry Tokens (IGTs) that capture high-level geometric priors from input videos, as well as complementary Explicit Geometry Tokens (EGTs) that encode detailed geometric structures from reconstructed 3D attributes. On top of that, VLM-IE3D comes with a 3D-aware adapter that effectively fuses the two types of geometric representations with 2D visual cues. This RGB-only design injects strong 3D inductive biases for fine-grained spatial understanding and reasoning without requiring any additional 3D inputs. Extensive experiments show that VLM-IE3D achieves superior performance consistently across various 3D tasks including 3D video detection, 3D visual grounding, 3D dense captioning, and spatial reasoning. Code and models are available at https://github.com/Vegetebird/VLM-IE3D.
Wenhao Li, Xueying Jiang, Quanhao Qian +4
1Nanyang Technological University · 2DAMO Academy, Alibaba Group · 3HuPan Lab +1
Multimodal Large Language Models (MLLMs) have made impressive progress in connecting vision and language, but they still struggle with spatial understanding and viewpoint-aware reasoning. Recent efforts aim to augment the input representations with geometric cues rather than explicitly teaching models to reason in 3D space. We introduce Loc3R-VLM, a framework that equips 2D Vision-Language Models with advanced 3D understanding capabilities from monocular video input. Inspired by human spatial cognition, Loc3R-VLM relies on two joint objectives: global layout reconstruction to build a holistic representation of the scene structure, and explicit situation modeling to anchor egocentric perspective. These objectives provide direct spatial supervision that grounds both perception and language in a 3D context. To ensure geometric consistency and metric-scale alignment, we leverage lightweight camera pose priors extracted from a pre-trained 3D foundation model. Loc3R-VLM achieves state-of-the-art performance in language-based localization and outperforms existing 2D- and video-based approaches on situated and general 3D question-answering benchmarks, demonstrating that our spatial supervision framework enables strong 3D understanding. Project page: https://kevinqu7.github.io/loc3r-vlm