Reinforcement Learning with Verifiable Rewards ( RLVR ) has emerged as a transformative paradigm for enhancing the reasoning capabilities of Large Language Models ( LLMs), yet its potential in 3D scene understanding remains under-explored. Existing approaches largely rely on Supervised Fine-Tuning ( SFT), where the token-level cross-entropy loss acts as an indirect proxy for optimization, leading to a misalignment between training objectives and task performances. To bridge this gap, we present Reinforcement Fine-Tuning for Video-based 3D Scene Understanding (3D-RFT ), the first framework to extend RLVR to video-based 3D perception and reasoning. 3D-RFT shifts the paradigm by directly optimizing the model towards evaluation metrics. 3D-RFT first activates 3D-aware Multi-modal Large Language Models ( MLLM s) via SFT, followed by reinforcement fine-tuning using Group Relative Policy Optimization ( GRPO) with strictly verifiable reward functions. We design task-specific reward functions directly from metrics like 3D IoU and F1-Score to provide more effective signals to guide model training. Extensive experiments demonstrate that 3D-RFT-4B achieves state-of-the-art performance on various video-based 3D scene understanding tasks. Notably, 3D-RFT-4B significantly outperforms larger models (e.g., VG LLM-8B) on 3D video detection, 3D visual grounding, and spatial reasoning benchmarks. We further reveal good properties of 3D-RFT such as robust efficacy, and valuable insights into training strategies and data impact. We hope 3D-RFT can serve as a robust and promising paradigm for future development of 3D scene understanding.
Figures & tables
Figure 1: Comparison between SFT and 3D-RFT training paradigms. Left: Standard Supervised Fine-Tuning (SFT) relies on a per-token Cross-Entropy loss which acts as an indirect proxy, leading to a gap between training objectives and final evaluation metrics. Middle: Identical output formats for 3D Video Detection, Grounding, and Spatial Reasoning tasks. Right: 3D-RFT utilizes a Scalar Reward (Policy Gradient) derived directly from evaluation metrics ( e.g . , F1-Score, 3D IoU, and Accuracy) through a decoding and parsing module, ensuring the model directly optimizes for the final task performance.
Figure 2: Overview of the 3D-RFT training framework. The process consists of two main stages: (1) SFT Warm Up : Initial training of the 3D-aware VLM using SFT data to establish a baseline policy. (2) RL Training : The Policy Model generates completions for video-based problems. A Verifiable Reward is calculated based on Format Reward (adherence to structured output) and Task Reward (performance in 3D Video Detection, 3D Visual Grounding, and Spatial Reasoning using metrics like 3D IoU and F1 Score). The model is optimized via Policy Gradient while maintaining a KL Divergence constraint relative to the frozen Reference Model.
Model
chair
cabinet
table
bin
couch
bed
bathtub
toilet
20 Common Classes
P25
R25
F125
4-Frame Setting
Qwen2.5-VL-3B ( Zheng et al., 2025b )
37.7
10.2
35.0
23.1
39.0
64.8
32.4
68.8
32.6
27.9
30.0
Qwen2.5-VL-7B ( Zheng et al., 2025b )
41.2
11.6
36.5
30.2
41.1
68.2
36.6
68.7
34.6
31.0
32.5
VG LLM-4B ( Zheng et al., 2025b )
49.7
13.1
41.3
39.2
44.6
71.2
33.5
83.4
41.7
35.7
38.2
VG LLM-8B ( Zheng et al., 2025b )
54.0
17.1
46.5
39.8
47.0
74.1
42.1
82.5
43.4
39.6
41.2
Table 1: Quantitative results on ScanNetDetection. In this table, we present the comparison with baseline models and report the performance improvement between the sft baseline (VG LLM-4B) and 3D-RFT-4B (ours).
Table 3: Ablation study of training strategies and 3D priors. We report accuracy at IoU thresholds 0.25 and 0.5.
Model
Avg.
Numerical Answer
Multiple-Choice Answer
Obj. Count
Abs. Dist.
Obj. Size
Room Size
Rel. Dist.
Rel. Dir.
Route Plan
Appr. Order
Base Models
GPT-4o ( Hurst et al., 2024 )
34.0
46.2
5.3
43.8
38.2
37.0
41.3
31.5
28.5
Gemini-2.5 Pro ( Comanici et al., 2025 )
51.5
43.8
34.9
64.3
42.8
61.1
47.8
45.9
71.3
LLaVA-Video-7B ( Zhang et al., 2025c )
35.6
48.5
14.0
47.8
24.2
43.5
42.4
34.0
30.6
Qwen2.5-VL-7B ( Bai et al., 2025b )
32.7
34.5
19.4
47.6
40.8
32.8
24.5
32.5
29.4
Table 4: Quantitative results on VSI-Bench. We include zero-shot results of base models, and test results of models after sft and rft .
Figure 7
Figure 6: Training Dynamics Analysis (3D Video Detection).
Figure 7: Training Dynamics Analysis (3D Spatial Reasoning).
Figure 8: Conceptual 1D Boundary Optimization Space. We illustrate a simplified 1D proxy to illustrate the fundamental mathematical mismatch between token classification and geometric optimization. The axes represent the predicted left ( x ) and right ( y ) boundaries of a 1D interval.
Figure 9: Distribution of 3D IoU in 3D Visual Grounding. A comparison of the raw 3D IoU distributions on the validation set between the baseline model trained with SFT and SFT.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
VSI-Bench
MindCube
MMSI-Bench
All-Angles
ERQA
VST-3B ( Yang et al., 2025b )
57.7
35.9
31.3
-
-
3D-RFT-4B (ours)
62.8
36.4
29.9
41.6
34.2
VST-7B ( Yang et al., 2025b )
61.2
39.7
34.8
-
-
3D-RFT-8B (ours)
64.0
42.6
28.0
43.1
38.5
Appendix
Table 5: Additional results across more spatial reasoning benchmarks and different model scales.
Figure 10: Training Dynamics of 3D Visual Grounding. We visualize the training curves for the total reward (left), the 3D IoU reward (middle), and the Frame Alignment reward (right). The curves show stable convergence and correlation with the optimization objectives.
Figure 11: Evaluation results on MindCube-Tiny .
Figure 12: Qualitative comparison of 3D object detection (20 common classes) results. We compare the predictions of the baseline VG LLM-4B against our 3D-RFT-4B model.
Figure 13: Qualitative results of 3D Visual Grounding using our 3D-RFT-4B . Each block shows the grounding text followed by 10 video frames visualizing the predicted 3D bounding boxes (green) and ground truth (red).
Figure 14: Qualitative example on VSI-Bench (absolute distance).
Figure 15: Qualitative example on VSI-Bench (relative direction).
Figure 16: Qualitative example on VSI-Bench (route plan).
Large language models are increasingly being applied to 3D scene understanding. However, existing 3D-aware LLMs largely operate as black boxes: they produce bounding boxes or textual responses without exposing the reasoning process, and they typically depend on pre-trained 3D detectors for object proposals. We present Scene-R1, a video-grounded framework for 3D scene reasoning that avoids both pre-trained 3D object detectors and dense point-cloud backbones. The framework combines reinforcement-learning-based reasoning with a two-stage grounding pipeline. In the first stage, Scene-R1 performs temporal grounding by identifying the video segments most relevant to an open-ended query. In the second stage, it grounds the target in image space by predicting a 2D bounding box. This prediction is then refined into a precise 3D localization using high-quality candidates from a zero-shot segmentation module, enabling fine-grained geometric reasoning without detector-based proposals. Scene-R1 can also be adapted to 3D visual question answering, generating free-form answers directly from RGB-D video. Our training pipeline uses lightweight 2D supervision projected from sparse scans, avoiding the computational overhead required to train 3D instance segmentation models. Across multiple datasets, Scene-R1 outperforms existing open-vocabulary baselines while also providing interpretable intermediate reasoning steps. These results suggest that reinforcement-learning-based reasoning over RGB-D video can support accurate 3D scene understanding with task-level supervision. The training, inference, and 3D refinement code is available at https://github.com/819346675/Scene-R1.
Junjie Wang, Zhihao Yuan, Shuyi Jiang +5
FNii-Shenzhen, CUHKSZ · SSE, CUHKSZ · Singapore University of Technology and Design +1
Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning. To bridge this gap, we present VLM-IE3D, a unified framework that enhances the 3D spatial awareness of VLMs by equipping them with both implicit and explicit 3D geometries learned from RGB videos. Our VLM-IE3D introduces Implicit Geometry Tokens (IGTs) that capture high-level geometric priors from input videos, as well as complementary Explicit Geometry Tokens (EGTs) that encode detailed geometric structures from reconstructed 3D attributes. On top of that, VLM-IE3D comes with a 3D-aware adapter that effectively fuses the two types of geometric representations with 2D visual cues. This RGB-only design injects strong 3D inductive biases for fine-grained spatial understanding and reasoning without requiring any additional 3D inputs. Extensive experiments show that VLM-IE3D achieves superior performance consistently across various 3D tasks including 3D video detection, 3D visual grounding, 3D dense captioning, and spatial reasoning. Code and models are available at https://github.com/Vegetebird/VLM-IE3D.
Wenhao Li, Xueying Jiang, Quanhao Qian +4
1Nanyang Technological University · 2DAMO Academy, Alibaba Group · 3HuPan Lab +1
Multimodal Large Language Models (MLLMs) have made impressive progress in connecting vision and language, but they still struggle with spatial understanding and viewpoint-aware reasoning. Recent efforts aim to augment the input representations with geometric cues rather than explicitly teaching models to reason in 3D space. We introduce Loc3R-VLM, a framework that equips 2D Vision-Language Models with advanced 3D understanding capabilities from monocular video input. Inspired by human spatial cognition, Loc3R-VLM relies on two joint objectives: global layout reconstruction to build a holistic representation of the scene structure, and explicit situation modeling to anchor egocentric perspective. These objectives provide direct spatial supervision that grounds both perception and language in a 3D context. To ensure geometric consistency and metric-scale alignment, we leverage lightweight camera pose priors extracted from a pre-trained 3D foundation model. Loc3R-VLM achieves state-of-the-art performance in language-based localization and outperforms existing 2D- and video-based approaches on situated and general 3D question-answering benchmarks, demonstrating that our spatial supervision framework enables strong 3D understanding. Project page: https://kevinqu7.github.io/loc3r-vlm