Reinforcement Learning with Verifiable Rewards ( RLVR ) has emerged as a transformative paradigm for enhancing the reasoning capabilities of Large Language Models ( LLMs), yet its potential in 3D scene understanding remains under-explored. Existing approaches largely rely on Supervised Fine-Tuning ( SFT), where the token-level cross-entropy loss acts as an indirect proxy for optimization, leading to a misalignment between training objectives and task performances. To bridge this gap, we present Reinforcement Fine-Tuning for Video-based 3D Scene Understanding (3D-RFT ), the first framework to extend RLVR to video-based 3D perception and reasoning. 3D-RFT shifts the paradigm by directly optimizing the model towards evaluation metrics. 3D-RFT first activates 3D-aware Multi-modal Large Language Models ( MLLM s) via SFT, followed by reinforcement fine-tuning using Group Relative Policy Optimization ( GRPO) with strictly verifiable reward functions. We design task-specific reward functions directly from metrics like 3D IoU and F1-Score to provide more effective signals to guide model training. Extensive experiments demonstrate that 3D-RFT-4B achieves state-of-the-art performance on various video-based 3D scene understanding tasks. Notably, 3D-RFT-4B significantly outperforms larger models (e.g., VG LLM-8B) on 3D video detection, 3D visual grounding, and spatial reasoning benchmarks. We further reveal good properties of 3D-RFT such as robust efficacy, and valuable insights into training strategies and data impact. We hope 3D-RFT can serve as a robust and promising paradigm for future development of 3D scene understanding.
Figures & tables
Figure 1: Comparison between SFT and 3D-RFT training paradigms. Left: Standard Supervised Fine-Tuning (SFT) relies on a per-token Cross-Entropy loss which acts as an indirect proxy, leading to a gap between training objectives and final evaluation metrics. Middle: Identical output formats for 3D Video Detection, Grounding, and Spatial Reasoning tasks. Right: 3D-RFT utilizes a Scalar Reward (Policy Gradient) derived directly from evaluation metrics ( e.g . , F1-Score, 3D IoU, and Accuracy) through a decoding and parsing module, ensuring the model directly optimizes for the final task performance.
Figure 2: Overview of the 3D-RFT training framework. The process consists of two main stages: (1) SFT Warm Up : Initial training of the 3D-aware VLM using SFT data to establish a baseline policy. (2) RL Training : The Policy Model generates completions for video-based problems. A Verifiable Reward is calculated based on Format Reward (adherence to structured output) and Task Reward (performance in 3D Video Detection, 3D Visual Grounding, and Spatial Reasoning using metrics like 3D IoU and F1 Score). The model is optimized via Policy Gradient while maintaining a KL Divergence constraint relative to the frozen Reference Model.
Model
chair
cabinet
table
bin
couch
bed
bathtub
toilet
20 Common Classes
P25
R25
F125
4-Frame Setting
Qwen2.5-VL-3B ( Zheng et al., 2025b )
37.7
10.2
35.0
23.1
39.0
64.8
32.4
68.8
32.6
27.9
30.0
Qwen2.5-VL-7B ( Zheng et al., 2025b )
41.2
11.6
36.5
30.2
41.1
68.2
36.6
68.7
34.6
31.0
32.5
VG LLM-4B ( Zheng et al., 2025b )
49.7
13.1
41.3
39.2
44.6
71.2
33.5
83.4
41.7
35.7
38.2
VG LLM-8B ( Zheng et al., 2025b )
54.0
17.1
46.5
39.8
47.0
74.1
42.1
82.5
43.4
39.6
41.2
Table 1: Quantitative results on ScanNetDetection. In this table, we present the comparison with baseline models and report the performance improvement between the sft baseline (VG LLM-4B) and 3D-RFT-4B (ours).
Table 3: Ablation study of training strategies and 3D priors. We report accuracy at IoU thresholds 0.25 and 0.5.
Model
Avg.
Numerical Answer
Multiple-Choice Answer
Obj. Count
Abs. Dist.
Obj. Size
Room Size
Rel. Dist.
Rel. Dir.
Route Plan
Appr. Order
Base Models
GPT-4o ( Hurst et al., 2024 )
34.0
46.2
5.3
43.8
38.2
37.0
41.3
31.5
28.5
Gemini-2.5 Pro ( Comanici et al., 2025 )
51.5
43.8
34.9
64.3
42.8
61.1
47.8
45.9
71.3
LLaVA-Video-7B ( Zhang et al., 2025c )
35.6
48.5
14.0
47.8
24.2
43.5
42.4
34.0
30.6
Qwen2.5-VL-7B ( Bai et al., 2025b )
32.7
34.5
19.4
47.6
40.8
32.8
24.5
32.5
29.4
Table 4: Quantitative results on VSI-Bench. We include zero-shot results of base models, and test results of models after sft and rft .
Figure 7
Figure 6: Training Dynamics Analysis (3D Video Detection).
Figure 7: Training Dynamics Analysis (3D Spatial Reasoning).
Figure 8: Conceptual 1D Boundary Optimization Space. We illustrate a simplified 1D proxy to illustrate the fundamental mathematical mismatch between token classification and geometric optimization. The axes represent the predicted left ( x ) and right ( y ) boundaries of a 1D interval.
Figure 9: Distribution of 3D IoU in 3D Visual Grounding. A comparison of the raw 3D IoU distributions on the validation set between the baseline model trained with SFT and SFT.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
VSI-Bench
MindCube
MMSI-Bench
All-Angles
ERQA
VST-3B ( Yang et al., 2025b )
57.7
35.9
31.3
-
-
3D-RFT-4B (ours)
62.8
36.4
29.9
41.6
34.2
VST-7B ( Yang et al., 2025b )
61.2
39.7
34.8
-
-
3D-RFT-8B (ours)
64.0
42.6
28.0
43.1
38.5
Appendix
Table 5: Additional results across more spatial reasoning benchmarks and different model scales.
Figure 10: Training Dynamics of 3D Visual Grounding. We visualize the training curves for the total reward (left), the 3D IoU reward (middle), and the Frame Alignment reward (right). The curves show stable convergence and correlation with the optimization objectives.
Figure 11: Evaluation results on MindCube-Tiny .
Figure 12: Qualitative comparison of 3D object detection (20 common classes) results. We compare the predictions of the baseline VG LLM-4B against our 3D-RFT-4B model.
Figure 13: Qualitative results of 3D Visual Grounding using our 3D-RFT-4B . Each block shows the grounding text followed by 10 video frames visualizing the predicted 3D bounding boxes (green) and ground truth (red).
Figure 14: Qualitative example on VSI-Bench (absolute distance).
Figure 15: Qualitative example on VSI-Bench (relative direction).
Figure 16: Qualitative example on VSI-Bench (route plan).