Humans are born with vision-based 4D spatial-temporal intelligence, which enables us to perceive and reason about the evolution of 3D space over time from purely visual inputs. Despite its importance, this capability remains a significant bottleneck for current multimodal large language models (MLLMs). To tackle this challenge, we introduce MLLM-4D, a comprehensive framework designed to bridge the gaps in training data curation and model post-training for spatiotemporal understanding and reasoning. On the data front, we develop a cost-efficient data curation pipeline that repurposes existing stereo video datasets into high-quality 4D spatiotemporal instructional data. This results in the MLLM4D-2M and MLLM4D-R1-30k datasets for Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT), alongside MLLM4D-Bench for comprehensive evaluation. Regarding model training, our post-training strategy establishes a foundational 4D understanding via SFT and further catalyzes 4D reasoning capabilities by employing Group Relative Policy Optimization (GRPO) with specialized Spatiotemporal Chain of Thought (ST-CoT) prompting and Spatiotemporal reward functions (ST-reward) without involving the modification of architecture. Extensive experiments demonstrate that MLLM-4D achieves state-of-the-art spatial-temporal understanding and reasoning capabilities from purely 2D RGB inputs. Project page: https://github.com/GVCLab/MLLM-4D.
Figures & tables
Figure 2: The components of our MLLM4D-Bench.
Figure 3: Our scalable curation pipeline for instructional spatiotemporal data. Our automated pipeline leverages several advanced vision techniques to extract 4D spatiotemporal information from stereoscopic videos, including per-frame camera poses, object-level 3D point clouds, and semantic descriptions. These data are then processed through a physics-based spatiotemporal relation solver to generate 4D QA pairs, and our specialized ST-CoT prompting strategy synthesizes the corresponding reasoning trajectories.
Figure 4: Our RFT pipeline. Given the input video and question, the MLLM-4D model generates multiple rollouts using the ST-CoT reasoning format. Within each group, relative advantages are computed based on accuracy reward, format reward and ST-reward. The model parameters are then updated via the GRPO objective, which incorporates a KL penalty relative to the frozen reference model.
Models
MLLM4D-Bench
Avg.
Camera
Object
Object & Camera
Abs. Dis
Rel. Dir.
Abs. Dis.
Abs. Dis.
Rel. Dis.
Rel. Dir.
Proprietary Models
GPT-4o ( Hurst et al., 2024 )
34.8
57.6
32.7
36.7
56.7
51.0
44.9
Gemini2.5-Flash ( Comanici et al., 2025 )
35.2
59.4
24.1
37.8
55.3
50.6
43.8
Gemini2.5-Pro ( Comanici et al., 2025 )
37.4
57.5
31.1
38.6
60.1
55.0
46.6
Table 1: Comparison of different models on MLLM4D-Bench. Bold font indicates the best performance.
Models
Real
Synthetic
Overall
Proprietary Models
GPT-4o
60.0
49.9
57.5
Gemini-2.5-Pro
63.5
57.3
62.0
Open-Source Models
Qwen2.5-VL-7B
43.3
45.6
43.8
Qwen2.5-VL-72B
53.1
52.6
53.0
Table 2: Evaluation on VLM4D benchmark.
Figure 5: Scalability of training data on SFT and RFT stage.
Models
MLLM4D-Bench
VLM4D
Baseline
35.3
52.1
SFT (w/o data in Sec. 3 )
59.9
56.2
SFT (w/ data in Sec. 3 ))
70.1
59.7
GRPO (w/o ST-reward)
70.5
61.4
GRPO (w/ ST-reward)
72.7
63.1
Table 3: Ablation study of MLLM-4D framework.
Figure 6: Illustration of the prompt used to identify all moving entities and extract their corresponding noun categories.
Figure 7: The components of our MLLM4D-2M.
Figure 8: The components of our MLLM4D-R1-30k.
Figure 9: An alternative data pipeline based on monocular videos.
Figure 10: Detailed system prompt and user prompt setting for our SFT, Cold Start and GRPO stage.
Figure 11: Detailed structure of the Spatiotemporal Chain of Thought (ST-CoT) Generation Prompt. The prompt is color-coded into four functional modules: ■ System Role & Task defines the Al persona and core mission; ■ Input Data & Mandatory Rules enforces visual-driven reasoning and constraints; ■ CoT Structure provides the step-by-step requirements for the reasoning chain; ■ Output Format specifies the data placeholders and JSON schema.
Figure 12: Qualitative comparison on MLLM4D-Bench.
Figure 13: Qualitative comparison on MLLM4D-Bench.
Figure 14: Qualitative comparison on MLLM4D-Bench.
Figure 15: Qualitative comparison on VLM4D benchmark.
Figure 16: Qualitative comparison on VLM4D benchmark.
Figure 17: Qualitative comparison on VLM4D benchmark.