Humans are born with vision-based 4D spatial-temporal intelligence, which enables us to perceive and reason about the evolution of 3D space over time from purely visual inputs. Despite its importance, this capability remains a significant bottleneck for current multimodal large language models (MLLMs). To tackle this challenge, we introduce MLLM-4D, a comprehensive framework designed to bridge the gaps in training data curation and model post-training for spatiotemporal understanding and reasoning. On the data front, we develop a cost-efficient data curation pipeline that repurposes existing stereo video datasets into high-quality 4D spatiotemporal instructional data. This results in the MLLM4D-2M and MLLM4D-R1-30k datasets for Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT), alongside MLLM4D-Bench for comprehensive evaluation. Regarding model training, our post-training strategy establishes a foundational 4D understanding via SFT and further catalyzes 4D reasoning capabilities by employing Group Relative Policy Optimization (GRPO) with specialized Spatiotemporal Chain of Thought (ST-CoT) prompting and Spatiotemporal reward functions (ST-reward) without involving the modification of architecture. Extensive experiments demonstrate that MLLM-4D achieves state-of-the-art spatial-temporal understanding and reasoning capabilities from purely 2D RGB inputs. Project page: https://github.com/GVCLab/MLLM-4D.
Figures & tables
Figure 2: The components of our MLLM4D-Bench.
Figure 3: Our scalable curation pipeline for instructional spatiotemporal data. Our automated pipeline leverages several advanced vision techniques to extract 4D spatiotemporal information from stereoscopic videos, including per-frame camera poses, object-level 3D point clouds, and semantic descriptions. These data are then processed through a physics-based spatiotemporal relation solver to generate 4D QA pairs, and our specialized ST-CoT prompting strategy synthesizes the corresponding reasoning trajectories.
Figure 4: Our RFT pipeline. Given the input video and question, the MLLM-4D model generates multiple rollouts using the ST-CoT reasoning format. Within each group, relative advantages are computed based on accuracy reward, format reward and ST-reward. The model parameters are then updated via the GRPO objective, which incorporates a KL penalty relative to the frozen reference model.
Models
MLLM4D-Bench
Avg.
Camera
Object
Object & Camera
Abs. Dis
Rel. Dir.
Abs. Dis.
Abs. Dis.
Rel. Dis.
Rel. Dir.
Proprietary Models
GPT-4o ( Hurst et al., 2024 )
34.8
57.6
32.7
36.7
56.7
51.0
44.9
Gemini2.5-Flash ( Comanici et al., 2025 )
35.2
59.4
24.1
37.8
55.3
50.6
43.8
Gemini2.5-Pro ( Comanici et al., 2025 )
37.4
57.5
31.1
38.6
60.1
55.0
46.6
Table 1: Comparison of different models on MLLM4D-Bench. Bold font indicates the best performance.
Models
Real
Synthetic
Overall
Proprietary Models
GPT-4o
60.0
49.9
57.5
Gemini-2.5-Pro
63.5
57.3
62.0
Open-Source Models
Qwen2.5-VL-7B
43.3
45.6
43.8
Qwen2.5-VL-72B
53.1
52.6
53.0
Table 2: Evaluation on VLM4D benchmark.
Figure 5: Scalability of training data on SFT and RFT stage.
Models
MLLM4D-Bench
VLM4D
Baseline
35.3
52.1
SFT (w/o data in Sec. 3 )
59.9
56.2
SFT (w/ data in Sec. 3 ))
70.1
59.7
GRPO (w/o ST-reward)
70.5
61.4
GRPO (w/ ST-reward)
72.7
63.1
Table 3: Ablation study of MLLM-4D framework.
Figure 6: Illustration of the prompt used to identify all moving entities and extract their corresponding noun categories.
Figure 7: The components of our MLLM4D-2M.
Figure 8: The components of our MLLM4D-R1-30k.
Figure 9: An alternative data pipeline based on monocular videos.
Figure 10: Detailed system prompt and user prompt setting for our SFT, Cold Start and GRPO stage.
Figure 11: Detailed structure of the Spatiotemporal Chain of Thought (ST-CoT) Generation Prompt. The prompt is color-coded into four functional modules: ■ System Role & Task defines the Al persona and core mission; ■ Input Data & Mandatory Rules enforces visual-driven reasoning and constraints; ■ CoT Structure provides the step-by-step requirements for the reasoning chain; ■ Output Format specifies the data placeholders and JSON schema.
Figure 12: Qualitative comparison on MLLM4D-Bench.
Figure 13: Qualitative comparison on MLLM4D-Bench.
Figure 14: Qualitative comparison on MLLM4D-Bench.
Figure 15: Qualitative comparison on VLM4D benchmark.
Figure 16: Qualitative comparison on VLM4D benchmark.
Figure 17: Qualitative comparison on VLM4D benchmark.
Recent large multimodal models (LMMs) have become increasingly capable on image and video understanding, yet still struggle to sustain 4D continuous spatiotemporal dynamic reasoning. To study this capability gap, we formulate trajectory-grounded multi-turn spatiotemporal dialogue, a new task in which a model must answer spatiotemporal queries while returning structured 3D target trajectories over an entire short clip or a specified segment of a longer clip, and introduce Track4D-Bench, a benchmark with 526 clip-level dialogue samples spanning 23.5k frames and 7.5k object annotations, for training and evaluation. Building on this task, we propose LMM-Track4D, which combines RTGE (Ray--Time Geometry Encoding), a dedicated streaming state token TRK for long-horizon dynamic propagation, and an Object-Slot Kinematic, Residual-Anchor (OSK-RA) decoder for stable 4-step 3D state estimation under occlusion and viewpoint variation. Experiments on Track4D-Bench show consistent improvements over strong baselines, suggesting that explicit dynamic state modeling is a useful design principle for eliciting 4D dynamic reasoning in LMMs. Our code and dataset will be publicly available at https://github.com/mikubaka88/LMM-Track4D.
Chaoyue Li, Yongxue Xu, Jie Feng +1
Huazhong University of Science and Technology · Sun Yat-sen University · Beihang University +1
Dynamic spatial reasoning from monocular video is essential for bridging visual intelligence and the physical world, yet remains challenging for vision-language models (VLMs). Prior approaches either verbalize spatial-temporal reasoning entirely as text, which is inherently verbose and imprecise for complex dynamics, or rely on external geometric modules that increase inference complexity without fostering intrinsic model capability. In this paper, we present 4DThinker, the first framework that enables VLMs to "think with 4D" through dynamic latent mental imagery, i.e., internally simulating how scenes evolve within the continuous hidden space. Specifically, we first introduce a scalable, annotation-free data generation pipeline that synthesizes 4D reasoning data from raw videos. We then propose Dynamic-Imagery Fine-Tuning (DIFT), which jointly supervises textual tokens and 4D latents to ground the model in dynamic visual semantics. Building on this, 4D Reinforcement Learning (4DRL) further tackles complex reasoning tasks via outcome-based rewards, restricting policy gradients to text tokens to ensure stable optimization. Extensive experiments across multiple dynamic spatial reasoning benchmarks demonstrate that 4DThinker consistently outperforms strong baselines and offers a new perspective toward 4D reasoning in VLMs. Our code is available at https://github.com/zhangquanchen/4DThinker.
Zhangquan Chen, Manyuan Zhang, Xinlei Yu +9
Tsinghua University, SIGS · Meituan · The Chinese University of Hong Kong +3
Humans inhabit a physical 4D world where geometric structure and semantic content evolve over time, constituting a dynamic 4D reality (spatial with temporal dimension). While current Multimodal Large Language Models (MLLMs) excel in static visual understanding, can they also be adept at "thinking in dynamics", i.e., perceive, track and reason about spatio-temporal dynamics in evolving scenes? To systematically assess their spatio-temporal reasoning and localized dynamics perception capabilities, we introduce Dyn-Bench, a large-scale benchmark built from diverse real-world and synthetic video datasets, enabling robust and scalable evaluation of spatio-temporal understanding. Through multi-stage filtering from massive 2D and 4D data sources, Dyn-Bench provides a high-quality collection of dynamic scenes, comprising 1k videos, 7k visual question answering (VQA) pairs, and 3k dynamic object grounding pairs. We probe general, spatial and region-level MLLMs to express how they think in dynamics both linguistically and visually, and find that existing models cannot simultaneously maintain strong performance in both spatio-temporal reasoning and dynamic object grounding, often producing inconsistent interpretations of motion and interaction. Notably, conventional prompting strategies (e.g., chain-of-thought or caption-based hints) provide limited improvement, whereas structured integration approaches, including Mask-Guided Fusion and Spatio-Temporal Textual Cognitive Map (ST-TCM), significantly enhance MLLMs' dynamics perception and spatio-temporal reasoning in the physical 4D world. Code and benchmark are available at https://dyn-bench.github.io/.