Humans inhabit a physical 4D world where geometric structure and semantic content evolve over time, constituting a dynamic 4D reality (spatial with temporal dimension). While current Multimodal Large Language Models (MLLMs) excel in static visual understanding, can they also be adept at "thinking in dynamics", i.e., perceive, track and reason about spatio-temporal dynamics in evolving scenes? To systematically assess their spatio-temporal reasoning and localized dynamics perception capabilities, we introduce Dyn-Bench, a large-scale benchmark built from diverse real-world and synthetic video datasets, enabling robust and scalable evaluation of spatio-temporal understanding. Through multi-stage filtering from massive 2D and 4D data sources, Dyn-Bench provides a high-quality collection of dynamic scenes, comprising 1k videos, 7k visual question answering (VQA) pairs, and 3k dynamic object grounding pairs. We probe general, spatial and region-level MLLMs to express how they think in dynamics both linguistically and visually, and find that existing models cannot simultaneously maintain strong performance in both spatio-temporal reasoning and dynamic object grounding, often producing inconsistent interpretations of motion and interaction. Notably, conventional prompting strategies (e.g., chain-of-thought or caption-based hints) provide limited improvement, whereas structured integration approaches, including Mask-Guided Fusion and Spatio-Temporal Textual Cognitive Map (ST-TCM), significantly enhance MLLMs' dynamics perception and spatio-temporal reasoning in the physical 4D world. Code and benchmark are available at https://dyn-bench.github.io/.
Figures & tables
Numerical Statistics
QA Types
Detailed Features
Benchmark
# Videos
# QA Pairs
# Masklets
Inter-Object
Object-Scene
Camera-Object
Object Grounding
Scene Type
Dynamic Type
Real-world?
Metric-scale?
EgoDynamic4D [ 12 ]
275
927,000
-
✓
✓
✓
✗
Indoor
S.fisheye
Mixed
Yes
Chat4D [ 58 ]
-
879,100
-
✗
✓
✗
✗
Mixed
Realistic
Reak-world
Yes
DynSuperCLEVR [ 43 ]
1200
11,589
-
✓
✓
✗
✗
Outdoor
Open-field
Synthetic
Yes
VideoSTR [ 42 ]
-
205,000
-
✓
✓
✗
✗
Mixed
Realistc
Real-world
Yes
STI-Bench [ 19 ]
300
2,000
-
✓
✓
✓
✗
Mixed
Driving
Real-world
Yes
Table 1: Comparison of Dyn-Bench with existing spatio-temporal benchmarks. Dyn-Bench offers a unified assessment covering three levels of dynamic object grounding and reasoning while spanning more diverse data domains than prior works.
Figure 2: Model performance on Dyn-Bench . Left / Center radar charts show general and spatial MLLMs accuracy on nine spatio-temporal tasks; Right radar chart shows region-level MLLMs performance on spatio-temporal reasoning and dynamic object grounding.
Figure 3: Benchmark curation pipeline. The pipeline integrates dynamic video datasets from multiple sources, followed by multimodal completion with geometry and mask extraction. Data filter strategy ensures consistency and motion stability, complemented by human validation for quality assurance. Finally, spatial-temporal VQA and grounding pairs are generated with structured textual cognitive maps.
Table 2: Dataset statistics across Dyn-Bench filtering stages.
4D spatio-temporal reasoning, jointly modeling 3D spatial structure and temporal evolution, is essential for understanding dynamic worlds and enabling embodied interaction. While current Multimodal Large Language Models (MLLMs) show strong capabilities in static scene understanding and coarse-grained 4D tasks, they still have notable limitations in continuous dynamic scene perception, especially in tracking dynamic object evidence for coherent 4D spatio-temporal reasoning. This shortcoming stems mainly from relying on sparse frame-level observations, fragmenting continuous dynamic cues and leaving models unable to disentangle genuine object dynamics from camera-induced apparent motion. Inspired by humans tracking dynamic cues while compensating for viewpoint changes, we propose DynTrace, a training-free framework for 4D spatio-temporal reasoning with two complementary components. Dynamic Trajectory Visualization (DTV) reprojects world-coordinate trajectories onto the image plane, providing geometry-informed visual priors that disentangle genuine object dynamics from camera-induced apparent motion. Meanwhile, the Dynamic Trace Token (DT-Token), organized into a Dynamic Trace Graph (DTG), tracks object-level dynamic cues, trace evolution, and key moments, maintaining continuous dynamic object evidence for coherent 4D reasoning. Together, these two components equip MLLMs with continuously tracked dynamic object evidence, grounded in geometry-informed visual priors and structured spatio-temporal traces. DynTrace consistently improves open-source MLLMs, achieving state-of-the-art results on Dyn-Bench, VLM4D, and DSI-Bench, validating the importance of tracking dynamic object evidence for robust 4D spatio-temporal reasoning.
Rongxin Gao, Yuzhi Huang, Dongxuan Liu +8
1Xiamen University · 2Shenzhen International Graduate School, Tsinghua University · 3China University of Mining and Technology +1
Recent large multimodal models (LMMs) have become increasingly capable on image and video understanding, yet still struggle to sustain 4D continuous spatiotemporal dynamic reasoning. To study this capability gap, we formulate trajectory-grounded multi-turn spatiotemporal dialogue, a new task in which a model must answer spatiotemporal queries while returning structured 3D target trajectories over an entire short clip or a specified segment of a longer clip, and introduce Track4D-Bench, a benchmark with 526 clip-level dialogue samples spanning 23.5k frames and 7.5k object annotations, for training and evaluation. Building on this task, we propose LMM-Track4D, which combines RTGE (Ray--Time Geometry Encoding), a dedicated streaming state token TRK for long-horizon dynamic propagation, and an Object-Slot Kinematic, Residual-Anchor (OSK-RA) decoder for stable 4-step 3D state estimation under occlusion and viewpoint variation. Experiments on Track4D-Bench show consistent improvements over strong baselines, suggesting that explicit dynamic state modeling is a useful design principle for eliciting 4D dynamic reasoning in LMMs. Our code and dataset will be publicly available at https://github.com/mikubaka88/LMM-Track4D.
Chaoyue Li, Yongxue Xu, Jie Feng +1
Huazhong University of Science and Technology · Sun Yat-sen University · Beihang University +1
Multimodal Large Language Models (MLLMs) have achieved remarkable success across diverse expert-level tasks, but they still struggle with fundamental abilities that humans naturally develop through continuous observation of the real world, such as spatial perception and dynamic reasoning. Recent studies have recognized this gap and introduced dedicated benchmarks to evaluate the spatial-temporal capabilities of MLLMs. However, existing benchmarks mostly focus on static scenes or require exact quantitative predictions, leaving intuitive reasoning from temporal cues largely underexplored. In this paper, we introduce the Visual Spatial-Temporal Reasoning Benchmark (ViSTR-Bench), a novel evaluation suite designed to systematically assess whether MLLMs can perform qualitative reasoning from continuous visual cues in dynamic scenes. Guided by the principles of temporal emphasis, reasoning orientation, and qualitative evaluation, ViSTR-Bench establishes a comprehensive four-dimensional evaluations covering Motion Perception, Spatial Relations, Outcome Prediction, and Physical Dynamics. The benchmark comprises 15 distinct subtasks and 1,340 high-quality video question-answer pairs spanning diverse tabletop, indoor, and outdoor scenarios. Extensive evaluations of a broad spectrum of state-of-the-art proprietary, open-source, and specialized spatial MLLMs reveal that, despite their strong general video understanding capabilities, current models still face substantial bottlenecks in complex spatial-temporal reasoning and remain far below human performance.
Han Li, Si Liu, Zehao Huang +6
School of Artificial Intelligence, Beihang University, Beijing 100191, China · Zhongguancun Academy, Beijing 100094, China · School of Engineering, Westlake University, Hangzhou 310030, China