Humans inhabit a physical 4D world where geometric structure and semantic content evolve over time, constituting a dynamic 4D reality (spatial with temporal dimension). While current Multimodal Large Language Models (MLLMs) excel in static visual understanding, can they also be adept at "thinking in dynamics", i.e., perceive, track and reason about spatio-temporal dynamics in evolving scenes? To systematically assess their spatio-temporal reasoning and localized dynamics perception capabilities, we introduce Dyn-Bench, a large-scale benchmark built from diverse real-world and synthetic video datasets, enabling robust and scalable evaluation of spatio-temporal understanding. Through multi-stage filtering from massive 2D and 4D data sources, Dyn-Bench provides a high-quality collection of dynamic scenes, comprising 1k videos, 7k visual question answering (VQA) pairs, and 3k dynamic object grounding pairs. We probe general, spatial and region-level MLLMs to express how they think in dynamics both linguistically and visually, and find that existing models cannot simultaneously maintain strong performance in both spatio-temporal reasoning and dynamic object grounding, often producing inconsistent interpretations of motion and interaction. Notably, conventional prompting strategies (e.g., chain-of-thought or caption-based hints) provide limited improvement, whereas structured integration approaches, including Mask-Guided Fusion and Spatio-Temporal Textual Cognitive Map (ST-TCM), significantly enhance MLLMs' dynamics perception and spatio-temporal reasoning in the physical 4D world. Code and benchmark are available at https://dyn-bench.github.io/.
Figures & tables
Numerical Statistics
QA Types
Detailed Features
Benchmark
# Videos
# QA Pairs
# Masklets
Inter-Object
Object-Scene
Camera-Object
Object Grounding
Scene Type
Dynamic Type
Real-world?
Metric-scale?
EgoDynamic4D [ 12 ]
275
927,000
-
✓
✓
✓
✗
Indoor
S.fisheye
Mixed
Yes
Chat4D [ 58 ]
-
879,100
-
✗
✓
✗
✗
Mixed
Realistic
Reak-world
Yes
DynSuperCLEVR [ 43 ]
1200
11,589
-
✓
✓
✗
✗
Outdoor
Open-field
Synthetic
Yes
VideoSTR [ 42 ]
-
205,000
-
✓
✓
✗
✗
Mixed
Realistc
Real-world
Yes
STI-Bench [ 19 ]
300
2,000
-
✓
✓
✓
✗
Mixed
Driving
Real-world
Yes
Table 1: Comparison of Dyn-Bench with existing spatio-temporal benchmarks. Dyn-Bench offers a unified assessment covering three levels of dynamic object grounding and reasoning while spanning more diverse data domains than prior works.
Figure 2: Model performance on Dyn-Bench . Left / Center radar charts show general and spatial MLLMs accuracy on nine spatio-temporal tasks; Right radar chart shows region-level MLLMs performance on spatio-temporal reasoning and dynamic object grounding.
Figure 3: Benchmark curation pipeline. The pipeline integrates dynamic video datasets from multiple sources, followed by multimodal completion with geometry and mask extraction. Data filter strategy ensures consistency and motion stability, complemented by human validation for quality assurance. Finally, spatial-temporal VQA and grounding pairs are generated with structured textual cognitive maps.
Table 2: Dataset statistics across Dyn-Bench filtering stages.
School of Artificial Intelligence, Beihang University, Beijing 100191, China · Zhongguancun Academy, Beijing 100094, China · School of Engineering, Westlake University, Hangzhou 310030, China