Dynamic spatial reasoning from monocular video enables vision-language models (VLMs) to understand how 3D environments evolve over time, supporting physical world understanding and embodied intelligence. Existing efforts primarily address observed-dynamics understanding, whereas agents in dynamic environments also require future-dynamics prediction beyond the observation boundary. Moreover, external geometric processing and textual descriptions can increase inference complexity or lose information when representing continuous dynamics. These limitations motivate a shared internal state that preserves observed dynamics and remains predictive beyond the input video. Accordingly, we present Pros4D, a framework for modeling evolving 4D worlds in latent space that unifies retrospective understanding of observed-dynamics with prospective reasoning for future-dynamics prediction. Specifically, Pros4D-Engine derives geometry-grounded supervision and constructs paired examples for both tasks, while Pros4D-Bench evaluates them across eight spatial task categories spanning dynamic cameras, objects, and scenes. At the model level, Pros4D compresses observed dynamics into latent observed states and autoregressively rolls them forward into latent future states. 4D Latent Alignment (4DLA) grounds these states through complementary visual-semantic and geometric supervision, while joint language supervision connects them to task reasoning. Experiments across benchmarks show consistent improvements over strong baselines in both capabilities.
Figures & tables
Figure 2: Overview of Pros4D-Engine . Given raw videos, Stage 1 grounds and verifies entities to obtain reliable object-mask alignments. Stage 2 recovers 3D geometry and constructs question-answer pairs for camera motion, object trajectories, and relation evolution. Stage 3 synthesizes 4D latent-state reasoning traces, pairing textual reasoning with observed- and future-state supervision.
Figure 3: 4D Latent Alignment (4DLA) . Visual-semantic and geometric supervision ground the interleaved observed and future states used by 4D-LSR. Alignment is used only during training. Both encoders stay frozen while the decoder and alignment head are trained.
Figure 4: Pros4D-Bench task composition across dynamic camera, object, and scene reasoning categories.
Retrospective
Prospective
Cam. Dir.
Cam.-Obj. Dist.
Obj. Dir.
Inter-Obj. Dist.
Obj. Speed
Obj. Dist.
Scene Rel.
Scene Traj.
Methods
Avg.
Temporal
Dynamic Camera
Dynamic Objects
Dynamic Scene
Spatial Understanding Models
SpaceR-7B [ 40 ]
46.9
49.5
42.6
86.3
39.1
41.4
24.5
53.6
29.9
33.9
19.4
VST-7B-RL [ 63 ]
41.1
44.2
35.8
74.9
37.9
32.6
25.2
45.8
23.3
31.9
23.0
Spatial-SSRL-7B [ 35 ]
39.3
43.2
32.6
73.4
32.8
35.0
25.8
30.0
31.4
34.7
18.3
Table 1: Pros4D-Bench results (accuracy, %) . Retrospective and Prospective evaluate observed-dynamics understanding and future-dynamics prediction from the same observed video prefix. Eight task columns cover dynamic camera, object, and scene reasoning across the two temporal subsets, with averages over all spatial tasks. Bold and underlined values mark the best and second best scores.
Methods
Avg.
Spatial Reasoning
Goal Reasoning
Interaction Reasoning
RS
OS
SU
MV
TS-G
TS-S
TS-GL
AU
IP
TU
Qwen2.5-VL-3B
30.48
37.09
80.11
23.23
13.47
37.46
40.63
2.10
24.05
26.81
46.94
4D-RGPT-8B
29.70
45.20
42.70
18.60
17.70
30.40
38.10
19.90
18.90
32.80
34.00
MLLM-4D-8B
46.30
70.30
98.50
22.60
20.00
39.70
70.50
20.90
37.90
41.00
46.20
Pros4D-3B
51.39 [-0.1em] (+20.91)
47.09 [-0.1em] (+10.00)
100.00 [-0.1em] (+19.89)
30.09 [-0.1em] (+6.86)
29.51 [-0.1em] (+16.04)
39.80 [-0.1em] (+2.34)
37.01 [-0.1em] (−3.62)
47.90 [-0.1em] (+45.80)
30.76 [-0.1em] (+6.71)
79.41 [-0.1em] (+52.60)
43.54 [-0.1em] (−3.40)
Table 5: OpenX-VQA results (accuracy, %) . The ten tasks cover spatial, goal, and interaction reasoning in robot manipulation videos. Parentheses in the Pros4D-3B row give percentage-point changes from Qwen2.5-VL-3B. Task abbreviations are defined in Appendix C.4 .
Time
LSR
Target Distance
Target Direction
Ego Speed
Target Speed
t0
14.67 (+3.18)
43.03 (+9.78)
35.70 (-0.49)
42.30 (+12.72)
35.70 (+5.14)
t1
44.74 (+2.93)
44.74 (+2.93)
44.74 (+2.93)
44.74 (+10.75)
38.63 (+4.64)
t2
42.30 (+2.94)
42.30 (+2.94)
42.30 (+2.94)
44.50 (+11.25)
35.94 (+6.11)
t3
39.85 (+5.62)
39.85 (+5.62)
39.85 (+5.62)
52.57 (+17.36)
34.23 (+4.65)
Table 6: STRIDE-QA-Bench results (success rate, %) . Cells report Pros4D-3B success rates, with percentage-point changes from Qwen2.5-VL-3B in parentheses. At each future horizon ( t1 – t3 ), the same answers meet the distance and direction criteria, so both rates equal the joint localization success rate (LSR, Appendix C.4 ).
Figure 5: Latent state analysis on Pros4D-8B. (a) Cosine similarity to future visual and geometric targets; GT-Neg. Margin is matched-minus-negative future similarity. (b) Pros4D-Bench accuracy (%) under alignment-loss ablations; CE denotes cross entropy. (c) Future-dynamics prediction accuracy drops relative to the unmodified model, measured in percentage points (pp).
Dominant axis of accumulated translation; six directions
Camera-Object Distance
∥ct−pt∥2
Robust distance trend; five change categories
Object Motion Direction
Object trajectory in the camera view
Robust view-plane displacement; eight directions or nearly stationary
Inter-Object Distance
∥pt−qt∥2
Robust distance trend; five change categories
Object Speed Change
Displacement per elapsed second
Smoothed speed changes; constant, monotonic, or two-phase trends
Object Travel Distance
Smoothed object trajectory
Robust endpoint separation, rounded to 0.1 m
Table 13: Task definitions for observed-dynamics understanding and future-dynamics prediction.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Table 15: Comparison of dynamic spatial datasets and benchmarks. Cam.-, Obj.-, and Scn.-Level denote camera, object, and scene coverage; Future Pred. indicates future-dynamics prediction.
Methods
Dyn-Bench
VLM4D
Avg.
Inter-Obj.
Obj.-Scene
Cam.-Obj.
Avg.
Real
Syn.
Spatial Understanding Models
SpaceR-7B [ 40 ]
56.5
54.1
71.9
47.8
47.4
49.2
41.8
Spatial-SSRL-7B [ 35 ]
48.4
45.5
68.9
36.7
52.5
52.2
53.3
SpatialReasoner [ 38 ]
54.5
50.1
68.0
48.4
42.4
42.8
41.3
SpatialLadder-3B [ 28 ]
52.3
48.0
69.7
43.5
42.4
44.0
37.5
Appendix
Table 19: Complete Dyn-Bench and VLM4D comparison (accuracy, %). Relation and source subsets follow the main tables.
Column
Task
Queried information
RS
Robot State
Arm or gripper configuration
OS
Object State
Reachability or manipulability
SU
Scene Understanding
Spatial layout and depth relations
MV
Multiple View
Correspondence across viewpoints
TS-G
Task State, grasp
Grasp state and stability
TS-S
Task State, success
Whether manipulation succeeded
Appendix
Table 20: OpenX-VQA task abbreviations. Scores are multiple-choice accuracy (%).
Figure 6: Camera-Object Distance templates for observed-dynamics understanding and 4D-LSR.
Figure 7: Camera-Object Distance templates for future-dynamics prediction and 4D-LSR.
Figure 8: Inter-Object Distance templates for observed-dynamics understanding and 4D-LSR.
Figure 9: Inter-Object Distance templates for future-dynamics prediction and 4D-LSR.
Figure 10: Scene Spatial Relationship templates for observed-dynamics understanding and 4D-LSR.
Figure 11: Scene Spatial Relationship templates for future-dynamics prediction and 4D-LSR.
Figure 12: Pros4D-Bench examples across eight spatial task categories in a shared scene, covering dynamic camera, object, and scene reasoning.
Figure 13: Camera-Object Distance on Pros4D-Bench. The comparison concerns the decreasing distance between the camera and the seated person.
Figure 14: Object Motion Direction on Pros4D-Bench. The queried person moves leftward in the camera view.
Figure 15: Object-scene relations on Dyn-Bench. The bus becomes increasingly occluded by trees as it moves through the scene.
Figure 16: Observer-relative motion on VLM4D. The queried player moves away from the camera.
Spatial reasoning studies how systems represent geometry, relations, viewpoints, and motion to understand scenes, plan actions, and solve spatial tasks.
Dynamic spatial reasoning from monocular video is essential for bridging visual intelligence and the physical world, yet remains challenging for vision-language models (VLMs). Prior approaches either verbalize spatial-temporal reasoning entirely as text, which is inherently verbose and imprecise for complex dynamics, or rely on external geometric modules that increase inference complexity without fostering intrinsic model capability. In this paper, we present 4DThinker, the first framework that enables VLMs to "think with 4D" through dynamic latent mental imagery, i.e., internally simulating how scenes evolve within the continuous hidden space. Specifically, we first introduce a scalable, annotation-free data generation pipeline that synthesizes 4D reasoning data from raw videos. We then propose Dynamic-Imagery Fine-Tuning (DIFT), which jointly supervises textual tokens and 4D latents to ground the model in dynamic visual semantics. Building on this, 4D Reinforcement Learning (4DRL) further tackles complex reasoning tasks via outcome-based rewards, restricting policy gradients to text tokens to ensure stable optimization. Extensive experiments across multiple dynamic spatial reasoning benchmarks demonstrate that 4DThinker consistently outperforms strong baselines and offers a new perspective toward 4D reasoning in VLMs. Our code is available at https://github.com/zhangquanchen/4DThinker.
Zhangquan Chen, Manyuan Zhang, Xinlei Yu +9
Tsinghua University, SIGS · Meituan · The Chinese University of Hong Kong +3
Humans inhabit a physical 4D world where geometric structure and semantic content evolve over time, constituting a dynamic 4D reality (spatial with temporal dimension). While current Multimodal Large Language Models (MLLMs) excel in static visual understanding, can they also be adept at "thinking in dynamics", i.e., perceive, track and reason about spatio-temporal dynamics in evolving scenes? To systematically assess their spatio-temporal reasoning and localized dynamics perception capabilities, we introduce Dyn-Bench, a large-scale benchmark built from diverse real-world and synthetic video datasets, enabling robust and scalable evaluation of spatio-temporal understanding. Through multi-stage filtering from massive 2D and 4D data sources, Dyn-Bench provides a high-quality collection of dynamic scenes, comprising 1k videos, 7k visual question answering (VQA) pairs, and 3k dynamic object grounding pairs. We probe general, spatial and region-level MLLMs to express how they think in dynamics both linguistically and visually, and find that existing models cannot simultaneously maintain strong performance in both spatio-temporal reasoning and dynamic object grounding, often producing inconsistent interpretations of motion and interaction. Notably, conventional prompting strategies (e.g., chain-of-thought or caption-based hints) provide limited improvement, whereas structured integration approaches, including Mask-Guided Fusion and Spatio-Temporal Textual Cognitive Map (ST-TCM), significantly enhance MLLMs' dynamics perception and spatio-temporal reasoning in the physical 4D world. Code and benchmark are available at https://dyn-bench.github.io/.
Understanding how objects and their spatial relationships evolve in 3D space over time remains challenging for vision-language models (VLMs), particularly under changing viewpoints. Recent approaches incorporate geometric information into reasoning, but explicit supervision for temporally ordered 4D state evolution and its use as intermediate evidence remains limited. We introduce Trace4D, which trains VLMs to predict 4D state sequences and reason over their inter-timestep changes alongside temporal visual observations. Its 4D-grounded chain-of-thought (CoT) explicitly represents 4D states, analyzes temporal visual context, and connects both as evidence for the answer. We construct Trace4D-Train from in-the-wild videos with 600K 4D perception examples and approximately 70K grounded QA examples, and train with joint supervised fine-tuning followed by geometry-aware GRPO. At inference, Trace4D predicts 4D states and reasoning directly from video without reference annotations or an external geometric model. Experiments on Trace4D-Bench and VLM4D show improved QA accuracy over the base VLMs and benefits from reasoning over 4D state changes and temporal visual observations.