cs.CVNov 2, 2026
SaveTrace4D: Grounded Spatiotemporal Reasoning via 4D State Sequences
Organizations: KAIST AI · Microsoft Research Asia
Abstract
Understanding how objects and their spatial relationships evolve in 3D space over time remains challenging for vision-language models (VLMs), particularly under changing viewpoints. Recent approaches incorporate geometric information into reasoning, but explicit supervision for temporally ordered 4D state evolution and its use as intermediate evidence remains limited. We introduce Trace4D, which trains VLMs to predict 4D state sequences and reason over their inter-timestep changes alongside temporal visual observations. Its 4D-grounded chain-of-thought (CoT) explicitly represents 4D states, analyzes temporal visual context, and connects both as evidence for the answer. We construct Trace4D-Train from in-the-wild videos with 600K 4D perception examples and approximately 70K grounded QA examples, and train with joint supervised fine-tuning followed by geometry-aware GRPO. At inference, Trace4D predicts 4D states and reasoning directly from video without reference annotations or an external geometric model. Experiments on Trace4D-Bench and VLM4D show improved QA accuracy over the base VLMs and benefits from reasoning over 4D state changes and temporal visual observations.
Figures & tables
Figure 1: Trace4D connects structured 4D states with temporal visual observations for video reasoning. In the boxer–referee example, centroid-distance changes are matched to the visible approach and subsequent separation, supporting answer B. Baseline reasoning is from Qwen3-VL (top) and MLLM-4D (bottom).
Figure 2: Overview of Trace4D . Reference 4D states support template-based QA construction and teacher-generated CoT that connects state changes with temporal visual observations. Joint SFT learns state prediction and grounded QA together, followed by geometry-aware GRPO using answer, format, position, displacement, and bounding-box rewards.
Figure 3: Example of 4D reference annotations. Colored boxes identify the same two entities across four frames; labels report their bounding-box coordinates and 3D centroids.
Figure 4: Dataset statistics. (a) Joint distribution of object and camera dynamics. (b) Question-type proportions for relative and absolute QA. (c) Mean reasoning token counts by question type.
| Models | Relative Spatial Relations | Absolute Spatial Relations | Overall | ||||||||||||
| Spd | Spd Comp | Dist Cam | Dist Obj | Dir Cam | Dir Obj | Avg. | Spd | Spd Comp | Dist Cam | Dir Cam | Dir Obj | Motion | Avg. | ||
| Proprietary Models | |||||||||||||||
| GPT-4o ( Hurst et al., 2024 ) | 33.0 | 32.0 | 21.0 | 13.0 | 47.0 | 32.0 | 29.67 | 31.0 | 32.0 | 36.0 | 38.0 | 35.0 | 54.0 | 37.67 | 33.67 |
| GPT-5 ( Singh et al., 2025 ) | 32.0 | 37.0 | 58.0 | 47.0 | 68.0 | 54.0 | 49.33 | 35.0 | 36.0 | 50.0 | 40.0 | 43.0 | 60.0 | 44.00 | 46.67 |
| General VLMs | |||||||||||||||
| Qwen2.5-VL-3B ( Bai et al., 2025b ) | 40.0 | 47.0 | 25.0 | 35.0 | 21.0 | 22.0 | 31.67 | 42.0 | 49.0 | 31.0 | 24.0 | 29.0 | 33.0 | 34.67 | 33.17 |
Table 1: Comparison of vision-language models on Trace4D-Bench, covering relative and absolute spatial and motion understanding.
| Model | Syn. | Real. | Avg. |
|---|---|---|---|
| Proprietary Models | |||
| GPT-4o ( Hurst et al., 2024 ) | 54.38 | 56.02 | 55.62 |
| GPT-5 ( Singh et al., 2025 ) | 62.70 | 74.69 | 71.75 |
| General VLMs | |||
| Qwen2.5-VL-3B ( Bai et al., 2025b ) | 35.96 | 47.34 | 44.55 |
| Qwen3-VL-4B ( Bai et al., 2025a ) | 48.76 | 57.55 | 55.40 |
Table 2: Performance on VLM4D. Bold and underline denote the best and second-best scores among non-proprietary models.
| CoT Variant | Trace4D-Bench | VLM4D |
|---|---|---|
| State-only | 74.83 | 52.86 |
| 4D-state CoT | 79.42 | 54.68 |
| Visual-only CoT | 73.75 | 59.25 |
| 4D + Visual CoT | 77.25 | 60.52 |
Table 3: CoT ablation. Bold and underlined values indicate the best and second-best results, respectively.
Figure 5: Predicted 3D centroid trajectories and 2D bounding boxes across timesteps.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Relative vs. Absolute frames for the same clip and car (top-down view). Relative uses each frame’s own camera as the origin; Absolute fixes the origin at the camera of .
| Question type | Setting | Question template |
|---|---|---|
| Camera–object distance | Rel./Abs. | [Prefix] how does the distance between [Object 1] and [Camera] change? |
| Object–object distance | Rel. | [Prefix] how does the distance between [Object 1] and [Object 2] change? |
| Camera–object direction | Rel./Abs. | [Prefix] how does the direction from [Camera] to [Object 1] change? |
| Object–object direction | Rel./Abs. | [Prefix] how does the direction from [Object 1] to [Object 2] change? |
| Speed | Rel./Abs. | [Prefix] how does the speed of [Object 1] change? |
| Speed comparison | Rel./Abs. | [Prefix] compare the speed between [Object 1] and [Object 2] . |
Table 7: Question templates for the 12 benchmark categories. Shared templates are grouped across coordinate settings. The object-motion row shows one of six paraphrases used in the data.
| Question family | Reference quantity | Example choice text |
|---|---|---|
| Camera–object distance | Distance from the object to the current or anchor-camera origin. | It moves closer to the camera, then moves farther from the camera. |
| Object–object distance | Distance between two object centroids. | They move closer together, then move farther apart. |
| Direction | Directed 3D offset from the first entity to the second. | It is to the front, then to the front-right. |
| Speed | Magnitude of consecutive coordinate increments. | It slows down, then moves at a steady pace. |
| Speed comparison | Two objects’ increment magnitudes over corresponding intervals. | The first is faster, then the second is faster. |
| Object motion | Translation along the fixed anchor camera’s forward and lateral axes. | First moves toward the camera, then left |
Table 8: Geometric quantities and example answer-option forms. Each option describes one or two ordered phases; the reference answer identifies one of the four options. The examples are drawn from the stored QA records.
| Question type | How the answer is determined |
|---|---|
| Distance 3D separation between the queried entities. | Nearly constant: within of the phase’s starting distance. Increasing: each distance exceeds the preceding one by more than . Decreasing: each distance is more than below the preceding one. |
| Speed 3D displacement magnitude per interval. | Steady pace: within of the phase’s starting displacement. Speeding up: each displacement exceeds the preceding one by more than . Slowing down: each displacement is more than below the preceding one. |
| Speed comparison Two objects over the same interval. | Faster: an object’s displacement is more than times the other’s. Nearly equal: neither object meets this condition. |
| Direction Unit direction: first to second entity. | Front / left / above: forward / left / up component above . Behind / right / below: that component is below . Combine directions when multiple axes qualify. |
| Moving Scene-normalized path in the anchor frame. | Stationary: both forward–lateral and vertical path lengths are at most . Moving: dominant forward, backward, left, or right displacement in each valid phase. |
Table 9: Main criteria for determining reference answers. Distance, speed, speed comparison, and direction criteria are adapted from DSR-Suite ( Zhou et al., 2026 ) (Table 8 and implementation). Moving questions use our additional criteria for fixed-frame translation.
Figure 7: Teacher prompt for generating 4D-grounded CoT supervision, shown for relative distance-to-camera questions.
Figure 8: Instructions for 4D perception supervision: relative (left) and absolute (right).
Figure 9: QA suffixes for grounded QA training and inference: relative (left) and absolute (right).
| Training / feedback | QA accuracy (%) | 4D state prediction | |||
|---|---|---|---|---|---|
| Overall | VLM4D | Centroid L2 | Trajectory L2 | IoU | |
| Joint SFT | 77.83 | 62.39 | 0.1873 | 0.1178 | 0.7353 |
| + GRPO: accuracy only | 78.33 | 62.67 | 0.1831 | 0.1164 | 0.7362 |
| + GRPO: position only | 78.83 | 63.11 | 0.1805 | 0.1151 | 0.7403 |
| Trace4D (Joint SFT + GRPO) | 79.25 | 64.32 | 0.1775 | 0.1123 | 0.7377 |
Table 10: QA accuracy and 4D state prediction in the GRPO ablation. State metrics use all 1,200 Trace4D-Bench questions, with valid-track averaging.
Figure 10: Trace4D predictions for camera-relative distance (top) and direction (bottom). Each example connects temporal visual observations with 4D state changes to support the answer. The displayed <4d> blocks are abbreviated.
Figure 11: Trace4D predictions for absolute speed comparison (top) and distance to the anchor camera (bottom). Both examples use the fixed anchor-camera frame specified in the question. Ellipses indicate omitted portions of the responses.
Figure 12: Trace4D predictions on VLM4D: robot-arm movement direction (top) and swan motion from its own perspective (bottom). The displayed <4d> blocks are abbreviated.
More in this topic
Spatiotemporal Reasoning
Spatiotemporal reasoning studies how systems connect objects, events, locations, and changes across space and time.
See all 73 papers