Motion-o: Trajectory-Grounded Video Reasoning
Organizations: Northeastern University
Abstract
Recent video reasoning models increasingly produce spatio-temporal evidence chains that localize objects at specific timestamps. While these traces improve interpretability by grounding where and when evidence appears, they often leave the motion connecting observations, the how, implicit. This makes dynamic and trajectory-dependent claims difficult to supervise, verify, or penalize when unsupported by the video. We formalize this missing component as Spatial-Temporal-Trajectory (STT) reasoning and introduce Motion-o, a motion-centric extension to vision-language models (VLMs) that makes trajectories explicit and verifiable. Motion-o augments evidence chains with Motion Chain of Thought (MCoT), a structured pathway that represents object motion through a discrete <motion/> tag summarizing direction, speed, and scale change. To supervise MCoT, we densify sparse spatio-temporal annotations into object tracks and derive motion descriptors from centroid displacement and box-area change. We then train with complementary rewards for trajectory consistency and visual grounding, including a perturbation-based signal that penalizes motion descriptions that remain unchanged when temporal evidence is removed. Across multiple video understanding benchmarks, Motion-o consistently improves trajectory-faithful reasoning without architectural modifications. These results suggest that an explicit motion interface can complement existing VLM pipelines by converting implicit dynamics into verifiable evidence. Code is available at~\faGithub\ ostadabbas/Motion-o.
Figures & tables
| Abl. | Config | mAM | mLGM | VMME |
|---|---|---|---|---|
| Dens. | Sparse/Disc. | 31.2 | 44.5 | 68.1 |
| Dense/Disc. | 35.5 | 49.4 | 67.5 | |
| Tag | Dense/Num. | 2.1 | 5.4 | 14.4 |
| Dense/Disc. | 35.5 | 49.4 | 67.5 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Prompt | Motion Tag Acc (%) | V-STAR | |||||
|---|---|---|---|---|---|---|---|---|
| Dir | Speed | Scale | Avg | Temp tIoU | [email protected] | Spatial mIoU | ||
| GPT-4o | w/o <motion/> | 0.0 | 0.0 | 0.0 | 0.0 | 23.6 | 28.0 | 10.3 |
| GPT-4o | w/ <motion/> | 31.9 | 29.8 | 30.5 | 30.7 | 25.4 | 29.0 | 12.5 |
| Qwen2.5-VL-7B | w/o <motion/> | 0.0 | 0.0 | 0.0 | 0.0 | 14.6 | 18.2 | 9.3 |
| Qwen2.5-VL-7B | w/ <motion/> | 24.1 | 22.3 | 23.8 | 23.4 | 17.1 | 21.4 | 12.7 |
| Component | Training / Inference (Trajectory-Grounded) | Benchmark Evaluation (Letter-Only MCQ) |
|---|---|---|
| Frames | uniformly sampled, temporal order | uniformly sampled, temporal order |
| System message | Task-conditioned grammar (Sec. F.2 ) or unified inference prompt (Sec. F.3 ) | “Watch carefully … answer with only the letter” |
| <think> requirements | <obj> / <box> / <t> grounding + <motion/> tags | Free (not enforced at parse time) |
| <answer> content | Plain text, option letter, or From <t> </t>s to <t> </t>s (task-dependent) | Single option letter |
| Output target | Free-form generation with structured tags | Single letter A, B, C, D |
| Scoring | (Sec. 3.3 ) | Exact-match top-1 accuracy |
Video Reasoning
Video reasoning studies how AI systems use video evidence to infer events, actions, temporal relations, causes, and future states.
See all 200 papers