Spatiotemporal Reasoning

Follow this topic Ask about this topic

Spatiotemporal reasoning is the ability to identify where things are, track how they move or change, and infer relationships between events over time. Work combines video-language models with object tracking, 3D geometry, persistent scene graphs and memory, as well as symbolic tools such as temporal logic; these approaches support tasks in robotics, driving, surgery, and geospatial analysis. Benchmarks increasingly test more than recognition, including long-horizon tracking, unseen past or future states, causal inference, streaming video, and reasoning under changing viewpoints. Models have improved with structured representations and targeted training, but reliable visual grounding, memory, temporal extrapolation, and evaluation that distinguishes genuine reasoning from answer shortcuts remain open challenges.

Latest papers

71 papers11 papers in the last four weeks, up 38% on the four weeks before. 0.1% of all new papers.

Spatio-Temporal Grounding of Large Language Models from Perception Streams
Apr 8, 2026Jacob Anderson, Bardh Hoxha, Georgios Fainekos +2Spatiotemporal Reasoning LLM Grounding
Toyota Motor North America Research & Development, Ann Arbor, MI, USA
Differentiable SpaTiaL: Symbolic Learning and Reasoning with Geometric Temporal Logic for Manipulation Tasks
Apr 3, 2026Licheng Luo, Kaier Liang, Cristian-Ioan Vasile +1Differentiable Logic Spatiotemporal Reasoning
UC Riverside · Lehigh University
A 4D Representation for Training-Free Agentic Reasoning from Monocular Laparoscopic Video
Apr 1, 2026Maximilian Fehrentz, Nicolas Stellwag, Robert Wiebe +7Spatiotemporal Reasoning Surgical Video Understanding
Computer Aided Medical Procedures, TU Munich, Munich, Germany · TUM.ai, Munich, Germany · Munich Center for Machine Learning, Munich, Germany +1
Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence
Mar 13, 2026Seunghwan Bang, Hwanjun SongSpatiotemporal Reasoning VLM Reasoning
UNIST · KAIST
RoboStream: Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Robotics
Mar 13, 2026Yuzhi Huang, Jie Wu, Weijue Bu +9Long-Horizon Robotic Manipulation Robot Task Planning
Shenzhen International Graduate School, Tsinghua University · YuanxingGuangnian Robotics · China University of Mining and Technology +3
Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World
Mar 13, 2026Yuzhi Huang, Kairun Wen, Rongxin Gao +14Spatiotemporal Reasoning Multimodal Large Language Models
XMU · SUSTech · THU +4
SCP: Spatial Causal Prediction in Video
Mar 4, 2026Yanguang Zhao, Jie Yang, Shengqiong Wu +9Spatiotemporal Reasoning Causal Reasoning in Videos
National University of Singapore · Shenzhen University · Sichuan University +1
TimeSpot: Benchmarking Geo-Temporal Understanding in Vision-Language Models in Real-World Settings
Mar 4, 2026Azmine Toushik Wasi, Shahriyar Zaman Ridoy, Koushik Ahamed Tonmoy +5Spatiotemporal Reasoning Geospatial Reasoning Benchmarks
Computational Intelligence and Operations Laboratory (CIOL), Bangladesh · Shahjalal University of Science and Technology (SUST), Sylhet, Bangladesh · North South University (NSU), Dhaka, Bangladesh +1
MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence
Feb 28, 2026Xingyilang Yin, Chengzhengxu Li, Jiahao Chang +2Spatiotemporal Reasoning Multimodal Large Language Models
University of Macau · GVC Lab, Great Bay University · Xi’an Jiaotong University +1
UDVideoQA: A Traffic Video Question Answering Dataset for Multi-Object Spatio-Temporal Reasoning in Urban Dynamics
Feb 24, 2026Joseph Raj Vishal, Nagasiri Poluri, Katha Naik +7Video QA Visual Question Answering
Arizona State University
Making Space for Time: The Special Galilean Group and Its Application to Some Robotics Problems
Sep 22, 2024Jonathan KellyLie Group Methods Spatiotemporal Reasoning
Space & Terrestrial Autonomous Robotic Systems (STARS) Laboratory, University of Toronto Institute for Aerospace Studies (UTIAS), Toronto, Canada

Apr 8, 2026J/K move · Enter open · S save

Spatio-Temporal Grounding of Large Language Models from Perception Streams

Jacob Anderson, Bardh Hoxha, Georgios Fainekos, Hideki Okamoto+1

Toyota Motor North America Research & Development, Ann Arbor, MI, USA

Spatiotemporal Reasoning LLM Grounding Temporal Video Grounding

Open paper AskPDF arXiv

Embodied-AI agents must reason about how objects move and interact in 3-D space over time, yet existing smaller frontier Large Language Models (LLMs) still mis-handle fine-grained spatial relations, metric distances, and temporal orderings. We introduce the general framework Formally Explainable Spatio-Temporal Scenes (FESTS) that injects verifiable spatio-temporal supervision into an LLM by compiling natural-language queries into Spatial Regular Expression (SpRE) -- a language combining regular expression syntax with S4u spatial logic and extended here with universal and existential quantification. The pipeline matches each SpRE against any structured video log and exports aligned (query, frames, match, explanation) tuples, enabling unlimited training data without manual labels. Training a 3-billion-parameter model on 27k such tuples boosts frame-level F1 from 48.5% to 87.5%, matching GPT-4.1 on complex spatio-temporal reasoning while remaining two orders of magnitude smaller, and, hence, enabling spatio-temporal intelligence for Video LLM.