Spatiotemporal Reasoning

Follow this topic Ask about this topic

Spatiotemporal reasoning is the ability to identify where things are, track how they move or change, and infer relationships between events over time. Work combines video-language models with object tracking, 3D geometry, persistent scene graphs and memory, as well as symbolic tools such as temporal logic; these approaches support tasks in robotics, driving, surgery, and geospatial analysis. Benchmarks increasingly test more than recognition, including long-horizon tracking, unseen past or future states, causal inference, streaming video, and reasoning under changing viewpoints. Models have improved with structured representations and targeted training, but reliable visual grounding, memory, temporal extrapolation, and evaluation that distinguishes genuine reasoning from answer shortcuts remain open challenges.

Latest papers

71 papers11 papers in the last four weeks, up 38% on the four weeks before. 0.1% of all new papers.

SpaceAct: Benchmarking 3D Spatial and Temporal Reasoning via Closed-Loop Action and Active Perception
Oct 28, 2026Chenglong Shi, Zekai Yuan, Yue Song +2Spatial Reasoning Benchmarks Spatiotemporal Reasoning
Tencent HY
Gaussian Episodic Memory for Long-Term Spatiotemporal Reasoning
Oct 23, 2026Ling-Chia Chen, Abdelrhman Werby, Kaan Aytekin +1Spatiotemporal Reasoning Long-Term Memory for Embodied Agents
Socially Intelligent Robotics Lab, Institute for Artificial Intelligence, University of Stuttgart, Germany
Causal Spatiotemporal State Evolution for Monocular 4D Reconstruction
Oct 20, 2026Qiaowei Miao, Yijing Guo, Lipei Zhang +44D Reconstruction Dynamic Scene Reconstruction
Zhejiang University · ShanghaiTech University · University of Cambridge +2
SENRI: Understanding the Space-Event Nexus in Real-Life Videos
Oct 18, 2026Kehan Lan, Kaining Ying, Henghui Ding +1Egocentric Video Understanding Spatiotemporal Reasoning
Fudan University · Alaya Lab
STRIDE: Evaluating Spatiotemporal Reasoning in Driving Edge Cases
Oct 16, 2026Lieqi Liu, Rui gao, Jia-Chen Gu +6Autonomous Driving Benchmarks Spatiotemporal Reasoning
University of California, Los Angeles · Toyota InfoTech Labs
Event-STVG: Spatio-Temporal Video Grounding for Long-Form Event Reasoning
Oct 15, 2026Yizhe Zhang, Xujia Wang, Songqi Tang +10Video Reasoning Temporal Video Grounding
Tsinghua University · Beijing University of Posts and Telecommunications
Anticipatory Commonsense with Moving Objects: An ASP-Based Framework for Predictive Reasoning with Interacting Trajectories
Oct 14, 2026Julius Monsen, Jakob Suchan, Mehul BhattSpatiotemporal Reasoning Multi-Agent Trajectory Prediction
Örebro University, Sweden · CoDesign Lab EU · Constructor University Bremen, Germany
DynThinker: Unifying Static 3D and Dynamic 4D Spatial Reasoning
Oct 14, 2026Qihang Cao, Haoyuan Li, Zihan Guo +8Spatiotemporal Reasoning Spatial Reasoning
1Nanyang Technological University · 2Shenzhen campus of Sun Yat-sen University · 3Yinwang Intelligent Technology Co. Ltd +1
SceneForesee: Learning Scene Evolution for Spatiotemporal Reasoning
Oct 10, 2026Chaoyue Li, Shoudong Han, Haoyang Wu +4Spatiotemporal Reasoning 4D Scene Understanding
Huazhong University of Science and Technology · Shanghai Jiao Tong University
SpaTime: Streaming Vision-Language Models for Spatio-temporal Reasoning
Oct 6, 2026Hairong Yin, Huangying Zhan, Shin-Fang Chng +2Vision-Language Models Spatiotemporal Reasoning
Department of Computer Science, Purdue University, USA · Goertek Alpha Labs, USA
Why Do Conventional World Models Fail to Learn Cellular Automata?
Sep 30, 2026Shaoyang Guo, Ziming LiuWorld Model Learning Spatiotemporal Reasoning
Meta Circle(元环智能) · Peking University · Tsinghua University
Long Time No See: Benchmarking VLMs for Out-of-Sight Spatiotemporal Reasoning in Egocentric Videos
Sep 28, 2026Fangzhou Ma, Ivo Alexander Ban, Eren Homburg +5Temporal Video Grounding Visual Spatial Reasoning
ETH Zurich · Microsoft Spatial AI Lab · Bocconi University
MinkowskiPE: Minkowski Positional Encoding for Spatiotemporal Perception
Sep 27, 2026Yuhao Li, Louie Hong Yao, Tianyi Shi +4Spatiotemporal Reasoning Positional Encoding
Wave Intelligence Lab · Department of Computer Science and Engineering, The Chinese University of Hong Kong · Shanghai Artificial Intelligence Laboratory
OCC4M: Object-Centric 4D Memory for Spatiotemporal Reasoning in Long-Horizon Manipulation
Sep 23, 2026Jack B. Jedlicki, Tanguy Dieudonné, Heng YangLong-Horizon Robotic Manipulation Spatiotemporal Reasoning
Harvard University · ETH Zürich
EgoTSR++: Egocentric Spatiotemporal Reasoning for Task Progress Understanding
Sep 23, 2026Xiaoda Yang, Can Wang, Yuxiang Liu +3Robot Task Progress Estimation Temporal Reasoning in VLMs
Zhejiang University, Hangzhou, China · Qingdao University, Qingdao, China · Tianjin University, Tianjin, China +1
ReSTI: A Source-Grounded Audit and Repair of STI-Bench
Sep 21, 2026Pengzhan Sun, Ramanathan Rajaraman, Shiu-Hong Kao +2Spatial Reasoning Benchmarks Spatiotemporal Reasoning
National University of Singapore · University of Science and Technology of China
Do LiDAR Language Models Really Understand Spatio-temporal Relationships?
Sep 21, 2026Runyi Yang, Murat Akkoyun, Di Wen +8VLM Evaluation Hallucination Detection in VLMs
INSAIT, Sofia University “St. Kliment Ohridski”, Bulgaria · Karlsruhe Institute of Technology, Germany · Hunan University, China
Unified Vision-Centric Pedestrian Crossing Action Prediction via Adaptive Patch Projection and Proactive Spatial Rectification
Sep 7, 2026Yao Tian, Le Yang, Binglu WangPedestrian Intention Prediction VLMs for Autonomous Driving
School of Electronics and Control Engineering, Chang’an University, Xi’an 710064, China · School of Astronautics, Northwestern Polytechnical University, Xi’an 710072, China
CASCADE: A Spatio-Temporal-Causal Reasoning Representation and Dataset for Driving
Sep 7, 2026Jenny Schmalfuss, Despoina Paschalidou, Simon Gerstenecker +2Spatiotemporal Reasoning Autonomous Driving Datasets
NVIDIA · ETH Zürich · Work done during NVIDIA Internship
Continuous Token-Level Spatio-Temporal Context Modeling for Visual Object Tracking
Sep 7, 2026Ding Xia, Meiqin Liu, Jing Zhou +1Spatial Context Modeling Visual Object Tracking
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’an 710049, China. · College of Electrical Engineering, Zhejiang University, Hangzhou 310027, China. · School of Electronics and Information Engineering, Xi’an Jiaotong University, Xi’an 710049, China.
STEGNav: Spatio-Temporal Event Graph Reasoning for Multimodal Lifelong Object Navigation
Aug 28, 2026Yang Chen, Zhenyu Huang, Wenbo Fu +4Vision-Language Navigation Object-Goal Navigation
State Key Laboratory of Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · School of Artificial Intelligence, Nanjing University
CoAnchor: Robust Collaborative Perception under Spatio-Temporal Misalignment via Object-Level Anchors
Aug 21, 2026Chi Li, Rui Lin, Aobo Ji +1Cooperative Perception Spatiotemporal Reasoning
State Key Laboratory of Networking and Switching Technology Beijing University of Posts and Telecommunications Beijing, China
GESTO: Human-Centric Spatio-Temporal Memory for Reasoning in Dynamic Scenes
Aug 11, 2026Ermanno Bartoli, Buwei He, Dennis Rotondi +6Spatiotemporal Reasoning Graph-Based Agent Memory
Department of Robotics, Perception and Learning, KTH Royal Institute of Technology, Sweden · Socially Intelligent Robotics Lab at the Institute for Artificial Intelligence, University of Stuttgart, Germany · Intl. Max Planck Research School for Intelligent Systems (IMPRS-IS) +2
PULSE: An Executable Contract Language for Spatiotemporal Knowledge Graph Engineering
Jul 26, 2026Dongxu Yang, Ziyi LiangKG Construction Spatiotemporal Reasoning
DeepLethe
Structure over Depth: A Single-Block Spatio-Temporal Transformer for Multi-Entity Reasoning
Jul 25, 2026Narthana Sivalingam, Santhirarajah Sivasthigan, Buddhi Wijenayake +3Spatiotemporal Reasoning Human Activity Recognition
Department of Electrical and Electronic Engineering University of Peradeniya Peradeniya, Sri Lanka
From Scalars to Time Series: Rethinking Implicit Neural Representations for Time-Varying Volumetric Data
Jul 23, 2026Weihan Zhang, Xuan Zhao, Yenwen Peng +2Implicit Neural Representations Spatiotemporal Reasoning
School of Computer Science and Engineering, Sun Yat-sen University · Department of Computer Science, Hong Kong Baptist University · School of Computer Science and Engineering, Sun Yat-sen University, Guangdong Province Key Laboratory of Computational Science, and National Supercomputer Center in Guangzhou
ViSTR-Bench: Can MLLMs Reason from Continuous Visual Cues in Dynamic Scenes?
Jul 23, 2026Han Li, Si Liu, Zehao Huang +6VLM Evaluation Spatial Reasoning Benchmarks
School of Artificial Intelligence, Beihang University, Beijing 100191, China · Zhongguancun Academy, Beijing 100094, China · School of Engineering, Westlake University, Hangzhou 310030, China
Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMs
Jul 17, 2026Like Liu, Zhengzheng Xu, Haitao He +3Multimodal Grounding Aerial Robotics
Northwestern Polytechnical University Xi’an, Shaanxi, China
GeoChrono: Benchmarking and Rethinking Long-Term Temporal Understanding in Remote Sensing
Jul 17, 2026Yujie Li, Jiancheng Pan, Zhiwei Wei +3Spatiotemporal Reasoning Geospatial Representation Learning
Beijing University of Posts and Telecommunications Haidian, Beijing, China · Tsinghua University Haidian, Beijing, China · Hunan Normal University Changsha, Hunan, China +1
From Continuous Deployment to Queryable Dataset: Terabyte-Scale AIS-Aligned Passive Acoustic Labelling
Jul 15, 2026Wayne Renaud, Priyanka Aravindan, Gabriel SpadonSpatiotemporal Reasoning Passive Acoustic Monitoring
Faculty of Computer Science, Dalhousie University, Halifax, Canada

Oct 28, 2026J/K move · Enter open · S save

SpaceAct: Benchmarking 3D Spatial and Temporal Reasoning via Closed-Loop Action and Active Perception

Chenglong Shi, Zekai Yuan, Yue Song, Jon Pu+1

Tencent HY

Spatial Reasoning Benchmarks Spatiotemporal Reasoning Multimodal Spatial Reasoning Active Perception 3D Visual Grounding

Open paper AskPDF arXiv

To solve problems in 3D space rather than merely describe images, multimodal large language models (MLLMs) need 3D spatiotemporal reasoning: reconstructing 3D geometry from occluded 2D views, anticipating physical dynamics, and executing precise, closed-loop actions. Prior studies test passive VQA on static images or symbolic planning, requiring no execution or supplying privileged coordinates while ignoring collision, statics, and timing. We introduce SpaceAct, an interactive benchmark evaluating 3D spatiotemporal reasoning via closed-loop physical action and active perception, with SAPA, a Spatiotemporal Active-Probing Agent framework. Built on interlocking bricks and mechanical puzzles, it has 108 verified instances across 36 task families in six categories, without privileged coordinates, and diagnostic modes (+plan, +state, no-camera, blind, L7 levers). In brick-bridging or aperture-threading, models rotate the camera to resolve occlusion, read in-scene rulers, output 6-DoF poses under swept-collision and statics checks, and self-correct via split-screen rollback feedback (dynamic tasks freeze time during deliberation and give five strobe frames). Across 10 frontier MLLMs on the 44 hardest instances, scores span 4.8-79.1 (seven solve at most 27.3% of 6-DoF tasks); for the top three, a reference plan (+plan) closes 44% of the gap while leaving collisions at 5.2 per episode, whereas adding exact 3D state (+state) closes 90% and cuts collisions to 0.5; for four others, +plan lifts the mean score from 10.9 to 35.3 and +state to 81.8; and 46% of failures are collision or observation loops. SpaceAct and SAPA advance evaluation to closed-loop physical interaction, showing that even with a valid plan, MLLMs remain bottlenecked by 3D visual grounding and action refinement, while offering a testbed for active multi-view exploration and explicit 3D grounding.