Temporal Video Understanding

Follow this topic Ask about this topic

Momentum

23 papers in the last four weeks, up 109% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 220

All topics

SpaTime: Streaming Vision-Language Models for Spatio-temporal Reasoning
Oct 6, 2026Hairong Yin, Huangying Zhan, Shin-Fang Chng +2Vision-Language Models Spatiotemporal Reasoning
Department of Computer Science, Purdue University, USA · Goertek Alpha Labs, USA
EC-RAG: Event Chain Retrieval-Augmented Generation for Long Video Understanding
Oct 6, 2026Yuhao Qin, Junbo Wang, Yuke Li +1Multimodal RAG Video-Language Models
Northwestern Polytechnical University Xi’an, China
Multi-Label Perceptual Bug Detection in Video Games using Deep Learning on Gameplay Footage
Oct 6, 2026Nahian Rifaat, Felix Morosov, Loutfouz ZamanAutomated Software Testing Multi-Label Classification
Ontario Tech University Oshawa, Ontario, Canada · Universität Osnabrück Osnabrück, Niedersachsen, Germany
Joint Class-Time Learning for Video Classification with Multi-Instance Partial-Label Learning
Oct 5, 2026Lingyu Shen, Wei Tang, Fakhri Karray +1Weakly Supervised Learning Multiple Instance Learning
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, United Arab Emirates
Cog-VADU: A Training-Free Cognitive Reasoning Framework for Video Anomaly Detection and Understanding
Oct 1, 2026Mohd Ubaid Wani, Sara Atito, Josef Kittler +1CoT Reasoning Interpretable Anomaly Detection
Centre for Vision, Speech and Signal Processing (CVSSP) University of Surrey, UK · Surrey Institute for People-Centred AI (PAI) University of Surrey, UK
FlashBack: Knowing When to Remember in Streaming Vision-Language Models
Oct 1, 2026Yi Chen, MingMing Yu, Rui-Qi Wang +5Memory-Augmented VLMs Vision-Language Models
Rightly Robotics, A4X
Video-Index: A Curated Meta-Benchmark for Video Understanding
Oct 1, 2026Enxin Song, Yinuo Xu, Shusheng Yang +2Video Understanding Video-Language Model Evaluation
University of Pennsylvania · New York University · Princeton University
RESUME: Recurrent State Updates from Motion and Residual Signals for Efficient Video Language Modeling
Sep 30, 2026Can Zhang, Xiaotian Han, Junyuan Shang +5Video-Language Models Video Representation Learning
Beijing University of Chemical Technology · Baidu, Inc.
TSMD: Temporal-Stream Modality Dropout for Robust Video Highlight Detection
Sep 30, 2026Bo-Yuan Cheng, Kuan-Yu Chen, Po-Han Huang +2Missing-Modality Learning Multimodal Robustness
AI Research Center, Inventec Corporation, Taiwan · Graduate Institute of Communication Engineering, National Taiwan University, Taiwan
SYNCR: Diagnosing and Learning Cross-Video Reasoning from Simulation
Sep 29, 2026Sara Ghazanfari, Siddharth Garg, Prashanth Krishnamurthy +1Video-Language Model Evaluation Video Reasoning
New York University
EGSD: Event-Grounded Self-Distillation for Streaming Video Understanding
Sep 29, 2026Yuwei Miao, Xuesheng Zhang, Wenhao Zou +5Streaming Video Understanding Self-Distillation
Baidu · University of Chinese Academy of Sciences · China University of Petroleum, Beijing
RoboChrono: A Real Robot Benchmark for Streaming Task Understanding
Sep 29, 2026Yuzhou Wu, Longteng Fan, Zimeng Li +22Next-Action Prediction Robot Manipulation Benchmarks
Tianji Tec. · Shenzhen University · General Intelligence Machine +7
Foresight at the Event Boundary: Evaluating Physical Prediction in Video World Models
Sep 29, 2026Estela Monserrat Arriaga Santana, Julian Rosas Scull, Ehécatl Sacamch'en Núñez Rico +1Physical Consistency in Video Generation World Model Evaluation
National Autonomous University of Mexico · National Autonomous University of Mexico, Mexico · University of Texas at El Paso +1
ReVA: A Scene-Centric Dataset Beyond Repetition for Remote Sensing Video Question Answering
Sep 28, 2026Zhen Yao, Likai Wang, Yuming Yang +9Remote Sensing VQA Video QA
Lehigh University · University of Southern California · Qualcomm AI Research
ActionLens: Diagnosing Spatial-Temporal Binding Failures in Vision-Language Models
Sep 28, 2026Gueter Josmy Faure, Min-Hung Chen, Hao Ping Wang +3Video-Language Model Evaluation Temporal Video Understanding
National Taiwan University · NVIDIA
A Hierarchy-Aware Video-Language Model Evaluation and Hyperbolic Baseline for Surgery
Sep 22, 2026Ana Manzano Rodríguez, Pascal Mettes, Marlies P. Schijven +1Video-Language Model Evaluation Hierarchical Representation Learning
Data Science Center HAVA-Lab, University of Amsterdam · Video & Image Sense Lab, Informatics Institute, University of Amsterdam · Amsterdam UMC Location University of Amsterdam, Surgery +2
Not Another Text Benchmark: Putting the "Visual" Back in Visual Question Answering for Large Video Models
Sep 15, 2026Rwiddhi Chakraborty, Yinong, Wang +7Video-Language Model Evaluation Video QA
Oliver · University of Copenhagen · Carnegie Mellon University +4
BVB: Benchmarking Agentic Video Understanding via Programmatic Reconstruction in Blender
Sep 14, 2026Yolo Y. Tang, Daiki Shimada, Jiayue Meng +14AI Agent Benchmarks Temporal Video Understanding
University of Rochester · Sony Group Corporation · Carnegie Mellon University +1
CapsuleMotion: A Lightweight Real-Time Visual Motion Predictor for Capsule Endoscopy
Sep 14, 2026Oliver Bause, Julia Werner, Oliver BringmannEndoscopy Medical Imaging
University of Tübingen, Faculty of Science, Department of Computer Science, Embedded Systems Group
OphBiWSSD: Scaling Temporal Action Localization in Ophthalmic Surgeries with Bidirectional Weight-tied State Space Duality
Sep 14, 2026Yang Liu, Qionghong Ma, Joongwon Chae +10Surgical Video Understanding Temporal Action Detection
Institute of Biopharmaceutics and Health Engineering, Tsinghua Shenzhen International Graduate School, Shenzhen 518055, China · Department of Ophthalmology, The Fifth Affiliated Hospital of Wenzhou Medical University, Wenzhou Medical University, Lishui 323000, China · School of Optometry, The Hong Kong Polytechnic University +4
Zero-shot video highlight detection based on text descriptions and synthetic images
Sep 13, 2026Michal Byra, Alberto Presta, Grzegorz Stefanski +1Video Understanding Temporal Video Understanding
Samsung AI Center, Warsaw, Poland · Institute of Fundamental Technological Research, Polish Academy of Sciences, Warsaw, Poland
LiveProBench: Can Streaming Video Models Really Interact Like Humans?
Sep 11, 2026Kaixuan Du, Xin Wan, Hang Zhang +4Streaming Video Understanding Video-Language Model Evaluation
School of Automation Science and Electrical Engineering, Beihang University · Qwen Business Unit of Alibaba
VLX-VR: An Agentic-Aware Video Reasoning Model
Sep 9, 2026Sheng Li, Peng Liu, Qianqian Zhang +1Video Reasoning Memory-Augmented Video Understanding
Om AI Research
MotionBlind: Probing the Illusion of Motion Understanding in Video-LLMs
Sep 8, 2026Dhairya Bhatia, Bishoy Galoaa, Oliver Fritsche +7Video-Language Model Evaluation Video-Language Models
Electrical and Computer Engineering Department Northeastern University, Boston, MA, USA
EgoSIS: From Factorized Visual Ego-Transitions to Motion-Canonical Spatial Evidence for UAV Reasoning
Sep 8, 2026Jingpu Yang, Fengxian Ji, Mingxuan Cui +4Visual Spatial Reasoning Egocentric Video QA
Beihang University, Beijing, China · Zhongguancun Academy, Beijing, China · Northeastern University, Shenyang, China +1
Kairos: A Dataset for Fine-Grained Video-Language Modeling over Space, Time, and Dynamics
Sep 8, 2026Ruibo Ming, Lei Sun, Deheng Zhang +8Video-Language Model Evaluation Video-Language Models
1INSAIT, Sofia University “St. Kliment Ohridski” · 2Adobe Research · 3Snap Research
STSG-VQA: Evidence-Grounded Temporal Question Answering from Surgical Spatio-Temporal Scene Graphs
Sep 8, 2026Jing Li, Duygu SarikayaMedical VQA Question Answering
School of Computer Science, University of Leeds, UK
Temporal Self-Distillation: Learning Visual State Tracking in Videos Without Supervision
Sep 3, 2026Shravan Venkatraman, Wenshuai Zhao, Mohammad Hassan Vali +1Self-Supervised Learning State Tracking
Mohamed bin Zayed University of Artificial Intelligence, UAE · Aalto University, Finland · ELLIS Institute Finland & Department of Computer Science, Aalto University, Finland
The Shape of Time: Video-Token Contrast for Temporal Understanding in VideoLMs
Sep 3, 2026Yumeng Shi, Quanyu Long, Yin Wu +1Video-Language Models Video Representation Learning
Nanyang Technological University
TAME: Temporal-Aware Mixture-of-Experts for Text-Video Retrieval
Sep 2, 2026Uicheol Jung, Juyoung Hong, Hojung Kwon +1Temporal Video Understanding Mixture of Experts
Sejong University, Seoul, Republic of Korea · Wisenut, Seongnam, Republic of Korea

Oct 6, 2026, cs.CVJ/K move · Enter open · S save

SpaTime: Streaming Vision-Language Models for Spatio-temporal Reasoning

Hairong Yin, Huangying Zhan, Shin-Fang Chng, Yi Xu+1

Department of Computer Science, Purdue University, USA · Goertek Alpha Labs, USA

Vision-Language Models Spatiotemporal Reasoning VLM Reasoning 3D VLMs Temporal Video Understanding

Open paper AskPDF arXiv

Embodied agents must reason about 3D space while the video is still arriving, answering questions as soon as they have observed enough of the scene. VLMs that incorporate 3D geometric priors achieve strong spatial reasoning, but they operate offline, i.e., the full video must be available before they produce an answer. Streaming VLMs process frames causally and decide for themselves when to respond, yet they lack explicit 3D representations. We present SpaTime, a streaming VLM that fuses causal geometry tokens into the language model at every frame, using only the frames observed so far. To supervise when the model answers, we propose a response-time loss that maps per-frame response probabilities to a differentiable expected response time and penalizes the distance from the ground-truth frame. For evaluation, we construct StreamVSTI-Bench and StreamVSI-Bench, streaming adaptations of VSTI-Bench and VSI-Bench. On StreamVSTI-Bench, SpaTime reaches 49.2% overall accuracy and reduces the mean response-time error by 66% relative to the strongest streaming baseline.

Momentum

Related topics

Latest papers 220