Temporal Video Understanding

Ask about this topic

Overview

Temporal video understanding studies how actions and events unfold, how objects and states change, and how evidence across frames supports tasks such as action localization, moment retrieval, video question answering, and future prediction. Approaches include temporal transformers and state-space models, motion and trajectory representations, event graphs and memory, and adaptive frame or token selection; vision-language models are also trained or guided to reason over video evidence. Evaluation spans action-localization datasets, long-video question answering, temporal-order and counting tests, and streaming settings where models must decide when to respond. A central tension is that models can perform well on broad benchmarks while relying on static cues or language priors rather than temporal evidence, and efficient sampling or compression must retain the brief events needed for reliable reasoning.

Latest papers

CardsList

371 papers45 papers in the last four weeks, up 45% on the four weeks before. 0.3% of all new papers.

  1. Denoise and Align: Diffusion-Driven Foreground Knowledge Prompting for Open-Vocabulary Temporal Action Detection

    Apr 20, 2026Sa Zhu, Wanqian Zhang, Lin Wang +3Cross-Modal AlignmentTemporal Action Detection

    Institute of Information Engineering, Chinese Academy of Sciences, School of Cyber Security, University of Chinese Academy of Sciences, State Key Laboratory of Cyberspace Security Defense, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · Hangzhou Dianzi University, Hangzhou, China +1

  2. LiquidTAD: Efficient Temporal Action Detection via Parallel Liquid-Inspired Temporal Relaxation

    Apr 20, 2026Zepeng Sun, Naichuan Zheng, Hailun Xia +3Liquid Neural NetworksHuman Activity Recognition

  3. Lost in Motion: Vision Language Models Fail the Dynamic Gauges Test

    Apr 19, 2026Tairan Fu, Francisco Javier Santos-Martín, Javier Conde +2VLM EvaluationIndustrial Inspection

  4. Relaxing Anchor-Frame Dominance for Mitigating Hallucinations in Video Large Language Models

    Apr 14, 2026Zijian Liu, Sihan Cao, Pengcheng Zheng +5Video-Language ModelsAttention Mechanisms

    University of Electronic Science and Technology of China · Institute of Software Chinese Academy of Sciences · Harbin Institute of Technology, Shenzhen +1

  5. Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Grounding

    Apr 13, 2026Tencent Hunyuan TeamVideo CaptioningTemporal Video Understanding

  6. Do Thought Streams Matter? Evaluating Reasoning in Gemini Vision-Language Models for Video Scene Understanding

    Apr 13, 2026Shivam Sharma, Sankalp Nagaonkar, Ashish Choithani +1VLM ReasoningVLM Evaluation

    VideoDB∗

  7. BoxTuning: Directly Injecting the Object Box for Multimodal Model Fine-Tuning

    Apr 13, 2026Zekun Qian, Ruize Han, Wei FengMultimodal Large Language ModelsMultimodal QA

    School of Computer Science and Technology, Tianjin University, Tianjin, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China

  8. OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video

    Apr 13, 2026Junfu Pu, Yuxin Chen, Teng Wang +1Long-Video UnderstandingMultimodal Large Language Models

    ARC Lab, Tencent

  9. A Progressive Training Strategy for Embodied Vision-Language Models to Mitigate Spatio-Temporal Hallucinations

    Apr 12, 2026Xiaoda Yang, Shuai Yang, Can Wang +9VLM AdaptationVLM Hallucination

    The Hong Kong University of Science and Technology · The Chinese University of Hong Kong

  10. Attention-Guided Dual-Stream Learning for Group Engagement Recognition: Fusing Transformer-Encoded Motion Dynamics with Scene Context via Adaptive Gating

    Apr 11, 2026Saniah Kayenat Chowdhury, Muhammad E. H. ChowdhuryAI in EducationTemporal Video Understanding

    Department of Robotics and Mechatronics Engineering, University of Dhaka, Dhaka, Bangladesh · Department of Electrical Engineering, Qatar University, Doha, 2713, Qatar

  11. Efficient Spatial-Temporal Focal Adapter with SSM for Temporal Action Detection

    Apr 10, 2026Yicheng Qiu, Keiji YanaiTemporal Action DetectionTemporal Action Localization

    Department of Informatics The University of Electro-Communications Chofu, Tokyo, Japan

  12. InstrAct: Towards Action-Centric Understanding in Instructional Videos

    Apr 9, 2026Zhuoyi Yang, Jiapeng Yu, Reuben Tan +2Video UnderstandingVideo-Language Models

    Pennsylvania State University, University Park, USA · Microsoft Research · Nanyang Technological University, Singapore

  13. CrashSight: A Phase-Aware, Infrastructure-Centric Video Benchmark for Traffic Crash Scene Understanding and Reasoning

    Apr 9, 2026Rui Gan, Junyi Ma, Pei Li +4Autonomous Driving BenchmarksAutonomous Driving Safety Evaluation

    University of Wisconsin–Madison · University of Wyoming · Columbia University

  14. HST-HGN: Heterogeneous Spatial-Temporal Hypergraph Networks with Bidirectional State Space Models for Global Fatigue Assessment

    Apr 9, 2026Changdao Chen, Qinqiuhong Ye, Hao Chen +1Heterogeneous GNNsHypergraph Neural Networks

    School of Computer Science and Technology, Xi’an Jiaotong University Xi’an, Shaanxi, China · School Of Mechanical Engineering, Xi’an Jiaotong University Xi’an, Shaanxi, China · Xi’an Jiaotong University Health Science Center Xi’an, Shaanxi, China

  15. ABMAMBA: Multimodal Large Language Model with Aligned Hierarchical Bidirectional Scan for Efficient Video Captioning

    Apr 9, 2026Daichi Yashima, Shuhei Kurita, Yusuke Oda +3Video CaptioningMultimodal Large Language Models

    Keio University, 3-14-1, Kohoku Ward, Yokohama, Kanagawa 223-8522, Japan · National Institute of Informatics Research and Development Center for Large Language Models, 1-1-1, Hitotsubashi, Chiyoda, Tokyo 100-0003, Japan · National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda, Tokyo 101-8430, Japan

  16. Event-Level Detection of Surgical Instrument Handovers in Videos with Interpretable Vision Models

    Apr 8, 2026Katerina Katsarou, George Zountsas, Karam Tomotaki-Dawoud +6Surgical Video UnderstandingTemporal Video Understanding

    Fraunhofer HHI, Berlin, Germany · Technical University of Berlin, Germany · Universit´e de Perpignan, Perpignan, France

  17. Watch Before You Answer: Learning from Visually Grounded Post-Training

    Apr 6, 2026Yuxuan Zhang, EunJeong Hwang, Huaisong Zhang +8VLM Post-TrainingVision-Language Models

    University of British Columbia · Vector Institute · Etude AI +4

  18. Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding

    Apr 6, 2026Chaoyou Fu, Haozhi Yuan, Yuhao Dong +16Video UnderstandingVideo-Language Model Evaluation

  19. When Rewards Disagree: Reward Weighting for Token-Efficient Video Reasoning

    Apr 6, 2026Zinuo Li, Yongxin Guo, Jun Liu +6Video ReasoningVLM Reasoning

    University of Western Australia · Tencent Youtu Lab · CUHKSZ +2

  20. Single-agent vs. Multi-agents for Automated Video Analysis of On-Screen Collaborative Learning Behaviors

    Apr 4, 2026Likai Peng, Shihui FengVision-Language ModelsMulti-Agent Collaboration

    Faculty of Education, The University of Hong Kong, Pokfulam, Hong Kong · Institute of Data Science, The University of Hong Kong, Pokfulam, Hong Kong

  21. Reinforcing Consistency in Video MLLMs with Structured Rewards

    Apr 1, 2026Yihao Quan, Zeru Shi, Jinman Zhao +1Video-Language ModelsObject Hallucination in VLMs

    Rutgers University · University of Toronto

  22. An Approach to Enriching Surgical Video Datasets for Fine-Grained Spatial-Temporal Understanding of Vision-Language Models

    Apr 1, 2026Lennart Maack, Alexander SchlaeferMedical VQASurgical Video Understanding

    Hamburg University of Technology, Germany

  23. TTA-Vid: Generalized Test-Time Adaptation for Video Reasoning

    Apr 1, 2026Soumya Shamarao Jahagirdar, Edson Araujo, Anna Kukleva +7Test-Time AdaptationVideo Reasoning

    University of Tübingen · Max Planck Institute for Informatics, SIC · MIT +3

  24. SurgTEMP: Temporal-Aware Surgical Video Question Answering with Text-guided Visual Memory for Laparoscopic Cholecystectomy

    Mar 31, 2026Shi Li, Vinkle Srivastav, Shih-Min Yin +7Medical VQASurgical Video Understanding

    University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France · IHU Strasbourg, Strasbourg, France · Department of General Surgery, Kaohsiung Chang Gung Memorial Hospital, Chang Gung University College of Medicine, Kaohsiung, Taiwan +1

  25. Video-Oasis: Rethinking Evaluation of Video Understanding

    Mar 31, 2026Geuntaek Lim, Sungjune Park, Jaeyun Lee +5VLM EvaluationTemporal Video Understanding

    Sejong University, South Korea · NAVER Cloud, South Korea

  26. VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inference

    Mar 31, 2026Anmin Liu, Ruixuan Yang, Huiqiang Jiang +5Transformer AttentionSparse Attention

    SCS, Peking University, Beijing, China · Key Lab of HCST (PKU), MOE, Beijing, China · Fudan University, Shanghai, China +2

  27. CLIP-AUTT: Test-Time Personalization with Action Unit Prompting for Fine-Grained Video Emotion Recognition

    Mar 30, 2026Muhammad Osama Zeeshan, Masoumeh Sharafi, Benoit Savary +3VLM AdaptationFacial Expression Recognition

    LIVIA, ILLS, Dept. of Systems Engineering, ETS Montreal, Canada · Dept. of Computer Science, École Polytechnique, Paris, France · LIVIA, Dept. of Software and IT Engineering, ETS Montreal, Canada

  28. VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation

    Mar 28, 2026Jihwan Hong, Jaeyoung DoVideo Object SegmentationReferring Video Object Segmentation

    AIDAS Laboratory, 1IPAI, Seoul National University · AIDAS Laboratory, 2ECE, Seoul National University

  29. PerceptionComp: A Video Benchmark for Complex Perception-Centric Reasoning

    Mar 27, 2026Shaoxuan Li, Zhixuan Zhao, Hanze Deng +9Video ReasoningTemporal Video Understanding

    Tsinghua University · Nanyang Technological University · University of Washington

  30. Dynamic Token Compression for Efficient Video Understanding through Reinforcement Learning

    Mar 27, 2026Shida Wang, YongXiang Hua, Zhou Tao +2Video Token CompressionTemporal Video Understanding

    University of Science and Technology of China · State Key Laboratory of Cognitive Intelligence