Video-Language Models

Ask about this topic

Overview

Video-language models study how systems can interpret video and use language to answer questions, describe events, locate moments, and assist during live streams. Common approaches combine pretrained vision encoders with large language models, using frame sampling or temporal representations, instruction tuning and reinforcement learning, and retrieval or tools to handle long videos. Token compression and selective frame processing help control the high cost of video inference, while benchmarks such as VideoMME, MVBench, and ActivityNet test general understanding and temporal localization. A central open challenge is ensuring that answers genuinely reflect visual and temporal evidence: models can recognize scenes yet miss motion, event order, or brief details, and may hallucinate or rely on language priors.

Latest papers

CardsList

167 papers27 papers in the last four weeks, up 125% on the four weeks before. 0.2% of all new papers.

  1. Leveraging Vision-Language Models to Detect Attention in Educational Videos

    Apr 20, 2026Gabriel Becquet, Sébastien Lallé, Vanda Luengo +1VLM EvaluationEducational Technology

    LIP6, CNRS, SU · Sorbonne University, CNRS, LIP6 & PHENIX, F-75005 Paris, France · CNRS, LIP6, SU +2

  2. EasyVideoR1: Easier RL for Video Understanding

    Apr 18, 2026Chuanyu Qin, Chenxu Yang, Qingyi Si +6Reinforcement LearningVideo-Language Models

    Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

  3. Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translation

    Apr 16, 2026Yiyang Jiang, Li Zhang, Xiao-Yong Wei +1Sign Language TranslationVideo-Language Models

    The Hong Kong Polytechnic University · Sichuan University

  4. Distorted or Fabricated? A Survey on Hallucination in Video LLMs

    Apr 14, 2026Yiyang Huang, Yitian Zhang, Yizhou Wang +4VLM HallucinationVideo-Language Models

    Department of Electrical and Computer Engineering, Northeastern University · Khoury College of Computer Science, Northeastern University

  5. Relaxing Anchor-Frame Dominance for Mitigating Hallucinations in Video Large Language Models

    Apr 14, 2026Zijian Liu, Sihan Cao, Pengcheng Zheng +5Video-Language ModelsAttention Mechanisms

    University of Electronic Science and Technology of China · Institute of Software Chinese Academy of Sciences · Harbin Institute of Technology, Shenzhen +1

  6. ViLL-E: Video LLM Embeddings for Retrieval

    Apr 13, 2026Rohit Gupta, Jayakrishnan Unnikrishnan, Fan Fei +3Video-Language ModelsVision-Language Models

  7. Lost in Adaptation: Layer-Selective Recovery of Temporal Reasoning in Video-Language Models

    Apr 13, 2026Zihang Fu, Haonan Wang, Jian Kang +2Video-Language ModelsVLM Adaptation

    National University of Singapore · MBZUAI

  8. Tango: Taming Visual Signals for Efficient Video Large Language Models

    Apr 10, 2026Shukang Yin, Sirui Zhao, Hanchao Wang +4Video-Language ModelsVisual Token Pruning

    University of Science and Technology of China · Reconova AI Lab · Nanjing University

  9. InstrAct: Towards Action-Centric Understanding in Instructional Videos

    Apr 9, 2026Zhuoyi Yang, Jiapeng Yu, Reuben Tan +2Video UnderstandingVideo-Language Models

    Pennsylvania State University, University Park, USA · Microsoft Research · Nanyang Technological University, Singapore

  10. See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs

    Apr 7, 2026Yicheng Ji, Jun Zhang, Jinpeng Chen +4Speculative DecodingLLM Inference Acceleration

    ZJU · BUPT

  11. AURA: Always-On Understanding and Real-Time Assistance via Video Streams

    Apr 5, 2026Xudong Lu, Yang Bo, Jinpeng Chen +9Video-Language ModelsStreaming Video Understanding

    CUHK MMLab · Huawei Research

  12. KiToke: Distribution-Aware Token Compression for Video Large Language Models

    Apr 3, 2026Haifeng Huang, Biyin Xu, Yang LiVideo Token CompressionVisual Token Compression

    Department of Computer Science, Iowa State University.

  13. STEAR: Layer-Aware Spatiotemporal Evidence Intervention for Hallucination Mitigation in Video Large Language Models

    Apr 3, 2026Linfeng Fan, Yuan Tian, Ziwei Li +1VLM Hallucination MitigationVideo-Language Models

    Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · King Abdullah University of Science and Technology, Thuwal, Saudi Arabia

  14. GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding

    Apr 2, 2026Rong Fan, Kaiyan Xiao, Minghao Zhu +3Temporal Video GroundingVideo-Language Models

    Newcapec AI Research · Fudan University · Tongji University

  15. HieraVid: Hierarchical Token Pruning for Fast Video Large Language Models

    Apr 2, 2026Yansong Guo, Chaoyang Zhu, Jiayi Ji +2Token PruningVideo-Language Models

    Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China Xiamen University Xiamen, Fujian, China

  16. Reinforcing Consistency in Video MLLMs with Structured Rewards

    Apr 1, 2026Yihao Quan, Zeru Shi, Jinman Zhao +1Video-Language ModelsObject Hallucination in VLMs

    Rutgers University · University of Toronto

  17. GIFT: Global Irreplaceability Frame Targeting for Efficient Video Understanding

    Mar 26, 2026Junpeng Ma, Sashuai Zhou, Guanghao Li +9Video Frame SelectionVideo Understanding

    Institute of Science and Technology for Brain-inspired Intelligence, Fudan University · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Alibaba Group Holding Limited +2

  18. CliPPER: Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognition

    Mar 25, 2026Florian Stilz, Vinkle Srivastav, Nassir Navab +1Surgical Video UnderstandingVideo-Language Models

    University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France · IHU Strasbourg, France · Technical University of Munich, Germany

  19. ViKey: Enhancing Temporal Understanding in Videos via Visual Prompting

    Mar 24, 2026Yeonkyung Lee, Dayun Ju, Youngmin Kim +2Temporal Video UnderstandingVideo-Language Models

    Yonsei University

  20. Cluster-Wise Spatio-Temporal Masking for Efficient Video-Language Pretraining

    Mar 24, 2026Weijun Zhuang, Yuqing Huang, Weikang Meng +5Video-Language ModelsVision-Language Alignment

    Harbin Institute of Technology · Pengcheng Laboratory

  21. ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modeling

    Mar 24, 2026Shaobo Ju, Baiyang Song, Tao Chen +6Video Token CompressionVisual Token Pruning

    Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · National University of Defense Technology.

  22. Leum-VL Technical Report

    Mar 20, 2026Yuxuan He, Chaiming Huang, Yifan Wu +4Video UnderstandingVideo-Language Models

    Hainan Sihe Data Technology Co., Ltd.

  23. ParallelVLM: Lossless Video-LLM Acceleration with Visual Alignment Aware Parallel Speculative Decoding

    Mar 20, 2026Quan Kong, Yuhao Shen, Yicheng Ji +2Efficient VLM InferenceVideo-Language Models

    Zhejiang University, Hangzhou, China

  24. Do Vision Language Models Understand Human Engagement in Games?

    Mar 19, 2026Ziyi Wang, Qizan Guo, Rishitosh Singh +1VLM EvaluationVideo-Language Models

    Arizona State University

  25. Seeking Universal Shot Language Understanding Solutions

    Mar 19, 2026Haoxin Liu, Harshavardhan Kamarthi, Zhiyuan Zhao +2Video UnderstandingVideo-Language Model Evaluation

    Georgia Institute of Technology · Dolby Laboratories

  26. Video-CoE: Reinforcing Video Event Prediction via Chain of Events

    Mar 16, 2026Qile Su, Jing Tang, Rui Chen +2Video PredictionTemporal Video Understanding

    AMAP, Alibaba Group

  27. Geometry-Guided Camera Motion Understanding in VideoLLMs

    Mar 13, 2026Haoan Feng, Sri Harsha Musunuri, Guan-Ming SuCamera Motion UnderstandingVideo-Language Models

    University of Maryland, College Park · Dolby Laboratories Inc.

  28. Unleashing Video Language Models for Fine-grained HRCT Report Generation

    Mar 12, 2026Yingying Fang, Huichi Zhou, KinHei Lee +4CT Report GenerationRadiology Report Generation

    Bioengineering Department, Imperial College London, London, UK · School of Biomedical Engineering & lmaging Sciences, King’s College London, London,UK

  29. INFACT: A Diagnostic Benchmark for Induced Faithfulness and Factuality Hallucinations in Video-LLMs

    Mar 12, 2026Junqi Yang, Yuecong Min, Jie Zhang +2Video-Language Model EvaluationVideo-Language Models

    State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, UCAS

  30. Real-Time Generation of Game Video Commentary with Multimodal LLMs: Pause-Aware Decoding Approaches

    Mar 3, 2026Anum Afzal, Yuki Saito, Hiroya Takamura +5Language Model DecodingMultimodal Large Language Models

    School of CIT, Technical University of Munich · National Institute of Advanced Industrial Science and Technology (AIST) · The University of Tokyo +3