Video-Language Models

Follow this topic Ask about this topic

Video-language models study how systems can interpret video and use language to answer questions, describe events, locate moments, and assist during live streams. Common approaches combine pretrained vision encoders with large language models, using frame sampling or temporal representations, instruction tuning and reinforcement learning, and retrieval or tools to handle long videos. Token compression and selective frame processing help control the high cost of video inference, while benchmarks such as VideoMME, MVBench, and ActivityNet test general understanding and temporal localization. A central open challenge is ensuring that answers genuinely reflect visual and temporal evidence: models can recognize scenes yet miss motion, event order, or brief details, and may hallucinate or rely on language priors.

Latest papers

167 papers27 papers in the last four weeks, up 125% on the four weeks before. 0.2% of all new papers.

DynaTokens: Controlling Token Dynamics for Continual Video-Language Understanding
Mar 2, 2026Toan Nguyen, Yang Liu, Celso De Melo +1Continual Learning Video QA
School of Computer Science and Engineering, University of New South Wales, Australia · DEVCOM Army Research Laboratory, USA
VidDoS: Universal Denial-of-Service Attack on Video-based Large Language Models
Mar 2, 2026Duoxun Tang, Dasen Dai, Jiyao Wang +3Adversarial Attacks on VLMs Adversarial Attacks
Shenzhen International Graduate School, Tsinghua University · The Chinese University of Hong Kong, Hong Kong SAR · McGill University +3
Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Models
Mar 2, 2026Jinlong Li, Liyuan Jiang, Haonan Zhang +1Video Token Compression Video-Language Models
University of Trento · Tsinghua University · University of Electronic Science and Technology of China
Unified Vision-Language Modeling via Concept Space Alignment
Mar 1, 2026Yifu Qiu, Paul-Ambroise Duquenne, Holger SchwenkVision-Language Models Vision-Language Alignment
University of Edinburgh · FAIR at Meta
Stateful Token Reduction for Long-Video Hybrid VLMs
Feb 27, 2026Jindong Jiang, Amala Sanjay Deshmukh, Kateryna Chumachenko +7Hybrid Mamba-Transformer Architectures Efficient VLM Inference
NVIDIA
MovieTeller: Tool-augmented Movie Synopsis with ID Consistent Progressive Abstraction
Feb 26, 2026Yizhi Li, Xiaohan Chen, Miao Jiang +2Video Summarization Video-Language Models
Zhejiang University Hangzhou, China · China Media Group Beijing, China · Watch AI Group Beijing, China
WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMs
Feb 25, 2026Yulin Zhang, Cheng Shi, Sibei YangVideo-Language Models Streaming Video Understanding
ShanghaiTech University · School of Computing and Data Science, The University of Hong Kong · School of Computer Science and Engineering, Sun Yat-sen University
QuickGrasp: Responsive Video-Language Querying Service via Accelerated Tokenization and Edge-Augmented Inference
Feb 23, 2026Miao Zhang, Ruixiao Zhang, Jianxin Shi +3Efficient VLM Inference VLM Inference
School of Computing Science, Simon Fraser University, Canada · Computer Science Department, University of Illinois Urbana-Champaign, United States · College of Cryptology and Cyber Science, Nankai University, China +1
Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models
Feb 11, 2026Sethuraman T, Savya Khosla, Aditi Tiwari +12VLM Robustness Video-Language Model Evaluation
University of Illinois Urbana-Champaign, USA · Adobe Research, USA · Qualtrics, USA +6
VideoSTF: Stress-Testing Output Repetition in Video Large Language Models
Feb 11, 2026Yuxin Cao, Wei Song, Shangzhi Xu +2VLM Evaluation VLM Robustness
National University of Singapore, Singapore · University of New South Wales, Australia · CSIRO’s Data61, Australia
Cattle-CLIP: A Multimodal Framework for Dairy Cattle Behaviour Recognition from Video
Oct 10, 2025Huimin Liu, Jing Gao, Daria Baran +3Animal Behavior Analysis Video-Language Models
Bristol Veterinary School, University of Bristol, Langford House, Dolberry, Churchill, Bristol, BS40 5DU, United Kingdom · School of Computer Science, University of Bristol, Merchant Venturers Building, Woodland Road, Bristol, BS8 1UB, United Kingdom
Oracle-RLAIF: An Improved Fine-Tuning Framework for Multi-modal Video Models using Reinforcement Learning from Ranking Feedback
Oct 2, 2025Derek Shi, Ruben Glatt, Christine Klymko +5Video-Language Models RL Fine-Tuning
Stanford University, CA, USA · Lawrence Livermore National Laboratory, CA, USA · Microsoft, WA, USA
Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy
Sep 22, 2025Geewook Kim, Minjoon SeoVLM Evaluation Audio-Visual Understanding
NAVER Cloud AI, South Korea · KAIST AI, South Korea
Diffusion-grounded VideoLLM for Entity-aware temoporal grounding
Aug 21, 2025Pengcheng Fang, Yuxia Chen, Yong Wang +2Temporal Video Grounding Video-Language Models
VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR
May 23, 2025Shenghui Chen, Po-han Li, Sandeep Chinchali +1Information Bottleneck Video Summarization
The University of Texas at Austin
Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questions
Feb 18, 2025Aditya K Surikuchi, Raquel Fernández, Sandro PezzelleVisual Storytelling Multimodal Learning
Institute for Logic, Language and Computation (ILLC), University of Amsterdam, Netherlands
Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models
Jan 14, 2025Yifang Xu, Yunzhuo Sun, Benxiang Zhai +4Video Moment Retrieval Temporal Video Grounding
Nanjing University · Dalian University of Technology · Nanjing University of Information Science and Technology

Mar 2, 2026J/K move · Enter open · S save

DynaTokens: Controlling Token Dynamics for Continual Video-Language Understanding

Toan Nguyen, Yang Liu, Celso De Melo, Flora D. Salim

School of Computer Science and Engineering, University of New South Wales, Australia · DEVCOM Army Research Laboratory, USA

Continual Learning Video QA Video-Language Models Prompt Learning

Open paper AskPDF arXiv

Continual VideoQA with multimodal LLMs remains challenging because sequential adaptation induces task interference, while storing task-specific prompts becomes impractical as task sequences grow. We introduce DynaTokens, a transformer-based token generator that dynamically produces fine-tuning tokens on demand, enabling task-adaptive prompt updates through shared generation weights. To mitigate forgetting, we introduce meta-learning-inspired regularisers that look ahead to avoid task-specific sharp update directions while anchoring the evolving generator to prior-task behaviours. We theoretically connect this objective to sharpness-aware optimisation, showing how it favours flatter cross-task minima and improves retention. DynaTokens combines gradient-free routing based on robust pretrained token and visual embeddings with lightweight auxiliary multimodal supervision, reducing router drift during continual adaptation. Across standard continual VideoQA benchmarks, DynaTokens achieves higher average accuracy and substantially lower forgetting than strong baselines. It also improves zero-shot generalisation and remains effective in longer domain-incremental sequences with extended task shifts. Finally, we introduce a challenging ImageQA->VideoQA protocol and show that DynaTokens enables robust cross-modal continual transfer.