Video-Language Models
Overview
Video-language models study how systems can interpret video and use language to answer questions, describe events, locate moments, and assist during live streams. Common approaches combine pretrained vision encoders with large language models, using frame sampling or temporal representations, instruction tuning and reinforcement learning, and retrieval or tools to handle long videos. Token compression and selective frame processing help control the high cost of video inference, while benchmarks such as VideoMME, MVBench, and ActivityNet test general understanding and temporal localization. A central open challenge is ensuring that answers genuinely reflect visual and temporal evidence: models can recognize scenes yet miss motion, event order, or brief details, and may hallucinate or rely on language priors.
Latest papers
167 papers27 papers in the last four weeks, up 125% on the four weeks before. 0.2% of all new papers.