Progressive Video Condensation with MLLM Agent for Long-form Video Understanding

Yufei Yin; Yuchen Xing; Qianke Meng; Minghao Chen; Yan Yang; Zhou Yu

cs.CVApr 3, 2026

Progressive Video Condensation with MLLM Agent for Long-form Video Understanding

Authors: Yufei Yin, Yuchen Xing, Qianke Meng, Minghao Chen, Yan Yang, Zhou Yu

Organizations: Zhejiang Key Laboratory of Space Information Sensing and Transmission, Hangzhou Dianzi University, China

Abstract

Understanding long videos requires extracting query-relevant information from long sequences under tight compute budgets. Existing text-then-LLM pipelines lose fine-grained visual cues, while video-based multimodal large language models (MLLMs) can keep visual details but are too frame-hungry and computationally expensive. In this work, we aim to harness MLLMs for efficient video understanding. We propose ProVCA, a progressive video condensation agent that iteratively locates key video frames at multiple granularities. ProVCA first adopts a segment localization module to identify the video segment relevant to the query, then a snippet selection module to select important snippets based on similarity, and finally a keyframe refinement module to pinpoint specific keyframes in those snippets. By progressively narrowing the scope from coarse segments to fine frames, ProVCA identifies a small set of keyframes for MLLM-based reasoning. ProVCA achieves state-of-the-art zero-shot accuracies of 69.3% on EgoSchema, 80.5% on NExT-QA, and 77.7% on IntentQA, while using fewer frames than previous training-free methods.

Figures & tables

Fig. 1 : Conceptual comparisons of three Video understanding methods based on LLM (MLLM). Existing approaches either (a) directly reason with pre-trained video-based MLLMs at high computational cost, or (b) use LLM agents that convert frames to captions and rely on VLMs (e.g., CLIP) to filter key frames, discarding fine-grained visual cues. In contrast, our ProVCA (c) progressively condenses videos by using an MLLM to narrow down query-relevant content from coarse segments to fine frames, then feeds the selected key frames into the MLLM to generate the final answer.

Fig. 2 : Overview of ProVCA in video understanding based on MLLM.

Method	LLM/MLLM	NExT-QA				EgoSchema		IntentQA
Method	LLM/MLLM	Tem.	Cau.	Des.	Avg.	Sub.	Full	Acc.
methods based on open-source MLLMs
SeViLA [ 30 ]	Flan-T5 XL	61.3	61.5	75.6	63.6	25.7	22.7	-
MVU [ 17 ]	Mistral-13B	55.4	48.1	64.1	55.2	60.3	37.6	-
LangRepo [ 5 ]	Mixtral-8×7B	51.4	64.4	69.1	60.9	66.2	41.2	59.1
methods based on proprietary LLMs

TABLE I : Comparison with other training-free methods on EgoSchema, NExT-QA, and IntentQA.

TABLE II : Ablation experiments for ProVCA . All results are on the val subset of NExT-QA. Best results are bolded, and default settings are highlighted in gray .

Method	MLLM	Input	Frames	Acc
VideoAgent [ 22 ]	GPT-4o	Image	7.7	73.2
VideoTree [ 24 ]	GPT-4o	Image	6.4	72.4
ProVCA	GPT-4o	Image	7.1	74.2

TABLE III : Comparison of ProVCA, VideoAgent, and VideoTree on the EgoSchema subset.

Explore similar work

VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs
Sep 15, 2026Haoyu Guo, Yuan Feng, Junlin Lv +3Efficient VLM Inference Long-Video Understanding
School of Biomedical Engineering, University of Science and Technology of China · Data Darkness Lab, MIRACLE Center, Suzhou Institute for Advanced Research · China +1
MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video Understanding
Feb 26, 2026Wenhui Tan, Xiaoyi Yu, Jiaze Li +5Long-Video QA Multimodal Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China · Tongji University · MiLM Plus, Xiaomi Inc.
Query-Conditioned Evidential Keyframe Sampling for MLLM-Based Long-Form Video Understanding
Apr 1, 2026Yiheng Wang, Lichen Zhu, Yueqian Lin +4Query-Aware Video Frame Selection Long-Video Understanding
Duke University, Durham, North Carolina, USA · Independent Researcher

Sep 15, 2026J/K move · Enter open · S save

VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs

Haoyu Guo, Yuan Feng, Junlin Lv, Mingjun Xiao+2

School of Biomedical Engineering, University of Science and Technology of China · Data Darkness Lab, MIRACLE Center, Suzhou Institute for Advanced Research · China+1

Efficient VLM Inference Long-Video Understanding Visual Token Compression

Open paper AskPDF arXiv

Scaling Multimodal Large Language Models (MLLMs) to long-form video understanding is bottlenecked by the explosion of visual tokens, which saturates context windows and incurs prohibitive costs. Current solutions predominantly rely on auxiliary models for token reduction but face a fundamental dilemma: lightweight encoder-driven approaches often overlook critical semantic information, whereas heavyweight MLLM-driven reduction negates the efficiency gains. {In this work, we identify a more fundamental inefficiency underlying this dilemma: while fine-grained visual details are essential for detailed understanding, they are largely redundant for the preliminary task of selecting semantically relevant regions. } Motivated by this, we introduce VideoMM, which marks a paradigm shift from model-centric downsizing to adaptive perceptual granularity. Specifically, our framework {decouples selection from reasoning} by executing semantic filtering on a cost-effective Macro Proxy (derived from downscaled frames), and projecting the selected regions onto high-fidelity Micro Tokens for detailed understanding only when necessary. Extensive evaluations show that VideoMM significantly outperforms existing solutions. It achieves a 6.13

\times

speedup and a 7.4% accuracy gain over full-context baselines on LongVideoBench, and further accelerates inference by 2.73

\times

over current leading methods, establishing a highly scalable paradigm for long-video understanding. Our code is available at: https://github.com/adfh917k/VideoMM.

Abstract

Explore similar work

VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs

MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video Understanding

Query-Conditioned Evidential Keyframe Sampling for MLLM-Based Long-Form Video Understanding