Multimodal Large Language Models (MLLMs) have shown strong performance on video question answering, but their application to long-form videos is constrained by limited context length and computational cost, making keyframe sampling essential. Existing approaches typically rely on semantic relevance or reinforcement learning, which either fail to capture evidential clues or suffer from inefficient combinatorial optimization. In this work, we propose an evidence-driven keyframe sampling framework grounded in information bottleneck theory. We formulate keyframe selection as maximizing the conditional mutual information between selected frames and the query, providing a principled objective that reflects each frame's contribution to answering the question. To make this objective tractable, we exploit its structure to derive a decomposed optimization that reduces subset selection to independent frame-level scoring. We further introduce a query-conditioned evidence scoring network trained with a contrastive objective to estimate evidential importance efficiently. Experiments on long-form video understanding benchmarks show that our method consistently outperforms prior sampling strategies under strict token budgets, while significantly improving training efficiency.
Figures & tables
Figure 1: Semantic relevance vs. evidential usefulness in keyframe selection. Frames with the highest semantic similarity ( blue ) predicted by CLIP [ 12 ] , describing poem content (left) or general match scenes (right), are not always informative for answering the query. In contrast, frames with the highest evidence scores ( red ) predicted by our method better capture answer-critical information. The green regions indicate the temporal segments containing the key evidence needed for answering.
Figure 2: Overview of the query-conditioned evidence scoring network. The input video is uniformly sampled into frames, which are encoded into frame embeddings and scored to obtain frame-level evidence scores conditioned on the query. Frames with the highest evidence scores are then selected and fed into an MLLM to generate the final response.
Method
Frames
LVBench
Overall
ER
EU
KIR
TG
Rea
Sum
Agentic MLLMs
VideoTree [ 31 ]
-
28.8
30.3
25.1
26.5
27.7
31.9
25.5
VideoAgent [ 32 ]
-
29.3
28.0
30.3
28.0
29.3
28.0
36.4
VCA [ 33 ]
-
41.3
43.7
40.7
37.8
38.0
46.2
27.3
SFT/RL MLLMs
Table 1: Comparison with the state-of-the-art methods on LVBench [ 16 ] . ER, EU, KIR, TG, Rea, and Sum stand for entity recognition, event understanding, key information retrieval, temporal grounding, reasoning, and summarization subtasks. † denotes our reproduced results.
Method
Architecture
Frames
VideoMME
Long
Average
Qwen2-VL-7B †
-
32
48.7
58.0
+ AKS [ 9 ]
BLIP-0.5B
32
-
59.9
+ FOCUS [ 11 ]
BLIP-0.5B
32
-
59.7
+ Q-Frame [ 10 ]
CLIP-0.4B
44
48.3
58.3
+ MLLM-Selector [ 20 ]
MLLM-1.5B
32
-
58.7
Table 2: Comparison with existing keyframe sampling methods on VideoMME [ 29 ] . † denotes our reproduced results.
Method
Sampling
Encoder
VideoMME
LVBench
Qwen2-VL-7B
Uniform
-
58.0
39.1
Ours
CLIP
60.1
46.6
Ours
SigLIP
59.9
45.8
Ours
SigLIP2
59.0
44.2
Qwen2.5-VL-7B
Uniform
-
60.7
37.6
Ours
CLIP
63.6
47.7
Table 3: Ablation study on pretrained vision-language encoders. We evaluate the impact of different encoders within our sampling framework across MLLMs. Our approach consistently surpasses uniform sampling, while the choice of encoder has a moderate impact on the final performance.
Frames
LVBench
Uniform
Ours
8
35.8
45.1 ( ↑9.3 )
16
37.1
46.0 ( ↑8.9 )
32
40.9
48.3 ( ↑7.4 )
64
41.7
49.4 ( ↑7.7 )
Table 4: Ablation study on frame budget. We compare uniform sampling with our method under different numbers of frames on LVBench with LLaVa-Video-7B.
Duration
Frames
Sampling
MLLM
2 min
64
3.0
2.1
8 min
64
7.6
2.1
30 min
64
18.6
2.1
Table 5: Keyframe sampling and MLLM inference latency tested on one NVIDIA L40S GPU under varying video durations. The reported MLLM latency corresponds to a minimal setting where only a single-token response is generated, thus representing a lower bound on the actual inference cost.
Figure 3: Comparison between uniform sampling and our keyframe sampling on long-form video examples. Red boxes highlight the query-relevant frames containing the critical visual evidence. Our method successfully captures the query-relevant frames, enabling Qwen2-VL-7B to identify the correct visual evidence and produce the correct answer, whereas uniform sampling misses these critical moments.
Figure 4: Comparison between uniform sampling and our keyframe sampling. Red boxes indicate the key evidence frames required to answer the query. Our method selects these informative moments, allowing Qwen2.5-VL-7B to answer correctly, while uniform sampling overlooks them.
Figure 5: Comparison between uniform sampling and our keyframe sampling. Red boxes denote the critical frames that contain the necessary visual cues. By capturing these brief but informative moments, our method enables LLaVA-Video-7B to arrive at the correct answer, whereas uniform sampling fails.
Figure 6: Failure case on the LVBench summarization task using Qwen2-VL-7B. The question requires global understanding of the video content. Uniform sampling provides broader temporal coverage and captures more representative frames, leading to the correct answer, whereas our keyframe sampling produces temporally clustered frames and fails due to insufficient coverage.
Efficiently understanding long-form videos remains a fundamental challenge for multimodal large language models (MLLMs). In this paper, we present MLLM-Sampler Joint Evolution (MSJoE), a novel framework that jointly evolves the MLLM and a lightweight key-frame sampler for efficient long-form video understanding. MSJoE builds upon a key assumption that only a small subset of key-frames is truly informative for answering each question to a video. Specifically, MSJoE first reasons out several queries, which describe diverse visual perspectives relevant to the question. Then, these queries interact with a frozen CLIP model to produce a query-frame similarity matrix. Finally, a lightweight sampler predicts key-frame sampling weights from this matrix, selecting a compact set of informative frames, which are then fed into the MLLM for answer generation. Both the MLLM and sampler are jointly optimized through reinforcement learning, enabling co-adaptation of query-reasoning, frame-sampling, and key-frame understanding. A new long-video QA dataset containing 2.8K videos with 7K question-answer pairs is collected to support the training process. Extensive experiments on VideoMME, LongVideoBench, LVBench, and MLVU show that MSJoE achieves 8.0% accuracy gain upon the base MLLM, and 1.1% higher accuracy than strongest baseline method.
Wenhui Tan, Xiaoyi Yu, Jiaze Li +5
Gaoling School of Artificial Intelligence, Renmin University of China · Tongji University · MiLM Plus, Xiaomi Inc.
Despite rapid progress in Multi-modal Large Language Models (MLLMs), understanding long-form videos is still bottlenecked by limited context windows. While recent keyframe sampling methods attempt to mitigate this by distilling video inputs into a compact set of query-relevant frames, navigating the vast spatio-temporal search space remains challenging, as spatial detail and temporal coverage often conflict. To address this, we introduce LENS, a training-free keyframe sampling framework that dynamically decides when to zoom in for fine-grained details and when to zoom out for broader context based on the text query. Concretely, LENS adaptively allocates a limited frame budget between spatial zoom-ins, which highlight query-relevant regions within individual frames, and temporal zoom-outs, which expand the temporal scope through multi-frame aggregation, enabling the model to reason across multiple granularities while capturing both high-fidelity details and long-range context. Across diverse long-form video benchmarks, LENS consistently outperforms prior state-of-the-art keyframe sampling methods and delivers substantial gains over uniform sampling, improving Video-MME accuracy from 53.3% to 60.7% with Qwen2.5-VL.Code is available at https://github.com/zhangce01/LENS.
Long-video understanding remains challenging for Multimodal Large Language Models (MLLMs) due to limited context length. Uniform sampling may miss crucial moments, while agent-based frame video understanding methods often evaluate frames independently, overlooking the temporal organization of videos. Ideally, evidence selection should mimic how humans answer questions about long videos: first locating the relevant segment from the global context, then zooming into local objects and details. We propose Temporal Tree of Thought T^3, a training-free framework for adaptive coarse-to-fine long-video understanding. T^3 constructs a question-agnostic hierarchical temporal tree via recursive temporally constrained clustering, where each node represents a contiguous segment with an informative key frame. During inference, T^3 performs an answer-retrieve-explore loop: it reasons over coarse representative frames, generates a search statement when evidence is insufficient, and expands relevant branches for finer-grained evidence. This process adaptively shifts the search target from temporal regions to specific objects and visual details to help video understanding. Experiments on VideoMME, LongVideoBench, and LVBench show that T^3 improves Qwen2.5-VL-7B by 0.5%, 4.6%, and 4.4%, respectively, under the same frame budget, demonstrating the effectiveness of structured temporal reasoning.