Multimodal Large Language Models (MLLMs) have shown strong performance on video question answering, but their application to long-form videos is constrained by limited context length and computational cost, making keyframe sampling essential. Existing approaches typically rely on semantic relevance or reinforcement learning, which either fail to capture evidential clues or suffer from inefficient combinatorial optimization. In this work, we propose an evidence-driven keyframe sampling framework grounded in information bottleneck theory. We formulate keyframe selection as maximizing the conditional mutual information between selected frames and the query, providing a principled objective that reflects each frame's contribution to answering the question. To make this objective tractable, we exploit its structure to derive a decomposed optimization that reduces subset selection to independent frame-level scoring. We further introduce a query-conditioned evidence scoring network trained with a contrastive objective to estimate evidential importance efficiently. Experiments on long-form video understanding benchmarks show that our method consistently outperforms prior sampling strategies under strict token budgets, while significantly improving training efficiency.
Figures & tables
Figure 1: Semantic relevance vs. evidential usefulness in keyframe selection. Frames with the highest semantic similarity ( blue ) predicted by CLIP [ 12 ] , describing poem content (left) or general match scenes (right), are not always informative for answering the query. In contrast, frames with the highest evidence scores ( red ) predicted by our method better capture answer-critical information. The green regions indicate the temporal segments containing the key evidence needed for answering.
Figure 2: Overview of the query-conditioned evidence scoring network. The input video is uniformly sampled into frames, which are encoded into frame embeddings and scored to obtain frame-level evidence scores conditioned on the query. Frames with the highest evidence scores are then selected and fed into an MLLM to generate the final response.
Method
Frames
LVBench
Overall
ER
EU
KIR
TG
Rea
Sum
Agentic MLLMs
VideoTree [ 31 ]
-
28.8
30.3
25.1
26.5
27.7
31.9
25.5
VideoAgent [ 32 ]
-
29.3
28.0
30.3
28.0
29.3
28.0
36.4
VCA [ 33 ]
-
41.3
43.7
40.7
37.8
38.0
46.2
27.3
SFT/RL MLLMs
Table 1: Comparison with the state-of-the-art methods on LVBench [ 16 ] . ER, EU, KIR, TG, Rea, and Sum stand for entity recognition, event understanding, key information retrieval, temporal grounding, reasoning, and summarization subtasks. † denotes our reproduced results.
Method
Architecture
Frames
VideoMME
Long
Average
Qwen2-VL-7B †
-
32
48.7
58.0
+ AKS [ 9 ]
BLIP-0.5B
32
-
59.9
+ FOCUS [ 11 ]
BLIP-0.5B
32
-
59.7
+ Q-Frame [ 10 ]
CLIP-0.4B
44
48.3
58.3
+ MLLM-Selector [ 20 ]
MLLM-1.5B
32
-
58.7
Table 2: Comparison with existing keyframe sampling methods on VideoMME [ 29 ] . † denotes our reproduced results.
Method
Sampling
Encoder
VideoMME
LVBench
Qwen2-VL-7B
Uniform
-
58.0
39.1
Ours
CLIP
60.1
46.6
Ours
SigLIP
59.9
45.8
Ours
SigLIP2
59.0
44.2
Qwen2.5-VL-7B
Uniform
-
60.7
37.6
Ours
CLIP
63.6
47.7
Table 3: Ablation study on pretrained vision-language encoders. We evaluate the impact of different encoders within our sampling framework across MLLMs. Our approach consistently surpasses uniform sampling, while the choice of encoder has a moderate impact on the final performance.
Frames
LVBench
Uniform
Ours
8
35.8
45.1 ( ↑9.3 )
16
37.1
46.0 ( ↑8.9 )
32
40.9
48.3 ( ↑7.4 )
64
41.7
49.4 ( ↑7.7 )
Table 4: Ablation study on frame budget. We compare uniform sampling with our method under different numbers of frames on LVBench with LLaVa-Video-7B.
Duration
Frames
Sampling
MLLM
2 min
64
3.0
2.1
8 min
64
7.6
2.1
30 min
64
18.6
2.1
Table 5: Keyframe sampling and MLLM inference latency tested on one NVIDIA L40S GPU under varying video durations. The reported MLLM latency corresponds to a minimal setting where only a single-token response is generated, thus representing a lower bound on the actual inference cost.
Figure 3: Comparison between uniform sampling and our keyframe sampling on long-form video examples. Red boxes highlight the query-relevant frames containing the critical visual evidence. Our method successfully captures the query-relevant frames, enabling Qwen2-VL-7B to identify the correct visual evidence and produce the correct answer, whereas uniform sampling misses these critical moments.
Figure 4: Comparison between uniform sampling and our keyframe sampling. Red boxes indicate the key evidence frames required to answer the query. Our method selects these informative moments, allowing Qwen2.5-VL-7B to answer correctly, while uniform sampling overlooks them.
Figure 5: Comparison between uniform sampling and our keyframe sampling. Red boxes denote the critical frames that contain the necessary visual cues. By capturing these brief but informative moments, our method enables LLaVA-Video-7B to arrive at the correct answer, whereas uniform sampling fails.
Figure 6: Failure case on the LVBench summarization task using Qwen2-VL-7B. The question requires global understanding of the video content. Uniform sampling provides broader temporal coverage and captures more representative frames, leading to the correct answer, whereas our keyframe sampling produces temporally clustered frames and fails due to insufficient coverage.