Organizations: Institute of Science and Technology for Brain-inspired Intelligence, Fudan University · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Alibaba Group Holding Limited · Zhejiang University · Future Living Lab of Alibaba
Video Large Language Models (VLMs) have achieved remarkable success in video understanding, but the significant computational cost from processing dense frames severely limits their practical application. Existing methods alleviate this by selecting keyframes, but their greedy decision-making, combined with a decoupled evaluation of relevance and diversity, often falls into local optima and results in erroneously selecting irrelevant noise frames. To address these challenges, we propose GIFT: Global Irreplaceability Frame Targeting, a novel training-free framework that selects frames by assessing their intrinsic irreplaceability. Specifically, we first introduce Directed Diversity to quantify a frame's uniqueness conditioned on relevance, which allows us to formulate a unified irreplaceability score. Subsequently, our Budget-Aware Refinement strategy employs a adaptive iterative process that first secures a core set of frames with the highest irreplaceability, and then shifts its priority to building crucial temporal context around these selections as the budget expands. Extensive experiments demonstrate that GIFT achieves a maximum average improvement of 12.5% across long-form video benchmarks on LLaVA-Video-7B compared to uniform sampling. Code is available at https://github.com/ylnxxts/GIFT.
Figures & tables
Figure 1: Average accuracy of LLaVA-Video-7B on Video-MME, LongVideoBench and MLVU for different methods: uniform sampling, BOLT [ 30 ] , AKS [ 40 ] , and our GIFT.
Figure 2: The overall framework of our GIFT. Given an input video and user query, GIFT first calculates each frame’s query-relevance( r ) and directed diversity( d ) to quantify its irreplaceability(detailed in the left panel). The core of GIFT is a two-stage selection process (detailed in the right panel). Stage 1 performs the initial selection by identifying frames with high scores ( s=r×d ). If the sampling budget ( K ) exceeds batch-size ( B ), the Budget-Aware Refinement in Stage 2 is triggered. This stage iteratively Selects a batch of B frames, Removes them, and Updates the d for all remaining frames until the budget is met. This iterative update process continuously Releases previously suppressed frames, progressively building a rich temporal context.
Method
MVBench
LongVideo Bench
MLVU
VideoMME(w.o. sub.)
Average
Overall
Short
Medium
Long
Duration
16s
1 ∼ 60min
3 ∼ 120min
1 ∼ 60min
1 ∼ 3min
3 ∼ 30min
30 ∼ 60min
Score
%
LLaVA-Video(64Frames)
59.2
58.9
67.4
64.3
77.3
62.3
53.2
62.5
100.0
Frame Budget: 32Frames
Uniform Sampling
59.2
58.0
61.2
62.6
76.3
59.2
52.3
60.3
96.5
BOLT (CVPR2025)
59.7
59.8
66.5
64.1
75.7
62.7
54.0
62.5
100.0
Table 1: Comparison of state-of-the-art methods across video understanding benchmarks on LLaVA-Video-7B. Our GIFT consistently outperforms competing approaches across all frame budgets, demonstrating superior performance especially under tight budget constraints. Best results are highlighted in bold, second best underlined.
Model
LongVideoBench
MLVU +0.0 ↑
VideoMME(w.o. sub.)
Overall
Short
Medium
Long
VILA-V1.5 8 frame
47.1
49.8
48.4
59.4
44.7
41.1
VILA-V1.5 8 frame +GIFT
50.9 +3.8 ↑
56.7 +6.9 ↑
52.8 +4.4 ↑
65.7 +6.3 ↑
49.8 +5.1 ↑
42.9 +1.8 ↑
LLaVA-OneVision 8 frame
54.3
58.5
53.9
64.9
52.0
44.7
LLaVA-OneVision 8 frame +GIFT
59.6 +5.3 ↑
67.3 +8.8 ↑
58.8 +4.9 ↑
69.9 +5.0 ↑
57.9 +5.9 ↑
48.7 +4.0 ↑
Qwen2.5-VL 8 frame
52.7
53.8
53.6
62.7
50.7
47.6
Table 2: Performance comparison of state-of-the-art VLMs with and without our GIFT enhancement. GIFT delivers consistent and significant performance gains across all base models, benchmarks and frame budgets (8 and 32), demonstrating its strong generalizability as a model-agnostic, plug-and-play solution.
B in BAR
LVB
VideoMME(w.o. sub.)
average
Overall
Short
Medium
Long
Vanilla
58.0
62.6
76.3
59.2
52.3
60.3
B=6
59.8
65.2
78.1
63.0
54.6
62.5
B=7
60.0
64.8
76.7
63.9
53.9
62.4
B=8
59.7
65.1
76.8
64.8
53.9
62.4
B=9
60.2
65.0
77.3
64.1
53.6
62.6
Table 3: Ablation study on the refinement batch size B for our Budget-Aware Refinement (BAR) module . Frames are selected iteratively in batches of B in BAR module. Vanilla represents the performance of uniform sampling for comparison.
Figure 3: Ablation study of GIFT’s modules on LLaVA-Video (Frame Budget:32) . The y-axis shows performance relative to Uniform Sampling (100%). “Standard Diversity” refers to using the decoupled evaluation criteria with standard diversity instead of our proposed directed diversity. “Without BAR” refers to disabling the Budget-Aware Refinement module.
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University · College of Computer and Data Science, Fuzhou University