MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs
Authors: Junpeng Ma, Qizhe Zhang, Ming Lu, Zhibin Wang, Qiang Zhou, Jun Song, Shanghang Zhang
Organizations: State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Fudan University · Taobao & Tmall Group of Alibaba
Video Large Language Models (VLLMs) excel in video understanding, but their excessive visual tokens pose a significant computational challenge for real-world applications. Current methods aim to enhance inference efficiency by visual token pruning. However, they do not consider the dynamic characteristics and temporal dependencies of video frames, as they perceive video understanding as a multi-frame task. To address these challenges, we propose MMG-Vid, a novel training-free visual token pruning framework that removes redundancy by Maximizing Marginal Gains at both segment-level and token-level. Specifically, we first divide the video into segments based on frame similarity, and then dynamically allocate the token budget for each segment to maximize the marginal gain of each segment. Subsequently, we propose a temporal-guided DPC algorithm that jointly models inter-frame uniqueness and intra-frame diversity, thereby maximizing the marginal gain of each token. By combining both stages, MMG-Vid can maximize the utilization of the limited token budget, significantly improving efficiency while maintaining strong performance. Extensive experiments demonstrate that MMG-Vid can maintain over 99.5% of the original performance, while effectively reducing 75% visual tokens and accelerating the prefilling stage by 3.9x on LLaVA-OneVision-7B. Code is available at https://github.com/ylnxxts/MMG-Vid.
Figures & tables
Figure 1: Comparison of different budget allocation strategies. (a) Static uniform budgeting overlooks varying segment significance. (b) Static importance-based budgeting acknowledges the significance of segments but wastes resources by allocating redundant budgets to two visually similar static segments (the first and third). (c) Our marginal gain-based segment budgeting further reduces redundancy by penalizing a segment’s budget if its information is already included in previously selected segments. This results in an optimal allocation of the budget that takes into account the dynamic characteristics.
Figure 2: Overall framework. Segment-level (Bottom-Right): We iteratively calculate the marginal gain (a combination of representativeness and diversity) for each segment to dynamically allocate budget, prioritizing more informative segments. Token-level (Top-Right): Our proposed TG-DPC progressively prunes each frame by selecting tokens that are both salient within the frame and novel across the temporal dimension, guided by the set of previously selected tokens
Method
MVBench
LongVideo Bench
MLVU
VideoMME
Average
Overall
Short
Medium
Long
Duration
16s
1 ∼ 60min
3 ∼ 120min
1 ∼ 60min
1 ∼ 3min
3 ∼ 30min
30 ∼ 60min
Score
%
LLaVA-Video
59.2
58.9
67.4
64.3
77.3
62.3
53.2
62.4
100.0
Retention Ratio: 25%
FastV (ECCV24)
52.1
54.8
57.8
58.6
68.7
58.4
48.7
55.8
89.4
VisionZip (CVPR2025)
56.0
58.0
64.4
61.9
73.4
60.6
51.6
60.1
96.3
Table 1: Comparison of state-of-the-art methods across video understanding benchmarks on LLaVA-Video-7B. Best results are highlighted in bold, second best underlined.
Retention Ratio
Time (ms)
Acc%
Prefill
Generate
LLaVA-OV
207.5 (1.0x)
329.7 (1.0x)
100
25%
52.8 (3.9x)
107.2 (3.1x)
99.5
15%
34.6 (6.0x)
78.3 (4.2x)
98.3
LLaVA-Video
277.5 (1.0x)
411.6 (1.0x)
100
25%
65.0 (4.3x)
142.9 (2.9x)
98.1
Table 2: Efficiency comparison of different Retention Ratios. “Prefill Time”: Time for model to generate first token; “Generate Time”: Time for model to generate response.
Method
MVBench
LongVideo Bench
MLVU
VideoMME
Average
Overall
Short
Medium
Long
Duration
16s
1 ∼ 60min
3 ∼ 120min
1 ∼ 60min
1 ∼ 3min
3 ∼ 30min
30 ∼ 60min
Score
%
LLaVA-OneVision
57.6
56.6
63.1
58.5
70.1
56.6
48.8
59.0
100.0
Retention Ratio: 25%
FastV (ECCV24)
54.8
56.8
59.3
55.9
66.0
54.6
47.2
56.7
96.1
VisionZip (CVPR2025)
56.9
56.0
62.9
58.0
68.9
57.4
47.6
58.5
99.2
Table 3: Comparison of state-of-the-art methods across video understanding benchmarks on LLaVA-OneVision-7B.
λ in Budgeting
MLVU
VideoMME
Acc %
Overall
Short
Medium
Long
Vanilla
67.4
64.3
77.3
62.3
53.2
100.0
1.0
65.2
62.0
74.3
60.2
51.3
96.6
0.8
65.5
61.9
74.2
60.2
51.1
96.7
0.6
66.2
62.0
74.3
59.7
51.9
97.3
0.5
66.2
62.3
74.7
60.0
52.1
97.6
Table 4: Ablation study on λ for marginal gain-based segment budgeting on LLaVA-Video (Retention Ratio: 25%). Our method consists of Representiveness and Diversity, λ determines the proportion between them.
Figure 3: Ablation study of MMG-Vid’s modules on LLaVA-Video (Retention Ratio: 25%). “DPC-KNN” refers to using the standard DPC-KNN algorithm instead of our proposed TG-DPC. “Uniform Budget” refers to the conventional method of assigning a fixed budget to each frame.