HieraVid: Hierarchical Token Pruning for Fast Video Large Language Models
Authors: Yansong Guo, Chaoyang Zhu, Jiayi Ji, Jianghang Lin, Liujuan Cao
Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China Xiamen University Xiamen, Fujian, China
Video Large Language Models (VideoLLMs) have demonstrated impressive capabilities in video understanding, yet the massive number of input video tokens incurs a significant computational burden for deployment. Existing methods mainly prune video tokens at input level while neglecting the inherent information structure embedded in videos and large language models (LLMs). To address this, we propose HieraVid, a hierarchical pruning framework that progressively and dynamically reduces visual redundancy. Based on two observations that videos possess the segment-frame structure and LLMs internally propagate multi-modal information unidirectionally, we decompose pruning into three levels: 1) segment-level, where video tokens are first temporally segmented and spatially merged; 2) frame-level, where similar frames within the same segment are jointly pruned to preserve diversity; 3) layer-level, redundancy gradually shrinks as LLM layer increases w/o compromising performance. We conduct extensive experiments on four widely used video understanding benchmarks to comprehensively evaluate the effectiveness of HieraVid. Remarkably, with only 30% of tokens retained, HieraVid achieves new state-of-the-art performance, while maintaining over 98% and 99% of the performance of LLaVA-Video-7B and LLaVA-OneVision-7B, respectively.
Figures & tables
Figure 1. Comparison of existing VideoLLM pruning methods and our approach under different pruning ratios on LLaVA-Video (top) and LLaVA-OneVision (bottom). “mc” and “oe” represent the multiple choice and open ended settings in NExT-QA. “w/o sub.” and “w/ sub.” are the VideoMME settings without and with subtitles. Our model HieraVid is superior across all benchmarks with diverse durations, complexities, and pruning ratios.
Figure 2. Framework of HieraVid. To balance visual information loss from shallow-layer pruning and inefficiency from deep-layer pruning, we apply layer-level multi-stage pruning to the LLM. HieraVid divides the pruning process into three stages. (i) Merge Ratio-guided Segmentation: At the LLM input layer, static tokens at corresponding positions across merged frames are temporally combined by merging similar tokens at their first occurrence. Furthermore, we partition frames into multiple segments based on the merge ratio between adjacent frames to ensure inter-segment diversity and intra-segment continuity (see Figure 3 ). (ii) Segment Budget Allocation dynamically assigns pruning ratios to each segment to maximize visual diversity after pruning. (iii) Frame-level DPP Pruning integrates the DPP kernel matrix with instruction features to balance visual diversity and instruction relevance in the pruned tokens.
Figure 3. Visualization of frame segment results by our HieraVid. The horizontal dashed line represents the segment threshold. Lower bars indicate lower similarity with preceding frames. Black borders mark the segmentation boundaries, where frames within each segment exhibit high similarity, while those across segments maintain distinctiveness.
Method
Prefilling FLOPs (T)
FLOPs Ratio
MVBench
EgoSchema
NExT-QA
VideoMME
Avg.
mc
oe_test
w/o sub.
w/ sub.
score
%
LLaVA-Video
80.2
100%
60.4
59.4
80.2
34.6
64.1
71.4
61.7
100.0%
30% relative token budget
StreamingLLM (ICLR2024) ( Xiao et al., 2023 )
60.3
75.2%
53.8
51.6
76.1
32.4
55.5
65.7
55.9
90.6%
FastV (ECCV2024) ( Chen et al., 2024b )
23.5
39.3%
56.6
55.1
77.2
33.2
59.3
66.7
58.0
94.0%
PruMerge (ICCV2025) ( Shang et al., 2025a )
19.1
23.8%
54.6
55.7
76.4
32.5
60.9
67.7
58.0
94.0%
Table 1. Comparison of state-of-the-art methods across video understanding benchmarks on LLaVA-Video-7B. Best results are in bold, second best underlined. “Avg.” indicates the mean performance across all benchmarks, where “score” represents the average score over all benchmarks, and “%” denotes the ratio of this average score to that of the unpruned LLaVA-Video model.
Method
Prefilling FLOPs (T)
FLOPs Ratio
MVBench
EgoSchema
NExT-QA
VideoMME
Avg.
mc
oe_test
w/o sub.
w/ sub.
score
%
LLaVA-OneVision
40.8
100%
58.4
62.2
79.4
18.2
58.5
61.9
56.4
100.0%
30% relative token budget
StreamingLLM (ICLR2024) ( Xiao et al., 2023 )
30.6
75%
54.4
56.4
73.5
15.7
52.0
54.9
51.1
90.6%
FastV (ECCV2024) ( Chen et al., 2024b )
12.7
31.2%
57.0
59.6
76.2
16.1
56.3
58.1
53.9
95.5%
PruMerge (ICCV2025) ( Shang et al., 2025a )
10.6
25.9%
56.6
59.3
75.8
15.8
56.6
58.8
53.8
95.3%
Table 2. Comparison of state-of-the-art methods across video understanding benchmarks on LLaVA-Onevision-7B. Best results are in bold, second best underlined. “Avg.” indicates the mean performance across all benchmarks, where “score” represents the average score over all benchmarks, and “%” denotes the ratio of this average score to that of the unpruned LLaVA-Video model.
Method
Prefilling FLOPs (T)
FLOPs Ratio
VideoMME
Avg.
w/o sub.
w/ sub.
score
%
Vanilla
27.1
100.0
57.6
61.0
59.3
100.0
PruneVid
7.3
27.0
53.8
57.3
55.6
93.7
Framefusion
7.3
27.0
55.4
59.7
57.6
97.0
HieraVid
7.4
27.2
56.1
60.1
58.1
98.0
Table 3. Comparison of state-of-the-art methods on the VideoMME benchmark using Qwen2-VL (Relative Token Budget: 30%). Vanilla denotes the original Qwen2-VL before pruning. “Avg.” indicates the mean performance across all benchmarks, where “score” represents the average score over all benchmarks, and “%” denotes the ratio of this average score to that of the unpruned LLaVA-Video model.
Method
MVBench
EgoSchema
NExT-QA
VideoMME
Avg.
mc
oe_test
w/o sub.
w/ sub.
score
%
HieraVid
58.3
59.2
79.9
33.7
62.3
70.8
60.7
98.4%
- Layer-level Balancing
57.8 ↓
58.8 ↓
78.8 ↓
33.8 ↑
61.1 ↓
70.4 ↓
60.1 ↓
94.2% ↓
- Merge Ratio-guided Segmentation
57.8 ↓
58.6 ↓
78.5 ↓
33.6 ↓
59.0 ↓
68.1 ↓
59.3 ↓
92.8% ↓
- Frame-level DPP Pruning
51.5 ↓
50.7 ↓
53.6 ↓
25.5 ↓
52.2 ↓
56.1 ↓
48.3 ↓
78.2% ↓
Table 4. Ablation study on components of HieraVid on LLaVA-Video (Retention Ratio: 30%).
Method
EgoSchema
VideoMME
Avg.
w/o sub.
w/ sub.
score
%
Vanilla
59.4
64.1
71.4
65.0
100.0
No segment
56.2
60.6
70.1
62.3
95.9
Avg-pool
56.6
61.0
70.4
62.7
96.5
Merge Ratio-based segmentation*
59.2
62.3
70.8
64.1
98.6
Table 5. Ablation on segment strategy. Vanilla denotes the original LLaVA-Video before pruning. * means our HieraVid.
Figure 4. Ablation Experiments on segment threshold β .
Method
EgoSchema
VideoMME
Avg.
w/o sub.
w/ sub.
score
%
Vanilla
59.4
64.1
71.4
65.0
100.0
0
57.5
61.9
68.9
62.8
96.6
0.2
58.0
62.1
70.1
63.4
97.6
0.4
58.4
62.2
70.8
63.8
98.2
0.5
59.2
62.3
70.8
64.1
98.6
Table 6. Ablation study on λ for the budget of segment dynamic ratio allocation on LLaVA-Video (Relative Token Budget: 30%). Vanilla denotes the original LLaVA-Video before pruning.
Method
MVBench
NExT-QA
Avg.
mc
oe_test
score
%
Vanilla
60.4
80.2
34.6
58.4
100.0
Average Instruction Tokens
57.4
79.0
33.2
56.5
96.8
Last Instruction Token
58.3
79.6
33.7
57.3
98.2
Table 7. Ablation on the relevance score . We compare using the last token against the conventional approach that averages the attention of instruction tokens to visual tokens. Vanilla denotes the original LLaVA-Video before pruning.
Video Large Language Models (VLLMs) excel in video understanding, but their excessive visual tokens pose a significant computational challenge for real-world applications. Current methods aim to enhance inference efficiency by visual token pruning. However, they do not consider the dynamic characteristics and temporal dependencies of video frames, as they perceive video understanding as a multi-frame task. To address these challenges, we propose MMG-Vid, a novel training-free visual token pruning framework that removes redundancy by Maximizing Marginal Gains at both segment-level and token-level. Specifically, we first divide the video into segments based on frame similarity, and then dynamically allocate the token budget for each segment to maximize the marginal gain of each segment. Subsequently, we propose a temporal-guided DPC algorithm that jointly models inter-frame uniqueness and intra-frame diversity, thereby maximizing the marginal gain of each token. By combining both stages, MMG-Vid can maximize the utilization of the limited token budget, significantly improving efficiency while maintaining strong performance. Extensive experiments demonstrate that MMG-Vid can maintain over 99.5% of the original performance, while effectively reducing 75% visual tokens and accelerating the prefilling stage by 3.9x on LLaVA-OneVision-7B. Code is available at https://github.com/ylnxxts/MMG-Vid.
Junpeng Ma, Qizhe Zhang, Ming Lu +4
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Fudan University · Taobao & Tmall Group of Alibaba
Video large language models (Video LLMs) encode each frame into hundreds of visual tokens, making video inference expensive. Existing methods reduce this cost through similarity-based merging or attention-based selection but face two challenges: Hierarchy-blind Pruning, which overlooks regional compressibility and can discard critical evidence, and Attention Drift, where positional effects and attention sinks shift text-to-vision attention away from task relevance. We propose FoveaTree, a two-stage, training-free visual-token pruning framework. First, Merge-Gain Density Estimation measures the feature distortion caused by regional merging and constructs a quadtree-derived density map to guide token dropping, merging, and retention. Second, Attention Tree Pruning uses the surviving units and inherited tree topology to refine text-to-vision attention in intermediate LLM layers, preserving relevant regional evidence while removing low-relevance content. We evaluate FoveaTree on InternVL3.5-8B and Qwen3-VL-8B across five video question-answering benchmarks and one captioning benchmark at 50%, 30%, and 10% visual-token retention. FoveaTree consistently achieves overall state-of-the-art performance across the evaluated settings, offering a strong accuracy-efficiency trade-off.
Zihan Meng, Weidong Chen, Dexiang Hong +2
University of Science and Technology of China · Harbin Institute of Technology
Video Large Language Models (VideoLLMs) have achieved strong video understanding capabilities but incur substantial inference overhead due to the large number of visual tokens. Existing VideoLLM token compression methods largely rely on selection-independent scoring, overlooking cross-frame complementarity and consequently retaining redundant evidence across frames. Instead, we view video token selection as a progressive evidence accumulation process. It aims to retain visual evidence that is individually informative and collectively complementary under a limited token budget. Building on this insight, we introduce GleanVID, a training-free inference acceleration framework for VideoLLMs. Specifically, GleanVID first allocates the global token budget across frames according to temporal novelty and then selects tokens by jointly considering local representativeness and subspace complementarity, thereby preserving richer and less redundant visual evidence. Extensive experiments across diverse VideoLLMs and benchmarks demonstrate that GleanVID consistently achieves state-of-the-art performance. Notably, with only 25% of visual tokens, GleanVID preserves 98.6% of Qwen3-VL's original performance while reducing its prefill latency by 44.7%. On LLaVA-OV-7B, GleanVID at a 25% retention ratio even slightly surpasses the original model.
Shuo Yang, Changbai Li, Rui Tang +3
Beihang University · Communication University of China