HieraVid: Hierarchical Token Pruning for Fast Video Large Language Models
Authors: Yansong Guo, Chaoyang Zhu, Jiayi Ji, Jianghang Lin, Liujuan Cao
Organizations: Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China Xiamen University Xiamen, Fujian, China
Video Large Language Models (VideoLLMs) have demonstrated impressive capabilities in video understanding, yet the massive number of input video tokens incurs a significant computational burden for deployment. Existing methods mainly prune video tokens at input level while neglecting the inherent information structure embedded in videos and large language models (LLMs). To address this, we propose HieraVid, a hierarchical pruning framework that progressively and dynamically reduces visual redundancy. Based on two observations that videos possess the segment-frame structure and LLMs internally propagate multi-modal information unidirectionally, we decompose pruning into three levels: 1) segment-level, where video tokens are first temporally segmented and spatially merged; 2) frame-level, where similar frames within the same segment are jointly pruned to preserve diversity; 3) layer-level, redundancy gradually shrinks as LLM layer increases w/o compromising performance. We conduct extensive experiments on four widely used video understanding benchmarks to comprehensively evaluate the effectiveness of HieraVid. Remarkably, with only 30% of tokens retained, HieraVid achieves new state-of-the-art performance, while maintaining over 98% and 99% of the performance of LLaVA-Video-7B and LLaVA-OneVision-7B, respectively.
Figures & tables
Figure 1. Comparison of existing VideoLLM pruning methods and our approach under different pruning ratios on LLaVA-Video (top) and LLaVA-OneVision (bottom). “mc” and “oe” represent the multiple choice and open ended settings in NExT-QA. “w/o sub.” and “w/ sub.” are the VideoMME settings without and with subtitles. Our model HieraVid is superior across all benchmarks with diverse durations, complexities, and pruning ratios.
Figure 2. Framework of HieraVid. To balance visual information loss from shallow-layer pruning and inefficiency from deep-layer pruning, we apply layer-level multi-stage pruning to the LLM. HieraVid divides the pruning process into three stages. (i) Merge Ratio-guided Segmentation: At the LLM input layer, static tokens at corresponding positions across merged frames are temporally combined by merging similar tokens at their first occurrence. Furthermore, we partition frames into multiple segments based on the merge ratio between adjacent frames to ensure inter-segment diversity and intra-segment continuity (see Figure 3 ). (ii) Segment Budget Allocation dynamically assigns pruning ratios to each segment to maximize visual diversity after pruning. (iii) Frame-level DPP Pruning integrates the DPP kernel matrix with instruction features to balance visual diversity and instruction relevance in the pruned tokens.
Figure 3. Visualization of frame segment results by our HieraVid. The horizontal dashed line represents the segment threshold. Lower bars indicate lower similarity with preceding frames. Black borders mark the segmentation boundaries, where frames within each segment exhibit high similarity, while those across segments maintain distinctiveness.
Method
Prefilling FLOPs (T)
FLOPs Ratio
MVBench
EgoSchema
NExT-QA
VideoMME
Avg.
mc
oe_test
w/o sub.
w/ sub.
score
%
LLaVA-Video
80.2
100%
60.4
59.4
80.2
34.6
64.1
71.4
61.7
100.0%
30% relative token budget
StreamingLLM (ICLR2024) ( Xiao et al., 2023 )
60.3
75.2%
53.8
51.6
76.1
32.4
55.5
65.7
55.9
90.6%
FastV (ECCV2024) ( Chen et al., 2024b )
23.5
39.3%
56.6
55.1
77.2
33.2
59.3
66.7
58.0
94.0%
PruMerge (ICCV2025) ( Shang et al., 2025a )
19.1
23.8%
54.6
55.7
76.4
32.5
60.9
67.7
58.0
94.0%
Table 1. Comparison of state-of-the-art methods across video understanding benchmarks on LLaVA-Video-7B. Best results are in bold, second best underlined. “Avg.” indicates the mean performance across all benchmarks, where “score” represents the average score over all benchmarks, and “%” denotes the ratio of this average score to that of the unpruned LLaVA-Video model.
Method
Prefilling FLOPs (T)
FLOPs Ratio
MVBench
EgoSchema
NExT-QA
VideoMME
Avg.
mc
oe_test
w/o sub.
w/ sub.
score
%
LLaVA-OneVision
40.8
100%
58.4
62.2
79.4
18.2
58.5
61.9
56.4
100.0%
30% relative token budget
StreamingLLM (ICLR2024) ( Xiao et al., 2023 )
30.6
75%
54.4
56.4
73.5
15.7
52.0
54.9
51.1
90.6%
FastV (ECCV2024) ( Chen et al., 2024b )
12.7
31.2%
57.0
59.6
76.2
16.1
56.3
58.1
53.9
95.5%
PruMerge (ICCV2025) ( Shang et al., 2025a )
10.6
25.9%
56.6
59.3
75.8
15.8
56.6
58.8
53.8
95.3%
Table 2. Comparison of state-of-the-art methods across video understanding benchmarks on LLaVA-Onevision-7B. Best results are in bold, second best underlined. “Avg.” indicates the mean performance across all benchmarks, where “score” represents the average score over all benchmarks, and “%” denotes the ratio of this average score to that of the unpruned LLaVA-Video model.
Method
Prefilling FLOPs (T)
FLOPs Ratio
VideoMME
Avg.
w/o sub.
w/ sub.
score
%
Vanilla
27.1
100.0
57.6
61.0
59.3
100.0
PruneVid
7.3
27.0
53.8
57.3
55.6
93.7
Framefusion
7.3
27.0
55.4
59.7
57.6
97.0
HieraVid
7.4
27.2
56.1
60.1
58.1
98.0
Table 3. Comparison of state-of-the-art methods on the VideoMME benchmark using Qwen2-VL (Relative Token Budget: 30%). Vanilla denotes the original Qwen2-VL before pruning. “Avg.” indicates the mean performance across all benchmarks, where “score” represents the average score over all benchmarks, and “%” denotes the ratio of this average score to that of the unpruned LLaVA-Video model.
Method
MVBench
EgoSchema
NExT-QA
VideoMME
Avg.
mc
oe_test
w/o sub.
w/ sub.
score
%
HieraVid
58.3
59.2
79.9
33.7
62.3
70.8
60.7
98.4%
- Layer-level Balancing
57.8 ↓
58.8 ↓
78.8 ↓
33.8 ↑
61.1 ↓
70.4 ↓
60.1 ↓
94.2% ↓
- Merge Ratio-guided Segmentation
57.8 ↓
58.6 ↓
78.5 ↓
33.6 ↓
59.0 ↓
68.1 ↓
59.3 ↓
92.8% ↓
- Frame-level DPP Pruning
51.5 ↓
50.7 ↓
53.6 ↓
25.5 ↓
52.2 ↓
56.1 ↓
48.3 ↓
78.2% ↓
Table 4. Ablation study on components of HieraVid on LLaVA-Video (Retention Ratio: 30%).
Method
EgoSchema
VideoMME
Avg.
w/o sub.
w/ sub.
score
%
Vanilla
59.4
64.1
71.4
65.0
100.0
No segment
56.2
60.6
70.1
62.3
95.9
Avg-pool
56.6
61.0
70.4
62.7
96.5
Merge Ratio-based segmentation*
59.2
62.3
70.8
64.1
98.6
Table 5. Ablation on segment strategy. Vanilla denotes the original LLaVA-Video before pruning. * means our HieraVid.
Figure 4. Ablation Experiments on segment threshold β .
Method
EgoSchema
VideoMME
Avg.
w/o sub.
w/ sub.
score
%
Vanilla
59.4
64.1
71.4
65.0
100.0
0
57.5
61.9
68.9
62.8
96.6
0.2
58.0
62.1
70.1
63.4
97.6
0.4
58.4
62.2
70.8
63.8
98.2
0.5
59.2
62.3
70.8
64.1
98.6
Table 6. Ablation study on λ for the budget of segment dynamic ratio allocation on LLaVA-Video (Relative Token Budget: 30%). Vanilla denotes the original LLaVA-Video before pruning.
Method
MVBench
NExT-QA
Avg.
mc
oe_test
score
%
Vanilla
60.4
80.2
34.6
58.4
100.0
Average Instruction Tokens
57.4
79.0
33.2
56.5
96.8
Last Instruction Token
58.3
79.6
33.7
57.3
98.2
Table 7. Ablation on the relevance score . We compare using the last token against the conventional approach that averages the attention of instruction tokens to visual tokens. Vanilla denotes the original LLaVA-Video before pruning.
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Fudan University · Taobao & Tmall Group of Alibaba