Token pruning has emerged as a mainstream approach for developing efficient Video Large Language Models (Video LLMs). This work revisits and advances the two predominant token-pruning paradigms: attention-based selection and similarity-based clustering. Our study reveals two critical limitations in existing methods: (1) conventional top-k selection strategies fail to fully account for the attention distribution, which is often spatially multi-modal and long-tailed in magnitude; and (2) direct similarity-based clustering frequently generates fragmented clusters, resulting in distorted representations after pooling. To address these bottlenecks, we propose Tango, a novel framework designed to optimize the utilization of visual signals. Tango integrates a diversity-driven strategy to enhance attention-based token selection, and introduces Spatio-temporal Rotary Position Embedding (ST-RoPE) to preserve geometric structure via locality priors. Comprehensive experiments across various Video LLMs and video understanding benchmarks demonstrate the effectiveness and generalizability of our approach. Notably, when retaining only 10% of the video tokens, Tango preserves 98.9% of the original performance on LLaVA-OV while delivering a 1.88× inference speedup.
Figures & tables
Figure 1: Limitations of two typical pre-LLM token pruning approaches. (Top Right) Top-k selection fails to fully capture the attention distribution, which is spatially multimodal and long-tailed in magnitude. (Bottom Right) Direct similarity-based clustering can result in noisy representations.
Figure 2: Illustration of attention distribution and similarity-based clustering. Attention Heatmaps: The spatial distribution of attention scores exhibits multiple modes ( i.e . , local maxima of attention scores) corresponding to distinct semantic regions ( e.g . , subtitle and head in the first frame). Clustering Results: Direct similarity-based cluster assignment (Baseline) leads to spatially fragmented results, where a coherent semantic entity ( e.g . , the head or body) is split into different clusters or mixed with background. Pooling on these clusters may result in noisy representations. In comparison, our method (bottom row) mitigates this issue and exhibits superior geometric shape preservation. Visualization is extracted using the SigLIP [ 11 ] ViT of LLaVA-OV-7B on a Video-MME sample ( vid: lmlw4NNZBbw ). More cases and details are provided in the Appendix.
Figure 3: Distribution of sorted attention scores. The scores exhibit a distinct long-tailed distribution with a narrow variance band, demonstrating highly stable attention patterns across videos. Results are averaged over 100 video samples from ActivityNet [ 15 ] .
Figure 4: Overview of our method. It comprises three modules: Temporal Video Segmentation (TVS), Salient Token Selection (STS), and Spatio-Temporal Merging (STM). Our technical contributions lie in (1) an enhanced salient token selection strategy in the STS module, and (2) position-aware clustering with ST-RoPE, used in STS and STM modules.
Figure 5: Illustration of ST-RoPE. (Bottom left) It encodes spatio-temporal position information by applying a rotation matrix to vision tokens. (Bottom right) The pairwise distance on the hypersphere, d(⋅,⋅) , is not only determined by cosine similarity, but also modulated by Δp , where spatio-temporally distant tokens are assigned a larger distance penalty. Consequently, semantically similar and spatio-temporally neighboring tokens are clustered together.
Methods
Video-MME (w/o sub)
MVBench
LongVideo Bench
MLVU
Avg.
Perf.
Short
Medium
Long
Overall
LLaVA-OneVision-7B [ 26 ]
71.9
56.3
49.6
59.3
55.3
56.8
64.7
59.0
100.0
Retention Ratio=10%
FastV [ 1 ] [ECCV 2024]
54.7
49.0
43.7
49.1
48.3
47.3
54.6
49.8
84.4
DART [ 2 ] [EMNLP 2025]
54.4
49.0
44.0
49.1
46.2
46.3
55.4
49.3
83.5
VisionZip [ 3 ] [CVPR 2025]
61.2
52.3
47.2
53.6
50.7
51.7
59.0
53.8
91.1
Table 1: Comparison of state-of-the-art token pruning methods on four mainstream video understanding benchmarks. We experiment with three retention ratios (10%, 15%, 20%) and 32 sampled frames on the LLaVA-OneVision-7B model. Our method consistently achieves the best overall performance across different retention ratios. Bold and u nderlined denote best and second-best performance, respectively. Avg.: Average accuracy on four benchmarks. Perf.: Ratio of performance retained relative to the original model. All baseline results are reproduced under the same settings.
Methods
Video-MME (w/o sub)
MVBench
LongVideo Bench
MLVU
Avg.
Perf.
Short
Medium
Long
Overall
LLaVA-Video-7B [ 27 ]
76.7
62.0
51.4
63.4
60.7
57.5
71.4
63.3
100.0
Retention Ratio=10%
FastV [ 1 ] [ECCV 2024]
56.1
49.6
44.4
50.0
47.6
45.8
52.2
48.9
77.3
VisionZip [ 3 ] [CVPR 2025]
67.0
54.6
46.8
56.1
54.2
52.4
62.9
56.4
89.1
VidCom 2 [ 4 ] [EMNLP 2025]
65.6
55.7
45.9
55.7
52.1
50.4
59.7
54.5
86.1
Table 2: Cross-model evaluation on LLaVA-Video-7B and Qwen2.5-VL-7B , with 64 and 32 input frames, respectively. † : Combined with intra-LLM method (FastV variant [ 9 ] ), our method can achieve better overall performance.
Figure 6: Results with different input frames. Our method maintains stable scaling with frame number, clearly outperforming other SOTA methods. “w/o M” denotes pre-LLM-only pruning [ 9 ] . The performance is averaged across three video datasets: Video-MME, LongVideoBench, and MLVU.
Figure 7: Results with different salient token selection strategy . Top-k (attn) underperforms, and Top-k (p-attn) is only comparable to a simple uniform sampling baseline. Plugging in our proposed diversity-driven selection method (“w/ div.”) brings remarkable gains. attn and p-attn denote attention scores extracted by pooling attention weights on the query dimension [ 3 , 9 ] and calculating with a global query [ 5 ] , respectively.
w/ div.
ST Merge
ST-RoPE
TSA
V-MME
LVB
Perf.
Effect on Token Selection
✗
✗
✗
✗
55.3
50.7
92.7
✓
✗
✗
✗
57.1
54.1
97.0
Effect on ST Merging
✗
✓
✗
✗
56.9
54.2
96.7
✗
✓
✓
✗
57.3
55.1
97.3
Table 3: Ablation study of the proposed components. w/ div.: Diversity-driven salient token selection strategy. ST Merge: Spatio-temporal merging. TSA: Timestamp alignment for each video frame, otherwise a fixed increment is used.
Figure 8: Impact of expansion coefficient. Overall accuracy is relatively stable w.r.t. varying coefficients. Since excess candidate tokens can also introduce background noise, a moderate value ( 1.5 ) yields the overall best results.
θbase(t)
θbase(s)
Video-MME
MLVU
Short
Medium
Long
Overall
1e4
1e2
67.8
56.0
49.7
57.8
63.3
1e4
1e3
68.9
56.8
49.8
58.5
63.5
1e4
1e4
67.6
56.4
48.9
57.6
63.3
1e3
1e3
67.2
56.0
49.1
57.4
63.4
1e5
1e3
67.4
55.7
49.9
57.7
63.4
Table 4: Impact of rotational base frequencies. Smaller frequencies induce a relatively larger decay effect. The best configuration is highlighted . Note that θbase(s)=θbase(h)=θbase(w) .
Figure 9: Latency and relative performance under different retention ratios. Our method maintains 99.7% performance relative to the full-token setting, with a 1.63 × speedup. “Other” includes the projector, pooling, and additional pruning-related computations. Wall-clock time is measured on the Video-MME benchmark.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Illustration of SigLIP ViT backbone. The image is transformed by a patch embedding layer, added with position embedding, then sent into a stack of Transformer blocks, comprising layer normalization, self-attention/MLP, and residual connection.
Metric
Sink Token
Normal Token
Top 1
212.5
5.9
Top 2
35.9
5.6
Top 3
33.5
3.7
Top 1%
11.6
2.7
Top 10%
1.4
0.8
Top 25%
0.7
0.5
Appendix
Table 5: Statistics of activation magnitudes at a hidden state after layer 0. The distribution of sink tokens differs from that of normal tokens, with a larger magnitude scale.
Figure 11: Visualization of spatial distributions of attention scores extracted with SigLIP ViT. We observe that (1) sink tokens are distributed around the corners (“attn” and “p-attn” show similar patterns), usually corresponding to backgrounds; and (2) “p-attn” shows more robustness to attention sinks. The indices of the top-10 highest-scoring tokens are annotated. Results are averaged over 100 video samples from ActivityNet [ 15 ] .
Figure 12: Qualitative case of intermediate variables within the attention block. We observe that (1) the sink tokens are initially induced by adding with the position embedding, where a few outlier tokens exhibit exceptionally high norms; (2) the skewed distribution induced by outliers is passed across layers through residual connection, and temporarily reverted by layer normalization (LM); (3) post-attention norms in shallow layers better preserve low-level geometric than attention score, which is overshadowed by attention sink.
Figure 13: Qualitative cases of attention patterns. We observe that (1) “attn” shows more severe attention sinks, failing to attend to salient objects; (2) “p-attn” shows a strong tendency to attend to text-related regions. For instance, the subtitle in the first case, and the leaderboard in the second and third cases.
Figure 14: Qualitative cases of clustering image features. Our method better preserves the geometric structure of objects and better separates different semantic entities. We note that compressing complex scenes remains a significant challenge ( e.g . , crowded scenes in the last frame of case 2 or an e-sports event in case 4, which involve intricate or abstract semantics).
Token pruning is essential for enhancing the computational efficiency of vision-language models (VLMs), particularly for video-based tasks where temporal redundancy is prevalent. Prior approaches typically prune tokens either (1) within the vision transformer (ViT) exclusively for unimodal perception tasks such as action recognition and object segmentation, without adapting to downstream vision-language tasks; or (2) only within the LLM while leaving the ViT output intact, often requiring complex text-conditioned token selection mechanisms. In this paper, we introduce Spatio-Temporal Token Scoring (STTS), a simple and lightweight module that prunes vision tokens across both the ViT and the LLM without text conditioning or token merging, and is fully compatible with end-to-end training. By learning how to score temporally via an auxiliary loss and spatially via LLM downstream gradients, aided by our efficient packing algorithm, STTS prunes 50% of vision tokens throughout the entire architecture, resulting in a 62% improvement in efficiency during both training and inference with only a 0.7% drop in average performance across 13 short and long video QA tasks. Efficiency gains increase with more sampled frames per video. Applying test-time scaling for long-video QA further yields performance gains of 0.5-1% compared to the baseline. Overall, STTS represents a novel, simple yet effective technique for unified, architecture-wide vision token pruning.
Jianrui Zhang, Yue Yang, Rohun Tripathi +5
University of Wisconsin-Madison · Allen Institute for AI
Video Large Language Models (VLLMs) excel in video understanding, but their excessive visual tokens pose a significant computational challenge for real-world applications. Current methods aim to enhance inference efficiency by visual token pruning. However, they do not consider the dynamic characteristics and temporal dependencies of video frames, as they perceive video understanding as a multi-frame task. To address these challenges, we propose MMG-Vid, a novel training-free visual token pruning framework that removes redundancy by Maximizing Marginal Gains at both segment-level and token-level. Specifically, we first divide the video into segments based on frame similarity, and then dynamically allocate the token budget for each segment to maximize the marginal gain of each segment. Subsequently, we propose a temporal-guided DPC algorithm that jointly models inter-frame uniqueness and intra-frame diversity, thereby maximizing the marginal gain of each token. By combining both stages, MMG-Vid can maximize the utilization of the limited token budget, significantly improving efficiency while maintaining strong performance. Extensive experiments demonstrate that MMG-Vid can maintain over 99.5% of the original performance, while effectively reducing 75% visual tokens and accelerating the prefilling stage by 3.9x on LLaVA-OneVision-7B. Code is available at https://github.com/ylnxxts/MMG-Vid.
Junpeng Ma, Qizhe Zhang, Ming Lu +4
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Fudan University · Taobao & Tmall Group of Alibaba
Video large language models (Video LLMs) encode each frame into hundreds of visual tokens, making video inference expensive. Existing methods reduce this cost through similarity-based merging or attention-based selection but face two challenges: Hierarchy-blind Pruning, which overlooks regional compressibility and can discard critical evidence, and Attention Drift, where positional effects and attention sinks shift text-to-vision attention away from task relevance. We propose FoveaTree, a two-stage, training-free visual-token pruning framework. First, Merge-Gain Density Estimation measures the feature distortion caused by regional merging and constructs a quadtree-derived density map to guide token dropping, merging, and retention. Second, Attention Tree Pruning uses the surviving units and inherited tree topology to refine text-to-vision attention in intermediate LLM layers, preserving relevant regional evidence while removing low-relevance content. We evaluate FoveaTree on InternVL3.5-8B and Qwen3-VL-8B across five video question-answering benchmarks and one captioning benchmark at 50%, 30%, and 10% visual-token retention. FoveaTree consistently achieves overall state-of-the-art performance across the evaluated settings, offering a strong accuracy-efficiency trade-off.
Zihan Meng, Weidong Chen, Dexiang Hong +2
University of Science and Technology of China · Harbin Institute of Technology