Tango: Taming Visual Signals for Efficient Video Large Language Models

Shukang Yin; Sirui Zhao; Hanchao Wang; Baozhi Jia; Xianquan Wang; Chaoyou Fu; Enhong Chen

cs.CVApr 10, 2026

Tango: Taming Visual Signals for Efficient Video Large Language Models

Authors: Shukang Yin, Sirui Zhao, Hanchao Wang, Baozhi Jia, Xianquan Wang, Chaoyou Fu, Enhong Chen

Organizations: University of Science and Technology of China · Reconova AI Lab · Nanjing University

Abstract

Token pruning has emerged as a mainstream approach for developing efficient Video Large Language Models (Video LLMs). This work revisits and advances the two predominant token-pruning paradigms: attention-based selection and similarity-based clustering. Our study reveals two critical limitations in existing methods: (1) conventional top-k selection strategies fail to fully account for the attention distribution, which is often spatially multi-modal and long-tailed in magnitude; and (2) direct similarity-based clustering frequently generates fragmented clusters, resulting in distorted representations after pooling. To address these bottlenecks, we propose Tango, a novel framework designed to optimize the utilization of visual signals. Tango integrates a diversity-driven strategy to enhance attention-based token selection, and introduces Spatio-temporal Rotary Position Embedding (ST-RoPE) to preserve geometric structure via locality priors. Comprehensive experiments across various Video LLMs and video understanding benchmarks demonstrate the effectiveness and generalizability of our approach. Notably, when retaining only 10% of the video tokens, Tango preserves 98.9% of the original performance on LLaVA-OV while delivering a 1.88 $\times$ inference speedup.

Figures & tables

Figure 1: Limitations of two typical pre-LLM token pruning approaches. (Top Right) Top-k selection fails to fully capture the attention distribution, which is spatially multimodal and long-tailed in magnitude. (Bottom Right) Direct similarity-based clustering can result in noisy representations.

Figure 2: Illustration of attention distribution and similarity-based clustering. Attention Heatmaps: The spatial distribution of attention scores exhibits multiple modes ( i.e . , local maxima of attention scores) corresponding to distinct semantic regions ( e.g . , subtitle and head in the first frame). Clustering Results: Direct similarity-based cluster assignment (Baseline) leads to spatially fragmented results, where a coherent semantic entity ( e.g . , the head or body) is split into different clusters or mixed with background. Pooling on these clusters may result in noisy representations. In comparison, our method (bottom row) mitigates this issue and exhibits superior geometric shape preservation. Visualization is extracted using the SigLIP [ 11 ] ViT of LLaVA-OV-7B on a Video-MME sample ( vid: lmlw4NNZBbw ). More cases and details are provided in the Appendix.

Figure 3: Distribution of sorted attention scores. The scores exhibit a distinct long-tailed distribution with a narrow variance band, demonstrating highly stable attention patterns across videos. Results are averaged over 100 video samples from ActivityNet [ 15 ] .

Figure 4: Overview of our method. It comprises three modules: Temporal Video Segmentation (TVS), Salient Token Selection (STS), and Spatio-Temporal Merging (STM). Our technical contributions lie in (1) an enhanced salient token selection strategy in the STS module, and (2) position-aware clustering with ST-RoPE, used in STS and STM modules.

Figure 5: Illustration of ST-RoPE. (Bottom left) It encodes spatio-temporal position information by applying a rotation matrix to vision tokens. (Bottom right) The pairwise distance on the hypersphere, $\text{d}(\cdot,\cdot)$ , is not only determined by cosine similarity, but also modulated by $\Delta p$ , where spatio-temporally distant tokens are assigned a larger distance penalty. Consequently, semantically similar and spatio-temporally neighboring tokens are clustered together.

Methods	Video-MME (w/o sub)				MVBench	LongVideo Bench	MLVU	Avg.	Perf.
	Short	Medium	Long	Overall	MVBench	LongVideo Bench		Avg.	Perf.
LLaVA-OneVision-7B [ 26 ]	71.9	56.3	49.6	59.3	55.3	56.8	64.7	59.0	100.0
Retention Ratio=10%
FastV [ 1 ] [ECCV 2024]	54.7	49.0	43.7	49.1	48.3	47.3	54.6	49.8	84.4
DART [ 2 ] [EMNLP 2025]	54.4	49.0	44.0	49.1	46.2	46.3	55.4	49.3	83.5
VisionZip [ 3 ] [CVPR 2025]	61.2	52.3	47.2	53.6	50.7	51.7	59.0	53.8	91.1

Table 1: Comparison of state-of-the-art token pruning methods on four mainstream video understanding benchmarks. We experiment with three retention ratios (10%, 15%, 20%) and 32 sampled frames on the LLaVA-OneVision-7B model. Our method consistently achieves the best overall performance across different retention ratios. Bold and u nderlined denote best and second-best performance, respectively. Avg.: Average accuracy on four benchmarks. Perf.: Ratio of performance retained relative to the original model. All baseline results are reproduced under the same settings.

Methods	Video-MME (w/o sub)				MVBench	LongVideo Bench	MLVU	Avg.	Perf.
	Short	Medium	Long	Overall	MVBench	LongVideo Bench		Avg.	Perf.
LLaVA-Video-7B [ 27 ]	76.7	62.0	51.4	63.4	60.7	57.5	71.4	63.3	100.0
Retention Ratio=10%
FastV [ 1 ] [ECCV 2024]	56.1	49.6	44.4	50.0	47.6	45.8	52.2	48.9	77.3
VisionZip [ 3 ] [CVPR 2025]	67.0	54.6	46.8	56.1	54.2	52.4	62.9	56.4	89.1
VidCom 2 [ 4 ] [EMNLP 2025]	65.6	55.7	45.9	55.7	52.1	50.4	59.7	54.5	86.1

Table 2: Cross-model evaluation on LLaVA-Video-7B and Qwen2.5-VL-7B , with 64 and 32 input frames, respectively.

\dagger

: Combined with intra-LLM method (FastV variant [ 9 ] ), our method can achieve better overall performance.

Figure 6: Results with different input frames. Our method maintains stable scaling with frame number, clearly outperforming other SOTA methods. “w/o M” denotes pre-LLM-only pruning [ 9 ] . The performance is averaged across three video datasets: Video-MME, LongVideoBench, and MLVU.

Figure 7: Results with different salient token selection strategy . Top-k (attn) underperforms, and Top-k (p-attn) is only comparable to a simple uniform sampling baseline. Plugging in our proposed diversity-driven selection method (“w/ div.”) brings remarkable gains. attn and p-attn denote attention scores extracted by pooling attention weights on the query dimension [ 3 , 9 ] and calculating with a global query [ 5 ] , respectively.

w/ div.	ST Merge	ST-RoPE	TSA	V-MME	LVB	Perf.
Effect on Token Selection
✗	✗	✗	✗	55.3	50.7	92.7
✓	✗	✗	✗	57.1	54.1	97.0
Effect on ST Merging
✗	✓	✗	✗	56.9	54.2	96.7
✗	✓	✓	✗	57.3	55.1	97.3

Table 3: Ablation study of the proposed components. w/ div.: Diversity-driven salient token selection strategy. ST Merge: Spatio-temporal merging. TSA: Timestamp alignment for each video frame, otherwise a fixed increment is used.

Figure 8: Impact of expansion coefficient. Overall accuracy is relatively stable w.r.t. varying coefficients. Since excess candidate tokens can also introduce background noise, a moderate value (

1.5

) yields the overall best results.

$\theta_{\mathrm{base}}^{(t)}$	$\theta_{\mathrm{base}}^{(s)}$	Video-MME				MLVU
$\theta_{\mathrm{base}}^{(t)}$		Short	Medium	Long	Overall	MLVU
1e4	1e2	67.8	56.0	49.7	57.8	63.3
1e4	1e3	68.9	56.8	49.8	58.5	63.5
1e4	1e4	67.6	56.4	48.9	57.6	63.3
1e3	1e3	67.2	56.0	49.1	57.4	63.4
1e5	1e3	67.4	55.7	49.9	57.7	63.4

Table 4: Impact of rotational base frequencies. Smaller frequencies induce a relatively larger decay effect. The best configuration is highlighted . Note that

\theta_{\mathrm{base}}^{(s)}=\theta_{\mathrm{base}}^{(h)}=\theta_{\mathrm{base}}^{(w)}

Figure 9: Latency and relative performance under different retention ratios. Our method maintains 99.7% performance relative to the full-token setting, with a 1.63

\times

speedup. “Other” includes the projector, pooling, and additional pruning-related computations. Wall-clock time is measured on the Video-MME benchmark.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Illustration of SigLIP ViT backbone. The image is transformed by a patch embedding layer, added with position embedding, then sent into a stack of Transformer blocks, comprising layer normalization, self-attention/MLP, and residual connection.

Metric	Sink Token	Normal Token
Top 1	212.5	5.9
Top 2	35.9	5.6
Top 3	33.5	3.7
Top 1%	11.6	2.7
Top 10%	1.4	0.8
Top 25%	0.7	0.5

Appendix

Table 5: Statistics of activation magnitudes at a hidden state after layer 0. The distribution of sink tokens differs from that of normal tokens, with a larger magnitude scale.

Panel of Figure 11 — Figure 11: Visualization of spatial distributions of attention scores extracted with SigLIP ViT. We observe that (1) sink tokens are distributed around the corners (“attn” and “p-attn” show similar patterns), usually corresponding to backgrounds; and (2) “p-attn” shows more robustness to attention sinks. The indices of the top-10 highest-scoring tokens are annotated. Results are averaged over 100 video samples from ActivityNet [ 15 ] .

Figure 12: Qualitative case of intermediate variables within the attention block. We observe that (1) the sink tokens are initially induced by adding with the position embedding, where a few outlier tokens exhibit exceptionally high norms; (2) the skewed distribution induced by outliers is passed across layers through residual connection, and temporarily reverted by layer normalization (LM); (3) post-attention norms in shallow layers better preserve low-level geometric than attention score, which is overshadowed by attention sink.

Figure 13: Qualitative cases of attention patterns. We observe that (1) “attn” shows more severe attention sinks, failing to attend to salient objects; (2) “p-attn” shows a strong tendency to attend to text-related regions. For instance, the subtitle in the first case, and the leaderboard in the second and third cases.

Figure 14: Qualitative cases of clustering image features. Our method better preserves the geometric structure of objects and better separates different semantic entities. We note that compressing complex scenes remains a significant challenge ( e.g . , crowded scenes in the last frame of case 2 or an e-sports event in case 4, which involve intricate or abstract semantics).

Video-Language Models

Video-language models connect visual content over time with natural language for understanding, describing, retrieving, and interacting with videos.

See all 167 papers

Unified Spatio-Temporal Token Scoring for Efficient Video VLMs
Mar 18, 2026Jianrui Zhang, Yue Yang, Rohun Tripathi +5Visual Token Pruning Efficient VLM Inference
University of Wisconsin-Madison · Allen Institute for AI
MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs
Aug 28, 2025Junpeng Ma, Qizhe Zhang, Ming Lu +4Efficient VLM Inference Video Token Compression
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Fudan University · Taobao & Tmall Group of Alibaba
FoveaTree: Information-Density Trees for Query-Guided Visual Token Pruning in Video Large Language Models
Oct 12, 2026Zihan Meng, Weidong Chen, Dexiang Hong +2Efficient VLM Inference Visual Token Pruning
University of Science and Technology of China · Harbin Institute of Technology

Mar 18, 2026J/K move · Enter open · S save

Unified Spatio-Temporal Token Scoring for Efficient Video VLMs

Jianrui Zhang, Yue Yang, Rohun Tripathi, Winson Han+4

University of Wisconsin-Madison · Allen Institute for AI

Visual Token Pruning Efficient VLM Inference Temporal Video Understanding

Open paper AskPDF arXiv

Token pruning is essential for enhancing the computational efficiency of vision-language models (VLMs), particularly for video-based tasks where temporal redundancy is prevalent. Prior approaches typically prune tokens either (1) within the vision transformer (ViT) exclusively for unimodal perception tasks such as action recognition and object segmentation, without adapting to downstream vision-language tasks; or (2) only within the LLM while leaving the ViT output intact, often requiring complex text-conditioned token selection mechanisms. In this paper, we introduce Spatio-Temporal Token Scoring (STTS), a simple and lightweight module that prunes vision tokens across both the ViT and the LLM without text conditioning or token merging, and is fully compatible with end-to-end training. By learning how to score temporally via an auxiliary loss and spatially via LLM downstream gradients, aided by our efficient packing algorithm, STTS prunes 50% of vision tokens throughout the entire architecture, resulting in a 62% improvement in efficiency during both training and inference with only a 0.7% drop in average performance across 13 short and long video QA tasks. Efficiency gains increase with more sampled frames per video. Applying test-time scaling for long-video QA further yields performance gains of 0.5-1% compared to the baseline. Overall, STTS represents a novel, simple yet effective technique for unified, architecture-wide vision token pruning.

Abstract

Video-Language Models

Explore similar work

Unified Spatio-Temporal Token Scoring for Efficient Video VLMs

MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs

FoveaTree: Information-Density Trees for Query-Guided Visual Token Pruning in Video Large Language Models