Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention

Junhao Du; Jialong Xue; Anqi Li; Jincheng Dai; Guo Lu

cs.CVMar 23, 2026

Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention

Authors: Junhao Du, Jialong Xue, Anqi Li, Jincheng Dai, Guo Lu

Organizations: Shanghai Jiao Tong University · Beijing University of Posts and Telecommunications

Abstract

Video large language models (Video-LLMs) face high computational costs due to large volumes of visual tokens. Existing token compression methods typically adopt a two-stage spatiotemporal compression strategy, relying on stage-specific metrics and an implicit assumption of spatiotemporal separability. Under extremely low retention ratios, however, such approaches often result in unbalanced allocation and loss of visual evidence essential for question answering. We reformulate token compression as a spatiotemporal allocation task within a global token retention pool. We propose a unified selection mechanism that integrates attention weights and semantic similarity to globally select tokens with high contribution and low redundancy. Unselected tokens are merged via clustering and refilled, preserving information integrity. Inside the LLM, we further introduce text-aware merging to perform secondary compression based on query relevance. Without requiring retraining, our method serves as a plug-and-play module compatible with existing Video-LLMs. Experiments show that retaining only about 2% of visual tokens preserves 90.1% of baseline performance across multiple benchmarks, while reducing FLOPs to roughly 2.6%. These benefits generalize across diverse backbones, decreasing end-to-end inference latency and memory consumption. Our unified spatiotemporal token compression strategy establishes the state-of-the-art in video understanding under ultra-low token retention.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Previous method remove redundant tokens via separate spatial and temporal stages. (b) Our method maintains a global retention pool for unified spatiotemporal redundancy removal. (c) Response of LLaVA-OneVision-7B to video content understanding. (d) At 5% token retention, HoliTom retains redundant tokens and misses critical ones, leading to misunderstanding ( red ). (e) Our method effectively mitigates these issues and yields the correct response ( green ).

Figure 2: Overview of our method. 1) The Unified Spatiotemporal Compression module filters tokens with high contribution and low semantic redundancy, incorporating them into the retention pool. At the same time, it performs clustering and merging on tokens in the recycle pool to preserve the integrity of visual semantic information. 2) The Text-Aware Merging mechanism further enhances answer accuracy by guiding the LLM to focus on visual tokens that are most relevant to the input query.

Figure 3: The Text-Aware Merging mechanism further enhances answer accuracy by guiding the LLM to focus on visual tokens that are most relevant to the input query.

Method	FLOPs (T) $\downarrow$	FLOPs Ratio $\downarrow$	Retention Ratio	MVBench $\uparrow$	EgoSchema $\uparrow$	MLVU $\uparrow$	LongVideo Bench $\uparrow$	VideoMME $\uparrow$	Avg. $\uparrow$
Method	FLOPs (T) $\downarrow$	FLOPs Ratio $\downarrow$	Retention Ratio	MVBench $\uparrow$	EgoSchema $\uparrow$	MLVU $\uparrow$	LongVideo Bench $\uparrow$	VideoMME $\uparrow$	Score	%
LLaVA-OV-7B [ 11 ]	41.4	100%	100%	58.3	60.4	47.7	56.4	58.6	56.3	100
FsatV [ 4 ]	7.9	19.1%	100%/10%	53.2	55.9	41.6	52.1	52.7	51.1	90.8
VisionZip [ 38 ]	4.0	9.6%	10%	53.5	58.0	42.5	49.3	53.4	51.3	91.2
LLaVA-Scissor [ 27 ]	4.0	9.6%	10%	-	57.5	-	-	55.8	-	-
FastVID [ 24 ]	4.0	9.6%	10%	55.9	58.7	42.6	56.3	57.3	54.2	96.2

Table 1: Comparison of State-of-The-Art Methods on LLaVA-OneVision-7B. The A%/B% retention ratio indicates that A% of the LLM input tokens are retained, and subsequently compressed to B% during the LLM forward pass. Best results are in bold, second best underlined. “(w/o M)” means our method without inner-LLM merging.

Method	FLOPs (T) $\downarrow$	FLOPs Ratio $\downarrow$	Retention Ratio	MVBench $\uparrow$	EgoSchema $\uparrow$	MLVU $\uparrow$	LongVideo Bench $\uparrow$	VideoMME $\uparrow$	Avg. $\uparrow$
Method	FLOPs (T) $\downarrow$	FLOPs Ratio $\downarrow$	Retention Ratio	MVBench $\uparrow$	EgoSchema $\uparrow$	MLVU $\uparrow$	LongVideo Bench $\uparrow$	VideoMME $\uparrow$	Score	%
LLaVA-Video-7B [ 47 ]	80.9	100%	100%	60.4	57.2	53.3	58.9	64.3	58.8	100
FsatV [ 4 ]	10.2	12.6%	100%/2%	42.9	41.0	34.3	45.6	48.1	42.4	72.1
VisionZip [ 38 ]	1.5	1.9%	2%	43.8	39.3	33.5	45.8	48.4	42.2	71.8
FastVid [ 24 ]	1.5	1.9%	2%	48.3	45.7	37.3	49.3	53.6	46.8	79.6
HoliTom [ 23 ]	1.4	1.7%	2%/1%	50.2	46.5	39.9	50.7	55.3	48.5	82.5

Table 2: Cross-backbone Method Comparison. Performance comparison of our method against state-of-the-art methods across different backbones, demonstrating consistent effectiveness.

Panel of Figure 4 — Figure 4: Comparative Analysis of Attention Distribution and Semantic Similarity Distribution. Compared to the HoliTom, our approach more effectively preserves visual tokens with high attention scores but low semantic relevance, thereby eliminating semantic redundancy while retaining visually informative tokens.

Method	Frames	FLOPs (T)	VideoMME
Method	Frames	FLOPs (T)	Short	Medium	Long	Total
Vanilla [ 11 ]	16	19.0	68.2	53.8	48.2	56.7
HoliTom [ 23 ]	64	1.7	63.0	53.9	46.3	54.4
Ours	64	1.7	62.4	53.3	47.6	54.5
HoliTom [ 23 ]	96	2.2	64.1	52.4	45.9	54.1
Ours	96	2.2	65.8	55.0	46.8	55.9

Table 3: Performance of LLaVA-OneVision-7B under different frame sampling rates at 2% token retention.

Method	Retention Ratio	MVBench	LongVideo Bench	Avg. Score (%)
Vanilla [ 11 ]	100%	58.3	56.4	57.4 (100.0%)
FastVid [ 24 ]	25%	56.5	56.3	56.4 (98.3%)
VisionZip [ 38 ]	25%	57.9	56.5	57.2 (99.7%)
HoliTom [ 23 ]	15%	58.1	56.4	57.3 (99.8%)
Ours	15%	58.3	57.5	57.9 (100.9%)

Table 4: Performance comparison of LLaVA-OneVision-7B with different methods at high token retention ratios.

Attn	Sim	Cluster	MVBench	EgoSchema	MLVU	LongVideo Bench	VideoMME	Avg. $\uparrow$
Attn	Sim	Cluster	MVBench	EgoSchema	MLVU	LongVideo Bench	VideoMME	Score	%
✓			41.4	46.6	30.8	44.7	45.3	41.8	74.2
	✓		51.4	55.1	38.8	50.0	50.9	49.2	87.5
		✓	52.2	55.0	36.3	48.4	51.3	48.6	86.4
✓	✓		52.1	56.8	38.4	49.9	51.2	49.7	88.2
✓		✓	48.3	51.4	33.2	46.4	49.5	45.8	81.3

Table 5: Module Ablations for Unified Spatiotemporal Token Compression.

Panel of Figure 5 — Figure 5: Ablation Experiment Results for Each Parameter.

Method	Process	Prefill	TTFT	Throughput	Score
Vanilla [ 11 ]	-	701.2 (1×)	1104.7	27.6	56.3
VisionZip [ 38 ]	35.8	42.1 (17×)	451.6	35.4	45.8
FastVid [ 24 ]	8.6	43.1 (16×)	446.2	33.8	46.9
HoliTom [ 23 ]	88.7	40.5 (17×)	497.4	34.3	49.4
Ours	74.0	31.3 (22×)	473.8	35.7	50.7

Table 6: Comparison of Efficiency and Performance. Process, Prefill, and Time to First Token (TTFT) are measured in milliseconds; throughput is reported in tokens per second (token/s).

Figure 6: Distributions of Selected Token Indices for Text-Aware and Last Token Strategies on the MVBench.

Method	Retention Ratio	MVBench	EgoSchema	MLVU	LongVideo Bench	VideoMME	Avg. $\uparrow$
Method	Retention Ratio	MVBench	EgoSchema	MLVU	LongVideo Bench	VideoMME	Score	%
Qwen2.5-VL [ 2 ]	100%	63.0	52.8	38.5	55.0	57.5	53.4	100.0
VisionZip [ 38 ]	2%	50.3	45.3	29.7	43.4	44.1	42.6	79.8
FastVID [ 24 ]	2%	51.8	46.2	30.6	45.0	46.4	44.0	82.5
Ours(w/o M)	2%	54.1	47.8	32.7	45.4	48.1	45.6	85.5
Ours	2%	53.7	48.4	31.8	45.6	48.1	45.5	85.3

Table S1 : Comparison results based on the Qwen2.5-VL-7B model.

Method	Retention Ratio	MVBench	EgoSchema	MLVU	LongVideo Bench	VideoMME	Avg. $\uparrow$
Method	Retention Ratio	MVBench	EgoSchema	MLVU	LongVideo Bench	VideoMME	Score	%
LLaVA-OV-7B [ 11 ]	100%	58.3	60.4	47.7	56.4	58.6	56.3	100
VisionZip [ 38 ]	15%	56.5	59.8	43.3	54.4	56.1	54.0	95.9
DyToK [ 16 ]	15%	56.1	59.5	44.6	53.7	56.1	54.0	95.9
FastVID [ 24 ]	15%	56.0	58.8	43.3	56.2	57.7	54.4	96.6
HoliTom [ 23 ]	15%/7.5%	58.1	61.2	46.7	56.4	57.3	56.0	99.5

Table S2 : Additional Comparison with DyToK under Higher Retention Ratios

Method	Retain Ratio	ActivityNet-QA
Method	Retain Ratio	Accuracy $\uparrow$	Score $\uparrow$
LLaVA-OV-7B	100%	54.5	3.55
+ VisionZip [ 38 ]	2%	41.7	3.13
+ FastVid [ 24 ]	2%	49.6	3.35
+ HoliTom [ 23 ]	2%	49.9	3.31
+ Ours	2%	50.2	3.37

Table S3 : Comparison on ActivityNet-QA. The two metrics ( Accuracy and Score ) are sub-scores under the ActivityNet-QA benchmark.

Method	Frames	MVBench	EgoSchema	MLVU	LongVideo Bench	VideoMME	Avg. Score
Vanilla [ 11 ]	16	58.1	59.5	41.8	55.7	56.7	54.4
HoliTom [ 23 ]	64	54.8	59.5	38.6	51.0	54.4	51.7
Ours	64	54.1	59.1	41.3	52.5	54.5	52.3
HoliTom [ 23 ]	96	55.9	60.4	41.0	51.4	54.1	52.6
Ours	96	55.7	59.5	42.9	54.2	55.9	53.6
HoliTom [ 23 ]	128	55.9	60.9	41.3	53.8	55.4	53.5

Table S4 : Comparison across different frame settings at 2% token retention.

Method	K	MVBench	EgoSchema	MLVU	LongVideo Bench	VideoMME	Avg.
Method	K	MVBench	EgoSchema	MLVU	LongVideo Bench	VideoMME	Score	%
Vanilla [ 11 ]	-	58.3	60.4	47.7	56.4	58.6	56.3	100
Ours	2	48.7	52.7	36.7	47.8	49.4	47.1	83.6
Ours	7	50.3	54.0	36.6	47.8	49.5	47.6	84.6
Ours	14	52.9	55.7	39.7	51.1	51.4	50.2	89.1
Ours	18	52.8	57.6	40.3	50.8	51.8	50.7	90.1

Table S5 : Performance comparison of different pruning layer K.

Method	$\tau$	MVBench	EgoSchema	MLVU	LongVideo Bench	VideoMME	Avg.
Method	$\tau$	MVBench	EgoSchema	MLVU	LongVideo Bench	VideoMME	Score	%
Vanilla [ 11 ]	-	58.3	60.4	47.7	56.4	58.6	56.3	100
Ours	0.5	49.6	52.9	39.4	49.1	52.3	48.7	86.4
Ours	0.6	51.7	55.9	40.0	50.7	52.7	50.3	89.4
Ours	0.7	52.9	57.2	39.5	51.0	51.3	50.4	89.5
Ours	0.8	53.1	56.4	38.2	49.4	51.3	49.7	88.2

Table S6 : Comparison of

\tau

Method	Ratio	MVBench	EgoSchema	MLVU	LongVideo Bench	VideoMME	Avg.
Method	Ratio	MVBench	EgoSchema	MLVU	LongVideo Bench	VideoMME	Score	%
Vanilla	-	58.3	60.4	47.7	56.4	58.6	56.3	100
Ours	0.1	51.1	56.5	39.0	50.7	51.6	49.7	88.4
Ours	0.2	51.6	56.8	38.6	49.9	51.3	49.6	88.2
Ours	0.3	52.9	57.2	39.5	51.0	51.3	50.4	89.5
Ours	0.4	52.2	56.5	38.4	49.4	52.0	49.7	88.3

Table S7 : Comparison of Cluster Ratio at 2% Tokens.

Method	$\lambda$	MVBench	EgoSchema	MLVU	LongVideo Bench	VideoMME	Avg.
Method	$\lambda$	MVBench	EgoSchema	MLVU	LongVideo Bench	VideoMME	Score	%
Vanilla [ 11 ]	-	58.3	60.4	47.7	56.4	58.6	56.3	100
Ours	0	52.6	57.5	40.4	50.6	51.5	50.5	89.7
Ours	0.25	52.6	57.5	40.1	50.7	51.5	50.5	89.7
Ours	0.5	52.8	57.6	40.3	50.8	51.8	50.7	90.1
Ours	0.75	52.5	57.5	39.9	50.5	51.5	50.4	89.5

Table S8 : Comparison of

\lambda

Figure S1 : Performance of different pruning ratios across different layers

Figure S2 : Tokens selected by our text-aware module within the LLM.

Panel of Figure 23 — Figure S3 : Token reductions outside the LLM. Green boxes mark retained tokens, Red boxes mark tokens discarded by similarity filtering, and Purple boxes mark tokens merged via clustering.

Figure S4 : Qualitative generation comparison between our method and other approaches.

Explore similar work

KiToke: Distribution-Aware Token Compression for Video Large Language Models
Apr 3, 2026Haifeng Huang, Biyin Xu, Yang LiVideo Token Compression Visual Token Compression
Department of Computer Science, Iowa State University.
InfoMerge: Information-aware Token Compression for Efficient Video Large Language Models
Jun 1, 2026Xinxin Liu, Shiwei Gan, Xiao Liu +3Efficient VLM Inference Video-Language Models
State Key Laboratory of Novel Software Technology, Nanjing University Nanjing 210023, China
DynaTok: Temporally Adaptive and Positional Bias-Aware Token Compression for Video-LLMs
May 19, 2026Minyoung Park, Taehun Kong, Sangjun AhnVideo-Language Models Video Token Compression
LG Electronics, Seoul, South Korea

Apr 3, 2026J/K move · Enter open · S save

KiToke: Distribution-Aware Token Compression for Video Large Language Models

Haifeng Huang, Biyin Xu, Yang Li

Department of Computer Science, Iowa State University.

Video Token Compression Visual Token Compression Efficient VLM Inference Video-Language Models Visual Token Merging

Open paper AskPDF arXiv

Video Large Language Models (Video LLMs) achieve strong performance on video understanding tasks but suffer from high inference costs due to the large number of visual tokens. We study training-free, query-agnostic video token compression, where heuristic importance rules can over-represent large redundant visual groups and miss less frequent visual patterns, particularly under aggressive token budgets. We propose KiToke, a distribution-aware compression framework that treats visual tokens as samples from the video's spatiotemporal token distribution. KiToke uses global embedding density as a proxy for representational redundancy and inverse-density sampling to balance token allocation across visual patterns of different frequencies. It further aggregates unselected tokens into retained anchors within detected temporal intervals to reduce cross-transition mixing. Across evaluated backbones, KiToke achieves the highest average accuracy among compared training-free methods and remains robust down to 1% retention, where most evaluated baselines underperform uniform random token sampling. On LLaVA-OneVision, it preserves 98.5% and 89.2% of the uncompressed model's four-benchmark mean accuracy at 10% and 1% token retention, with 6.8x and 12.5x prefill speedups, respectively, including compression overhead.

Abstract

Explore similar work

KiToke: Distribution-Aware Token Compression for Video Large Language Models

InfoMerge: Information-aware Token Compression for Efficient Video Large Language Models

DynaTok: Temporally Adaptive and Positional Bias-Aware Token Compression for Video-LLMs