Video Large Language Models (Video LLMs) achieve strong performance on video understanding tasks but suffer from high inference costs due to the large number of visual tokens. We study training-free, query-agnostic video token compression, where heuristic importance rules can over-represent large redundant visual groups and miss less frequent visual patterns, particularly under aggressive token budgets. We propose KiToke, a distribution-aware compression framework that treats visual tokens as samples from the video's spatiotemporal token distribution. KiToke uses global embedding density as a proxy for representational redundancy and inverse-density sampling to balance token allocation across visual patterns of different frequencies. It further aggregates unselected tokens into retained anchors within detected temporal intervals to reduce cross-transition mixing. Across evaluated backbones, KiToke achieves the highest average accuracy among compared training-free methods and remains robust down to 1% retention, where most evaluated baselines underperform uniform random token sampling. On LLaVA-OneVision, it preserves 98.5% and 89.2% of the uncompressed model's four-benchmark mean accuracy at 10% and 1% token retention, with 6.8x and 12.5x prefill speedups, respectively, including compression overhead.
Figures & tables
Figure 1: Distribution-aware token compression improves robustness across retention ratios. (a) KiToke uses global token density as a proxy for redundancy and inverse-density sampling to promote broad visual coverage. (b) Normalized performance vs. retention ratio for representative token compression methods, including Random. All prior methods shown at 1% retention fall below Random. Performance is normalized by each uncompressed backbone before averaging over LLaVA-OneVision, LLaVA-Video, and Qwen3-VL, using four benchmarks. (c) Normalized average accuracy versus prefill latency on LLaVA-OneVision-7B at 1% retention. Accuracy is the four-benchmark mean normalized to the uncompressed model. Latency includes compression overhead.
Figure 2: Overview of KiToke. (a) Visual tokens are compressed between the vision projector and the LLM. (b) Anchors are sampled using inverse-density scores computed from the full-video token distribution. (c) Unselected tokens are merged into anchors within content-adaptive temporal intervals using the same scores.
Figure 3: Performance comparison with state-of-the-art methods across video understanding benchmarks on LLaVA-Video and Qwen3-VL . Mean accuracy over four video understanding benchmarks is normalized to the uncompressed backbone (100%). Detailed benchmark-level results are reported in Appendix B .
Anchor selector
Mean accuracy (%)
Uniform sampling
50.4
Density-proportional, full video
48.8
k -center Gonzalez (1985)
50.1
DivPrune Alvar et al. (2025)
50.8
FastVID Shen et al. (2025a)
50.4
FLoC Cho et al. (2026)
50.3
Table 3: Controlled query-agnostic token allocation. We vary density scope only for selection, keeping intervals and inverse-density merge weights fixed for all selectors. Four-benchmark mean accuracy (%) on LLaVA-OneVision-7B at 1% retention is reported.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Hyperparameter ablations on LLaVA-OneVision-7B at 1% visual-token retention. Panels (a)–(c) vary Gaussian bandwidth α , the threshold on adjacent-frame change ct , and the threshold on its absolute increase Δt , respectively. Filled markers identify the defaults α=800 and (τc,τΔ)=(110,70) . The vertical axis is mean accuracy over MVBench, LongVideoBench, MLVU, and VideoMME without subtitles, normalized by the uncompressed LLaVA-OneVision-7B mean (100%).
Figure 5: Model-size ablation at 1% visual-token retention. Comparison at 0.5B, 7B, and 72B model sizes. The vertical axis reports normalized average accuracy: 100Aˉcompressed/Aˉuncompressed , where Aˉ is the mean accuracy over MVBench, LongVideoBench, MLVU, and VideoMME without subtitles. Each compressed result is normalized by the uncompressed result for the same model variant.
Figure 6: Qualitative comparison of temporal interval construction. Each example (a)–(e) shows 12 frames in temporal order, with zero-based display positions 0–11. FastVID and KiToke boundaries are shown in the corresponding rows, marked by red separators and downward triangles. In these examples, KiToke produces boundaries that align more closely with the illustrated content changes than FastVID.
Figure 7: Qualitative comparison of token selection (case 1). All methods show the same five frames, labeled with zero-based indices 1, 5, 6, 7, and 31. The bars show choice-normalized probabilities over A–D, and “Pred” identifies the option parsed from the generated answer. Compressed methods use γ=1% retention, with LLaVA-OneVision as the uncompressed reference. Anchors are fully opaque, while non-anchor locations are faded and, for KiToke, may contribute through merging. Green marks the ground-truth choice.
Figure 8: Qualitative comparison of token selection (case 2). All methods show the same five frames, labeled with zero-based indices 0, 4, 11, 15, and 26. The bars show choice-normalized probabilities over A–D, and “Pred” identifies the option parsed from the generated answer. Compressed methods use γ=1% retention, with LLaVA-OneVision as the uncompressed reference. Anchors are fully opaque, while non-anchor locations are faded and, for KiToke, may contribute through merging. Green marks the ground-truth choice.
Figure 9: Qualitative comparison of token selection (case 3). All methods show the same five frames, labeled with zero-based indices 0, 13, 14, 16, and 18. The bars show choice-normalized probabilities over A–D, and “Pred” identifies the option parsed from the generated answer. Compressed methods use γ=1% retention, with LLaVA-OneVision as the uncompressed reference. Anchors are fully opaque, while non-anchor locations are faded and, for KiToke, may contribute through merging. Green marks the ground-truth choice.
Figure 10: Qualitative comparison of token selection (case 4). All methods show the same five frames, labeled with zero-based indices 7, 18, 27, 28, and 31. The bars show choice-normalized probabilities over A–D, and “Pred” identifies the option parsed from the generated answer. Compressed methods use γ=1% retention, with LLaVA-OneVision as the uncompressed reference. Anchors are fully opaque, while non-anchor locations are faded and, for KiToke, may contribute through merging. Green marks the ground-truth choice.
Figure 11: Qualitative comparison of token selection (case 5). All methods show the same five frames, labeled with zero-based indices 7, 8, 12, 21, and 24. The bars show choice-normalized probabilities over A–D, and “Pred” identifies the option parsed from the generated answer. Compressed methods use γ=1% retention, with LLaVA-OneVision as the uncompressed reference. Anchors are fully opaque, while non-anchor locations are faded and, for KiToke, may contribute through merging. Green marks the ground-truth choice.
Video Large Language Models (Video-LLMs) achieve strong performance in video understanding, but their excessive visual tokens bring substantial computational overhead. Existing training-free compression methods improve inference efficiency by reducing visual tokens, yet they often rely on local adjacent-frame similarity for temporal redundancy estimation or allocate token budgets mainly according to segment length. Such designs are sensitive to frame-level noise and fail to capture the non-uniform information distribution of real-world videos. To address these challenges, we propose InfoMerge, a training-free visual token compression method that improves token utilization through robust redundancy estimation and content-aware budget allocation. Specifically, we propose the Temporal Fingerprint Difference: a segment-level second-order temporal redundancy estimation strategy, which models the temporal similarity structure of tokens at the same spatial positions within each segment. We further introduce Content-Aware Budget Allocation (CABA), which dynamically allocates segment-level token budgets based on segment uniqueness and spectral-entropy-based representational richness. By reducing repeated preservation of redundant static regions and allocating more tokens to informative segments, InfoMerge makes better use of the limited token budget while maintaining strong performance. Extensive experiments show that InfoMerge achieves strong efficiency--accuracy trade-offs across multiple benchmarks and backbones, with more pronounced advantages under aggressive compression. On LLaVA-OneVision-7B, InfoMerge retains 98.8% of the original average performance while reducing 85% of visual tokens and achieving a 4.24-fold speedup in the prefill stage.
Xinxin Liu, Shiwei Gan, Xiao Liu +3
State Key Laboratory of Novel Software Technology, Nanjing University Nanjing 210023, China
Video large language models (Video-LLMs) face high computational costs due to large volumes of visual tokens. Existing token compression methods typically adopt a two-stage spatiotemporal compression strategy, relying on stage-specific metrics and an implicit assumption of spatiotemporal separability. Under extremely low retention ratios, however, such approaches often result in unbalanced allocation and loss of visual evidence essential for question answering. We reformulate token compression as a spatiotemporal allocation task within a global token retention pool. We propose a unified selection mechanism that integrates attention weights and semantic similarity to globally select tokens with high contribution and low redundancy. Unselected tokens are merged via clustering and refilled, preserving information integrity. Inside the LLM, we further introduce text-aware merging to perform secondary compression based on query relevance. Without requiring retraining, our method serves as a plug-and-play module compatible with existing Video-LLMs. Experiments show that retaining only about 2% of visual tokens preserves 90.1% of baseline performance across multiple benchmarks, while reducing FLOPs to roughly 2.6%. These benefits generalize across diverse backbones, decreasing end-to-end inference latency and memory consumption. Our unified spatiotemporal token compression strategy establishes the state-of-the-art in video understanding under ultra-low token retention.
Junhao Du, Jialong Xue, Anqi Li +2
Shanghai Jiao Tong University · Beijing University of Posts and Telecommunications
Video large language models (VLLMs) rely on long visual token sequences to understand complex temporal content, which makes visual token compression important for efficient inference. Existing training-free compression methods mainly reduce the number of tokens sent to the language model, but the compression process itself can still require many visual feature comparisons, especially for intra-frame token protection and inter-frame temporal token merging. This paper proposes a video codec-guided training-free token compression method for VLLMs that uses coding information already available in compressed video as a low-cost search prior. The method aligns reference-frame dependencies, motion vectors, and residuals with the visual token grid. Residual information forms compact candidate pools for token protection, while motion information is propagated along reference-frame chains to construct localized candidates for temporal token merging between sparsely sampled frames. Local visual feature similarity comparison is then used within these candidates to make the final compression decisions, without modifying the base VLLM. Experiments on three representative VLLMs and four video-understanding benchmarks show that the proposed method achieves performance comparable to the state-of-the-art training-free baseline FlashVID, and in some settings surpasses it, while substantially reducing the feature comparisons required by token protection and temporal matching. These results demonstrate that codec information can effectively improve the accuracy-complexity tradeoff of video token compression.
Xihua Sheng, Ruijie Liu, Chang Wen Chen
Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China