Video large language models (VLLMs) rely on long visual token sequences to understand complex temporal content, which makes visual token compression important for efficient inference. Existing training-free compression methods mainly reduce the number of tokens sent to the language model, but the compression process itself can still require many visual feature comparisons, especially for intra-frame token protection and inter-frame temporal token merging. This paper proposes a video codec-guided training-free token compression method for VLLMs that uses coding information already available in compressed video as a low-cost search prior. The method aligns reference-frame dependencies, motion vectors, and residuals with the visual token grid. Residual information forms compact candidate pools for token protection, while motion information is propagated along reference-frame chains to construct localized candidates for temporal token merging between sparsely sampled frames. Local visual feature similarity comparison is then used within these candidates to make the final compression decisions, without modifying the base VLLM. Experiments on three representative VLLMs and four video-understanding benchmarks show that the proposed method achieves performance comparable to the state-of-the-art training-free baseline FlashVID, and in some settings surpasses it, while substantially reducing the feature comparisons required by token protection and temporal matching. These results demonstrate that codec information can effectively improve the accuracy-complexity tradeoff of video token compression.
Figures & tables
Fig. 1: Illustration of video codec information extraction and codec-guided local token matching. (a) Video codec information extraction. (b) Global visual token matching and codec-guided local token matching.
Fig. 2: Comparison of the performance and feature-comparison costs of our method and the state-of-the-art baseline FlashVID [ 13 ] .
Fig. 3: Overview of our proposed video codec-guided training-free token compression framework for video large language models.
Fig. 4: Illustration of the procedure for aligning block-level coding information to the visual token grid.
Fig. 5: Illustration of the residual-guided token protection method.
Fig. 6: Illustration of motion-guided temporal merging through reference-chain propagation.
Fig. 7: Subjective comparison results between FlashVID and our method.
ρ
Selection Reduction (%)
Temporal Reduction (%)
Video-MME
25%
79.8
64.7
59.0 (58.1)
20%
87.2
64.7
58.3 (58.0)
15%
93.0
64.7
57.0 (57.5)
10%
96.6
64.7
56.7 (57.4)
TABLE III: Reduction of feature comparisons relative to FlashVID on Video-MME with LLaVA-OneVision. The Video-MME column reports our score, with the FlashVID score shown in parentheses.
Method
Short
Medium
Long
Overall
Motion-Guided Only
71.4
56.5
48.9
59.0
Residual-Guided Only
70.3
55.8
49.0
58.4
Combined
70.4
56.8
49.8
59.0
TABLE IV: Effectiveness of motion and residual guidance on Video-MME with LLaVA-OneVision ( ρ=25% ).
α
Short
Medium
Long
Overall
0
70.9
55.9
48.9
58.6
0.1
70.9
55.4
49.3
58.6
0.2
70.4
56.6
49.8
59.0
TABLE V: Effectiveness of temporal residual increase on Video-MME with LLaVA-OneVision ( ρ=25% ).
r
Short
Medium
Long
Overall
1
70.3
56.3
49.7
58.8
2
70.4
56.8
49.8
59.0
3
70.3
55.6
49.9
58.6
4
70.9
56.8
49.8
59.1
TABLE VI: Effectiveness of the local search radius on Video-MME with LLaVA-OneVision ( ρ=25% ).
Fig. 8: Effectiveness of temporal residual increase and local search radius. The top part shows the protected tokens under different temporal residual increase weights. Adjusting α changes the protected tokens while keeping the protection budget fixed. The bottom part shows the candidate regions for local matching under different search radii r .
Video Large Language Models (Video LLMs) achieve strong performance on video understanding tasks but suffer from high inference costs due to the large number of visual tokens. We study training-free, query-agnostic video token compression, where heuristic importance rules can over-represent large redundant visual groups and miss less frequent visual patterns, particularly under aggressive token budgets. We propose KiToke, a distribution-aware compression framework that treats visual tokens as samples from the video's spatiotemporal token distribution. KiToke uses global embedding density as a proxy for representational redundancy and inverse-density sampling to balance token allocation across visual patterns of different frequencies. It further aggregates unselected tokens into retained anchors within detected temporal intervals to reduce cross-transition mixing. Across evaluated backbones, KiToke achieves the highest average accuracy among compared training-free methods and remains robust down to 1% retention, where most evaluated baselines underperform uniform random token sampling. On LLaVA-OneVision, it preserves 98.5% and 89.2% of the uncompressed model's four-benchmark mean accuracy at 10% and 1% token retention, with 6.8x and 12.5x prefill speedups, respectively, including compression overhead.
Haifeng Huang, Biyin Xu, Yang Li
Department of Computer Science, Iowa State University.
Video Large Language Models (Video-LLMs) achieve strong performance in video understanding, but their excessive visual tokens bring substantial computational overhead. Existing training-free compression methods improve inference efficiency by reducing visual tokens, yet they often rely on local adjacent-frame similarity for temporal redundancy estimation or allocate token budgets mainly according to segment length. Such designs are sensitive to frame-level noise and fail to capture the non-uniform information distribution of real-world videos. To address these challenges, we propose InfoMerge, a training-free visual token compression method that improves token utilization through robust redundancy estimation and content-aware budget allocation. Specifically, we propose the Temporal Fingerprint Difference: a segment-level second-order temporal redundancy estimation strategy, which models the temporal similarity structure of tokens at the same spatial positions within each segment. We further introduce Content-Aware Budget Allocation (CABA), which dynamically allocates segment-level token budgets based on segment uniqueness and spectral-entropy-based representational richness. By reducing repeated preservation of redundant static regions and allocating more tokens to informative segments, InfoMerge makes better use of the limited token budget while maintaining strong performance. Extensive experiments show that InfoMerge achieves strong efficiency--accuracy trade-offs across multiple benchmarks and backbones, with more pronounced advantages under aggressive compression. On LLaVA-OneVision-7B, InfoMerge retains 98.8% of the original average performance while reducing 85% of visual tokens and achieving a 4.24-fold speedup in the prefill stage.
Xinxin Liu, Shiwei Gan, Xiao Liu +3
State Key Laboratory of Novel Software Technology, Nanjing University Nanjing 210023, China
As Video Large Language Models (Video-LLMs) scale to longer and more complex videos, their inference cost grows rapidly due to the large volume of visual tokens accumulated across frames. Training-free token compression has emerged as a practical solution to this bottleneck. However, existing temporal compression methods rely primarily on cross-frame token similarity or segmentation heuristics, overlooking each token's semantic role within its frame and failing to adapt compression strength to the compressibility of each frame pair. In this work, we propose OTT-Vid, a transport-derived allocation framework for temporal token compression. Our approach consists of two stages: spatial pruning identifies representative content within each frame, and optimal transport (OT) is then solved between neighboring frames to estimate temporal compressibility. We formulate this OT with non-uniform token mass, which protects semantically important tokens from aggressive compression, and a locality-aware cost that captures both feature and spatial disparities. The resulting transport plan jointly balances token importance and matching cost, while its total cost defines the transport difficulty of each frame pair, which we use to allocate compression budgets dynamically. Experiments on six benchmarks spanning video question answering and temporal grounding show that OTT-Vid preserves 95.8% of VQA and 73.9% of VTG performance while retaining only 10% of tokens, consistently outperforming existing state-of-the-art training-free compression methods.