Organizations: Key Laboratory of Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology · Baidu, Inc
Video language models process videos as dense visual-token sequences with substantial representational redundancy. Compressing these sequences is therefore essential for reducing the visual-token burden on language-model decoding. The central challenge is to preserve visual information dispersed across frames under such compression. To this end, we introduce Aggregating Visual Information with Optimal Transport (AVIOT), which casts video token compression as transporting a dense empirical measure of frame observations onto a compact target measure. The resulting source-to-target coupling induces a distribution over source observations for each target support, directly specifying how the compressed video representation is constructed. We further adapt this construction along task and spatial axes. Question conditioning modulates the transport cost between source frames and target supports, while influencing how many supports are allocated to each temporal segment, thereby directing representation capacity toward question-relevant content. At multiple spatial granularities, AVIOT computes region-specific temporal transport plans and adaptively fuses the representations they yield, allowing different regions within the same compact representation to draw from different moments. Evaluations across varying compression ratios show that AVIOT matches or outperforms the uncompressed baseline on multiple video-understanding benchmarks while retaining strong performance at higher compression ratios.
Figures & tables
Figure 1: Motivation for AVIOT. Common compression paradigms reduce source observations through selection or pruning, or synthesize compact outputs through learned feature mapping. AVIOT constructs the output set through a joint source-to-target optimal-transport coupling, organizing how information distributed across frames is represented collectively by the compact supports.
Figure 2: Overview of the proposed AVIOT framework. Dense video features are augmented with time–height–width (THW) positional encodings and progressively compressed to the requested cardinality. At each stage, the question modulates the transport cost and, together with preliminary transport responses, guides support allocation across temporal segments. Within each segment, source-to-target couplings construct aligned representations at multiple spatial granularities through barycentric aggregation; these representations are adaptively fused before entering the language model.
Method
Token ret.
Prefill FLOPs (T)
Acti- vity Net- QA
Per- cep- tion Test
MV Bench
Video- MME w/o sub.
Ego Sche- ma
LLaVA- Video-7B
100.0%
106.69
64.10
67.90
58.60
63.30
57.30
CoPE- VideoLM
varying
–
64.80 92%
70.30 20%
61.90 16%
61.90 289%
–
Uniform Keep
50.0%
44.28
67.01
68.34
60.75
62.11
57.21
25.0%
19.87
65.79
67.01
58.40
60.30
54.98
10.9%
8.14
63.42
64.46
55.12
56.26
51.38
Segment Mean
50.0%
44.28
66.40
67.74
60.48
61.22
55.34
Table 1: Controlled comparison with a 64-frame input at compression ratios 2, 4, and 10. The second column reports token retention; bold denotes the best score in each benchmark column.
Figure 3: Generalization across compression ratios.
Figure 4: Question-conditioned temporal allocation and provenance across spatial granularities. (a) Support allocations for two questions. (b) Global, medium, and local provenance for one source frame and target support; percentages denote fused mass, and boxes mark the top 20% positions.
Figure 5: Hierarchical source-frame and spatial provenance of selected target supports.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Group
Setting
Value
Representation
ViT feature dim.
1152
Patch layout
27×27
Cost embedding dim.
256
Temporal segments
4
Transport
ε
0.10
ρs,ρt
0.5, 5.0
Appendix
Table S1: Final AVIOT architecture and numerical configuration.
Group
Setting
Value
Initialization
VideoLM
LLaVA-Video-7B-Qwen2
Visual encoder
SigLIP-SO400M-patch14-384
Data
Training set
LLaVA-Video-178K
Optimization
Steps
10,903
Global batch size
128
Learning rate
1×10−5
Appendix
Table S2: Optimization details for AVIOT training.
Method
Ratio
Multi-choice
Yes/no
Caption matching
Captioning
Task avg.
Uniform Keep
2
67.15
68.94
75.45
53.29
66.21
Uniform Keep
4
65.44
67.88
74.78
51.85
64.99
Uniform Keep
10
63.54
65.80
72.46
12.33
53.53
Segment Mean
2
66.52
68.98
75.32
53.54
66.09
Segment Mean
4
64.81
66.82
73.72
51.50
64.21
Segment Mean
10
60.32
63.72
68.80
14.07
51.73
Appendix
Table S3: Extended TempCompass results across four task types. Multi-choice follows the metric used in the main comparison; Task avg. is the unweighted mean of the four task accuracies. Bold marks the best compressed method at each ratio.
Dataset
r=2
r=3.5
r=4
r=5
r=5.5
r=6
r=6.5
r=7
r=8
r=10
LLaVA- Video-7B
CoPE
Video-MME w/o sub.
65.11
64.70
63.89
63.66
63.13
63.84
63.98
62.45
62.40
61.95
63.3
61.9
Video-MMMU
38.22
39.00
39.78
38.33
38.00
38.78
38.78
37.78
38.33
37.89
36.1
38.2
ActivityNet-QA
68.54
68.05
68.23
67.40
67.39
67.49
67.14
67.11
67.09
66.26
64.1
64.8
NExT-QA
84.01
83.51
83.49
83.48
83.30
83.03
82.85
82.60
82.44
82.11
83.2
82.1
Perception Test
71.84
71.48
71.45
71.03
70.86
70.57
70.52
70.48
70.27
69.80
67.9
70.3
TempCompass
69.56
68.04
68.35
68.35
68.16
67.78
67.78
67.47
66.77
65.57
66.6
68.9
Appendix
Table S4: AVIOT across compression ratios with 2 FPS sampling and at most 224 frames. Scores are percentages, and TempCompass follows the multi-choice metric. The last two columns reproduce the published reference scores for LLaVA-Video-7B ( Zhang et al., 2025d ) and CoPE ( Sarkar et al., 2026 ) .
Dataset
r=12
r=14
r=15
r=16
r=18
r=20
Video-MME w/o sub.
60.64
60.12
59.96
59.36
60.16
60.00
Video-MMMU
36.44
36.89
36.22
36.89
36.56
36.78
MVBench
58.13
58.10
58.00
57.90
57.75
57.38
NExT-QA
81.71
81.90
81.80
81.66
81.48
81.26
Perception Test
69.17
69.00
68.86
68.77
68.48
68.30
Appendix
Table S5: AVIOT at inference-time compression ratios beyond the training range for the five benchmarks in the main-paper ratio analysis, using 2 FPS sampling and at most 224 frames. All operating points use the same checkpoint and target-cardinality construction as the main results.
Dataset
64 frames r=2
2 FPS, ≤ 224 r=2
64 frames r=4
2 FPS, ≤ 224 r=4
64 frames r=10
2 FPS, ≤ 224 r=10
ActivityNet-QA
68.22
68.54
67.06
68.23
65.67
66.26
Perception Test
71.50
71.84
70.72
71.45
69.41
69.80
MVBench
62.20
62.73
61.33
61.58
58.65
59.60
Video-MME w/o sub.
63.22
65.11
60.85
63.89
58.26
61.95
EgoSchema
57.86
58.04
56.33
58.14
53.61
56.47
Appendix
Table S6: AVIOT under fixed-64-frame and 2 FPS, at most 224-frame input sampling. Scores are reported at compression ratios 2, 4, and 10.
Task
r=2
r=4
r=10
Task
r=2
r=4
r=10
Action antonym
79.50
78.00
72.50
Fine-grained pose
58.50
57.00
49.50
Action count
59.50
60.00
58.50
Moving attribute
68.00
68.50
64.50
Action localization
63.50
60.50
57.50
Moving count
46.00
45.50
44.50
Action prediction
66.00
64.00
60.50
Moving direction
37.00
33.50
29.50
Action sequence
77.00
75.50
70.50
Object existence
58.00
58.50
57.50
Character order
79.00
80.00
72.50
Object interaction
82.00
79.50
80.00
Appendix
Table S7: AVIOT accuracy on the 20 MVBench tasks at three compression ratios.
Setting
K
Visual tokens
Retention
Prefill (T)
Uncompressed
64
13,440
100.0%
106.69
r=2
32
6,720
50.0%
44.28
r=4
16
3,360
25.0%
19.87
r=10
7
1,470
10.9%
8.14
Appendix
Table S8: Visual-token and decoder-prefill accounting for the controlled 64-frame setting. Prefill follows Eq. ( 37 ).
Figure S1: Controlled question-conditioned temporal allocation on a fixed video and support budget. Bar heights encode the support share assigned to each temporal segment at the first progressive stage; heatmap rows are the exact source-time distributions of ordered final target supports. The matched column supplies the displayed question to AVIOT, the rotated-query column supplies a different fixed question from the same video, and the no-query column disables all compressor-side question routes.
Figure S2: Exact provenance across spatial granularities for one source frame and final target support. Each branch map is normalized independently to expose its spatial pattern; the percentages report its contribution mass after adaptive fusion. Colored outlines in the rightmost panel mark the top-20% cells selected by the global, medium, and local paths, with concentric outlines indicating cross-path overlap.
Figure S3: Exact frame- and token-level provenance for a target support in a long first-person board-installation video. The temporal curve reports ωi over the displayed source-frame interval. The token-level views show the complete spatial mixture for the displayed source frame.
Figure S4: Exact provenance for a target support spanning the liquid-to-solid egg transition. The frame-level curve reports ωi over the displayed source-frame interval. The token-level views show its spatial construction within the displayed source frame.
Figure S5: Exact provenance for a target support representing fine-grained knife sharpening. The temporal curve reports ωi over the displayed source-frame interval. The token-level views preserve the geometry of the hands, knife, and sharpening tool within the displayed source frame.
As Video Large Language Models (Video-LLMs) scale to longer and more complex videos, their inference cost grows rapidly due to the large volume of visual tokens accumulated across frames. Training-free token compression has emerged as a practical solution to this bottleneck. However, existing temporal compression methods rely primarily on cross-frame token similarity or segmentation heuristics, overlooking each token's semantic role within its frame and failing to adapt compression strength to the compressibility of each frame pair. In this work, we propose OTT-Vid, a transport-derived allocation framework for temporal token compression. Our approach consists of two stages: spatial pruning identifies representative content within each frame, and optimal transport (OT) is then solved between neighboring frames to estimate temporal compressibility. We formulate this OT with non-uniform token mass, which protects semantically important tokens from aggressive compression, and a locality-aware cost that captures both feature and spatial disparities. The resulting transport plan jointly balances token importance and matching cost, while its total cost defines the transport difficulty of each frame pair, which we use to allocate compression budgets dynamically. Experiments on six benchmarks spanning video question answering and temporal grounding show that OTT-Vid preserves 95.8% of VQA and 73.9% of VTG performance while retaining only 10% of tokens, consistently outperforming existing state-of-the-art training-free compression methods.
Video Large Language Models (Video LLMs) achieve strong performance on video understanding tasks but suffer from high inference costs due to the large number of visual tokens. We study training-free, query-agnostic video token compression, where heuristic importance rules can over-represent large redundant visual groups and miss less frequent visual patterns, particularly under aggressive token budgets. We propose KiToke, a distribution-aware compression framework that treats visual tokens as samples from the video's spatiotemporal token distribution. KiToke uses global embedding density as a proxy for representational redundancy and inverse-density sampling to balance token allocation across visual patterns of different frequencies. It further aggregates unselected tokens into retained anchors within detected temporal intervals to reduce cross-transition mixing. Across evaluated backbones, KiToke achieves the highest average accuracy among compared training-free methods and remains robust down to 1% retention, where most evaluated baselines underperform uniform random token sampling. On LLaVA-OneVision, it preserves 98.5% and 89.2% of the uncompressed model's four-benchmark mean accuracy at 10% and 1% token retention, with 6.8x and 12.5x prefill speedups, respectively, including compression overhead.
Haifeng Huang, Biyin Xu, Yang Li
Department of Computer Science, Iowa State University.
Video Large Language Models (Video-LLMs) achieve strong performance in video understanding, but their excessive visual tokens bring substantial computational overhead. Existing training-free compression methods improve inference efficiency by reducing visual tokens, yet they often rely on local adjacent-frame similarity for temporal redundancy estimation or allocate token budgets mainly according to segment length. Such designs are sensitive to frame-level noise and fail to capture the non-uniform information distribution of real-world videos. To address these challenges, we propose InfoMerge, a training-free visual token compression method that improves token utilization through robust redundancy estimation and content-aware budget allocation. Specifically, we propose the Temporal Fingerprint Difference: a segment-level second-order temporal redundancy estimation strategy, which models the temporal similarity structure of tokens at the same spatial positions within each segment. We further introduce Content-Aware Budget Allocation (CABA), which dynamically allocates segment-level token budgets based on segment uniqueness and spectral-entropy-based representational richness. By reducing repeated preservation of redundant static regions and allocating more tokens to informative segments, InfoMerge makes better use of the limited token budget while maintaining strong performance. Extensive experiments show that InfoMerge achieves strong efficiency--accuracy trade-offs across multiple benchmarks and backbones, with more pronounced advantages under aggressive compression. On LLaVA-OneVision-7B, InfoMerge retains 98.8% of the original average performance while reducing 85% of visual tokens and achieving a 4.24-fold speedup in the prefill stage.
Xinxin Liu, Shiwei Gan, Xiao Liu +3
State Key Laboratory of Novel Software Technology, Nanjing University Nanjing 210023, China