Aggregating Visual Information with Optimal Transport for VideoLM Token Compression

Wenti Yin; Xiaotian Han; Junyuan Shang; Yuchen Ding; Shuohuan Wang; Dianhai Yu; Changxin Gao; Nong Sang

cs.CVAug 20, 2026

Aggregating Visual Information with Optimal Transport for VideoLM Token Compression

Authors: Wenti Yin, Xiaotian Han, Junyuan Shang, Yuchen Ding, Shuohuan Wang, Dianhai Yu, Changxin Gao, Nong Sang

Organizations: Key Laboratory of Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology · Baidu, Inc

Abstract

Video language models process videos as dense visual-token sequences with substantial representational redundancy. Compressing these sequences is therefore essential for reducing the visual-token burden on language-model decoding. The central challenge is to preserve visual information dispersed across frames under such compression. To this end, we introduce Aggregating Visual Information with Optimal Transport (AVIOT), which casts video token compression as transporting a dense empirical measure of frame observations onto a compact target measure. The resulting source-to-target coupling induces a distribution over source observations for each target support, directly specifying how the compressed video representation is constructed. We further adapt this construction along task and spatial axes. Question conditioning modulates the transport cost between source frames and target supports, while influencing how many supports are allocated to each temporal segment, thereby directing representation capacity toward question-relevant content. At multiple spatial granularities, AVIOT computes region-specific temporal transport plans and adaptively fuses the representations they yield, allowing different regions within the same compact representation to draw from different moments. Evaluations across varying compression ratios show that AVIOT matches or outperforms the uncompressed baseline on multiple video-understanding benchmarks while retaining strong performance at higher compression ratios.

Figures & tables

Figure 1: Motivation for AVIOT. Common compression paradigms reduce source observations through selection or pruning, or synthesize compact outputs through learned feature mapping. AVIOT constructs the output set through a joint source-to-target optimal-transport coupling, organizing how information distributed across frames is represented collectively by the compact supports.

Figure 2: Overview of the proposed AVIOT framework. Dense video features are augmented with time–height–width (THW) positional encodings and progressively compressed to the requested cardinality. At each stage, the question modulates the transport cost and, together with preliminary transport responses, guides support allocation across temporal segments. Within each segment, source-to-target couplings construct aligned representations at multiple spatial granularities through barycentric aggregation; these representations are adaptively fused before entering the language model.

Method	Token ret.	Prefill FLOPs (T)	Acti- vity Net- QA	Per- cep- tion Test	MV Bench	Video- MME w/o sub.	Ego Sche- ma
LLaVA- Video-7B	100.0%	106.69	64.10	67.90	58.60	63.30	57.30
CoPE- VideoLM	varying	–	64.80 92%	70.30 20%	61.90 16%	61.90 289%	–
Uniform Keep	50.0%	44.28	67.01	68.34	60.75	62.11	57.21
	25.0%	19.87	65.79	67.01	58.40	60.30	54.98
	10.9%	8.14	63.42	64.46	55.12	56.26	51.38
Segment Mean	50.0%	44.28	66.40	67.74	60.48	61.22	55.34

Table 1: Controlled comparison with a 64-frame input at compression ratios 2, 4, and 10. The second column reports token retention; bold denotes the best score in each benchmark column.

Model	Video- MME w/o sub.	Video- MMMU	ActivityNet- QA	NExT-QA	Perception Test	Temp Compass	LongVideoBench	LVBench	Ego Schema	MVBench
LongVA ( 2025a )	52.6	23.9	50.0	68.3	–	56.9	–	–	–	–
IXC-2.5 ( 2024 )	55.8	–	52.8	71.0	34.4	67.1	–	–	–	69.1
LLaVA-OV ( 2025a )	58.2	33.9	56.6	79.4	57.1	64.8	56.4	38.1	60.1	56.7
Apollo ( 2025 )	61.3	–	–	–	67.3	64.9	58.5	–	–	–
Oryx ( 2025 )	58.3	–	–	81.9	68.6	–	55.3	–	–	63.9
LLaVA-Video ( 2025d )	63.3	36.1	64.1	83.2	67.9	66.6	58.2	44.2	57.3	58.6

Table 2: Open-source 7B VideoLM comparison. IXC-2.5 is InternLM-XComposer-2.5; bold/underline: best/second best.

Question conditioning
Configuration	Score
Full AVIOT	65.11
w/o question-conditioned transport cost	64.85
w/o question-conditioned temporal allocation	64.59
w/o both conditioning paths	64.67
Transport across spatial granularities

Table 3: Component analysis of AVIOT.

Figure 3: Generalization across compression ratios.

Figure 4: Question-conditioned temporal allocation and provenance across spatial granularities. (a) Support allocations for two questions. (b) Global, medium, and local provenance for one source frame and target support; percentages denote fused mass, and boxes mark the top 20% positions.

Figure 5: Hierarchical source-frame and spatial provenance of selected target supports.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Setting	Value
Representation	ViT feature dim.	1152
	Patch layout	$27\times 27$
	Cost embedding dim.	256
	Temporal segments	4
Transport	$\varepsilon$	0.10
	$\rho_{s},\rho_{t}$	0.5, 5.0

Appendix

Table S1: Final AVIOT architecture and numerical configuration.

Group	Setting	Value
Initialization	VideoLM	LLaVA-Video-7B-Qwen2
	Visual encoder	SigLIP-SO400M-patch14-384
Data	Training set	LLaVA-Video-178K
Optimization	Steps	10,903
	Global batch size	128
	Learning rate	$1\times 10^{-5}$

Appendix

Table S2: Optimization details for AVIOT training.

Method	Ratio	Multi-choice	Yes/no	Caption matching	Captioning	Task avg.
Uniform Keep	2	67.15	68.94	75.45	53.29	66.21
Uniform Keep	4	65.44	67.88	74.78	51.85	64.99
Uniform Keep	10	63.54	65.80	72.46	12.33	53.53
Segment Mean	2	66.52	68.98	75.32	53.54	66.09
Segment Mean	4	64.81	66.82	73.72	51.50	64.21
Segment Mean	10	60.32	63.72	68.80	14.07	51.73

Appendix

Table S3: Extended TempCompass results across four task types. Multi-choice follows the metric used in the main comparison; Task avg. is the unweighted mean of the four task accuracies. Bold marks the best compressed method at each ratio.

Dataset	$r=2$	$r=3.5$	$r=4$	$r=5$	$r=5.5$	$r=6$	$r=6.5$	$r=7$	$r=8$	$r=10$	LLaVA- Video-7B	CoPE
Video-MME w/o sub.	65.11	64.70	63.89	63.66	63.13	63.84	63.98	62.45	62.40	61.95	63.3	61.9
Video-MMMU	38.22	39.00	39.78	38.33	38.00	38.78	38.78	37.78	38.33	37.89	36.1	38.2
ActivityNet-QA	68.54	68.05	68.23	67.40	67.39	67.49	67.14	67.11	67.09	66.26	64.1	64.8
NExT-QA	84.01	83.51	83.49	83.48	83.30	83.03	82.85	82.60	82.44	82.11	83.2	82.1
Perception Test	71.84	71.48	71.45	71.03	70.86	70.57	70.52	70.48	70.27	69.80	67.9	70.3
TempCompass	69.56	68.04	68.35	68.35	68.16	67.78	67.78	67.47	66.77	65.57	66.6	68.9

Appendix

Table S4: AVIOT across compression ratios with 2 FPS sampling and at most 224 frames. Scores are percentages, and TempCompass follows the multi-choice metric. The last two columns reproduce the published reference scores for LLaVA-Video-7B ( Zhang et al., 2025d ) and CoPE ( Sarkar et al., 2026 ) .

Dataset	$r=12$	$r=14$	$r=15$	$r=16$	$r=18$	$r=20$
Video-MME w/o sub.	60.64	60.12	59.96	59.36	60.16	60.00
Video-MMMU	36.44	36.89	36.22	36.89	36.56	36.78
MVBench	58.13	58.10	58.00	57.90	57.75	57.38
NExT-QA	81.71	81.90	81.80	81.66	81.48	81.26
Perception Test	69.17	69.00	68.86	68.77	68.48	68.30

Appendix

Table S5: AVIOT at inference-time compression ratios beyond the training range for the five benchmarks in the main-paper ratio analysis, using 2 FPS sampling and at most 224 frames. All operating points use the same checkpoint and target-cardinality construction as the main results.

Dataset	64 frames $r=2$	2 FPS, $\leq$ 224 $r=2$	64 frames $r=4$	2 FPS, $\leq$ 224 $r=4$	64 frames $r=10$	2 FPS, $\leq$ 224 $r=10$
ActivityNet-QA	68.22	68.54	67.06	68.23	65.67	66.26
Perception Test	71.50	71.84	70.72	71.45	69.41	69.80
MVBench	62.20	62.73	61.33	61.58	58.65	59.60
Video-MME w/o sub.	63.22	65.11	60.85	63.89	58.26	61.95
EgoSchema	57.86	58.04	56.33	58.14	53.61	56.47

Appendix

Table S6: AVIOT under fixed-64-frame and 2 FPS, at most 224-frame input sampling. Scores are reported at compression ratios 2, 4, and 10.

Task	$r=2$	$r=4$	$r=10$	Task	$r=2$	$r=4$	$r=10$
Action antonym	79.50	78.00	72.50	Fine-grained pose	58.50	57.00	49.50
Action count	59.50	60.00	58.50	Moving attribute	68.00	68.50	64.50
Action localization	63.50	60.50	57.50	Moving count	46.00	45.50	44.50
Action prediction	66.00	64.00	60.50	Moving direction	37.00	33.50	29.50
Action sequence	77.00	75.50	70.50	Object existence	58.00	58.50	57.50
Character order	79.00	80.00	72.50	Object interaction	82.00	79.50	80.00

Appendix

Table S7: AVIOT accuracy on the 20 MVBench tasks at three compression ratios.

Setting	$K$	Visual tokens	Retention	Prefill (T)
Uncompressed	64	13,440	100.0%	106.69
$r=2$	32	6,720	50.0%	44.28
$r=4$	16	3,360	25.0%	19.87
$r=10$	7	1,470	10.9%	8.14

Appendix

Table S8: Visual-token and decoder-prefill accounting for the controlled 64-frame setting. Prefill follows Eq. ( 37 ).

Figure S1: Controlled question-conditioned temporal allocation on a fixed video and support budget. Bar heights encode the support share assigned to each temporal segment at the first progressive stage; heatmap rows are the exact source-time distributions of ordered final target supports. The matched column supplies the displayed question to AVIOT, the rotated-query column supplies a different fixed question from the same video, and the no-query column disables all compressor-side question routes.

Figure S2: Exact provenance across spatial granularities for one source frame and final target support. Each branch map is normalized independently to expose its spatial pattern; the percentages report its contribution mass after adaptive fusion. Colored outlines in the rightmost panel mark the top-20% cells selected by the global, medium, and local paths, with concentric outlines indicating cross-path overlap.

Figure S3: Exact frame- and token-level provenance for a target support in a long first-person board-installation video. The temporal curve reports $\omega_{i}$ over the displayed source-frame interval. The token-level views show the complete spatial mixture for the displayed source frame.

Figure S4: Exact provenance for a target support spanning the liquid-to-solid egg transition. The frame-level curve reports $\omega_{i}$ over the displayed source-frame interval. The token-level views show its spatial construction within the displayed source frame.

Figure S5: Exact provenance for a target support representing fine-grained knife sharpening. The temporal curve reports $\omega_{i}$ over the displayed source-frame interval. The token-level views preserve the geometry of the hands, knife, and sharpening tool within the displayed source frame.

Video Token Compression

Video token compression reduces redundant spatial and temporal video representations to make understanding and generation more efficient.

See all 82 papers

OTT-Vid: Optimal Transport Temporal Token Compression for Video Large Language Models
May 12, 2026Minseok Kang, Minhyeok Lee, Jungho Lee +6Efficient VLM Inference Video-Language Models
Yonsei University · LG Electronics · KIST
KiToke: Distribution-Aware Token Compression for Video Large Language Models
Apr 3, 2026Haifeng Huang, Biyin Xu, Yang LiVideo Token Compression Visual Token Compression
Department of Computer Science, Iowa State University.
InfoMerge: Information-aware Token Compression for Efficient Video Large Language Models
Jun 1, 2026Xinxin Liu, Shiwei Gan, Xiao Liu +3Efficient VLM Inference Video-Language Models
State Key Laboratory of Novel Software Technology, Nanjing University Nanjing 210023, China

May 12, 2026J/K move · Enter open · S save

OTT-Vid: Optimal Transport Temporal Token Compression for Video Large Language Models

Minseok Kang, Minhyeok Lee, Jungho Lee, Minjung Kim+5

Yonsei University · LG Electronics · KIST

Efficient VLM Inference Video-Language Models Video Token Compression Optimal Transport Visual Token Compression

Open paper AskPDF arXiv

As Video Large Language Models (Video-LLMs) scale to longer and more complex videos, their inference cost grows rapidly due to the large volume of visual tokens accumulated across frames. Training-free token compression has emerged as a practical solution to this bottleneck. However, existing temporal compression methods rely primarily on cross-frame token similarity or segmentation heuristics, overlooking each token's semantic role within its frame and failing to adapt compression strength to the compressibility of each frame pair. In this work, we propose OTT-Vid, a transport-derived allocation framework for temporal token compression. Our approach consists of two stages: spatial pruning identifies representative content within each frame, and optimal transport (OT) is then solved between neighboring frames to estimate temporal compressibility. We formulate this OT with non-uniform token mass, which protects semantically important tokens from aggressive compression, and a locality-aware cost that captures both feature and spatial disparities. The resulting transport plan jointly balances token importance and matching cost, while its total cost defines the transport difficulty of each frame pair, which we use to allocate compression budgets dynamically. Experiments on six benchmarks spanning video question answering and temporal grounding show that OTT-Vid preserves 95.8% of VQA and 73.9% of VTG performance while retaining only 10% of tokens, consistently outperforming existing state-of-the-art training-free compression methods.

Abstract

Video Token Compression

Explore similar work

OTT-Vid: Optimal Transport Temporal Token Compression for Video Large Language Models

KiToke: Distribution-Aware Token Compression for Video Large Language Models

InfoMerge: Information-aware Token Compression for Efficient Video Large Language Models