Video Large Language Models (VLLMs) demonstrate strong video understanding but suffer from inefficiency due to redundant visual tokens. Existing pruning primary targets intra-frame spatial redundancy or prunes inside the LLM with shallow-layer overhead, yielding suboptimal spatiotemporal reduction and underutilizing long-context compressibility. All of them often discard subtle yet informative context from merged or pruned tokens. In this paper, we propose a new perspective that elaborates token Anchors within intra-frame and inter-frame to comprehensively aggregate the informative contexts via local-global Optimal Transport (AOT). Specifically, we first establish local- and global-aware token anchors within each frame under the attention guidance, which then optimal transport aggregates the informative contexts from pruned tokens, constructing intra-frame token anchors. Then, building on the temporal frame clips, the first frame within each clip will be considered as the keyframe anchors to ensemble similar information from consecutive frames through optimal transport, while keeping distinct tokens to represent temporal dynamics, leading to efficient token reduction in a training-free manner. Extensive evaluations show that our proposed AOT obtains competitive performances across various short- and long-video benchmarks on leading video LLMs, obtaining substantial computational efficiency while preserving temporal and visual fidelity. Project webpage: https://tyroneli.github.io/AOT.
Figures & tables
Figure 1: The top is the essential differences compared with common token reduction methods, instead of simply removing unimportant or merging very similar tokens, ours utilizes a global optimization strategy to further exploit and aggregate necessary semantic and context from these onto the remaining tokens. Bottom is our proposed pipeline to adopt Optimal Transport to aggregate information within intra- and inter-frame levels for video tokens.
Figure 2: Overall pipeline of our AOT . Our method compresses tokens of video LLMs across spatiotemporal through optimal transport, first establishing token anchors within each frame to cover semantically important and spatially diverse token candidates, then utilizing optimal transport to aggregate the necessary informative cues within Intra-Frame at phase I, and finally shifting the optimization strategy into temporal within Inter-Frame at phase II. The proposed AOT preserves both temporal and visual integrity by utilizing efficient Sinkhorn-Knopp Iteration to solve the optimal transport plan assignment.
Method
Prefilling FLOPs (T) ↓
FLOPs Ratio ↓
Before LLM Retained Ratio
MVBench ↑
EgoSchema ↑
LongVideo Bench ↑
VideoMME ↑
Avg. ↑
Score
%
LLaVA-OV-7B
40.8
100%
100%
58.3
60.4
56.4
58.6
58.4
100
FastV [ 8 ]
9.3
22.8%
100%
55.9
57.5
56.7
56.1
56.5
96.7
PDrop [ 64 ]
10.5
25.7%
100%
56.1
58.0
54.1
56.4
56.2
96.2
DyCoke [ 51 ]
8.7
21.3%
25%
53.1
59.5
49.5
54.3
54.1
92.6
VisionZip [ 68 ]
8.7
21.3%
25%
57.9
60.3
56.5
58.2
58.2
99.7
Table 1: Comparison of state-of-the-art methods on LLaVA-OneVision [ 22 ] across video benchmarks. The best performance among those with similar retention ratios is highlighted in bold , while the second best will be denoted as underlined .
Method
Prefilling FLOPs (T) ↓
FLOPs Ratio ↓
Before LLM Retained Ratio
MVBench ↑
EgoSchema ↑
LongVideo Bench ↑
VideoMME ↑
Avg. ↑
Score
%
LLaVA-Video-7B
80.2
100%
100%
60.4
57.2
58.9
64.3
60.2
100
FastV [ 8 ]
17.1
21.3%
100%
54.3
54.1
55.0
58.8
55.6
92.4
PDrop [ 64 ]
19.5
24.3%
100%
55.9
54.3
54.7
61.9
56.7
94.2
VisionZip [ 68 ]
9.3
18.9%
25%
56.7
54.7
54.7
60.7
56.7
94.2
DyCoke [ 68 ]
9.3
18.9%
25%
50.8
-
53.0
56.9
-
-
Table 2: Comparison of state-of-the-art methods on LLaVA-Video [ 80 ] across video benchmarks. The best performance among those is highlighted in bold , while the second best will be denoted as underlined , demonstrating consistent effectiveness.
Figure 3: Left: scaling with more frames leads to more efficient and effective visual information abstraction. Right: sensitivity analysis of weighting coefficient controlling contextual contribution with consistent configuration, λintra and λinter .
Iteration
Intra-Frame
Inter-Frame
Overall
%
50
0.50 ms
1.50 ms
2.00 ms
≤ 1%
100
0.51 ms
1.60 ms
2.11 ms
≤ 1%
Table 3: Computational overhead in terms of Intra- and Inter-Frame using Sinkhorn-Knopp Iteration (milliseconds).
Aggregation
MVBench
EgoSchema
LongVideo Bench
VideoMME
Avg. ↑
No Merging
56.1
60.2
53.5
55.8
56.4
Cosine Merging
51.5
55.8
51.1
51.3
52.4
Ours AOT
57.0
60.6
54.2
56.1
57.0
Table 4: Ablation: The effect of different aggregation strategy to obtain the final token anchors set to represent the video.
Method
MVBench
EgoSchema
LongVideo Bench
VideoMME
Avg.
Score ↑
% ↑
Vanilla
58.3
60.4
56.4
58.6
58.4
100
w/o Local Anchors
56.5
60.1
54.0
55.7
56.6
96.9
w/o Global Anchors
55.5
59.4
53.4
53.1
55.4
94.9
w/o OT
56.1
60.2
53.5
55.8
56.4
96.6
OT w/o Intra-frame
57.1
60.2
53.6
54.6
56.3
96.6
Table 5: Ablation: Contribution of each component by gradually removing the proposed component to demonstrate effectiveness.
Figure 4: Qualitative visualizations of our Local - Global token anchors evolution across consecutive frames while optimal transport is adopted to aggregate necessary information from unselected tokens to help LLM precess better.
Tinter(ℓ)∗←SinkhornOT(C(ℓ),λinter,Iter).
Algorithm 2 : Inter-Frame Inference of AOT with OT
Method
Prefilling FLOPs (T) ↓
FLOPs Ratio ↓
Before LLM Retained Ratio
MVBench ↑
EgoSchema ↑
LongVideo Bench ↑
VideoMME ↑
Avg. ↑
Score
%
LLaVA-OV-7B
40.8
100%
100%
58.3
60.4
56.4
58.6
58.4
100
FastV [ 8 ]
9.3
22.8%
100%
55.9
57.5
56.7
56.1
56.5
96.7
PDrop [ 64 ]
10.5
25.7%
100%
56.1
58.0
54.1
56.4
56.2
96.2
DyCoke [ 51 ]
8.7
21.3%
25%
53.1
59.5
49.5
54.3
54.1
92.6
VisionZip [ 68 ]
8.7
21.3%
25%
57.9
60.3
56.5
58.2
58.2
99.7
Table 6: Comparison of state-of-the-art methods on LLaVA-OneVision [ 22 ] across video benchmarks. The best performance among those with similar retention ratios is highlighted in bold , while the second best will be denoted as underlined . AOT w Dyn denotes we apply dynamic temporal segmentation to obtain adaptive frames within each clip, following FastVID [ 46 ] .
Method
Prefilling FLOPs (T) ↓
FLOPs Ratio ↓
Before LLM Retained Ratio
MVBench ↑
EgoSchema ↑
LongVideo Bench ↑
VideoMME ↑
Avg. ↑
Score
%
LLaVA-Video-7B
80.2
100%
100%
60.4
57.2
58.9
64.3
60.2
100
FastV [ 8 ]
17.1
21.3%
100%
54.3
54.1
55.0
58.8
55.6
92.4
PDrop [ 64 ]
19.5
24.3%
100%
55.9
54.3
54.7
61.9
56.7
94.2
VisionZip [ 68 ]
9.3
18.9%
25%
56.7
54.7
54.7
60.7
56.7
94.2
DyCoke [ 68 ]
9.3
18.9%
25%
50.8
-
53.0
56.9
-
-
Table 7: Comparison of state-of-the-art methods on LLaVA-Video [ 80 ] across video benchmarks. The best performance among those is highlighted in bold , while the second best will be denoted as underlined , demonstrating consistent effectiveness. AOT w Dyn denotes we apply dynamic temporal segmentation to obtain adaptive frames within each clip, following FastVID [ 46 ] .
Method
MVBench
EgoSchema
LongVideo Bench
VideoMME
Avg.
Score ↑
% ↑
Vanilla
58.3
60.4
56.4
58.6
58.4
100
w Random Anchors
55.1
59.3
52.6
53.3
55.1
94.3
AOT
57.0
60.6
54.2
56.1
57.0
97.6
Table 8: Ablation: the impact of random token selection.
Figure 5: Qualitative visualizations of our Local - Global token anchors evolution across consecutive frames on MVBench sample while optimal transport is adopted to aggregate necessary information from unselected tokens to help LLM precess better. The top is the original sampled frames while the bottom is the corresponding tokens visualization.
Figure 6: Qualitative visualizations of our Local - Global token anchors evolution across consecutive frames on VideoMME sample while optimal transport is adopted to aggregate necessary information from unselected tokens to help LLM precess better. The top is the original sampled frames while the bottom is the corresponding tokens visualization.
As Video Large Language Models (Video-LLMs) scale to longer and more complex videos, their inference cost grows rapidly due to the large volume of visual tokens accumulated across frames. Training-free token compression has emerged as a practical solution to this bottleneck. However, existing temporal compression methods rely primarily on cross-frame token similarity or segmentation heuristics, overlooking each token's semantic role within its frame and failing to adapt compression strength to the compressibility of each frame pair. In this work, we propose OTT-Vid, a transport-derived allocation framework for temporal token compression. Our approach consists of two stages: spatial pruning identifies representative content within each frame, and optimal transport (OT) is then solved between neighboring frames to estimate temporal compressibility. We formulate this OT with non-uniform token mass, which protects semantically important tokens from aggressive compression, and a locality-aware cost that captures both feature and spatial disparities. The resulting transport plan jointly balances token importance and matching cost, while its total cost defines the transport difficulty of each frame pair, which we use to allocate compression budgets dynamically. Experiments on six benchmarks spanning video question answering and temporal grounding show that OTT-Vid preserves 95.8% of VQA and 73.9% of VTG performance while retaining only 10% of tokens, consistently outperforming existing state-of-the-art training-free compression methods.
Video Large Language Models (Video LLMs) achieve strong performance on video understanding tasks but suffer from high inference costs due to the large number of visual tokens. We study training-free, query-agnostic video token compression, where heuristic importance rules can over-represent large redundant visual groups and miss less frequent visual patterns, particularly under aggressive token budgets. We propose KiToke, a distribution-aware compression framework that treats visual tokens as samples from the video's spatiotemporal token distribution. KiToke uses global embedding density as a proxy for representational redundancy and inverse-density sampling to balance token allocation across visual patterns of different frequencies. It further aggregates unselected tokens into retained anchors within detected temporal intervals to reduce cross-transition mixing. Across evaluated backbones, KiToke achieves the highest average accuracy among compared training-free methods and remains robust down to 1% retention, where most evaluated baselines underperform uniform random token sampling. On LLaVA-OneVision, it preserves 98.5% and 89.2% of the uncompressed model's four-benchmark mean accuracy at 10% and 1% token retention, with 6.8x and 12.5x prefill speedups, respectively, including compression overhead.
Haifeng Huang, Biyin Xu, Yang Li
Department of Computer Science, Iowa State University.
Processing long videos with Vision-Language Models (VLMs) is bottlenecked by the quadratic cost of visual tokens, making long-form inference prohibitively expensive. While single-axis compression methods mitigate this, they hit a hard efficiency floor because they treat spatial and temporal redundancy independently. We present VETO (Video Efficient Token Optimization for Vision-Language Models), a training-optional plug-in that eliminates this bottleneck through dual-axis compression: (i) an intra-frame compressor that merges semantically similar tokens within each frame via optimal-transport inspired matching, and (ii) an inter-frame compressor that identifies and merges temporally redundant frames. The key design insight is hierarchical ordering: by first compressing spatial dimensions, VETO drastically reduces the cost of subsequent global temporal matching, bypassing the efficiency wall of single-axis approaches, with an advantage that grows with modern fully-fused attention infrastructure. Empirically, VETO achieves up to 45% faster inference (e.g., on LLaVA-OneVision-7B) while preserving or improving accuracy. Under extreme token starvation (10% budget), VETO outperforms VFlowOpt (54.9%), VisionZip (52.6%), and FastV (47.9%) with 55.7% accuracy. We demonstrate universal applicability across LLaVA-OneVision, InternVL-2.5, and LongVA, with zero-shot accuracy preserved or improved in all cases.