Video Large Language Models (VLLMs) demonstrate strong video understanding but suffer from inefficiency due to redundant visual tokens. Existing pruning primary targets intra-frame spatial redundancy or prunes inside the LLM with shallow-layer overhead, yielding suboptimal spatiotemporal reduction and underutilizing long-context compressibility. All of them often discard subtle yet informative context from merged or pruned tokens. In this paper, we propose a new perspective that elaborates token Anchors within intra-frame and inter-frame to comprehensively aggregate the informative contexts via local-global Optimal Transport (AOT). Specifically, we first establish local- and global-aware token anchors within each frame under the attention guidance, which then optimal transport aggregates the informative contexts from pruned tokens, constructing intra-frame token anchors. Then, building on the temporal frame clips, the first frame within each clip will be considered as the keyframe anchors to ensemble similar information from consecutive frames through optimal transport, while keeping distinct tokens to represent temporal dynamics, leading to efficient token reduction in a training-free manner. Extensive evaluations show that our proposed AOT obtains competitive performances across various short- and long-video benchmarks on leading video LLMs, obtaining substantial computational efficiency while preserving temporal and visual fidelity. Project webpage: https://tyroneli.github.io/AOT.
Figures & tables
Figure 1: The top is the essential differences compared with common token reduction methods, instead of simply removing unimportant or merging very similar tokens, ours utilizes a global optimization strategy to further exploit and aggregate necessary semantic and context from these onto the remaining tokens. Bottom is our proposed pipeline to adopt Optimal Transport to aggregate information within intra- and inter-frame levels for video tokens.
Figure 2: Overall pipeline of our AOT . Our method compresses tokens of video LLMs across spatiotemporal through optimal transport, first establishing token anchors within each frame to cover semantically important and spatially diverse token candidates, then utilizing optimal transport to aggregate the necessary informative cues within Intra-Frame at phase I, and finally shifting the optimization strategy into temporal within Inter-Frame at phase II. The proposed AOT preserves both temporal and visual integrity by utilizing efficient Sinkhorn-Knopp Iteration to solve the optimal transport plan assignment.
Method
Prefilling FLOPs (T) ↓
FLOPs Ratio ↓
Before LLM Retained Ratio
MVBench ↑
EgoSchema ↑
LongVideo Bench ↑
VideoMME ↑
Avg. ↑
Score
%
LLaVA-OV-7B
40.8
100%
100%
58.3
60.4
56.4
58.6
58.4
100
FastV [ 8 ]
9.3
22.8%
100%
55.9
57.5
56.7
56.1
56.5
96.7
PDrop [ 64 ]
10.5
25.7%
100%
56.1
58.0
54.1
56.4
56.2
96.2
DyCoke [ 51 ]
8.7
21.3%
25%
53.1
59.5
49.5
54.3
54.1
92.6
VisionZip [ 68 ]
8.7
21.3%
25%
57.9
60.3
56.5
58.2
58.2
99.7
Table 1: Comparison of state-of-the-art methods on LLaVA-OneVision [ 22 ] across video benchmarks. The best performance among those with similar retention ratios is highlighted in bold , while the second best will be denoted as underlined .
Method
Prefilling FLOPs (T) ↓
FLOPs Ratio ↓
Before LLM Retained Ratio
MVBench ↑
EgoSchema ↑
LongVideo Bench ↑
VideoMME ↑
Avg. ↑
Score
%
LLaVA-Video-7B
80.2
100%
100%
60.4
57.2
58.9
64.3
60.2
100
FastV [ 8 ]
17.1
21.3%
100%
54.3
54.1
55.0
58.8
55.6
92.4
PDrop [ 64 ]
19.5
24.3%
100%
55.9
54.3
54.7
61.9
56.7
94.2
VisionZip [ 68 ]
9.3
18.9%
25%
56.7
54.7
54.7
60.7
56.7
94.2
DyCoke [ 68 ]
9.3
18.9%
25%
50.8
-
53.0
56.9
-
-
Table 2: Comparison of state-of-the-art methods on LLaVA-Video [ 80 ] across video benchmarks. The best performance among those is highlighted in bold , while the second best will be denoted as underlined , demonstrating consistent effectiveness.
Figure 3: Left: scaling with more frames leads to more efficient and effective visual information abstraction. Right: sensitivity analysis of weighting coefficient controlling contextual contribution with consistent configuration, λintra and λinter .
Iteration
Intra-Frame
Inter-Frame
Overall
%
50
0.50 ms
1.50 ms
2.00 ms
≤ 1%
100
0.51 ms
1.60 ms
2.11 ms
≤ 1%
Table 3: Computational overhead in terms of Intra- and Inter-Frame using Sinkhorn-Knopp Iteration (milliseconds).
Aggregation
MVBench
EgoSchema
LongVideo Bench
VideoMME
Avg. ↑
No Merging
56.1
60.2
53.5
55.8
56.4
Cosine Merging
51.5
55.8
51.1
51.3
52.4
Ours AOT
57.0
60.6
54.2
56.1
57.0
Table 4: Ablation: The effect of different aggregation strategy to obtain the final token anchors set to represent the video.
Method
MVBench
EgoSchema
LongVideo Bench
VideoMME
Avg.
Score ↑
% ↑
Vanilla
58.3
60.4
56.4
58.6
58.4
100
w/o Local Anchors
56.5
60.1
54.0
55.7
56.6
96.9
w/o Global Anchors
55.5
59.4
53.4
53.1
55.4
94.9
w/o OT
56.1
60.2
53.5
55.8
56.4
96.6
OT w/o Intra-frame
57.1
60.2
53.6
54.6
56.3
96.6
Table 5: Ablation: Contribution of each component by gradually removing the proposed component to demonstrate effectiveness.
Figure 4: Qualitative visualizations of our Local - Global token anchors evolution across consecutive frames while optimal transport is adopted to aggregate necessary information from unselected tokens to help LLM precess better.
Tinter(ℓ)∗←SinkhornOT(C(ℓ),λinter,Iter).
Algorithm 2 : Inter-Frame Inference of AOT with OT
Method
Prefilling FLOPs (T) ↓
FLOPs Ratio ↓
Before LLM Retained Ratio
MVBench ↑
EgoSchema ↑
LongVideo Bench ↑
VideoMME ↑
Avg. ↑
Score
%
LLaVA-OV-7B
40.8
100%
100%
58.3
60.4
56.4
58.6
58.4
100
FastV [ 8 ]
9.3
22.8%
100%
55.9
57.5
56.7
56.1
56.5
96.7
PDrop [ 64 ]
10.5
25.7%
100%
56.1
58.0
54.1
56.4
56.2
96.2
DyCoke [ 51 ]
8.7
21.3%
25%
53.1
59.5
49.5
54.3
54.1
92.6
VisionZip [ 68 ]
8.7
21.3%
25%
57.9
60.3
56.5
58.2
58.2
99.7
Table 6: Comparison of state-of-the-art methods on LLaVA-OneVision [ 22 ] across video benchmarks. The best performance among those with similar retention ratios is highlighted in bold , while the second best will be denoted as underlined . AOT w Dyn denotes we apply dynamic temporal segmentation to obtain adaptive frames within each clip, following FastVID [ 46 ] .
Method
Prefilling FLOPs (T) ↓
FLOPs Ratio ↓
Before LLM Retained Ratio
MVBench ↑
EgoSchema ↑
LongVideo Bench ↑
VideoMME ↑
Avg. ↑
Score
%
LLaVA-Video-7B
80.2
100%
100%
60.4
57.2
58.9
64.3
60.2
100
FastV [ 8 ]
17.1
21.3%
100%
54.3
54.1
55.0
58.8
55.6
92.4
PDrop [ 64 ]
19.5
24.3%
100%
55.9
54.3
54.7
61.9
56.7
94.2
VisionZip [ 68 ]
9.3
18.9%
25%
56.7
54.7
54.7
60.7
56.7
94.2
DyCoke [ 68 ]
9.3
18.9%
25%
50.8
-
53.0
56.9
-
-
Table 7: Comparison of state-of-the-art methods on LLaVA-Video [ 80 ] across video benchmarks. The best performance among those is highlighted in bold , while the second best will be denoted as underlined , demonstrating consistent effectiveness. AOT w Dyn denotes we apply dynamic temporal segmentation to obtain adaptive frames within each clip, following FastVID [ 46 ] .
Method
MVBench
EgoSchema
LongVideo Bench
VideoMME
Avg.
Score ↑
% ↑
Vanilla
58.3
60.4
56.4
58.6
58.4
100
w Random Anchors
55.1
59.3
52.6
53.3
55.1
94.3
AOT
57.0
60.6
54.2
56.1
57.0
97.6
Table 8: Ablation: the impact of random token selection.
Figure 5: Qualitative visualizations of our Local - Global token anchors evolution across consecutive frames on MVBench sample while optimal transport is adopted to aggregate necessary information from unselected tokens to help LLM precess better. The top is the original sampled frames while the bottom is the corresponding tokens visualization.
Figure 6: Qualitative visualizations of our Local - Global token anchors evolution across consecutive frames on VideoMME sample while optimal transport is adopted to aggregate necessary information from unselected tokens to help LLM precess better. The top is the original sampled frames while the bottom is the corresponding tokens visualization.