VETO: Video Efficient Token Optimization for Vision Language Models

Gueter Josmy Faure; Hao Ping Wang; Min-Hung Chen; Winston H. Hsu

cs.CVOct 1, 2026

VETO: Video Efficient Token Optimization for Vision Language Models

Authors: Gueter Josmy Faure, Hao Ping Wang, Min-Hung Chen, Winston H. Hsu

Organizations: National Taiwan University · NVIDIA

Abstract

Processing long videos with Vision-Language Models (VLMs) is bottlenecked by the quadratic cost of visual tokens, making long-form inference prohibitively expensive. While single-axis compression methods mitigate this, they hit a hard efficiency floor because they treat spatial and temporal redundancy independently. We present VETO (Video Efficient Token Optimization for Vision-Language Models), a training-optional plug-in that eliminates this bottleneck through dual-axis compression: (i) an intra-frame compressor that merges semantically similar tokens within each frame via optimal-transport inspired matching, and (ii) an inter-frame compressor that identifies and merges temporally redundant frames. The key design insight is hierarchical ordering: by first compressing spatial dimensions, VETO drastically reduces the cost of subsequent global temporal matching, bypassing the efficiency wall of single-axis approaches, with an advantage that grows with modern fully-fused attention infrastructure. Empirically, VETO achieves up to 45% faster inference (e.g., on LLaVA-OneVision-7B) while preserving or improving accuracy. Under extreme token starvation (10% budget), VETO outperforms VFlowOpt (54.9%), VisionZip (52.6%), and FastV (47.9%) with 55.7% accuracy. We demonstrate universal applicability across LLaVA-OneVision, InternVL-2.5, and LongVA, with zero-shot accuracy preserved or improved in all cases.

Figures & tables

Figure 2 : VETO architecture. Visual encoder features are first compressed by the Intra-Frame Compressor (semantic token merging within each frame) and then by the Inter-Frame Compressor (frame-level merging). The compressed representation maintains feature dimensionality and is fed to the frozen LLM without architectural changes.

Figure 3 : The VETO dual-axis pipeline and dimensionality reduction. First (Intra-Frame): Optimal-transport-inspired matching reduces spatial token count from $N$ to $N/k$ , preserving salient semantics while shrinking the feature tensor. Second (Inter-Frame): Global frame-level redundancy is eliminated. Because the temporal similarity matrix is computed on the already spatially compressed dimension ( $N/k$ ), computational overhead is minimized.

Table 1: Performance across benchmarks. All latency normalized to Vanilla

=1.00

(

\downarrow

is better).

\dagger

Latency on H100+FlashAttention-2.

\ddagger

Latency on V100; full H100 comparison in supplementary.

\star

VideoMME accuracy from unified re-evaluation [ 32 ] : FastV 55.9%, VisionZip 57.5%, VFlowOpt 58.0%—all degrade from the 58.9% Vanilla baseline.

Table 2: Hyperparameter settings.

k

: intra-frame factor (retains

1/k

spatial tokens per frame);

G

: frame retention fraction. Abbrevs: VME = VideoMME; LVB = LongVideoBench; ES = EgoSchema; MC = MovieChat-1K; BF = BreakFast.

Table 3: (a) MovieChat-1K long-form narrative understanding (all baselines 7B, zero-shot). (b) Procedural action recognition on BreakFast (BF) and COIN (Fully-supervised).

\dagger

Methods using the identical InstructBLIP (with vicuna-7B) backbone as VETO.

Table 4 : Ablations (LLaVA-OV 7B, V100 lat.). (a) Token merging strategy, revealing the latency floor of single-axis methods. (b) Intra-frame compression ratios. (c) Inter-frame compression ratios. (d) Token starvation (VideoMME) vs. single-axis baselines.

Figure 4 : Qualitative visualizations of VETO’s dual-axis compression. Left: The inter-frame compressor retains scene-transition keyframes and discards redundant frames (7 $\to$ 3). Right: The intra-frame compressor; patches colored by merge group. Semantically similar background regions (sky, grass, textures) are merged while distinct foreground objects are preserved.

Operation	Seq. length	Rel. FLOPs
Vanilla attention	$46{,}656$	$1.000\times$
VETO intra (total)	—	$0.007\times$
VETO inter	—	${\ll}0.001\times$
VETO attention	$17{,}520$	$\mathbf{0.141\times}$

Table 5: Operation counts at

T=64

N=729

k=2

G=0.75

. Compression overhead is one to two orders of magnitude smaller than the attention saving it enables.

Method	Acc.	Lat.
InternVL-2.5 (1B)
Vanilla	39.8	1.00
VETO (Ours)	39.1	0.77

Table 6: EgoSchema on InternVL-2.5-1B (V100 lat.). Accuracy (%) and normalized latency (

\downarrow

better).

Method	Acc. (%)
TS ConvNets [ 23 ]	88.0
C3D [ 26 ]	82.3
VideoMAE [ 25 ]	91.3
TSN [ 28 ]	94.2
TSM [ 15 ]	95.9
VETO (Ours)	94.4

Table 7: UCF-101 action recognition. VETO uses the same frozen-backbone / linear-adapter setup as the main paper’s supervised experiments.

Explore similar work

TTF: Temporal Token Fusion for Efficient Video-Language Model
May 8, 2026Simin Huo, Ning LIVideo-Language Models Qwen3
Shanghai Jiao Tong University
CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models
Aug 3, 2026Yu Chen, Xiaohong Li, Xiaole Wang +3Token Compression Video-Language Models
1Evermind · 2Shanda Group · 3Peking University
InfoMerge: Information-aware Token Compression for Efficient Video Large Language Models
Jun 1, 2026Xinxin Liu, Shiwei Gan, Xiao Liu +3Token Compression Video-Language Models
State Key Laboratory of Novel Software Technology, Nanjing University Nanjing 210023, China

May 8, 2026, cs.CVJ/K move · Enter open · S save

TTF: Temporal Token Fusion for Efficient Video-Language Model

Simin Huo, Ning LI

Shanghai Jiao Tong University

Video-Language Models Qwen3

Open paper AskPDF arXiv

Video-language models (VLMs) face rapid inference costs as visual token counts scale with video length. For example, 32 frames at

448{\times}448

resolution already yield >8,000 visual tokens in Qwen3-VL, making LLM prefill the dominant throughput bottleneck. Existing methods often rely on global similarity or attention-guided compression, incurring offsets to their gains. We propose \textbf{Temporal Token Fusion (TTF)}, a training-free, plug-and-play pre-LLM token compression framework that exploits structured temporal redundancy in video. TTF automatically selects an anchor frame, then for each subsequent frame, performs a local window similarity search (e.g.,

3\times 3

), fusing tokens that exceed a threshold. The compressed sequence maintains positional consistency across both prefill and decoding through coordinate realignment, enabling seamless integration with existing VLM pipelines. On Qwen3-VL-8B with threshold t=0.70, TTF removes about 67% of visual tokens while retaining 99.5% of the baseline accuracy and introducing only

{\approx}0.16

,GFLOPs of matching overhead. Overall, TTF offers a practical, efficient solution for video understanding. The code is available at \href{https://github.com/Cominder/ttf}{https://github.com/Cominder/ttf}

Abstract

Explore similar work

TTF: Temporal Token Fusion for Efficient Video-Language Model

CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models

InfoMerge: Information-aware Token Compression for Efficient Video Large Language Models