Organizations: Sun Yat-sen University Guangzhou, Guangdong, China · The University of British Columbia Vancouver, British Columbia, Canada · Vast Intelligence Lab Sydney, New South Wales, Australia
Adapting video vision-language models (VLMs) is computationally expensive because video inputs produce a large number of visual tokens, making both fine-tuning and inference costly. Although visual token compression can reduce this overhead, direct adaptation on compressed inputs often causes semantic drift and noticeable performance degradation. We present Token-Budget Distillation (TBD), a parameter-efficient fine-tuning framework for adapting video VLMs under a fixed token budget. TBD freezes the pretrained backbone, updates only LoRA adapters, and integrates FlashVID-based visual token compression into the video pathway. To preserve full-token semantics under compression, TBD employs a dual-path teacher-student design, where a full-token teacher provides stable supervision and a compressed student is optimized with task loss, answer-region KL distillation, GT-anchored margin distillation, and reliability-aware KD control. This design enables the student to recover the semantic behavior of the full-token model while remaining efficient under aggressive token reduction. We evaluate TBD on three video VLM backbones, including LLaVA-Video, LLaVA-OneVision, and Qwen3-VL-8B-Instruct, across four video understanding benchmarks. TBD consistently outperforms compression-only baselines under both moderate and aggressive compression. On LLaVA-Video at retention ratio R = 10 percent, TBD preserves 97.0 percent of the Vanilla model's average accuracy; on LLaVA-OneVision at R = 10 percent, it achieves an average score of 58.4 and matches 100.0 percent relative accuracy.
Figures & tables
Figure 1. Overview of the Token-Budget Distillation (TBD) framework. A full-token teacher branch provides reliable guidance, while a compressed student branch with LoRA is optimized under a fixed token budget. Block diagram of the TBD training framework. A video and a question are processed by two branches sharing one frozen backbone: a teacher branch that consumes the full visual token sequence, and a student branch in which FlashVID compresses the tokens before the language model while LoRA adapters are trained. The output logits of the two branches are compared to compute the task cross-entropy loss, the answer-region KL distillation loss, and the GT-anchored margin loss, and only the LoRA parameters receive gradient updates.
Method
Retention Ratio R
VideoMME
EgoSchema
LongVideo Bench
MVBench
Avg.
Short
Medium
Long
Overall
Subset
Total
Score
Rel. Acc (%)
LLaVA-Video
Vanilla
100%
77.0
62.1
53.3
64.2
59.4
57.3
59.5
61.9
60.7
100.0
FastV
20%
69.3
58.3
49.9
59.2
54.8
54.1
56.0
58.4
56.9
93.7
VisionZip
72.3
59.6
53.3
61.7
59.0
56.4
58.0
59.8
59.0
97.2
FastVID
74.6
60.8
52.3
62.6
57.0
55.0
57.1
60.2
58.7
96.7
Table 1. Main results on four video understanding benchmarks under different token retention ratios. We compare TBD with the uncompressed Vanilla model and representative token reduction baselines on three backbones: LLaVA-Video , LLaVA-OneVision , and Qwen3-VL-8B-Instruct . “Avg. Score” denotes the average performance across benchmarks, and “Rel. Acc (%)” reports the average score normalized by the corresponding Vanilla model.
Variant
LoRA
Answer-region KL
GT-anchored Margin
Filtering & KD Cap
Score
Rel. Acc (%)
LoRA-only Student
✓
✗
✗
✗
58.3
96.0
+ Answer-region KL
✓
✓
✗
✗
58.6
96.5
+ GT-anchored Margin
✓
✓
✓
✗
58.8
96.9
Full TBD
✓
✓
✓
✓
58.9
97.0
Table 2. Ablation study on LLaVA-Video at retention ratio R=10% . “Rel. Acc (%)” is normalized by the corresponding Vanilla model in Table 1 .
Variant
Score
Rel. Acc (%)
TBD+FastV
55.0
90.6
TBD+VisionZip
57.1
94.1
TBD+FastVID
57.7
95.1
TBD (FlashVID)
58.9
97.0
Table 3. Effect of different compression modules within TBD on LLaVA-Video at retention ratio R=10% . FlashVID is the default compression module used in TBD unless otherwise specified. “Rel. Acc (%)” is normalized by the corresponding Vanilla model in Table 1 .
Figure 2. Logit distribution alignment on 100 randomly sampled VideoMME instances for LLaVA-Video at R=10% . Left: per-sample comparison of KL(PT∥Pbaseline) and KL(PT∥Pdistilled) on normalized A/B/C/D choice probabilities. Right: kernel density estimates of the KL distributions, where the distilled student shows a substantially smaller mean KL divergence to the teacher. Two-panel comparison of output alignment with the teacher. The left panel is a scatter plot in which each point is one sample, plotting the baseline's KL divergence against the distilled student's KL divergence; most points lie below the diagonal, meaning the student is closer to the teacher. The right panel shows density curves of the two KL distributions, with the student's distribution concentrated at clearly smaller values.
Figure 3. Decision boundary analysis on 200 randomly sampled VideoMME instances for LLaVA-Video at R=10% . Left: histogram and density of the GT-anchored margin Δ=Lgt−Lmax_wrong for the compression-only baseline and the distilled student. Right: summary statistics showing that the distilled student achieves a larger mean margin and a lower negative-margin ratio. Two-panel decision boundary comparison. The left panel overlays histograms and density curves of the ground-truth-anchored logit margin for the compression-only baseline and the distilled student, where the student's distribution is shifted toward larger positive margins. The right panel lists summary statistics showing the student attains a higher mean margin and a lower fraction of negative margins than the baseline.
Figure 4. Attention drift visualization on a randomly selected VideoMME example for LLaVA-Video at retention ratio R=10% . We compare the teacher and distilled student attention maps on the same frame. Side-by-side spatial attention heatmaps over the same video frame for the full-token teacher and the TBD-adapted compressed student. Both heatmaps concentrate on the same task-relevant foreground region of the frame, showing that the student's visual attention stays consistent with the teacher despite aggressive token compression.
Vision-language models (VLMs) have achieved remarkable multimodal understanding and reasoning capabilities, yet remain computationally expensive due to dense visual tokenization. Existing efficiency approaches either merge redundant visual tokens or drop them progressively in language backbone, often trading accuracy for speed. In this work, we propose DUET-VLM, a versatile plug-and-play dual compression framework that consists of (a) vision-only redundancy aware compression of vision encoder's output into information-preserving tokens, followed by (b) layer-wise, salient text-guided dropping of visual tokens within the language backbone to progressively prune less informative tokens. This coordinated token management enables aggressive compression while retaining critical semantics. On LLaVA-1.5-7B, our approach maintains over 99% of baseline accuracy with 67% fewer tokens, and still retains >97% even at 89% reduction. With this dual-stage compression during training, it achieves 99.7% accuracy at 67% and 97.6% at 89%, surpassing prior SoTA visual token reduction methods across multiple benchmarks. When integrated into Video-LLaVA-7B, it even surpasses the baseline -- achieving >100% accuracy with a substantial 53.1% token reduction and retaining 97.6% accuracy under an extreme 93.4% setting. These results highlight end-to-end training with DUET-VLM, enabling robust adaptation to reduced visual (image/video) input without sacrificing accuracy, producing compact yet semantically rich representations within the same computational budget. Our code is available at https://github.com/AMD-AGI/DUET-VLM.
In video understanding, vision-language models (VLMs) must ingest massive numbers of visual tokens, causing the computational and memory cost of the prefill stage to rise sharply. Such visual sequences are highly redundant along the spatio-temporal dimension, yet a high compression ratio is often accompanied by the loss of critical details. Existing token-compression methods either employ heuristic, training-free compression with limited content adaptivity or introduce additional modules that require expensive alignment training, leaving the trade-off between efficiency and adaptivity unresolved. To alleviate this limitation, we propose CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens. CRAFT recursively merges tokens by decoupling parameter-free token selection from learnable token fusion: global similarity determines which tokens to merge, while a position-aware weighting module and a content-adaptive channel-wise gate learn how to fuse them. The whole compression pipeline is query-agnostic. Because every retained token is a linear combination of the original tokens, CRAFT preserves their true spatio-temporal coordinates and stays aligned with the pre-trained language model's input distribution. Experiments on multiple representative video benchmarks show that CRAFT consistently outperforms prior state-of-the-art token-compression methods. At about 8× compression, it retains roughly 97% of the backbone's average accuracy and shows significant efficiency improvement.
Processing long videos with Vision-Language Models (VLMs) is bottlenecked by the quadratic cost of visual tokens, making long-form inference prohibitively expensive. While single-axis compression methods mitigate this, they hit a hard efficiency floor because they treat spatial and temporal redundancy independently. We present VETO (Video Efficient Token Optimization for Vision-Language Models), a training-optional plug-in that eliminates this bottleneck through dual-axis compression: (i) an intra-frame compressor that merges semantically similar tokens within each frame via optimal-transport inspired matching, and (ii) an inter-frame compressor that identifies and merges temporally redundant frames. The key design insight is hierarchical ordering: by first compressing spatial dimensions, VETO drastically reduces the cost of subsequent global temporal matching, bypassing the efficiency wall of single-axis approaches, with an advantage that grows with modern fully-fused attention infrastructure. Empirically, VETO achieves up to 45% faster inference (e.g., on LLaVA-OneVision-7B) while preserving or improving accuracy. Under extreme token starvation (10% budget), VETO outperforms VFlowOpt (54.9%), VisionZip (52.6%), and FastV (47.9%) with 55.7% accuracy. We demonstrate universal applicability across LLaVA-OneVision, InternVL-2.5, and LongVA, with zero-shot accuracy preserved or improved in all cases.