Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models

Xiaoyang Guo; Guoping Luo; Jusheng Zhang; Keze Wang; Wenhao Wang

cs.CVAug 28, 2026

Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models

Authors: Xiaoyang Guo, Guoping Luo, Jusheng Zhang, Keze Wang, Wenhao Wang

Organizations: Sun Yat-sen University Guangzhou, Guangdong, China · The University of British Columbia Vancouver, British Columbia, Canada · Vast Intelligence Lab Sydney, New South Wales, Australia

Abstract

Adapting video vision-language models (VLMs) is computationally expensive because video inputs produce a large number of visual tokens, making both fine-tuning and inference costly. Although visual token compression can reduce this overhead, direct adaptation on compressed inputs often causes semantic drift and noticeable performance degradation. We present Token-Budget Distillation (TBD), a parameter-efficient fine-tuning framework for adapting video VLMs under a fixed token budget. TBD freezes the pretrained backbone, updates only LoRA adapters, and integrates FlashVID-based visual token compression into the video pathway. To preserve full-token semantics under compression, TBD employs a dual-path teacher-student design, where a full-token teacher provides stable supervision and a compressed student is optimized with task loss, answer-region KL distillation, GT-anchored margin distillation, and reliability-aware KD control. This design enables the student to recover the semantic behavior of the full-token model while remaining efficient under aggressive token reduction. We evaluate TBD on three video VLM backbones, including LLaVA-Video, LLaVA-OneVision, and Qwen3-VL-8B-Instruct, across four video understanding benchmarks. TBD consistently outperforms compression-only baselines under both moderate and aggressive compression. On LLaVA-Video at retention ratio R = 10 percent, TBD preserves 97.0 percent of the Vanilla model's average accuracy; on LLaVA-OneVision at R = 10 percent, it achieves an average score of 58.4 and matches 100.0 percent relative accuracy.

Figures & tables

Figure 1. Overview of the Token-Budget Distillation (TBD) framework. A full-token teacher branch provides reliable guidance, while a compressed student branch with LoRA is optimized under a fixed token budget. Block diagram of the TBD training framework. A video and a question are processed by two branches sharing one frozen backbone: a teacher branch that consumes the full visual token sequence, and a student branch in which FlashVID compresses the tokens before the language model while LoRA adapters are trained. The output logits of the two branches are compared to compute the task cross-entropy loss, the answer-region KL distillation loss, and the GT-anchored margin loss, and only the LoRA parameters receive gradient updates.

Method	Retention Ratio $R$	VideoMME				EgoSchema		LongVideo Bench	MVBench	Avg.
Method	Retention Ratio $R$	Short	Medium	Long	Overall	Subset	Total	LongVideo Bench	MVBench	Score	Rel. Acc (%)
LLaVA-Video
Vanilla	100%	77.0	62.1	53.3	64.2	59.4	57.3	59.5	61.9	60.7	100.0
FastV	20%	69.3	58.3	49.9	59.2	54.8	54.1	56.0	58.4	56.9	93.7
VisionZip		72.3	59.6	53.3	61.7	59.0	56.4	58.0	59.8	59.0	97.2
FastVID		74.6	60.8	52.3	62.6	57.0	55.0	57.1	60.2	58.7	96.7

Table 1. Main results on four video understanding benchmarks under different token retention ratios. We compare TBD with the uncompressed Vanilla model and representative token reduction baselines on three backbones: LLaVA-Video , LLaVA-OneVision , and Qwen3-VL-8B-Instruct . “Avg. Score” denotes the average performance across benchmarks, and “Rel. Acc (%)” reports the average score normalized by the corresponding Vanilla model.

Variant	LoRA	Answer-region KL	GT-anchored Margin	Filtering & KD Cap	Score	Rel. Acc (%)
LoRA-only Student	✓	✗	✗	✗	58.3	96.0
+ Answer-region KL	✓	✓	✗	✗	58.6	96.5
+ GT-anchored Margin	✓	✓	✓	✗	58.8	96.9
Full TBD	✓	✓	✓	✓	58.9	97.0

Table 2. Ablation study on LLaVA-Video at retention ratio

R=10\%

. “Rel. Acc (%)” is normalized by the corresponding Vanilla model in Table 1 .

Variant	Score	Rel. Acc (%)
TBD+FastV	55.0	90.6
TBD+VisionZip	57.1	94.1
TBD+FastVID	57.7	95.1
TBD (FlashVID)	58.9	97.0

Table 3. Effect of different compression modules within TBD on LLaVA-Video at retention ratio

R=10\%

. FlashVID is the default compression module used in TBD unless otherwise specified. “Rel. Acc (%)” is normalized by the corresponding Vanilla model in Table 1 .

Figure 2. Logit distribution alignment on 100 randomly sampled VideoMME instances for LLaVA-Video at $R=10\%$ . Left: per-sample comparison of $\mathrm{KL}(P_{T}\|P_{\mathrm{baseline}})$ and $\mathrm{KL}(P_{T}\|P_{\mathrm{distilled}})$ on normalized A/B/C/D choice probabilities. Right: kernel density estimates of the KL distributions, where the distilled student shows a substantially smaller mean KL divergence to the teacher. Two-panel comparison of output alignment with the teacher. The left panel is a scatter plot in which each point is one sample, plotting the baseline's KL divergence against the distilled student's KL divergence; most points lie below the diagonal, meaning the student is closer to the teacher. The right panel shows density curves of the two KL distributions, with the student's distribution concentrated at clearly smaller values.

Figure 3. Decision boundary analysis on 200 randomly sampled VideoMME instances for LLaVA-Video at $R=10\%$ . Left: histogram and density of the GT-anchored margin $\Delta=L_{gt}-L_{max\_wrong}$ for the compression-only baseline and the distilled student. Right: summary statistics showing that the distilled student achieves a larger mean margin and a lower negative-margin ratio. Two-panel decision boundary comparison. The left panel overlays histograms and density curves of the ground-truth-anchored logit margin for the compression-only baseline and the distilled student, where the student's distribution is shifted toward larger positive margins. The right panel lists summary statistics showing the student attains a higher mean margin and a lower fraction of negative margins than the baseline.

Figure 4. Attention drift visualization on a randomly selected VideoMME example for LLaVA-Video at retention ratio $R=10\%$ . We compare the teacher and distilled student attention maps on the same frame. Side-by-side spatial attention heatmaps over the same video frame for the full-token teacher and the TBD-adapted compressed student. Both heatmaps concentrate on the same task-relevant foreground region of the frame, showing that the student's visual attention stays consistent with the teacher despite aggressive token compression.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 738 papers

DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference
Feb 21, 2026Aditya Kumar Singh, Hitesh Kandala, Pratik Prabhanjan Brahma +2Efficient VLM Inference Multimodal Token Compression
Advanced Micro Devices, Inc. (AMD)
CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models
Aug 3, 2026Yu Chen, Xiaohong Li, Xiaole Wang +3Efficient VLM Inference Large Vision-Language Models
1Evermind · 2Shanda Group · 3Peking University
VETO: Video Efficient Token Optimization for Vision Language Models
Oct 1, 2026Gueter Josmy Faure, Hao Ping Wang, Min-Hung Chen +1Efficient VLM Inference Large Vision-Language Models
National Taiwan University · NVIDIA

Feb 21, 2026J/K move · Enter open · S save

DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference

Aditya Kumar Singh, Hitesh Kandala, Pratik Prabhanjan Brahma, Zicheng Liu+1

Advanced Micro Devices, Inc. (AMD)

Efficient VLM Inference Multimodal Token Compression Visual Token Compression Visual Token Pruning Video Token Compression

Open paper AskPDF arXiv

Vision-language models (VLMs) have achieved remarkable multimodal understanding and reasoning capabilities, yet remain computationally expensive due to dense visual tokenization. Existing efficiency approaches either merge redundant visual tokens or drop them progressively in language backbone, often trading accuracy for speed. In this work, we propose DUET-VLM, a versatile plug-and-play dual compression framework that consists of (a) vision-only redundancy aware compression of vision encoder's output into information-preserving tokens, followed by (b) layer-wise, salient text-guided dropping of visual tokens within the language backbone to progressively prune less informative tokens. This coordinated token management enables aggressive compression while retaining critical semantics. On LLaVA-1.5-7B, our approach maintains over 99% of baseline accuracy with 67% fewer tokens, and still retains >97% even at 89% reduction. With this dual-stage compression during training, it achieves 99.7% accuracy at 67% and 97.6% at 89%, surpassing prior SoTA visual token reduction methods across multiple benchmarks. When integrated into Video-LLaVA-7B, it even surpasses the baseline -- achieving >100% accuracy with a substantial 53.1% token reduction and retaining 97.6% accuracy under an extreme 93.4% setting. These results highlight end-to-end training with DUET-VLM, enabling robust adaptation to reduced visual (image/video) input without sacrificing accuracy, producing compact yet semantically rich representations within the same computational budget. Our code is available at https://github.com/AMD-AGI/DUET-VLM.

Abstract

Vision-Language Models

Explore similar work

DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference

CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models

VETO: Video Efficient Token Optimization for Vision Language Models