DVTR-KD: Distilling Vision-Language Models via Intermediate Visual Token Reconstruction

Haolan Wei; Jiahui Wang; Junli Zha; Xinkai Lu; Yunxiao Zou; Chao Zhou; Xiang Song

cs.CVOct 13, 2026

DVTR-KD: Distilling Vision-Language Models via Intermediate Visual Token Reconstruction

Authors: Haolan Wei, Jiahui Wang, Junli Zha, Xinkai Lu, Yunxiao Zou, Chao Zhou, Xiang Song

Organizations: SF Technology Co., Ltd, Shenzhen, China

Abstract

Multimodal large language models (MLLMs) have achieved strong performance on diverse vision-language tasks, but their computational demands limit deployment under constrained budgets, particularly on edge devices. Many works focus on Knowledge Distillation (KD) and try to transfer knowledge from powerful teacher models to students. In addition to reducing model size, visual token compression provides a complementary route to efficiency, but introduces mismatched visual token counts that complicate direct token-wise knowledge transfer. To mitigate this, we propose DVTR-KD, a knowledge distillation framework based on Dense Visual Token Reconstruction, which reconstructs the teacher's dense visual tokens from the student's compressed tokens. We reconstruct visual states after the final LLM layer and at the input to an intermediate layer identified as important for visual-information readout. At the intermediate layer, reconstruction is supervised through the teacher's response predictions rather than by direct token-wise KL or MSE matching. An auxiliary attention-readout loss complements this response-level supervision by aligning how the student and the original teacher integrate contextual information when predicting responses. Evaluations on diverse benchmarks demonstrate that DVTR-KD improves the performance of token-compressed students while retaining the inference efficiency afforded by visual token compression.

Figures & tables

Figure 1: Layer-wise sensitivity to visual-value ablation after SFT and before distillation. We zero the value vectors of all visual tokens in one transformer layer at a time and report the change in response NLL (

\Delta\mathrm{NLL}

), averaged over the same diagnostic subset. In the legends,

1\times

denotes uncompressed models, while

4\times

and

16\times

denote models with nominal fourfold and sixteenfold reductions in visual-token count. The sample counts are summed over all training datasets before distillation.

Figure 2: Overview of DVTR-KD. (a) The intermediate reconstructor

R_{\mathrm{mid}}

reconstructs student visual states in the teacher’s dense layout at PVRL, substituting teacher’s visual states; this reconstruction is supervised through response predictions produced by the teacher from the PVRL onward. Native response distillation and an independent final-layer reconstructor

R_{\mathrm{last}}

provide additional supervision. (b) Readout alignment uses attention outputs from the original teacher and the student. (c) An illustration to the reconstructor.

Model	Stage 1	Stage 1.5	SFT	KD
Student	LLaVA-PreTrain-558K	–	LLaVA-NeXT-779K	LLaVA-NeXT-779K
Same-data teacher	LLaVA-PreTrain-558K	–	LLaVA-NeXT-779K	Frozen
Data-rich teacher	LLaVA-PreTrain-558K	LLaVA-ReCap-558K	LLaVA-OneVision-SI-3.3M	Frozen

Table 1: Training data for model preparation and knowledge distillation. “–” indicates a skipped stage; “Frozen” indicates that the teacher is used without parameter updates during KD.

Qwen3-0.6B teacher trained on 4.4M samples $\rightarrow$ Qwen3-0.6B compressed students
		General			Knowledge				OCR
Method	$\rho^{2}$	GQA	MME	RWQA	SQA	AI2D	MMMU	MMStar	ChartQA	DocVQA	TextVQA	OCRBench	$Avg_{11}$
Teacher	$1\times$	56.6	1862.9	61.7	88.2	73.6	37.0	50.2	78.8	87.5	77.3	74.3	68.3
SFT	$4\times$	57.8	1450.2	53.6	59.8	59.9	32.1	38.3	64.6	67.7	63.1	49.8	54.4
MiniLLM	$4\times$	58.3	1616.3	56.1	61.3	60.8	32.1	38.6	64.6	66.9	65.1	52.2	55.8
LLaVA-KD ∗	$4\times$	59.7	1636.3	58.0	62.6	64.9	32.3	40.5	70.6	74.4	68.0	56.3	58.7

Table 2: Distillation results across visual-token compression ratios, teacher data regimes and teacher scales. All methods are re-implemented for fair comparison and start from the corresponding SFT model.

\rho^{2}

denotes the nominal visual-token compression factor. LLaVA-KD ∗ means we use our reconstruction method to align different visual-token counts after last block. MME scores are divided by 28 when calculating the average scores.

Avg_{11}

is the mean over all 11 benchmarks.

$\mathcal{L}_{\mathrm{resp}}$	$\mathcal{L}_{\mathrm{vis}}$	$\mathcal{L}_{\mathrm{DVTR}}$	$\mathcal{L}_{\mathrm{readout}}$	$Avg_{11}$
✓	–	–	–	51.0
✓	✓	–	–	51.2
✓	✓	✓	–	52.6
✓	✓	✓	✓	53.0

Table 3: Ablation study on contributions from key components in our method.

$\rho^{2}$	type	$Avg_{11}$
$4\times$	MSE	60.2
$4\times$	response	59.9
$16\times$	MSE	51.9
$16\times$	response	53.0

Table 4: Ablation study on supervision type for reconstruction at PVRL.

$\ell$	$Avg_{G}$	$Avg_{K}$	$Avg_{O}$	$Avg_{11}$
1	55.5	49.7	51.6	52.0
11	56.6	50.8	51.6	52.7
13	55.9	50.5	52.3	52.6
12 ( $\ell^{*}$ )	56.6	51.0	52.4	53.0

Table 5: Reconstruction at different layers.

Avg_{G}

Avg_{K}

, and

Avg_{O}

denote the mean scores on the General, Knowledge, and OCR benchmarks, respectively.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Stage-1	Stage-1.5	Stage-2	KD
Train ViT Backbone	–	–	–	–
Train Projector	✓	✓	✓	✓
Train Compressor	✓	–	✓	✓
Train LLM	–	✓	✓	✓
Train Reconstructor	–	–	–	✓
Projector LR	$10^{-3}$	$2\times 10^{-5}$	$2\times 10^{-5}$	$2\times 10^{-5}$

Appendix

Table 6: Training hyperparameters across different stages.

Benchmark	Split	Count	Metric
GQA	Test-dev balanced	12,578	Exact match
MME	Test	2,374	Acc. + Acc. +
RWQA	Test	765	Accuracy
SQA	Test (image subset)	2,017	Accuracy
AI2D	Test	3,088	Accuracy
MMMU	Validation	900	Accuracy

Appendix

Table 7: Evaluation splits and metrics. Counts denote evaluated questions.

$\mathcal{L}_{\mathrm{resp}}$	$\mathcal{L}_{\mathrm{vis}}$	$\mathcal{L}_{\mathrm{rel}}$	$Avg_{11}$
✓	–	–	51.0
✓	✓	–	51.2
✓	✓	✓	50.6

Appendix

Table 8: Effect of relation distillation in our adapted LLaVA-KD baseline under

\rho^{2}=16

Figure 3: Layer-wise sensitivity to visual-value ablation after distillation. We compare DVTR-KD students with the data-rich teacher and the corresponding SFT students.

VLM Distillation

Vision-language model distillation transfers multimodal capabilities to smaller, faster models for efficient deployment.

See all 126 papers

DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference
Feb 21, 2026Aditya Kumar Singh, Hitesh Kandala, Pratik Prabhanjan Brahma +2Efficient VLM Inference Multimodal Token Compression
Advanced Micro Devices, Inc. (AMD)
ReTrace: Visual Backtracking for Efficient Multimodal Large Language Models
Oct 18, 2026Guangzong Si, Dong Wang, Yifan Yu +3Efficient VLM Inference Visual Token Compression
University of Science and Technology of China · Eastern Institute of Technology, Ningbo · Tsinghua University +2
Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation
Jul 1, 2026Jaehyun Jang, Eunseop Yoon, Hee Suk Yoon +3Cross-Modal Knowledge Distillation VLM Distillation
Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea · University of Illinois Urbana-Champaign, Champaign, USA

Feb 21, 2026J/K move · Enter open · S save

DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference

Aditya Kumar Singh, Hitesh Kandala, Pratik Prabhanjan Brahma, Zicheng Liu+1

Advanced Micro Devices, Inc. (AMD)

Efficient VLM Inference Multimodal Token Compression Visual Token Compression Visual Token Pruning Video Token Compression

Open paper AskPDF arXiv

Vision-language models (VLMs) have achieved remarkable multimodal understanding and reasoning capabilities, yet remain computationally expensive due to dense visual tokenization. Existing efficiency approaches either merge redundant visual tokens or drop them progressively in language backbone, often trading accuracy for speed. In this work, we propose DUET-VLM, a versatile plug-and-play dual compression framework that consists of (a) vision-only redundancy aware compression of vision encoder's output into information-preserving tokens, followed by (b) layer-wise, salient text-guided dropping of visual tokens within the language backbone to progressively prune less informative tokens. This coordinated token management enables aggressive compression while retaining critical semantics. On LLaVA-1.5-7B, our approach maintains over 99% of baseline accuracy with 67% fewer tokens, and still retains >97% even at 89% reduction. With this dual-stage compression during training, it achieves 99.7% accuracy at 67% and 97.6% at 89%, surpassing prior SoTA visual token reduction methods across multiple benchmarks. When integrated into Video-LLaVA-7B, it even surpasses the baseline -- achieving >100% accuracy with a substantial 53.1% token reduction and retaining 97.6% accuracy under an extreme 93.4% setting. These results highlight end-to-end training with DUET-VLM, enabling robust adaptation to reduced visual (image/video) input without sacrificing accuracy, producing compact yet semantically rich representations within the same computational budget. Our code is available at https://github.com/AMD-AGI/DUET-VLM.

Abstract

VLM Distillation

Explore similar work

DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference

ReTrace: Visual Backtracking for Efficient Multimodal Large Language Models

Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation