LinguDistill: Recovering Linguistic Ability in Vision-Language Models via Selective Cross-Modal Distillation

Patrick Amadeus Irawan; Erland Hilman Fuadi; Shanu Kumar; Alham Fikri Aji; Yova Kementchedjhieva

cs.CVApr 1, 2026

LinguDistill: Recovering Linguistic Ability in Vision-Language Models via Selective Cross-Modal Distillation

Authors: Patrick Amadeus Irawan, Erland Hilman Fuadi, Shanu Kumar, Alham Fikri Aji, Yova Kementchedjhieva

Organizations: Mohamed bin Zayed University of Artificial Intelligence

Abstract

Turning a pretrained language model (LM) into a vision-language model (VLM) through multimodal fine-tuning often erodes its native language ability, a form of catastrophic forgetting that shows up even on text-only tasks. This loss is hard to undo with further fine-tuning, and existing remedies add adapters or alignment modules that increase architectural complexity and inference cost. We propose LinguDistill, an adapter-free knowledge distillation method that uses the original frozen LM as the teacher during multimodal post-training. To let a text-only teacher supervise vision-conditioned outputs, we introduce layer-wise KV-cache sharing, which exposes the teacher to the student's multimodal representations without changing either architecture. We then apply distillation selectively, on language-heavy data only, so the teacher restores linguistic ability while the student keeps its visual grounding on document and OCR tasks. LinguDistill recovers the language and knowledge performance lost during multimodal fine-tuning, matching the original VLM on average over text-only benchmarks (ARC, HellaSwag) and exceeding it on ScienceQA, while keeping vision-heavy performance close to standard fine-tuning. Since the teacher is dropped after training, the final model adds no parameters and no inference cost. More broadly, our results show that a model's own pre-adaptation backbone is a practical teacher for undoing forgetting, suggesting a simple recipe for keeping language ability intact as models are extended to new modalities.

Figures & tables

Figure 2: KV sharing (a) and inference (b).

Category	Sources	KD signal	Reasoning
Language-heavy (8)	A-OKVQA, FigureQA, IconQA, RobustQA, ScienceQA, Visual7W, VQAv2, VSR	$\alpha>0$	Distill teacher’s linguistic priors on reasoning and knowledge tasks
OCR/doc-heavy (9)	Chart2Text, ChartQA, DocVQA, InfographicVQA, OCR-VQA, TextCaps, TextVQA, VisText, VisualMRC	$\alpha=0$	Preserve student’s native visual and OCR grounding ability

Table 1: Training source categorisation. Language-heavy sources receive the full KD signal (

\alpha>0

); OCR/document sources receive CE only (

\alpha=0

) in selective distillation variants.

Variant	KD	$\boldsymbol{\alpha}$	$\boldsymbol{T}$	Notes
Baselines
nanoVLM	–	–	–	Pretrained checkpoint
nanoVLM-full	$\times$	–	–	Fine-tune on all sources
nanoVLM-lang	$\times$	–	–	Fine-tune on language subset
Uniform KD
distill-full	$\checkmark$	0.5	2	KD on full subset

Table 2: Experimental variants: baselines (gray), uniform KD (yellow), selective KD (blue).

	Language		Doc & OCR	General multimodal
Variant	Language-heavy	Text-only	Avg	MME	Others
nanoVLM (original)	0.527	0.444	0.629	780.5	0.401
nanoVLM-full	0.471 (-10.6%)	0.382 (-14.0%)	0.592 (-5.9%)	628.0 (-19.5%)	0.390 (-2.7%)
nanoVLM-lang	0.481 (-8.7%)	0.384 (-13.5%)	0.587 (-6.7%)	665.5 (-14.7%)	0.383 (-4.5%)
distill-full	0.524 (-0.6%)	0.388 (-12.6%)	0.461 (-26.7%)	643.5 (-17.6%)	0.361 (-10.0%)
distill-lang	0.540 (+2.5%)	0.424 (-4.5%)	0.512 (-18.6%)	708.5 (-9.2%)	0.379 (-5.5%)

Table 3: Category averages across variants (full per-benchmark scores in Appendix A ). Language-heavy averages AI2D, COCO, ScienceQA, ARC-Easy, ARC-Challenge and HellaSwag; Text-only is its ARC/HellaSwag subset, which has no visual input. The original checkpoint (grey) is shown for reference; bold marks the best trained variant per column. Small coloured percentages give the relative change versus the original checkpoint.

Panel of Figure 3 — Figure 3: Training-loss analysis for the three selective distillation variants. Left: the CE term is lowest for the main LinguDistill setting, with High KD slightly above it and Low KD clearly worse. Right: the Soft loss uses the combined logged objective, which mixes the teacher-weighted distillation signal with the VLM loss in a balanced manner.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

		Fine-tuning		Uniform KD (Ours w/o Task Weighting)		Selective KD (Ours)
Task	nanoVLM	nanoVLM-full	nanoVLM-lang	distill-full	distill-lang	LinguDistill
Language-heavy
AI2D	0.440	0.416	0.413 (-0.7%)	0.490 (+17.8%)	0.502 (+20.7%)	0.507 (+21.9%)
COCO	0.800	0.673	0.742 (+10.3%)	0.843 (+25.3%)	0.787 (+16.9%)	0.866 (+28.7%)
ScienceQA	0.590	0.592	0.581 (-1.9%)	0.650 (+9.8%)	0.679 (+14.7%)	0.676 (+14.2%)
ARC Easy	0.605	0.540	0.542 (+0.4%)	0.548 (+1.5%)	0.601 (+11.3%)	0.621 (+15.0%)

Appendix

Table 4: Benchmark results across variants. Values are absolute scores; colored percentages denote change vs. nanoVLM-full . Average rows are manually color-scaled for perceptual consistency: small changes are subtle, larger changes progressively emphasized. Selective KD ( LinguDistill ) gives the strongest gains with controlled perception loss.

Parameter	Value
Vision encoder	SigLIP2-B/16-512 ( Tschannen et al., 2025 ) (frozen)
Language decoder	SmolLM2-360M-Instruct ( Allal et al., 2025 )
Image tokens	64 ( $4{\times}$ pixel shuffle)
Max images per example	1
Max sequence length	1024 tokens
Effective batch size	128

Appendix

Table 5: Shared training configuration across all variants.

1	Initialize $\mathcal{L}_{\mathrm{sum}}\leftarrow 0$ and $N\leftarrow 0$
2	Freeze $D_{\mathrm{right}}$ and $H_{\mathrm{right}}$
3	for $b\leftarrow 1$ to $B$ do
4	$\mathbf{v}_{b}\leftarrow E_{\mathrm{img}}(I_{b})$
5	$\mathbf{u}_{b}\leftarrow P(\mathbf{v}_{b})$
6	$\tilde{\mathbf{x}}_{b}\leftarrow\mathrm{concat}(x_{b},\mathbf{u}_{b})$
7	$(\mathbf{h}^{l}_{b},\mathrm{KV}^{l}_{b})\leftarrow D_{\mathrm{left}}(\tilde{\mathbf{x}}_{b},m_{b})$

Appendix

Figure 4: Pseudo-code for LinguDistill . The image is first mapped into the language subspace, the left decoder absorbs the full multimodal prompt and materializes prompt KV memory, and the frozen right decoder continues from that inherited state to produce detached supervision. The optimization objective combines source-selective distillation with hard next-token supervision, while updating only the left-side parameters.

Source	Cat.	Mean	Med.	P95	Max	% ${>}$ 1024
aokvqa	lang	62	60	100	116	0.0%
figureqa	lang	212	187	307	321	0.0%
iconqa	lang	37	35	56	664	0.0%
robut_sqa	lang	180	102	425	1974	2.4%
scienceqa	lang	493	118	549	15171	2.1%
visual7w	lang	251	197	705	972	0.0%

Appendix

Table 6: Token-length statistics per training source, sampled from The Cauldron and tokenized with SmolLM2-360M-Instruct. “Cat.” indicates language-heavy (lang) or OCR/document (ocr) sources. “%

{>}

1024” is the fraction of examples exceeding 1024 tokens.

Group	Sub-task	LinguDistill	nanoVLM-full	$\Delta$
Text Rec.	Regular /50	47	48	$-$ 1
	Irregular /50	40	44	$-$ 4
	Artistic /50	46	45	+1
	Subtotal /150	133	137	$-$ 4
Handwriting & Non-Semantic	Handwriting /50	30	36	$-$ 6
	Digit String /50	38	42	$-$ 4

Appendix

Table 7: OCRBench sub-task breakdown grouped by cognitive demand. LinguDistill vs. nanoVLM-full (460M, 4k steps). Text recognition (decoding-only) is largely preserved; the regression concentrates in extraction tasks requiring spatial grounding ( bold ).

		Fine-tuning		Uniform KD		Selective KD
Task	nanoVLM	nanoVLM-full	nanoVLM-lang	distill-full	distill-lang	LinguDistill	highKD	lowKD
Language and knowledge-heavy
AI2D	0.440	0.416	0.413	0.490	0.502	0.507	0.492	0.511
COCO 2017 Captioning2017_cap	0.800	0.673	0.742	0.843	0.787	0.866	0.876	0.842
ScienceQA	0.590	0.592	0.581	0.650	0.679	0.676	0.676	0.670
ARC Easy	0.605	0.540	0.542	0.548	0.601	0.621	0.614	0.627

Appendix

Table 8: Full benchmark results across all variants. Columns grouped by training strategy.

Task	nanoVLM	nanoVLM-full	nanoVLM-lang
Language and knowledge-heavy
AI2D	0.44	0.416	0.413
COCO 2017 Captioning	0.80	0.673	0.742
ScienceQA	0.59	0.592	0.581
ARC Easy	0.605	0.540	0.542
HellaSwag	0.405	0.326	0.329

Appendix

Table 9: Effect of data subset selection across all tasks.

Task	distill-full	distill-lang	LinguDistill
Language and knowledge-heavy
AI2D	0.490	0.502	0.507
COCO 2017 Captioning	0.843	0.787	0.866
ScienceQA	0.650	0.679	0.676
ARC Easy	0.548	0.601	0.621
HellaSwag	0.345	0.376	0.394

Appendix

Table 10: Uniform vs selective distillation across all tasks.

Task	LinguDistill	highKD	lowKD
Language and knowledge-heavy
AI2D	0.507	0.492	0.511
COCO 2017 Captioning	0.866	0.876	0.842
ScienceQA	0.676	0.676	0.670
ARC Easy	0.621	0.614	0.627
HellaSwag	0.394	0.392	0.393

Appendix

Table 11: Effect of distillation strength across all tasks.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 545 papers

From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers
Mar 11, 2026Ayan Sengupta, Shantanu Dixit, Md Shad Akhtar +1Cross-Modal Knowledge Distillation Knowledge Distillation
Indian Institute of Technology Delhi, India · Indraprastha Institute of Information Technology Delhi, India
Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models
Apr 16, 2026Haoyi Sun, Xiaoxiao Wang, Ning Mao +5Cross-Modal Knowledge Distillation Vision-Language Models
Li Auto Inc.
Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge
Jun 25, 2026Thomas Shih-Chao Liang, Zhuoran Yu, Yong Jae LeeCross-Modal Knowledge Distillation Fine-Grained Image Classification
University of Wisconsin-Madison

Mar 11, 2026J/K move · Enter open · S save

From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers

Ayan Sengupta, Shantanu Dixit, Md Shad Akhtar, Tanmoy Chakraborty

Indian Institute of Technology Delhi, India · Indraprastha Institute of Information Technology Delhi, India

Cross-Modal Knowledge Distillation Knowledge Distillation Language Model Distillation VLM Distillation

Open paper AskPDF arXiv

Knowledge distillation (KD) methods are pivotal in compressing large pre-trained language models into smaller models, ensuring computational efficiency without significantly dropping performance. Traditional KD techniques assume homogeneity in modalities between the teacher (source) and the student (target) models. On the other hand, existing multimodal knowledge distillation methods require modality-specific pre-training of the teacher model, which is computationally infeasible in most cases. In this paper, we introduce ARMADA, an efficient cross-modal knowledge distillation framework designed to transfer knowledge from large vision-language models, including black-box models, to language-only models. Unlike existing KD techniques that rely on the internal structures of multimodal teachers or require computationally expensive pre-training, ARMADA leverages novel alignment techniques to distil knowledge without altering the teacher model, ensuring efficiency and scalability. We empirically validate ARMADA on twelve natural language understanding, eight complex generative reasoning and five instruction-tuning tasks, demonstrating consistent performance improvements in large models such as DeBERTa-v2-1.4B, OPT-1.3B, LLaMA-{3B, 7B, 8B}. ARMADA achieves up to 3.4% improvement on language understanding tasks and 2.6% boost in generative reasoning, all without requiring expensive multimodal pre-training or fine-tuning of the teacher model. Our findings challenge conventional knowledge distillation paradigms by demonstrating that even vision-language models, despite lacking direct textual understanding, can significantly enhance language models when distilled appropriately.

Abstract

VLM Adaptation

Explore similar work

From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers

Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models

Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge