DARK: Diagonal-Anchored Repulsive Knowledge Distillation for Vision-Language Models under Extreme Compression

Numan Saeed; Asif Hanif; Fadillah Adamsyah Maani; Hussain Alasmawi; Mohammad Yaqub

cs.CVMar 5, 2026

DARK: Diagonal-Anchored Repulsive Knowledge Distillation for Vision-Language Models under Extreme Compression

Authors: Numan Saeed, Asif Hanif, Fadillah Adamsyah Maani, Hussain Alasmawi, Mohammad Yaqub

Organizations: Computer Vision, Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE

Abstract

Compressing vision-language models for on-device deployment is increasingly important in clinical settings, but knowledge distillation (KD) degrades sharply when the teacher-student capacity gap spans an order of magnitude or more. We argue that, under such gaps, strict imitation of the teacher is a poor objective: much of the teacher's pairwise similarity structure reflects its own architectural biases rather than information a compact student can efficiently represent. We propose Diagonal-Anchored Repulsive Knowledge Distillation (DARK), a contrastive KD framework that decomposes the distillation loss into a diagonal term (matched image-text pairs) and an off-diagonal term (non-target similarities). The diagonal term anchors matched-pair alignment throughout training; the off-diagonal term is annealed from positive to negative weighting, transitioning the student from imitating to repelling the teacher's non-target similarity structure. We instantiate DARK by distilling FetalCLIP, a 427M-parameter fetal ultrasound vision-language model, into MobileFetalCLIP, a 75M-parameter student model with a $26\times$ smaller visual encoder, running in 1.6,ms on an iPhone16Pro. The student matches or exceeds its teacher on three zero-shot benchmarks, including HC18 biometry validity (88.6% vs.\ 83.5%) and brain sub-plane F1 (0.784 vs.\ 0.702). Embedding-geometry and logit analyses show that DARK induces structured decorrelation: the student preserves teacher-aligned per-image confidence while diverging from inherited inter-class confusion, suggesting that controlled repulsion can be more efficient than imitation under extreme compression.

Figures & tables

Figure 1: Overview of the DARK framework. (A) Distillation setup: a frozen FetalCLIP teacher (ViT-L/14, 304M visual params) produces an $N\times N$ similarity matrix; a lightweight FastViT student (11.4M visual params) is trained via $\mathcal{L}_{\mathrm{CLIP}}$ and $\mathcal{L}_{\mathrm{KD}}$ . (B) Attraction-to-repulsion dynamics: the off-diagonal weight $\beta(t)$ linearly decays from positive $\beta_{\mathrm{start}}$ into a negative value $\beta_{\mathrm{stop}}$ ; the weight for $\mathcal{L}_{\mathrm{diag}}$ term remains fixed, preserving matched pair alignment throughout training. (C) Outcome: DARK produces structured decorrelation, resulting in better cluster separation and a higher VR HC18 and F1 3Brain with $26\times$ fewer visual parameters.

Models	Params.	F1 5Plane	F1 3Brain	F1 Pl_Br	VR HC18	Composite Score
Teacher Model
FetalCLIP (ViT-L/14)	427M	0.973	0.702	0.871	0.835	0.853
General VLMs
CLIP (ViT-L/14)	427M	0.308	0.206	0.270	0.110	0.190
BiomedCLIP (ViT-B/16)	150M	0.603	0.236	0.466	0.240	0.353
UniMed-CLIP (ViT-B/16)	150M	0.679	0.187	0.495	0.090	0.293

Table 1: Zero-shot comparison on fetal ultrasound benchmarks. Static Logit KD implements the CLIP-KD Yang et al. (2024) objective. Composite Score = (F1 Pl_Br + VR HC18 )/2.

Configurations	F1 5Plane	F1 3Brain	F1 Pl_Br	VR HC18	Composite Score
FetalCLIP (ViT-L/14) - Teacher	0.973	0.702	0.871	0.835	0.853
Using $\mathcal{L}_{\mathrm{CLIP}}$ only (without $\mathcal{L}_{\mathrm{KD}}$ )	0.889	0.712	0.823	0.713	0.768
Static KD ( $\mathcal{L}_{\mathrm{CLIP}}+\lambda\cdot\mathcal{L}_{\mathrm{KD}}$ ), $\lambda{=}1$	0.946	0.715	0.860	0.794	0.826
Partial KD Annealing ( $\lambda:\,1{\to}0.1$ )	0.902	0.713	0.831	0.746	0.788
Full KD Annealing ( $\lambda:1{\to}0$ )	0.842	0.742	0.805	0.731	0.768
Static KD + Feature KD	0.946	0.664	0.840	0.759	0.800

Table 2: Ablation on KD Strategies: KD strategies for MobileFetalCLIP (FastViT-MCI0, 11.4M visual encoder) distilled from FetalCLIP ViT-L/14 (304M). Composite Score = (F1 Pl_Br + VR HC18 )/2.

Table 4

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Learning rate ( $\eta_{\mathrm{peak}}$ )	$1\times 10^{-5}$
LR schedule	Cosine decay with linear warmup (500 warmup steps; world-size adjusted)
Weight decay	0.1
Batch size	512
Gradient accumulation steps	2
Effective batch size	1,024 (typical global batch)

Appendix

Table S1: Full training hyperparameter configuration. Values in the right column apply to all runs unless noted otherwise.

Figure S1: Row-normalized confusion matrices for zero-shot 5-plane classification, with an additional “other” column capturing predictions to off-set classes (kidney, lips/nose, profile, spine). Left: Static KD (

\lambda{=}1.0

) achieves high recall for cervix (96%), brain (95%), femur (98%), and thorax (93%), but lower abdomen recall (82%) and 9% off-class leakage. Right: DARK (

\beta_{\mathrm{start}}{=}2

\beta_{\mathrm{stop}}{=}{-}0.8

) improves brain (99%) and abdomen (89%) with minor trade-offs on cervix, femur, and thorax; the main shift is a thorax

\to

brain redistribution (1%

\to

7%).

Figure S2: Per-class F1 heatmap across fetal plane classes (abdomen, brain, femur, thorax, cervix) and brain sub-planes (transthalamic, transcerebellum, transventricular). Each row corresponds to the run’s best epoch (selected by Composite Score). DARK ( $\beta_{0}{=}2$ , $\beta_{\mathrm{stop}}{=}{-}0.8$ ) achieves the strongest brain sub-plane discrimination.

Figure S3: Logit distribution analysis: teacher–student decorrelation comparing Static KD and DARK. Row 1 : Softmax entropy distributions (lower = more confident; dashed lines denote empirical means). Row 2 : $N{\times}N$ logit matrix rank correlation. Row 3 : Per-image teacher-student probability agreement ( $1-\mathrm{JSD}$ ; higher = the student’s per-sample class distribution matches the teacher’s more closely; dashed lines denote empirical means).

Figure S4: Full training dynamics across all logged runs (4-panel overlay). Panels show: KD weight schedule, distill loss, learned logit scale, and the Composite Score metric over training epochs. DARK ( $\beta_{\mathrm{start}}{=}2$ , $\beta_{\mathrm{stop}}{=}{-}0.8$ ) achieves the highest Composite Score, consistent with the main-paper ablation.

Figure S5: Per-run Composite Score trajectories across training epochs. Static KD plateaus early, while repulsive variants continue improving through the repulsive phase. DARK ( $\beta_{\mathrm{start}}{=}2$ , $\beta_{\mathrm{stop}}{=}{-}0.8$ ) achieves the highest final composite score among the stable runs.

Aspect	DKD Zhao et al. [2022]	Ours
Task	$C$ -class classification	$N{\times}N$ contrastive similarity
TCKD (diagonal)	Binary KL on $[p_{t},1{-}p_{t}]$	Cross-entropy on diagonal entries ( $i{=}j$ )
NCKD (off-diagonal)	KL on renormalised non-target distribution	Cross-entropy on off-diagonal entries ( $i{\neq}j$ )
Weights	$\alpha{=}1$ , $\beta{=}8$ (both positive, static)	Diagonal fixed at 1.0; $\beta(t)$ follows a linear schedule that decays into negative values
Temperature	Shared $\tau$ on both teacher & student, with $T^{2}$ correction	Teacher-only $\tau_{\mathrm{KD}}$ ; student unscaled (see § A.3 )
Key contribution	Independent weighting of TCKD/NCKD	Time-varying schedule with negative $\beta$ for structured repulsion

Appendix

Table S2: Comparison of DARK with Zhao et al.’s DKD Zhao et al. [2022] . Both methods decompose the KD loss into target and non-target components, but differ in task formulation, weighting scheme, and, critically, the sign of the non-target weight.

Figure S6: Loss component analysis for representative decay and repulsive schedules. When the scheduled KD weight crosses zero (vertical dashed line), the effective KD contribution inverts. Moderate repulsion remains well behaved, whereas stronger repulsion produces substantially larger negative KD magnitude. The comparison clarifies the operating range in which repulsive KD remains stable.

Figure S7: Spectral analysis of learned embeddings (Planes DB, 5-plane, 8,187 images). (A) Effective dimensionality

d_{\text{eff}}

(participation ratio) across representative KD methods. (B) Singular value spectra for Static KD and a moderate repulsive run. (C)

d_{\text{eff}}

plotted against composite task performance, illustrating that spectral richness alone does not fully determine zero-shot quality.

Model	6-View (F1)	Brain (F1)	CHD (AUROC)
CLIP (ViT-L/14)	.867 (.866–.869)	.634 (.632–.637)	.679 (.650–.708)
BiomedCLIP (ViT-B/16)	.856 (.855–.858)	.582 (.577–.588)	.643 (.622–.665)
UniMed-CLIP (ViT-B/16)	.860 (.858–.861)	.607 (.603–.610)	.718 (.702–.734)
FetalCLIP (ViT-L/14)	.947 (.947–.948)	.820 (.818–.822)	.787 (.770–.804)
MobileFetalCLIP (FastViT)	.930 (.930–.931)	.799 (.797–.800)	.769 (.758–.779)
Retention	98.2%	97.4%	97.7%

Appendix

Table S3: Linear probing evaluation. Frozen encoder features + single linear layer. 95% CIs from 5-fold

\times

5 seeds. MobileFetalCLIP retains 97–98% of the teacher’s performance at 26

\times

fewer visual parameters.

Seed	F1 5Plane	F1 3Brain	VR HC18	Composite Score
42	0.946	0.784	0.886	0.886
123	0.945	0.656	0.882	0.860
7	0.933	0.764	0.845	0.857
Mean $\pm$ Std	$0.941\pm 0.007$	$0.735\pm 0.069$	$0.871\pm 0.022$	$0.867\pm 0.016$

Appendix

Table S4: Seed robustness for DARK (

\beta_{\mathrm{start}}{=}2

\beta_{\mathrm{stop}}{=}{-}0.8

). All three seeds surpass the Static KD baseline (Composite Score = 0.826) and the FetalCLIP teacher (Composite Score = 0.853).

VLM Distillation

Vision-language model distillation transfers multimodal capabilities to smaller, faster models for efficient deployment.

See all 126 papers

DVTR-KD: Distilling Vision-Language Models via Intermediate Visual Token Reconstruction
Oct 13, 2026Haolan Wei, Jiahui Wang, Junli Zha +4VLM Distillation Efficient VLM Inference
SF Technology Co., Ltd, Shenzhen, China
On-Policy Distillation for Vision-Language Model Adaptation, an Effective Paradigm on Low-Quality Multimodal Data
Sep 9, 2026Hongyuan Zhang, Xianda Guo, Yanlun Peng +6VLM Adaptation VLM Distillation
The University of Hong Kong, Hong Kong SAR, China · Great Wall Motor, China · School of Information Science and Technology, University of Science and Technology of China, China +4
What Do Students Learn? A Feature-Level Analysis of Dark Knowledge
Jun 2, 2026Seungu Kang, Songkuk KimRepresentation Learning Self-Distillation
Yonsei University, Seoul 03722, Republic of Korea

Oct 13, 2026J/K move · Enter open · S save

DVTR-KD: Distilling Vision-Language Models via Intermediate Visual Token Reconstruction

Haolan Wei, Jiahui Wang, Junli Zha, Xinkai Lu+3

SF Technology Co., Ltd, Shenzhen, China

VLM Distillation Efficient VLM Inference

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have achieved strong performance on diverse vision-language tasks, but their computational demands limit deployment under constrained budgets, particularly on edge devices. Many works focus on Knowledge Distillation (KD) and try to transfer knowledge from powerful teacher models to students. In addition to reducing model size, visual token compression provides a complementary route to efficiency, but introduces mismatched visual token counts that complicate direct token-wise knowledge transfer. To mitigate this, we propose DVTR-KD, a knowledge distillation framework based on Dense Visual Token Reconstruction, which reconstructs the teacher's dense visual tokens from the student's compressed tokens. We reconstruct visual states after the final LLM layer and at the input to an intermediate layer identified as important for visual-information readout. At the intermediate layer, reconstruction is supervised through the teacher's response predictions rather than by direct token-wise KL or MSE matching. An auxiliary attention-readout loss complements this response-level supervision by aligning how the student and the original teacher integrate contextual information when predicting responses. Evaluations on diverse benchmarks demonstrate that DVTR-KD improves the performance of token-compressed students while retaining the inference efficiency afforded by visual token compression.

Abstract

VLM Distillation

Explore similar work

DVTR-KD: Distilling Vision-Language Models via Intermediate Visual Token Reconstruction

On-Policy Distillation for Vision-Language Model Adaptation, an Effective Paradigm on Low-Quality Multimodal Data

What Do Students Learn? A Feature-Level Analysis of Dark Knowledge