Compressing vision-language models for on-device deployment is increasingly important in clinical settings, but knowledge distillation (KD) degrades sharply when the teacher-student capacity gap spans an order of magnitude or more. We argue that, under such gaps, strict imitation of the teacher is a poor objective: much of the teacher's pairwise similarity structure reflects its own architectural biases rather than information a compact student can efficiently represent. We propose Diagonal-Anchored Repulsive Knowledge Distillation (DARK), a contrastive KD framework that decomposes the distillation loss into a diagonal term (matched image-text pairs) and an off-diagonal term (non-target similarities). The diagonal term anchors matched-pair alignment throughout training; the off-diagonal term is annealed from positive to negative weighting, transitioning the student from imitating to repelling the teacher's non-target similarity structure. We instantiate DARK by distilling FetalCLIP, a 427M-parameter fetal ultrasound vision-language model, into MobileFetalCLIP, a 75M-parameter student model with a 26× smaller visual encoder, running in 1.6,ms on an iPhone16Pro. The student matches or exceeds its teacher on three zero-shot benchmarks, including HC18 biometry validity (88.6% vs.\ 83.5%) and brain sub-plane F1 (0.784 vs.\ 0.702). Embedding-geometry and logit analyses show that DARK induces structured decorrelation: the student preserves teacher-aligned per-image confidence while diverging from inherited inter-class confusion, suggesting that controlled repulsion can be more efficient than imitation under extreme compression.
Figures & tables
Figure 1: Overview of the DARK framework. (A) Distillation setup: a frozen FetalCLIP teacher (ViT-L/14, 304M visual params) produces an N×N similarity matrix; a lightweight FastViT student (11.4M visual params) is trained via LCLIP and LKD . (B) Attraction-to-repulsion dynamics: the off-diagonal weight β(t) linearly decays from positive βstart into a negative value βstop ; the weight for Ldiag term remains fixed, preserving matched pair alignment throughout training. (C) Outcome: DARK produces structured decorrelation, resulting in better cluster separation and a higher VR HC18 and F1 3Brain with 26× fewer visual parameters.
Models
Params.
F1 5Plane
F1 3Brain
F1 Pl_Br
VR HC18
Composite Score
Teacher Model
FetalCLIP (ViT-L/14)
427M
0.973
0.702
0.871
0.835
0.853
General VLMs
CLIP (ViT-L/14)
427M
0.308
0.206
0.270
0.110
0.190
BiomedCLIP (ViT-B/16)
150M
0.603
0.236
0.466
0.240
0.353
UniMed-CLIP (ViT-B/16)
150M
0.679
0.187
0.495
0.090
0.293
Table 1: Zero-shot comparison on fetal ultrasound benchmarks. Static Logit KD implements the CLIP-KD Yang et al. (2024) objective. Composite Score = (F1 Pl_Br + VR HC18 )/2.
Configurations
F1 5Plane
F1 3Brain
F1 Pl_Br
VR HC18
Composite Score
FetalCLIP (ViT-L/14) - Teacher
0.973
0.702
0.871
0.835
0.853
Using LCLIP only (without LKD )
0.889
0.712
0.823
0.713
0.768
Static KD ( LCLIP+λ⋅LKD ), λ=1
0.946
0.715
0.860
0.794
0.826
Partial KD Annealing ( λ:1→0.1 )
0.902
0.713
0.831
0.746
0.788
Full KD Annealing ( λ:1→0 )
0.842
0.742
0.805
0.731
0.768
Static KD + Feature KD
0.946
0.664
0.840
0.759
0.800
Table 2: Ablation on KD Strategies: KD strategies for MobileFetalCLIP (FastViT-MCI0, 11.4M visual encoder) distilled from FetalCLIP ViT-L/14 (304M). Composite Score = (F1 Pl_Br + VR HC18 )/2.
Table 4Figure 5
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Learning rate ( ηpeak )
1×10−5
LR schedule
Cosine decay with linear warmup (500 warmup steps; world-size adjusted)
Weight decay
0.1
Batch size
512
Gradient accumulation steps
2
Effective batch size
1,024 (typical global batch)
Appendix
Table S1: Full training hyperparameter configuration. Values in the right column apply to all runs unless noted otherwise.
Figure S1: Row-normalized confusion matrices for zero-shot 5-plane classification, with an additional “other” column capturing predictions to off-set classes (kidney, lips/nose, profile, spine). Left: Static KD ( λ=1.0 ) achieves high recall for cervix (96%), brain (95%), femur (98%), and thorax (93%), but lower abdomen recall (82%) and 9% off-class leakage. Right: DARK ( βstart=2 , βstop=−0.8 ) improves brain (99%) and abdomen (89%) with minor trade-offs on cervix, femur, and thorax; the main shift is a thorax → brain redistribution (1% → 7%).
Figure S2: Per-class F1 heatmap across fetal plane classes (abdomen, brain, femur, thorax, cervix) and brain sub-planes (transthalamic, transcerebellum, transventricular). Each row corresponds to the run’s best epoch (selected by Composite Score). DARK ( β0=2 , βstop=−0.8 ) achieves the strongest brain sub-plane discrimination.
Figure S3: Logit distribution analysis: teacher–student decorrelation comparing Static KD and DARK. Row 1 : Softmax entropy distributions (lower = more confident; dashed lines denote empirical means). Row 2 : N×N logit matrix rank correlation. Row 3 : Per-image teacher-student probability agreement ( 1−JSD ; higher = the student’s per-sample class distribution matches the teacher’s more closely; dashed lines denote empirical means).
Figure S4: Full training dynamics across all logged runs (4-panel overlay). Panels show: KD weight schedule, distill loss, learned logit scale, and the Composite Score metric over training epochs. DARK ( βstart=2 , βstop=−0.8 ) achieves the highest Composite Score, consistent with the main-paper ablation.
Figure S5: Per-run Composite Score trajectories across training epochs. Static KD plateaus early, while repulsive variants continue improving through the repulsive phase. DARK ( βstart=2 , βstop=−0.8 ) achieves the highest final composite score among the stable runs.
Aspect
DKD Zhao et al. [2022]
Ours
Task
C -class classification
N×N contrastive similarity
TCKD (diagonal)
Binary KL on [pt,1−pt]
Cross-entropy on diagonal entries ( i=j )
NCKD (off-diagonal)
KL on renormalised non-target distribution
Cross-entropy on off-diagonal entries ( i=j )
Weights
α=1 , β=8 (both positive, static)
Diagonal fixed at 1.0; β(t) follows a linear schedule that decays into negative values
Temperature
Shared τ on both teacher & student, with T2 correction
Teacher-only τKD ; student unscaled (see § A.3 )
Key contribution
Independent weighting of TCKD/NCKD
Time-varying schedule with negative β for structured repulsion
Appendix
Table S2: Comparison of DARK with Zhao et al.’s DKD Zhao et al. [2022] . Both methods decompose the KD loss into target and non-target components, but differ in task formulation, weighting scheme, and, critically, the sign of the non-target weight.
Figure S6: Loss component analysis for representative decay and repulsive schedules. When the scheduled KD weight crosses zero (vertical dashed line), the effective KD contribution inverts. Moderate repulsion remains well behaved, whereas stronger repulsion produces substantially larger negative KD magnitude. The comparison clarifies the operating range in which repulsive KD remains stable.
Figure S7: Spectral analysis of learned embeddings (Planes DB, 5-plane, 8,187 images). (A) Effective dimensionality deff (participation ratio) across representative KD methods. (B) Singular value spectra for Static KD and a moderate repulsive run. (C) deff plotted against composite task performance, illustrating that spectral richness alone does not fully determine zero-shot quality.
Model
6-View (F1)
Brain (F1)
CHD (AUROC)
CLIP (ViT-L/14)
.867 (.866–.869)
.634 (.632–.637)
.679 (.650–.708)
BiomedCLIP (ViT-B/16)
.856 (.855–.858)
.582 (.577–.588)
.643 (.622–.665)
UniMed-CLIP (ViT-B/16)
.860 (.858–.861)
.607 (.603–.610)
.718 (.702–.734)
FetalCLIP (ViT-L/14)
.947 (.947–.948)
.820 (.818–.822)
.787 (.770–.804)
MobileFetalCLIP (FastViT)
.930 (.930–.931)
.799 (.797–.800)
.769 (.758–.779)
Retention
98.2%
97.4%
97.7%
Appendix
Table S3: Linear probing evaluation. Frozen encoder features + single linear layer. 95% CIs from 5-fold × 5 seeds. MobileFetalCLIP retains 97–98% of the teacher’s performance at 26 × fewer visual parameters.
Seed
F1 5Plane
F1 3Brain
VR HC18
Composite Score
42
0.946
0.784
0.886
0.886
123
0.945
0.656
0.882
0.860
7
0.933
0.764
0.845
0.857
Mean ± Std
0.941±0.007
0.735±0.069
0.871±0.022
0.867±0.016
Appendix
Table S4: Seed robustness for DARK ( βstart=2 , βstop=−0.8 ). All three seeds surpass the Static KD baseline (Composite Score = 0.826) and the FetalCLIP teacher (Composite Score = 0.853).
Multimodal large language models (MLLMs) have achieved strong performance on diverse vision-language tasks, but their computational demands limit deployment under constrained budgets, particularly on edge devices. Many works focus on Knowledge Distillation (KD) and try to transfer knowledge from powerful teacher models to students. In addition to reducing model size, visual token compression provides a complementary route to efficiency, but introduces mismatched visual token counts that complicate direct token-wise knowledge transfer. To mitigate this, we propose DVTR-KD, a knowledge distillation framework based on Dense Visual Token Reconstruction, which reconstructs the teacher's dense visual tokens from the student's compressed tokens. We reconstruct visual states after the final LLM layer and at the input to an intermediate layer identified as important for visual-information readout. At the intermediate layer, reconstruction is supervised through the teacher's response predictions rather than by direct token-wise KL or MSE matching. An auxiliary attention-readout loss complements this response-level supervision by aligning how the student and the original teacher integrate contextual information when predicting responses. Evaluations on diverse benchmarks demonstrate that DVTR-KD improves the performance of token-compressed students while retaining the inference efficiency afforded by visual token compression.
Knowledge distillation offers an efficient route to transfer a task-adapted vision-language teacher to a compact student. The training target in current vision-language distillation methods is typically constructed from the teacher prediction and applied uniformly to all training samples, making it unreliable under class and domain shifts. In this paper, we argue that distillation target construction should be treated as a dynamic training decision rather than a fixed recipe. To this end, we propose OnPoKD, an on-policy distillation framework for vision-language model adaptation. To the best of our knowledge, OnPoKD is the first framework that applies on-policy distillation to vision-language model adaptation by learning target construction as a policy decision. OnPoKD learns a lightweight controller that constructs sample-wise adaptive targets using reliability and disagreement cues from the teacher model, student model, and zero-shot prior. Instead of relying on a fixed teacher prediction, the controller dynamically balances teacher supervision, zero-shot prior guidance, and hard-label anchoring through bounded policy actions, allowing the distillation target to adapt to varying sample reliability and training stages. The policy controller is updated with validation feedback, encouraging target construction to optimize transferability rather than merely fitting the training distribution. Since the controller is only used during training, OnPoKD can be seamlessly integrated into existing vision-language distillation pipelines while preserving the original inference architecture and test-time cost. Extensive experiments on Base-to-novel generalization and Cross-dataset transfer benchmarks show that OnPoKD consistently improves over strong vision-language distillation baselines.
Hongyuan Zhang, Xianda Guo, Yanlun Peng +6
The University of Hong Kong, Hong Kong SAR, China · Great Wall Motor, China · School of Information Science and Technology, University of Science and Technology of China, China +4
Knowledge Distillation (KD) is a powerful tool for model compression, yet the precise mechanisms by which student models acquire feature representations remain underexplored. In this work, we analyze student feature learning using the Interaction Tensor framework. Our analysis reveals that effective KD acts as a regularizer that prunes low-frequency, sample-specific features, encouraging the student to rely on a compact set of highly reusable features. Crucially, we observe that the dataset-level confusion matrix contains structural information analogous to the teacher's "Dark Knowledge." Leveraging this insight, we propose Confusion Distillation (CD), a teacher-free self-distillation method that utilizes the model's own evolving )confusion patterns as dynamic soft targets. CD achieves competitive performance on ResNet-34 and ResNet-50 for CIFAR-100, outperforming existing self-distillation methods like CS-KD and PS-KD by 1.2% while offering a computationally efficient alternative to standard KD.