Knowledge Distillation
Overview
Knowledge distillation studies how a student model can learn from a teacher’s outputs, internal representations, examples, or behavior, often to reduce inference cost or transfer capabilities. Common approaches match softened class probabilities, hidden features, attention or relational structure, and use teacher-generated data; newer language-model methods also train on student-generated rollouts. The field spans model compression, cross-architecture and multimodal transfer, and settings such as federated learning, robotics, and low-resource tasks. A central challenge is choosing supervision that fits the student and remains reliable: stronger or denser teacher signals can help, but can also cause negative transfer, forgetting, or loss of capabilities not captured by the benchmark.
Latest papers
887 papers87 papers in the last four weeks, down 49% on the four weeks before. 0.7% of all new papers.
What Survives Distillation? A Mechanistic Interpretability Study of GPT-2 and DistilGPT-2
Ground-to-Satellite Knowledge Distillation for Crop Classification under Temporal and Geographic Shift
Dynamic Merging Mutually-Distilled Multi-View Networks for RGB-Event-based Representation Learning
How Much Does the Hint Matter? Scope Conditions for Hindsight Self-Distillation in Financial Reasoning
Joint Latent Reasoning and Observation for Agentic Video Understanding
MuSID-KD: Multigranularity Knowledge Distillation for Generative Recommendation with Semantic IDs
DAC-Loop: Distillation-Aware Compression for Remote Sensing CNN
RSKD: Grounding Semantic Consistency in Driving VLMs via Distillation
Knowledge distillation across teacher architectures for compact diabetic retinopathy grading
TAD: Distilling Vision-Language-Action Policies from Teacher Trajectories Alone
PutOnPlateInScene25Main, \gad{} improves OpenVLA-OFT (7B) from to \successonce{} ( percentage points) and from to \successend{}. It trails white-box on-policy distillation (OPD) at / by / percentage points ( evaluation trajectories per checkpoint). The tested four-epoch continuation achieves the highest \gad{} result without additional teacher data. This best run uses OPD's student interaction budget and its complete-rollout batch-passes. The comparison demonstrates online VLA improvement without teacher scoring while quantifying unequal interaction and optimization budgets.