Self-Supervised Knowledge Distillation

Follow this topic Ask about this topic

Self-supervised knowledge distillation trains a student from signals produced by a teacher, often using unlabeled data or different views of the same input. Common approaches include DINO-style representation learning, predicting intermediate teacher features, and on-policy distillation where language models learn from their own generated attempts or revisions. The methods are used across vision, speech, audio, and multimodal learning, with evaluations spanning image classification and segmentation, speech recognition, and math or code reasoning. A central challenge is making the teacher signal useful rather than noisy or misleading, while balancing augmentation, model capacity, data reuse, and scaling.

Latest papers

22 papers7 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

The Locality Trade-off in Masked Patch Self-Distillation
Oct 23, 2026Xi Weng, Zhaoyu Liu, Lianyu Hu +1Self-Supervised Visual Representation Learning Vision Foundation Models
School of Computing, National University of Singapore · Nanyang Technological University
CODI: Improving Self-Supervised Vision Transformers with Cross Distillation
Oct 22, 2026Siran Dai, Qianqian Xu, Peisong Wen +3Vision Transformer Self-Supervised Visual Representation Learning
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · State Key Laboratory of AI Safety, Institute of Computing Technology, CAS +3
Data Density Scaling Laws for Image Self-Distillation
Oct 13, 2026Alvard Barseghyan, Ani Vanyan, Hakob Tamazyan +1AI Scaling Laws Self-Supervised Knowledge Distillation
YerevaNN and Yerevan State University, Yerevan, Armenia
RINO: Renormalization Group Distillation with No Labels
Oct 11, 2026Zichun Hao, Raghav Kansal, Abhijith Gandrakota +6Self-Supervised Learning High-Energy Physics
Division of Physics, Mathematics and Astronomy, California Institute of Technology, Pasadena, CA 91125 · Physics Division, Fermi National Accelerator Laboratory, Batavia, IL 60510
Recurrent Self-Improvement: Dynamic Cross-Loop On-Policy Distillation for Looped Language Models
Oct 7, 2026Yi Wang, Rui Qian, Yu Li +2Language Model Self-Improvement On-Policy Distillation
ShanghaiTech University · Fudan University · Southeast University +1
What Actually Makes Correlation-Based SSL Distillation Noise-Robust? A Mechanistic Correction
Oct 2, 2026Fabian Ritter-Gutierrez, Nancy F. Chen, Eng Siong ChngSelf-Supervised Knowledge Distillation Knowledge Distillation
Nanyang Technological University, Singapore · Institute for Infocomm Research (I2R), Singapore
Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
Sep 29, 2026Md. Ismail Hossain, Humaira Kousar, Isidora Chara TourniOn-Policy Self-Distillation Mathematical Reasoning
North South University · KAIST · Andria Labs
DINO Eats Geometry: Self-Supervised Sign Representations that Absorb Articulated Hand Geometry
Sep 5, 2026Junyi Hu, Zhewen He, Haomian Huang +4Self-Supervised Learning Sign Language Recognition
New York University Abu Dhabi
Biomechanical 3D Body: Self-Supervised Distillation of Biomechanical Pose from a 3D Body Foundation Model
Aug 30, 2026R. James Cotton, J. D. Peiffer, Lucinda Williamson +2Markerless Motion Capture Human Mesh Recovery
Shirley Ryan AbilityLab, Chicago, IL, USA · Biomedical Engineering, Northwestern University, Evanston, IL, USA · Biomedical Engineering, University of Illinois Chicago, Chicago, IL, USA +1
Knowing but Not Saying: Preventing Factual Access Failures in LLM SFT via Recall-Anchored Distillation
Aug 21, 2026Haodong Chen, Yadong Wang, Shengtao Wen +2Factual Knowledge in Language Models Domain Adaptation for LLMs
MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics
Self-Supervised Visual On-Policy Distillation
Aug 14, 2026Yijiang Li, Yijun Liang, Yunjie Tian +6On-Policy Distillation VLM Distillation
UC San Diego · University of Maryland, College Park · Independent Researcher +3
SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Features
Aug 11, 2026HyeonJun Lee, Hyeonsik Jo, Jinwoo Chung +1Quantization-Aware Training Quantization-Aware Distillation
Kookmin University
When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation
Aug 4, 2026Yinuo Jiang, Yongjie Ye, Zhou Tao +4VLM Distillation Language Model Distillation
Zhejiang University · ByteDance · Shanghai Artificial Intelligence Laboratory
Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation
Jul 1, 2026Jaehyun Jang, Eunseop Yoon, Hee Suk Yoon +3Cross-Modal Knowledge Distillation VLM Distillation
Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea · University of Illinois Urbana-Champaign, Champaign, USA
Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions
Jun 10, 2026José Medina, Paul Honeine, Abdelaziz Bensrhair +1Data Augmentation Model Calibration
ITS Lab, Institute of Computer Science, University of Tartu, Estonia. · LITIS, Universit´e de Rouen, France. · LITIS, INSA de Rouen, France.
Forgetting Has Neighbors: Localized Collateral Forgetting in Machine Unlearning
May 29, 2026Polina Dolgova, Sebastian U. StichSelf-Supervised Knowledge Distillation Machine Unlearning
CISPA Helmholtz Center for Information Security · Universität des Saarlandes · Saarbrücken, Germany
S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Models
Apr 27, 2026Mohammed Ali El Adlouni, Aurian Quelennec, Pierre Chouteau +2Model Compression Audio Representation Learning
LTCI, T´el´ecom Paris, Institut Polytechnique de Paris, Palaiseau, France
Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision
Apr 13, 2026Yinghui He, Simran Kaur, Adithya Bhaskar +7Language Model Self-Improvement Self-Supervised Knowledge Distillation
Princeton University · University of Toronto · Carnegie Mellon University
HARNESS: Lightweight Distilled Arabic Speech Foundation Models
Mar 31, 2026Vrunda N. Sukhadia, Shammur Absar ChowdhurySpeech Foundation Models Speech Processing
Amazon India · Qatar Computing Research Institute, HBKU, Qatar
Self-Distillation of Hidden Layers for Self-Supervised Representation Learning
Mar 16, 2026Scott C. Lowe, Anthony Fuller, Sageev Oore +2Self-Supervised Visual Representation Learning Predictive Representation Learning
Vector Institute · Carleton University · Dalhousie University +2
MAD: Microenvironment-Aware Distillation -- A Pretraining Strategy for Virtual Spatial Omics from Microscopy
Mar 11, 2026Jiashu Han, Kunzan Liu, Yeojin Kim +2Spatial Gene Expression Prediction Self-Supervised Knowledge Distillation
Research Laboratory of Electronics, Massachusetts Institute of Technology · Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology · The Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology +1
SlotMatch: Distilling Object-Centric Representations for Unsupervised Video Segmentation
Aug 5, 2025Diana-Nicoleta Grigore, Neelu Madan, Andreas Mogelmose +2Video Object Segmentation Video Object-Centric Learning
University of Bucharest, Romania · Aalborg University, Denmark

Oct 23, 2026J/K move · Enter open · S save

The Locality Trade-off in Masked Patch Self-Distillation

Xi Weng, Zhaoyu Liu, Lianyu Hu, Jin Song Dong

School of Computing, National University of Singapore · Nanyang Technological University

Self-Supervised Visual Representation Learning Vision Foundation Models Self-Supervised Knowledge Distillation

Open paper AskPDF arXiv

Masked patch self-distillation is a key component of modern vision foundation models, including DINOv2 and DINOv3, yet its implications for local visual information remain unclear. We show that its semantic benefits can coexist with a scale-dependent locality cost. Using two annotation-free diagnostics, we distinguish recoverable local appearance from spatial correspondence. In smaller models, adding iBOT substantially weakens patch correspondence while largely preserving appearance decodability. In larger models, correspondence improves, but pixel and texture evidence becomes less linearly accessible in deep features. Thus, scaling changes the form of the locality cost rather than uniformly eliminating it. Our analysis of Gram anchoring further shows that preserving patch relations does not guarantee simultaneous improvements in both properties. These findings motivate evaluating and designing visual representations around the complementary demands of semantic abstraction, appearance preservation, and spatial correspondence. Our code is available at https://github.com/winci-ai/jea.