Self Distillation

Self-distillation is a machine learning technique where a model learns from its own predictions, improving performance and efficiency without requiring a separate teacher model. Current research focuses on applying self-distillation to diverse tasks and model architectures, including spiking neural networks, transformers, and various deep learning models for image, point cloud, and natural language processing. This approach is particularly valuable for resource-constrained environments, enabling model compression and improved performance in scenarios with limited data or computational power, impacting fields like robotics, medical imaging, and natural language understanding.

Papers

December 10, 2023

Spatial-wise Dynamic Distillation for MLP-like Efficient Visual Fault Detection of Freight Trains
Yang Zhang, Huilin Pan, Mingying Li, An Wang, Yang Zhou, Hongliang Ren
Convolutional Neural Network Self Distillation

November 3, 2023

ProS: Facial Omni-Representation Learning via Prototype-based Self-Distillation
Xing Di, Yiyu Zheng, Xiaoming Liu, Yu Cheng
Self Distillation Synthetic Face Omni Supervised Unsupervised Face

November 2, 2023

Expanding Expressiveness of Diffusion Models with Limited Data via Self-Distillation based Fine-Tuning
Jiwan Hur, Jaehyun Choi, Gyojin Han, Dong-Jae Lee, Junmo Kim
Diffusion Model Fine Tuning Self Distillation Limited Data Expressive Speech Domain Translation

October 31, 2023

A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations
Hui Ma, Jian Wang, Hongfei Lin, Bo Zhang, Yijia Zhang, Bo Xu
Emotion Recognition Transformer Based Model Self Distillation Community Conversation Multimodal Emotion Recognition Modality Aware Transformer Emotion Recognition in Conversation Modal Representation

October 29, 2023

Towards Generalized Multi-stage Clustering: Multi-view Self-distillation
Jiatai Wang, Zhiwei Xu, Xin Wang, Tao Li
Pseudo Label Self Distillation Multi View Clustering Negative Pseudo Label Multi Stage Clustering

October 20, 2023

SILC: Improving Vision Language Pretraining with Self-Distillation
Muhammad Ferjad Naeem, Yongqin Xian, Xiaohua Zhai, Lukas Hoyer, Luc Van Gool, Federico Tombari
Vision Language Self Distillation Image Caption Open Vocabulary Detection Image Text Pretraining

October 3, 2023

Exploring Generalisability of Self-Distillation with No Labels for SAR-Based Vegetation Prediction
Laura Martínez-Ferrer, Anna Jungbluth, Joseph A. Gallego-Mejia, Matt Allen, Francisco Dorr, Freddie Kalaitzis, Raúl Ramos-Pollán
Jina Embeddings Remote Sensing Stronger Generalizability Synthetic Aperture Radar Self Distillation Self Supervised Model Label Information Small Scale Datasets

September 15, 2023

BROW: Better featuRes fOr Whole slide image based on self-distillation
Yuanfeng Wu, Shaojie Li, Zhiqiang Du, Wentao Zhu
Whole Slide Image Self Distillation Nucleus Instance Segmentation

August 28, 2023

Joint Multiple Intent Detection and Slot Filling with Supervised Contrastive Learning and Self-Distillation
Nguyen Anh Tu, Hoang Thi Thu Uyen, Tu Minh Phuong, Ngo Xuan Bach
Contrastive Learning Spoken Language Understanding Intent Detection Self Distillation Supervised Contrastive Learning Slot Filling Intent Assurance Joint Intent Detection

August 9, 2023

Self-supervised Learning of Rotation-invariant 3D Point Set Features using Transformer and its Self-distillation
Takahiko Furuya, Zhoujie Chen, Ryutarou Ohbuchi, Zhenzhong Kuang
Point Cloud Self Supervised Learning Transformer Based Self Distillation Rotation Invariant 3D Point

August 5, 2023

Self-Distillation Prototypes Network: Learning Robust Speaker Representations without Supervision
Yafeng Chen, Siqi Zheng, Hui Wang, Luyao Cheng, Qian Chen, Shiliang Zhang, Wen Wang
Self Distillation VoxCeleb Speaker Recognition Challenge Self Supervised Speaker Verification Robust Speaker Representation

July 22, 2023

Spatial Self-Distillation for Object Detection with Inaccurate Bounding Boxes
Di Wu, Pengfei Chen, Xuehui Yu, Guorong Li, Zhenjun Han, Jianbin Jiao
Object Detector Self Distillation Bounding Box Bounding Box Supervision

July 12, 2023

Towards Safe Self-Distillation of Internet-Scale Text-to-Image Diffusion Models
Sanghyun Kim, Seohyeon Jung, Balhae Kim, Moonseok Choi, Jinwoo Shin, Juho Lee
Diffusion Model Text to Image Diffusion Model Self Distillation Content Generation Harmful Content

July 11, 2023

The Staged Knowledge Distillation in Video Classification: Harmonizing Student Progress by a Complementary Weakly Supervised Framework
Chao Wang, Zheng Tang
Knowledge Distillation Weakly Supervised Self Distillation Multi Stage Video Classification Label Efficient

July 3, 2023

Efficient Visual Fault Detection for Freight Train Braking System via Heterogeneous Self Distillation in the Wild
Yang Zhang, Huilin Pan, Yang Zhou, Mingying Li, Guodong Sun
High Efficiency Wild Challenge Self Distillation Limited Fault

June 26, 2023

Federated Learning on Non-iid Data via Local and Global Distillation
Xiaolin Zheng, Senci Ying, Fei Zheng, Jianwei Yin, Longfei Zheng, Chaochao Chen, Fengqin Dong
Knowledge Distillation Self Distillation FedAvg Algorithm Non IID Data Global Distillation

June 5, 2023

Dual self-distillation of U-shaped networks for 3D medical image segmentation
Soumyanil Banerjee, Ming Dong, Carri Glide-Hurst
Self Distillation 3D Medical Image Segmentation Medical Image Segmentation Datasets U Shaped Network 3D Backbone

May 31, 2023

Adaptive Self-Distillation for Minimizing Client Drift in Heterogeneous Federated Learning
M. Yashwanth, Gaurav Kumar Nayak, Arya Singh, Yogesh Simmhan, Anirban Chakraborty
Regularization Model Self Distillation Heterogeneous Federated Learning Local Datasets Client Drift

May 20, 2023

Self-Distillation with Meta Learning for Knowledge Graph Completion
Yunshui Li, Junhao Liu, Chengming Li, Min Yang
Knowledge Graph Completion Self Distillation Dynamic Pruning

May 17, 2023

DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning
Alexander H. Liu, Heng-Jui Chang, Michael Auli, Wei-Ning Hsu, James R. Glass
Self Distillation Contextual Embeddings Self Supervised Speech Representation Learning DiNO Mix Online Clustering Representation Learner

Self Distillation

Papers

Spatial-wise Dynamic Distillation for MLP-like Efficient Visual Fault Detection of Freight Trains

ProS: Facial Omni-Representation Learning via Prototype-based Self-Distillation

Expanding Expressiveness of Diffusion Models with Limited Data via Self-Distillation based Fine-Tuning

A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations

Towards Generalized Multi-stage Clustering: Multi-view Self-distillation

SILC: Improving Vision Language Pretraining with Self-Distillation

Exploring Generalisability of Self-Distillation with No Labels for SAR-Based Vegetation Prediction

BROW: Better featuRes fOr Whole slide image based on self-distillation

Joint Multiple Intent Detection and Slot Filling with Supervised Contrastive Learning and Self-Distillation

Self-supervised Learning of Rotation-invariant 3D Point Set Features using Transformer and its Self-distillation

Self-Distillation Prototypes Network: Learning Robust Speaker Representations without Supervision

Spatial Self-Distillation for Object Detection with Inaccurate Bounding Boxes

Towards Safe Self-Distillation of Internet-Scale Text-to-Image Diffusion Models

The Staged Knowledge Distillation in Video Classification: Harmonizing Student Progress by a Complementary Weakly Supervised Framework

Efficient Visual Fault Detection for Freight Train Braking System via Heterogeneous Self Distillation in the Wild

Federated Learning on Non-iid Data via Local and Global Distillation

Dual self-distillation of U-shaped networks for 3D medical image segmentation

Adaptive Self-Distillation for Minimizing Client Drift in Heterogeneous Federated Learning

Self-Distillation with Meta Learning for Knowledge Graph Completion

DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning