Co Distillation

Co-distillation is a machine learning technique that enhances model performance and efficiency by mutually transferring knowledge between two models during simultaneous training. Current research focuses on applying co-distillation to diverse tasks, including depth estimation in 360° imagery, language model compression, and knowledge graph embedding, often employing transformer-based architectures and leveraging both labeled and unlabeled data. This approach offers significant advantages by improving the accuracy and speed of smaller models while also potentially boosting the performance of larger models, leading to more efficient and effective AI systems across various applications.

Papers

June 18, 2024

Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentation
Ning-Hsu Wang, Yu-Lun Liu
Depth Estimation Monocular Depth Estimation Large Depth Monocular Depth Camera Projection Co Distillation

November 6, 2023

Co-training and Co-distillation for Quality Improvement and Compression of Language Models
Hayeon Lee, Rui Hou, Jongpil Kim, Davis Liang, Hongbo Zhang, Sung Ju Hwang, Alexander Min
Language Model Knowledge Distillation Linear Compression Mutual Distillation Inference Speed Enhanced Quality Expensive Language Model Co Distillation

March 10, 2023

Robust Knowledge Distillation from RNN-T Models With Noisy Training Labels Using Full-Sum Loss
Mohammad Zeineldeen, Kartik Audhkhasi, Murali Karthick Baskar, Bhuvana Ramabhadran
Knowledge Distillation Recurrent Neural Network Transducer RNN T Model Co Distillation

August 21, 2022

I Know What You Do Not Know: Knowledge Graph Embedding via Co-distillation Learning
Yang Liu, Zequn Sun, Guangyao Li, Wei Hu
Knowledge Graph Pre Trained Language Model Entity Representation Co Distillation

Co Distillation

Papers

Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentation

Co-training and Co-distillation for Quality Improvement and Compression of Language Models

Robust Knowledge Distillation from RNN-T Models With Noisy Training Labels Using Full-Sum Loss

I Know What You Do Not Know: Knowledge Graph Embedding via Co-distillation Learning