Teacher-Student Learning

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 164

All topics

Logit Distillation on Manifolds: Mapping by Learning
May 30, 2026Yiru Yang, Junling Wang, Nishant Kumar Singh +2Teacher-Student Learning Representation Alignment
University of Zurich · ETH Zurich · Deutsche Bank Securities
Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance
May 29, 2026Yuxuan Jiang, Francis FerraroCoT Distillation Language Model Distillation
University of Maryland, Baltimore County
CAST: Non-Privileged Clipped Asymmetric Self-Teaching with Advantage Flipping for GRPO
May 29, 2026Yang Li, Gongle Xue, Yijia Guo +3RL for Language Model Reasoning Group Relative Policy Optimization
School of Software and Microelectronics, Peking University, Beijing · School of Artificial Intelligence, Peking University, Beijing · School of Computer Science, Peking University, Beijing +1
Student Capacity Moderates Knowledge Distillation Effectiveness: A Systematic Study Across ResNet Teacher-Student Pairs on CIFAR-10
May 29, 2026Umut Onur YasarTeacher-Student Learning Feature-Based Knowledge Distillation
Trust-Region Behavior Blending for On-Policy Distillation
May 29, 2026Daniil Plyusov, Alexey Gorbatovski, Alexey Malakhov +4Language Model Distillation Trust-Region Optimization
T-Tech
Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation
May 29, 2026Yanjiang Liu, Jie Lou, Xinyan Guan +7CoT Distillation Language Model Distillation
University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu
LARK: Learnability-Grounded Trajectory Selection for Efficient Reasoning Distillation
May 28, 2026Tianrun Yu, Kaixiang Zhao, Chih-Chun Chen +5CoT Distillation Language Model Distillation
Brigham Young University · The Pennsylvania State University · University of North Carolina at Chapel Hill
Bounded Behavioral Indistinguishability for Black-Box LLM Distillation
May 28, 2026Munawar HasanLLM Evaluation Language Model Distillation
Michigan Technological University, Houghton, MI, USA
Same Evidence, Different Answers: Canonical-Context On-Policy Distillation for Multi-Turn Language Models
May 28, 2026Zizhuo Lin, Quanling Liu, Jinsheng Quan +6LLM Grounding Reasoning Consistency in Language Models
Zhejiang University · University of Science and Technology of China
AgentSchool: An LLM-Powered Multi-Agent Simulation for Education
May 28, 2026Yulei Ye, Wenhao Li, Zhong Wen +23Multi-Agent LLM Systems Human Behavior Simulation
Shanghai Institute of AI for Education, East China Normal University · School of Computer Science and Technology, Tongji University · School of Computer Science and Technology, East China Normal University +3
Surfacing Isolated Learners with Outcome-Independent Mediation of Feedback between Teachers and Students Using AI
May 28, 2026Junsoo Park, Youssef Medhat, Htet Phyo Wai +2Explainable Artificial Intelligence AI in Education
Georgia Institute Of Technology, North Avenue, Atlanta, GA 30332, USA
Skill-Conditioned Gated Self-Distillation for LLM Reasoning
May 27, 2026Jiazhen Huang, Xiao Chen, Xiao Luo +3Mathematical Reasoning Self-Distillation
Tsinghua University · University of Wisconsin-Madison · X-Humanoid +2
ADWIN: Adaptive Windows for Horizon-Aware On-Policy Distillation
May 27, 2026Kun Liang, Chenming Tang, Clive Bai +3Language Model Distillation On-Policy Distillation
School of Computer Science, Peking University · 2National Key Laboratory for Multimedia Information Processing, Peking University · 3LLM Department, Tencent
Teacher-Student Representational Alignment for Reinforcement Learning-Driven Imitation Learning
May 27, 2026Meraj Mammadov, Pedro Zuidberg Dos Martires, Johannes Andreas StorkImitation Learning Robot Imitation Learning
Department of Computer Science Örebro University
Better Accuracies, Worse Reasoning: A Step-Level Audit of Medical Chain-of-Thought Distillation
May 27, 2026Zhaoyang Jiang, Xuanqi Peng, Fei Teng +5CoT Distillation CoT Evaluation
School of Health & Wellbeing, University of Glasgow, Glasgow, UK · Department of Respiratory and Critical Care Medicine, Shanghai Sixth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China · School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China +1
Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation
May 26, 2026Tianlei Chen, Jiao Ou, Ziyuan Liu +3Language Model Distillation Multi-Teacher Knowledge Distillation
Kuaishou Technology, Beijing, China
Less is More: Early Stopping Rollout for On-Policy Distillation
May 26, 2026Zhou Ziheng, Jiaqi Li, Huacong Tang +2Language Model Distillation On-Policy Distillation
University of California, Los Angeles · Beijing Institute of General Artificial Intelligence
When the Strongest Teacher Is Not the Best Teacher: Student-Centric Answer Selection
May 26, 2026Zhengyu Hu, Zheyuan Xiao, Linxin Song +10Language Model Distillation LLM Post-Training
University of Washington · University of Texas at Austin · University of Southern California +6
Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
May 26, 2026Yuanyi Wang, Su Lu, Yanggan Gu +6Selective Knowledge Distillation Language Model Distillation
The Hong Kong Polytechnic University, PolyU · Hong Kong Polytechnic University Daya Bay Technology and Innovation Research Institute
On-Policy Adversarial Flow Distillation for Autoregressive Video Generation
May 25, 2026Yang Luo, Shengju Qian, Xiaohang Tang +4Autoregressive Video Generation On-Policy Distillation
National Univesity of Singapore · LIGHTSPEED · University College London
Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap
May 23, 2026Tianlang Chen, Shirley Wu, Jure LeskovecLanguage Model Distillation Teacher-Student Learning
Stanford
Strong Teacher Not Needed? On Distillation in LLM Pretraining
May 22, 2026Taiming Lu, Zhuang LiuLanguage Model Pretraining Language Model Distillation
Princeton University
Learning Through Noise: Why Subliminal Learning Works and When It Fails
May 22, 2026Vincent C. Brockers, Roman D. Ventzke, Valentin Neuhaus +2Subliminal Learning Cross-Architecture Knowledge Distillation
Max Planck Institute for Dynamics and Self-Organization, Göttingen · Faculty of Physics, Institute for the Dynamics of Complex Systems, University of Göttingen
EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation
May 22, 2026Aristotelis Lazaridis, Dylan Bates, Aman Sharma +3On-Policy Self-Distillation On-Policy Distillation
EdgeRunner AI
Coupled Training with Privileged Information and Unlabeled Data
May 22, 2026Jiahao Shi, Omar Hagrass, Jason M. KlusowskiLearning Using Privileged Information Semi-Supervised Learning
Department of Electrical and Computer Engineering, Princeton University · Department of Operations Research and Financial Engineering, Princeton University
A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification
May 21, 2026Marcel Kühn, Yoon Thelge, Bernd RosenowStatistical Physics of Learning Multiclass Classification
Institute for Theoretical Physics, Leipzig University · 2ScaDS.AI Dresden/Leipzig
Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning
May 21, 2026Hongbin Zhang, Chaozheng Wang, Kehai Chen +4On-Policy Self-Distillation Mathematical Reasoning
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China · Keeta AI, Meituan, Beijing, China · Peng Cheng Laboratory, Shenzhen, China
Toward Understanding Adversarial Distillation: Why Robust Teachers Fail
May 21, 2026Hongsin Lee, Hye Won ChungAdversarial Training Neural Network Robustness
School of Electrical Engineering, KAIST, Daejeon, Korea.
Visual-Advantage On-Policy Distillation for Vision-Language Models
May 21, 2026Ruiqi Liu, Xiaolei Lv, Gengsheng Li +8Vision-Language Models VLM Distillation
Institute of Automation, CAS · School of Advanced Interdisciplinary Sciences, UCAS · 3Hello +1
X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
May 20, 2026Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Mingyu Yang +4Cross-Tokenizer Knowledge Distillation Language Model Distillation

May 30, 2026, cs.LGJ/K move · Enter open · S save

Logit Distillation on Manifolds: Mapping by Learning

Yiru Yang, Junling Wang, Nishant Kumar Singh, Luohong Wu+1

University of Zurich · ETH Zurich · Deutsche Bank Securities

Teacher-Student Learning Representation Alignment

Open paper AskPDF arXiv

A simple way to improve the performance of almost any machine learning model is not to train a single but several models with diverse algorithms which will make slightly distinct kinds of predictions and errors on the same data, and thus improve the average predictions and robustness. However, making predictions using a whole ensemble of models is cumbersome and computationally too expensive to allow deployment to a large number of users, especially if the models are large neural nets. In response to this, we introduce a layer and point wise projection mapping, which maps student and teacher representations into an aligned high-dimensional embedding space during training process. The proposed approach combined with LoRA injection reduces the student model trainable parameters to less than 1% of the teacher model, while significantly improving word error rate (WER) compared to other distillation methods, as demonstrated in ablation studies. Unlike a mixture of experts, our method can be trained rapidly and in parallel.

Momentum

Related topics

Latest papers 164