Teacher-Student Learning

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 162

All topics

Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning
Jun 14, 2026Yu Li, Shu Hong, Tian LanRL for Language Model Reasoning Credit Assignment in RL
Department of Electrical and Computer Engineering, George Washington University
Teacher-Student Structure for Domain Adaptation in Ensemble Audio-Visual Video Deepfake Detection
Jun 13, 2026Elham Abolhasani, Maryam Ramezani, Hamid R. RabieeAudio Deepfake Detection Domain Adaptation
Department of Computer Engineering, Sharif University of Technology.
Robust Fall Recovery for Armless Bipedal-Wheeled Robots Via Force-Guided Learning
Jun 12, 2026Haidong Hou, Zhangguo Yu, Tao Han +6Robot Failure Recovery Robot Locomotion
School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China
Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation
Jun 11, 2026Guo Yu, Wenlin Liu, Yulan Hu +3Sparse Fine-Tuning On-Policy Distillation
School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University · Amap, Alibaba Group
A solvable model for unsupervised federated learning
Jun 11, 2026Giovanni Catania, Aurélien Decelle, Gianluca Manzan +2Statistical Physics of Learning Unsupervised Learning
Institute for Cross-disciplinary Physics and Complex Systems IFISC (CSIC-UIB), Campus Universitat Illes Balears, 07122 Palma de Mallorca, Spain. · Escuela Técnica Superior de Ingenieros Industriales, Universidad Politécnica de Madrid, Calle de José Gutiérrez Abascal 2, Madrid 28006, Spain. · GISC - Grupo Interdisciplinar de Sistemas Complejos 28040 Madrid, Spain. +5
Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions
Jun 10, 2026José Medina, Paul Honeine, Abdelaziz Bensrhair +1Data Augmentation Model Calibration
ITS Lab, Institute of Computer Science, University of Tartu, Estonia. · LITIS, Universit´e de Rouen, France. · LITIS, INSA de Rouen, France.
Hey Chat, Can You Teach Me? Structuring Socratic Dialogue for Human Learning in the Wild
Jun 10, 2026Sidney Tio, Arunesh Sinha, Pradeep VarakanthamIntelligent Tutoring Systems AI in Education
School of Computing and Information Systems Singapore Management University · Department of Management Science and Information Systems Rutgers Business School
RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation
Jun 10, 2026Leyi Pan, Shuchang Tao, Yunpeng Zhai +5RL for Language Model Reasoning Language Model Distillation
1Tsinghua University · 2Tongyi Lab , Alibaba Group · 3Peking University
When Context Returns: Toward Robust Internalization in On-Policy Distillation
Jun 10, 2026Xun Wang, Ruishuo Chen, Zhuoran Li +2On-Policy Distillation Context Distillation
1IIIS, Tsinghua University
Quantifying Subliminal Behavioral Transfer Ratios in Language Model Distillation
Jun 9, 2026Uwe König, Hamza Kazmi, Ruizhe Li +1Subliminal Learning Language Model Distillation
1Technical University of Munich · 2Algoverse · University of Aberdeen +1
PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning
Jun 9, 2026Xinyue Peng, Yi Qian, Jiaojiao Lin +2Mixture-of-Experts Language Models Expert Routing
Intel Corporation, China · Zhejiang University, China
Escaping the KL Agreement Trap in On-Policy Distillation
Jun 8, 2026Haoran Xin, Anhao Zhao, Ying Sun +3On-Policy Distillation Teacher-Student Learning
Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong Polytechnic University · Eastern Institute of Technology, Ningbo +1
SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling
Jun 8, 2026Haoran Xu, Hongyu Wang, Yifei Gao +3Language Model Distillation On-Policy Distillation
Zhejiang University · Hunan University · Tianjin University +2
Trajectory-Refined Distillation
Jun 7, 2026Li Jiang, Haoran Xu, Yichuan Ding +1On-Policy Distillation LLM Post-Training
McGill University · Mila Quebec AI Institute · UT Austin
OPRD: On-Policy Representation Distillation
Jun 4, 2026Shenzhi Yang, Guangcheng Zhu, Bowen Song +8Cross-Architecture Knowledge Distillation Language Model Distillation
Zhejiang University · Ant Group
Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation
Jun 4, 2026Maxime Griot, Paul Steven Scotti, Tanishq Mathew AbrahamCoT Distillation Teacher-Student Learning
Université catholique de Louvain · Sophont Inc.
ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation
Jun 4, 2026Kanghui Tian, Siyuan Liu, Ziang Yan +3VLM Distillation Multimodal Reasoning
1Shanghai AI Laboratory · 3Nanjing University · 2Fudan University
SocraticPO: Policy Optimization via Interactive Guidance
Jun 3, 2026Zirui Liu, Jie Ouyang, Qi Liu +8RL for Language Model Reasoning LLM-Guided RL
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · iFLYTEK AI Research (Central China), iFLYTEK Co., Ltd
When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation
Jun 2, 2026Haowei Guo, Baolong Bi, Ruicheng Zhang +2On-Policy Self-Distillation Language Model Distillation
Peking University · University of Chinese Academy of Sciences · Tsinghua University
What Makes Interaction Trajectories Effective for Training Terminal Agents?
Jun 2, 2026Sidi Yang, Chaofan Tao, Jierun Chen +11AI Agent Benchmarks LLM Agent Training
1The University of Hong Kong · 2Huawei Technologies · 3Nanyang Technological University
What Do Students Learn? A Feature-Level Analysis of Dark Knowledge
Jun 2, 2026Seungu Kang, Songkuk KimRepresentation Learning Self-Distillation
Yonsei University, Seoul 03722, Republic of Korea
Why Are DMD Students Lazy? Understanding the Copying Behavior in Few-Step Distillation
Jun 1, 2026Shucheng Li, Iolo Jones, Alexander Tong +1Diffusion Model Distillation Few-Step Diffusion Sampling
Department of Computer Science, University of Oxford, Oxford, UK · AITHYRA, Research Institute for Biomedical AI, Vienna, AT
OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification
May 31, 2026Yuhang Zhou, Lizhu Zhang, Yifan Wu +5RL for Language Model Reasoning Language Model Distillation
Meta AI
What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regression
May 31, 2026Wendao Wu, Fangqing Zhang, Haihan Zhang +1Weak-to-Strong Generalization Teacher-Student Learning
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University, China · School of Mathematical Sciences, Peking University, China
OPD+: Rethinking the Advantage Design for On-Policy Distillation
May 31, 2026Hanyang Zhao, Haoxian Chen, Han Lin +3Language Model Distillation Policy Gradient Methods
Columbia University · Amazon · Meta +1
Trust Functions: Near-Lossless Weak-to-Strong Generalization by Learning When to Trust the Weak Teacher
May 31, 2026Arda Uzunoglu, Alvin Zhang, Daniel KhashabiData Selection Training Data Selection
Department of Computer Science, Johns Hopkins University, Maryland, United States.
Subliminal Learning Is Steering Vector Distillation
May 31, 2026Camila Blank, Agam Bhatia, Senthooran Rajamanoharan +2Language Model Steering Subliminal Learning
Stanford University
DASH: Dual-Branch Score Distillation for Guidance-Calibrated Compact Diffusion Models
May 30, 2026Abdullah Al Shafi, Kazi Saeed Alam, Sk Imran Hossain +1Model Compression Classifier-Free Guidance
Khulna University of Engineering & Technology, Bangladesh · University Clermont Auvergne, France
Logit Distillation on Manifolds: Mapping by Learning
May 30, 2026Yiru Yang, Junling Wang, Nishant Kumar Singh +2Teacher-Student Learning Representation Alignment
University of Zurich · ETH Zurich · Deutsche Bank Securities
Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance
May 29, 2026Yuxuan Jiang, Francis FerraroCoT Distillation Language Model Distillation
University of Maryland, Baltimore County

Jun 14, 2026, cs.LGJ/K move · Enter open · S save

Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning

Yu Li, Shu Hong, Tian Lan

Department of Electrical and Computer Engineering, George Washington University

RL for Language Model Reasoning Credit Assignment in RL Step-Level Credit Assignment in RL Reinforcement Learning with Verifiable Rewards On-Policy Distillation Teacher-Student Learning+1

Open paper AskPDF arXiv

Reinforcement learning from verifiable rewards assigns a single scalar to each rollout, leaving token-level credit assignment underspecified in long reasoning traces. On-policy self-distillation addresses this by letting the same model act as a teacher conditioned on privileged information, producing a dense per-token signal. But the common choice of a ground-truth answer is only an endpoint cue: on terse-answer tasks, the teacher falls silent at the intermediate positions where path-level guidance matters most. We propose Hindsight Self-Distillation (HSD), which conditions the teacher on a successful peer rollout drawn from the current training group. Such a peer is an exact sample from the success-conditioned policy, requiring no additional sampled rollouts. By providing a full successful continuation rather than only the final answer, the resulting credit signal concentrates at the divergence position between a failed rollout and a successful peer. Across Qwen3-8B and Qwen3-32B on math and code benchmarks, HSD obtains the best result against GRPO variants and on-policy distillation baselines, with the largest gains on terse-answer tasks such as AIME.

Momentum

Related topics

Latest papers 162