Language Model Distillation

Follow this topic Ask about this topic

Momentum

69 papers in the last four weeks, up 200% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 350

All topics

PRIDE: Privileged Information-enhanced Distillation for Empathetic Dialogue Generation
Jun 22, 2026Jiaqiang Wu, Zhouan Zhu, Shangfei WangLanguage Model Distillation Learning Using Privileged Information
Anhui Robot Technology Standard Innovation Base, School of Computer Science and Technology, University of Science and Technology of China, Hefei, Anhui 230027, PR China
ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillation
Jun 22, 2026Chen Lin, Kedi Chen, Wei ZhangCoT Distillation Language Model Distillation
East China Normal University · Shanghai Innovation Institute
Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails
Jun 22, 2026Xin Liu, Simin Ma, Shujian Liu +5Data-Free Knowledge Distillation Language Model Distillation
University of Michigan · 2Zoom Video Communications
A Formula-Driven Survey and Research Agenda for On-Policy Distillation
Jun 22, 2026Bowen ZhangLanguage Model Distillation On-Policy Distillation
Tsinghua University
On the Position Bias of On-Policy Distillation
Jun 21, 2026Yan Xie, Sijie Zhu, Tiansheng Wen +2Importance Weighting Language Model Distillation
Xidian University · Georgia Institute of Technology · Amazon AGI SF Lab
Context-Aware Distillation and Ablation for Text2DSL
Jun 21, 2026Alexander V. Kozachok, Alexander M. Nazimov, Shamil G. MagomedovLLM Prompting LLM-Based Program Synthesis
MIREA – Russian Technological University, Moscow, Russian Federation · Academy of the Federal Guard Service of the Russian Federation, Oryol, Russian Federation
Weight-Space Geometry of Offline Reasoning Training
Jun 21, 2026Aleksandr Nikolich, Igor Kiselev, Vladimir Platonov +1RL for Language Models Direct Preference Optimization
1Keenable AI · 2Accenture · 3Yandex
Channel Location Constrains the Auditability of Subliminal Learning
Jun 20, 2026Tamas MadlSubliminal Learning Language Model Distillation
Austrian Research Institute for Artificial Intelligence · University of Vienna
Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts
Jun 20, 2026Qingfei Zhao, Huan Song, Shuyu Tian +2Trajectory Generation CoT Distillation
TeleAI · Shanghai Jiao Tong University
TALAS: Teacher-Anchored Layer Alignment with Adaptive Sharpness-Aware Minimization for Embedding Distillation
Jun 20, 2026Quoc Phong Dao, Hoang Son Nguyen, Pham Khanh Chi +4LLM Compression Language Model Distillation
Hanoi University of Science and Technology · VNU University of Engineering and Technology · University of Oregon +1
Masked Self-Distillation: Internalizing the Chain-of-Thought in Language Models
Jun 18, 2026Durgesh Kalwar, Vardhan Palod, Jaya Adithya Pavuluri +1CoT Distillation Efficient Language Model Reasoning
SCAI, Arizona State University
Marginal Advantage Accumulation for Memory-Driven Agent Self-Evolution
Jun 18, 2026Mingyu Yang, Keye Zheng, Congchao Cheng +4Language Model Distillation Knowledge Distillation
Alibaba International Digital Commerce Group
Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation
Jun 17, 2026Siyi Gu, Jialin Chen, Sophia Zhou +2RL for Language Model Reasoning Language Model Distillation
Yale University
Efficient Financial Language Understanding via Distillation with Synthetic Data
Jun 17, 2026Wen-Fong, Huang, Edwin SimpsonFinancial Sentiment Analysis Language Model Distillation
Xavier · School of Engineering Mathematics and Technology, University of Bristol
Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation
Jun 15, 2026Zhongzhu Zhou, Qingyang Wu, Junxiong Wang +4Neural Network Initialization Linear Attention
1The University of Sydney · 2Together AI · 5Microsoft +2
The Quality-Utility Paradox: Why High-Reward Data Impairs Small Model Mathematical Reasoning
Jun 15, 2026Haolong Qian, Xianliang Yang, Yinuo ma +6CoT Distillation Language Model Distillation
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Microsoft Research Asia, Microsoft, Beijing, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China +1
On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents
Jun 14, 2026Gengsheng Li, Mao Zheng, Mingyang Song +8Language Model Distillation LLM Agent Training
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Large Language Model Department, Tencent +4
SHARD: Safe and Helpful Alignment via Self-Reframing Distillation
Jun 14, 2026Viswonathan Manoranjan, Amogh Gupta, Anvesh Rao Vijjini +2LLM Safety Alignment Language Model Distillation
UNC Chapel Hill
Be My Tutor: On-Policy Co-Distillation for Mutual LLM Improvement via Peer Feedback
Jun 12, 2026Woohyeon Byeon, Jiwon Jeon, Jeonghye Kim +1Language Model Distillation On-Policy Distillation
KAIST
Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers
Jun 10, 2026MohammadHossein Rezaei, Anas Mahmoud, Zihao Wang +6Rubric-Based RL Language Model Distillation
Scale AI
Closing the Social-Semantic Gap: SPSD for Edge-Based Prompt Compression in Cloud LLM Inference
Jun 10, 2026Abhinit Sen, Ajeet Kumar, Manaranjan PradhanLLM Inference Efficiency On-Device Language Model Inference
Indian School of Business, Hyderabad, India
RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation
Jun 10, 2026Leyi Pan, Shuchang Tao, Yunpeng Zhai +5RL for Language Model Reasoning Language Model Distillation
1Tsinghua University · 2Tongyi Lab , Alibaba Group · 3Peking University
The Role of Feedback Alignment in Self-Distillation
Jun 9, 2026Semih Kara, Oğuzhan ErsoyRL for Language Model Reasoning Language Model Distillation
Gensyn
AuRA: Internalizing Audio Understanding into LLMs as LoRA
Jun 9, 2026Bo Cheng, Lei Shi, Zhanyu Ma +5Spoken Language Understanding Audio Understanding
Meituan · Jilin University
K-Forcing: Joint Next-K-Token Decoding via Push-Forward Language Modeling
Jun 9, 2026Zhiwei Tang, Yuanyu He, Yizheng Han +4LLM Serving Language Model Decoding
1DAMO Academy, Alibaba Group · 2Hupan Lab · 3Zhejiang University +1
ParaBridge: Bridging Paralinguistic Perception and Dialogue Behavior in Speech Language Models
Jun 9, 2026Yuxiang Wang, Qinke Ni, Shengbo Cai +3On-Policy Self-Distillation Spoken Dialogue Systems
1The Chinese University of Hong Kong, Shenzhen · 2Tencent Hunyuan · 5Tsinghua University +2
Quantifying Subliminal Behavioral Transfer Ratios in Language Model Distillation
Jun 9, 2026Uwe König, Hamza Kazmi, Ruizhe Li +1Subliminal Learning Language Model Distillation
1Technical University of Munich · 2Algoverse · University of Aberdeen +1
Beyond Absolute Imitation: Anchored Residual Guidance for Privileged On-Policy Distillation
Jun 9, 2026Wenhao ZhangCoT Distillation Language Model Distillation
South China University of Technology
Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifacts
Jun 9, 2026Haoyu DongRL for Code Generation Code Generation
Microsoft
Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families
Jun 8, 2026Yifan Niu, Han Xiao, Dongyi Liu +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
The Hong Kong University of Science and Technology (Guangzhou) · Tencent · The Hong Kong University of Science and Technology

Jun 22, 2026, cs.CLJ/K move · Enter open · S save

PRIDE: Privileged Information-enhanced Distillation for Empathetic Dialogue Generation

Jiaqiang Wu, Zhouan Zhu, Shangfei Wang

Anhui Robot Technology Standard Innovation Base, School of Computer Science and Technology, University of Science and Technology of China, Hefei, Anhui 230027, PR China

Language Model Distillation Learning Using Privileged Information Knowledge Distillation Emotion-Aware Dialogue Generation

Open paper AskPDF arXiv

Large language models have demonstrated significant capabilities in generating diverse and context-aware responses for empathetic dialogue. However, their computational demands severely limit their deployment in resource-constrained environments. While knowledge distillation offers a promising compression solution, it often fails to transfer the nuanced understanding essential for empathy, as it overlooks the implicit contextual cues that guide human connection. To bridge this gap, we propose a \textbf{pr}ivileged \textbf{i}nformation-enhanced knowledge \textbf{d}istillation method for \textbf{e}mpathetic dialogue generation (PRIDE). Our method leverages privileged information, such as expert psychological annotations or future event summaries, which is available exclusively during training but unavailable at inference time. This allows us to transfer the teacher model's empathetic reasoning to smaller models without relying on extra inputs during deployment. Specifically, PRIDE has three key components: (1) An empathy-reasoning prompt that guides the teacher to explicitly decompose the empathetic process into understanding feelings and analyzing situations step-by-step; (2) A multi-source attention mechanism that directs the student to effectively integrate privileged information; (3) A dual-alignment loss that combines reversed Kullback-Leibler divergence and maximum mean discrepancy to ensure robust knowledge transfer at both logit and feature levels. Experiments on multi-modal and text-only datasets demonstrate that our method achieves competitive performance, and in some cases matches or even surpasses larger teacher models in terms of accuracy and semantic relevance.

Momentum

Related topics

Latest papers 350