Overview

On-policy distillation studies how to transfer a teacher’s capabilities by training a student on trajectories the student generates, rather than only on fixed teacher-written examples. Common approaches match teacher and student token distributions with KL-style objectives, often combining this dense feedback with verifier rewards, reinforcement learning, or feedback from successful peer attempts. Work spans language-model reasoning and tool-using agents as well as multimodal, speech, diffusion, and embodied models; methods increasingly select useful tokens, turns, or rollout lengths to reduce compute. A central tension is that student-generated states can make supervision more relevant, but teacher signals may be unreliable or mismatched, and repeated rollouts can be costly or destabilize training. 4 sentences total.

Latest papers

CardsList

557 papers70 papers in the last four weeks, down 63% on the four weeks before. 0.5% of all new papers.

  1. Recovering Reliable Supervision via Critique-Guided On-Policy Distillation

    Oct 14, 2026Kaixiang Zhao, Tianrun Yu, Shangzhe Li +4On-Policy DistillationKnowledge Distillation

  2. ECHOES: Seeing Beyond the Current Image via Recursive History Compression and Privileged On-Policy Distillation

    Oct 14, 2026Tengda Guo, Jiawang Hu, Yifei Tao +4Long-Horizon Agent TasksMemory-Augmented Agents

  3. RM-OPD: On-Policy Distillation for Generative Reward Models

    Oct 14, 2026Qinglin Su, Licheng Pan, Zhen Fang +5Reward ModelingRL for Generative Models

  4. ActionOPD: Distilling Decisions, Not Tokens

    Oct 14, 2026Huining Yuan, Xiangmin Yi, Zelai Xu +5LLM Agent TrainingKnowledge Distillation

  5. Function-Space Temporal Self-Distillation for Policy Optimization

    Oct 14, 2026Aryan Das, Arnab Bhattacharya, Vinay Kumar VermaRL for Language Model ReasoningSelf-Distillation

  6. Divergence-Routed On-Policy Distillation for Language Models

    Oct 14, 2026Tiantian Peng, Guannan Lv, Hongyang Li +5On-Policy DistillationKnowledge Distillation

  7. SERO: Learning from Executable Oracle Continuations for Long-Horizon Agentic Reinforcement Learning

    Oct 14, 2026Dingyi Rong, Yue Shi, Chaofan Ma +5Reinforcement LearningAgentic RL

  8. DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision

    Oct 13, 2026Qijia He, Zequn Li, Spencer Luo +6Long-Horizon ReasoningOn-Policy Distillation

  9. Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?

    Oct 13, 2026Yiping Li, Fan Zhao, Zhibo Hou +2On-Policy DistillationTeacher-Student Learning

  10. Simplifying and Improving On-Policy Distillation for Few-Step Diffusion

    Oct 13, 2026Jiawei Zhang, Ziyu Wan, Nanye Ma +5Diffusion Model DistillationDiffusion Distillation

  11. Correcting or Continuing? What On-Policy Distillation Learns from Student Errors

    Oct 13, 2026Shuo Yang, Zheyu Zhang, Mahito Sugiyama +1Language Model DistillationOn-Policy Distillation

  12. RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning

    Oct 13, 2026Kunwang Zheng, Xinyu Sun, Yu Cao +7Adaptive Knowledge DistillationRubric-Based RL

  13. JustOPD: On-Policy Distillation from Autoregressive to Diffusion Language Models

    Oct 12, 2026Zikun Zhang, Jiayuan Sheng, David D. Yao +1Diffusion Language Model DistillationOn-Policy Distillation

  14. LLM Value Functions for Credit Assignment in Agentic RL

    Oct 11, 2026Kaijie Zhu, Weixiang Yan, Omar Attia +3RL for LLM AgentsAgentic RL

  15. VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoning

    Oct 10, 2026Jiahe FengVLM DistillationVisually Grounded Reasoning

  16. Reward-Aligned Distillation

    Oct 10, 2026Yilan Chen, Zian Su, Manish Prajapat +4RL for Knowledge DistillationKnowledge Distillation

  17. Branching On-Policy Distillation

    Oct 9, 2026Dongyang Chen, Shudong Liu, Shubo Li +4On-Policy DistillationLanguage Model Distillation

  18. CV-OPD: Contrastive Vision On-policy Distillation

    Oct 9, 2026Haowei Wang, Hao Lin, Qi Zhu +4contrastive knowledge distillationVLM Distillation

  19. CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation

    Oct 9, 2026Jingru Yang, Jinhang Wu, Minjing Shi +4Cross-Tokenizer Knowledge DistillationLanguage Model Distillation

  20. State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents

    Oct 9, 2026Zhentao Zhang, Yanzhao Zheng, Yanlin Liu +9On-Policy Distillation

  21. Rethinking Relative Policy Differences Transfer of On-Policy Distillation

    Oct 9, 2026Rong Wu, Daocheng Fu, Yu Yang +9On-Policy Distillation

  22. MetaOPD: Meta-Learned Token Weighting for On-Policy Distillation

    Oct 8, 2026Zipeng Wang, Xinpeng Dong, Yuefan Wang +6On-Policy DistillationAdaptive Loss Weighting

  23. DIAL-OPD: Learning More from Fewer Tokens in On-Policy Distillation

    Oct 8, 2026Anhao Zhao, Haoran Xin, Junlong Tong +5Language Model DistillationOn-Policy Distillation

  24. Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards

    Oct 8, 2026Xiaobing Chen, Zhiqi PangRL for Language Model ReasoningRL Post-Training