CoT Distillation

CoT: Chain-of-Thought

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 57

All topics

Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance
May 29, 2026Yuxuan Jiang, Francis FerraroCoT Distillation Language Model Distillation
University of Maryland, Baltimore County
Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation
May 29, 2026Yanjiang Liu, Jie Lou, Xinyan Guan +7CoT Distillation Language Model Distillation
University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu
LARK: Learnability-Grounded Trajectory Selection for Efficient Reasoning Distillation
May 28, 2026Tianrun Yu, Kaixiang Zhao, Chih-Chun Chen +5CoT Distillation Language Model Distillation
Brigham Young University · The Pennsylvania State University · University of North Carolina at Chapel Hill
Better Accuracies, Worse Reasoning: A Step-Level Audit of Medical Chain-of-Thought Distillation
May 27, 2026Zhaoyang Jiang, Xuanqi Peng, Fei Teng +5CoT Distillation CoT Evaluation
School of Health & Wellbeing, University of Glasgow, Glasgow, UK · Department of Respiratory and Critical Care Medicine, Shanghai Sixth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China · School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China +1
Generative Spatiotemporal Intent Sequence Recommendation via Implicit Reasoning in Amap
May 27, 2026Sicong Wang, Ruiting Dong, Yue Liu +7Spatiotemporal Reasoning CoT Distillation
AMAP, Alibaba Group, Beijing, China
When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning
May 20, 2026Xiaogeng Liu, Xinyan Wang, Yingzi Ma +2CoT Distillation On-Policy Self-Distillation
1Johns Hopkins University · University of Wisconsin–Madison · 3Nanyang Technological University
Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation
May 19, 2026Bing Wang, Shaotian Yan, Chen Shen +7CoT Distillation Language Model Distillation
College of Computer Science and Technology, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering, MoE, Jilin University · Tongyi Lab, Alibaba Group +3
Distribution Corrected Offline Data Distillation for Large Language Models
May 13, 2026Yumeng Zhang, Zhengbang Yang, Yevin Nikhel Goonatilake +1CoT Distillation Efficient Language Model Reasoning
George Mason University, Fairfax, VA, USA
GateKD: Confidence-Gated Closed-Loop Distillation for Robust Reasoning
May 13, 2026Kasidit Sermsri, Teerapong PanboonyuenCoT Distillation Language Model Distillation
†Chulalongkorn University · ‡MARSAIL · §PBYAIL (Panboonyuen AI Lab)
Hide to See: Reasoning-prefix Masking for Visual-anchored Thinking in VLM Distillation
May 12, 2026Seonghoon Yu, Dongjun Nam, Byung-Kwan Lee +1Cross-Modal Knowledge Distillation CoT Distillation
1KAIST · 3POSTECH · 2NVIDIA
On-Policy Distillation with Best-of-N Teacher Rollout Selection
May 10, 2026Ke Zhang, Yunjie Tian, Dongdi Zhao +4CoT Distillation Best-of-N Sampling
Johns Hopkins University · TikTok · University of California, San Diego +1
LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering
May 10, 2026Runze Ma, Shunbo Jia, Haonan Lyu +2CoT Distillation Healthcare
School of Information Technology Monash University Malaysia · Faculty of Innovation Engineering Macau University of Science and Technology · Department of Bioelectronics Faculty of Biomedical Engineering Shenzhen University of Advanced Technology +1
Structural Rationale Distillation via Reasoning Space Compression
May 8, 2026Jialin Yang, Jiankun Wang, Jiajun Wu +3CoT Distillation Structured Reasoning
University of Calgary · Calgary, Canada · University of Michigan +1
Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decoding
May 4, 2026Taewon Yun, Jisu Shin, Jeonghwan Choi +2CoT Distillation Efficient Language Model Reasoning
KAIST · UNIST
Hiding in Plain Sight: Detectability-Aware Antidistillation of Reasoning Models
Apr 25, 2026Max Hartman, Vidhata Jayaraman, Moulik Choraria +2CoT Distillation Teacher-Student Learning
University of Illinois Urbana-Champaign · Carnegie Mellon University · Stony Brook University
Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
Apr 17, 2026Yao Chen, Jiawei Sheng, Wenyuan Zhang +1CoT Distillation Small Language Models
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences
Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective
Apr 10, 2026Tokio Kajitsuka, Ukyo Honda, Sho TakaseCoT Distillation Reasoning Distillation
The University of Tokyo, Tokyo, Japan · CyberAgent, Tokyo, Japan
Grounded Chess Reasoning in Language Models via Master Distillation
Mar 20, 2026Zhenwei Tang, Qianfeng Wen, Seth Grief-Albert +4LLM Grounding CoT Distillation
Department of Computer Science, University of Toronto · Queen’s University · Coolwei AI Lab
HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillation
Mar 11, 2026Wenjing Zhang, Jiangze Yan, Jieyun Huang +7CoT Distillation Language Model Distillation
Data Science & AI Research Institute, China Unicom · Unicom Data Intelligence, China Unicom
CRISP: Compressed Reasoning via Iterative Self-Policy Distillation
Mar 5, 2026Hejian Sang, Yuanda Xu, Zhengze Zhou +3CoT Distillation Efficient Language Model Reasoning
Measuring and Mitigating Post-hoc Rationalization in Reverse Chain-of-Thought Generation
Feb 16, 2026Guangyue Peng, Zongchao Chen, Wen Luo +9CoT Distillation CoT Reasoning
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Nanbeige Lab, BOSS Zhipin · University of Electronic Science and Technology of China
BRIDGE: Bridging Reasoning In Distillation Gap Elimination via Structure-Aware Masking
Feb 5, 2026Bowen Yu, Sheng Zhang, Binhao Wang +8CoT Distillation Mathematical Reasoning
1City University of Hong Kong · 2Mohamed Bin Zayed University of Artificial Intelligence
Distilling LLM Reasoning into Graph of Concept Predictors
Feb 3, 2026Ziyang Yu, Liang ZhaoCoT Distillation Active Learning
Department of Computer Science Emory University Atlanta, USA
Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation
Dec 24, 2025Wei-Rui Chen, Vignesh Kothapalli, Ata Fatahibaarzi +5CoT Distillation Selective Knowledge Distillation
1The University of British Columbia · 2LinkedIn · 3Canada Research Chair in NLP and ML
From Hindsight to Foresight: Self-Encouraged Hindsight Distillation for Knowledge-based Visual Question Answering
Nov 14, 2025Yu Zhao, Ying Zhang, Xuhui Sui +4Visual Question Answering CoT Distillation
College of Computer Science, VCIP, DISSec, Nankai University, Tianjin, China · School of Software, Tiangong University, Tianjin, China · Shenzhen Campus of Sun Yat-sen University, Shenzhen, China +1
Scale or Reason? A Compute-Equivalent Analysis of Reasoning Distillation
Sep 26, 2025Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Kevin El Haddad +2CoT Distillation LLM Fine-Tuning
1Diabolocom · 5MICS, CentraleSupélec, Université Paris-Saclay · 2Artefact Research Center +2
Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
Aug 13, 2025Xiaojun Wu, Xiaoguang Jiang, Huiyang Li +11CoT Distillation Language Model Distillation
1Zhongxing Telecom Equipment(ZTE), China · 2China Mobile Communications Group Co Ltd

May 29, 2026, cs.CLJ/K move · Enter open · S save

Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance

Yuxuan Jiang, Francis Ferraro

University of Maryland, Baltimore County

CoT Distillation Language Model Distillation On-Policy Distillation Teacher-Student Learning Knowledge Distillation

Open paper AskPDF arXiv

On-Policy Distillation (OPD) improves large language model reasoning by training a student model on trajectories sampled from its own policy under teacher supervision. Although OPD operates on trajectories, its learning signal remains token-level: it identifies deviations through high-loss tokens and repairs them through local reverse-KL correction. We show that this "trajectory-sampled but token-learned" mechanism cannot reliably bridge student trajectories toward teacher trajectories. About 30% of high-loss tokens fall into the low-divergence regime, indicating that many are surface-form mismatches rather than real reasoning forks. Moreover, even truly divergent tokens are difficult to repair with isolated token-level supervision, since reasoning failures often unfold as short-horizon distributional drift. We propose Trajectory-aware OPD (TOPD), which uses near-future trajectory information to identify real divergent states and distribute guidance across multiple future tokens. Experiments show that suppressing non-divergent high-loss tokens improves standard OPD from 47.8% to 48.2% average accuracy, while TOPD further improves performance to 52.2%, with gains on AIME24 from 60.0% to 63.3% and AIME25 from 46.7% to 53.3%.

Momentum

Related topics

Latest papers 57