CoT Distillation

CoT: Chain-of-Thought

Overview

Chain-of-thought distillation trains a student model using a teacher’s intermediate reasoning, aiming to transfer problem-solving behavior as well as correct answers. Common approaches include supervised fine-tuning on teacher-generated traces, matching teacher and student token distributions during student rollouts, and compressing or selecting traces to fit the learner. Methods are tested on mathematical reasoning benchmarks such as GSM8K, MATH, and AIME, as well as code, knowledge, and multimodal tasks. A central tension is that detailed traces can provide useful supervision but may be costly, incorrect, or mismatched to what a student can learn; answer accuracy alone also does not guarantee faithful reasoning.

Latest papers

CardsList

83 papers5 papers in the last four weeks, down 64% on the four weeks before. 0.0% of all new papers.

  1. Agents, Think Only When it Helps

    Nov 3, 2026Min-Seon Kim, Seungdong Yoa, Sanghyu Yoon +4CoT DistillationLLM Agent Efficiency

  2. Where Do Distilled Students Store Thinking?

    Nov 3, 2026Taeyeong Kim, Jaemin Jung, Ahhyun Kim +1CoT DistillationKnowledge Distillation

  3. Stepwise-Querying: Distilling Reasoning without Access to Teacher Chain-of-Thought

    Oct 30, 2026Dicong Yu, Wenhua Wang, Tian WangBlack-Box Knowledge DistillationLanguage Model Distillation

  4. From Solver Trajectories to Teaching Trajectories: Cognition-Aligned Reasoning Distillation

    Oct 27, 2026Yashuo Luo, Qianren Mao, Ziqi Qin +6Reasoning DistillationLanguage Model Distillation

  5. Beyond the Answer: Student-Conditioned Trace Construction for Reasoning Distillation

    Oct 16, 2026Shaoang Li, Yanhang Shi, Yufei Li +6CoT DistillationQuestion Answering

  6. CoTranslator: Reverse to Forward Chain-of-Thought Translator for Reasoning Distillation

    Oct 14, 2026Zinan Tang, Zhuoshi Pan, Kairui Chen +5CoT DistillationLanguage Model Distillation

  7. Rationale Distribution Distillation via Flow Matching for Distributional Continuous Latent Reasoning

    Oct 14, 2026Mingyeong Kim, Gyusang Cho, Juho Lee +1LLM ReasoningContinuous Latent Reasoning

  8. Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation

    Oct 13, 2026Yongcan Yu, Jian Liang, Lingxiao He +5CoT DistillationLanguage Model Distillation

  9. Audit Before You Distill: A Controlled Workflow for Reasoning Transfer into Compact Medical Language Models

    Oct 12, 2026Tiago Martinelli, João Paulo Papa, Adriano José PereiraLanguage Model DistillationMedical LLMs

  10. Overtaught, Underlearned: Learner-Centered Distillation from Multi-Agent Discussion

    Oct 12, 2026Mengtao Zhou, Jianxiao Wang, Xiupeng Chen +2Language Model DistillationKnowledge Distillation

  11. Learning from Repaired Reasoning: Root-Cause-Guided On-Policy Distillation

    Oct 2, 2026Chenglei Shen, Haoyang Yao, Weijie Yu +3Language Model DistillationOn-Policy Self-Distillation

  12. Hindsight-Guided Rationale Distillation for Rare Disease Diagnosis

    Oct 2, 2026Aarav Singh, Animesh Pathak, Navyansh SinghCoT DistillationKnowledge Distillation

  13. The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learning

    Sep 29, 2026Matthieu Zimmer, Xiaotong Ji, Tu Nguyen +1Language Model DistillationCoT Distillation

  14. Learning from Think-Mode Advantage via On-Policy Distillation

    Sep 29, 2026Wanqi Ren, Jianxiang Wang, Danxuan Liu +2CoT DistillationLanguage Model Distillation

  15. Distilling Visual Reasoning into Text Space

    Sep 28, 2026Wenhan Yang, Nilay Naharas, Ali Payani +1CoT DistillationVLM Distillation

  16. Learning from a Thoughtful Teacher: Adaptive On-Policy Self-Distillation for Mathematical Reasoning

    Sep 26, 2026Jiacheng Du, Weiwei Xie, Tianyi Du +3On-Policy Self-DistillationCoT Distillation

  17. Efficient Reasoning Distillation: Small Video-Language Models via Synthetic CoT and Difficulty-Aware Fine-Tuning

    Sep 14, 2026Mantek Singh, Jeshwanth Challagundla, Siddharth Raina +1CoT DistillationVLM Distillation

  18. Aha-Flow Distillation: Flow Markers Matter in LLM Reasoning

    Sep 7, 2026Xiaodong Wang, Peixi PengLLM ReasoningCoT Distillation

  19. MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation

    Aug 30, 2026Yangsong Lan, Renkai Hu, HongKai Zheng +4CoT DistillationLanguage Model Distillation

  20. Beyond Imitation: Auditing the Recoverability of Reasoning in Distilled Models

    Aug 11, 2026Ruitong Li, Binjie Guo, Aisheng Mo +4Language Model DistillationKnowledge Distillation

  21. Simple-OPD: Demystifying Warm-up for On-policy Distillation

    Aug 7, 2026Tao Liu, Taiqiang Wu, Mao Zheng +5CoT DistillationOn-Policy Distillation

  22. EKLAV: Elicited Key CoT Learning via Answer only superVision

    Jul 28, 2026Adarsh Singh, Aman Singh, Kushal Raj Bhandari +3Table QACoT Distillation

  23. DAIS: Dependency-Aware Intermediate QA Supervision for Complex Reasoning

    Jul 21, 2026Yu Wang, Ming Fan, Xicheng Zhang +5CoT DistillationCoT Supervision

  24. Better Starts, Better Ends: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoning

    Jul 17, 2026Leichao Dong, Dongxu Zhang, Yiding Sun +4CoT DistillationEfficient Language Model Reasoning

  25. Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models

    Jul 16, 2026Jungseob Lee, Seungyoon Lee, Suhyune Son +4CoT DistillationLanguage Model Distillation