CoT Distillation

CoT: Chain-of-Thought

Chain-of-thought distillation trains a student model using a teacher’s intermediate reasoning, aiming to transfer problem-solving behavior as well as correct answers. Common approaches include supervised fine-tuning on teacher-generated traces, matching teacher and student token distributions during student rollouts, and compressing or selecting traces to fit the learner. Methods are tested on mathematical reasoning benchmarks such as GSM8K, MATH, and AIME, as well as code, knowledge, and multimodal tasks. A central tension is that detailed traces can provide useful supervision but may be costly, incorrect, or mismatched to what a student can learn; answer accuracy alone also does not guarantee faithful reasoning.

Latest papers

80 papers13 papers in the last four weeks, up 225% on the four weeks before. 0.1% of all new papers.

Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
Apr 17, 2026Yao Chen, Jiawei Sheng, Wenyuan Zhang +1CoT Distillation Small Language Models
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences
Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective
Apr 10, 2026Tokio Kajitsuka, Ukyo Honda, Sho TakaseCoT Distillation Reasoning Distillation
The University of Tokyo, Tokyo, Japan · CyberAgent, Tokyo, Japan
State-of-the-Art Arabic Language Modeling with Sparse MoE Fine-Tuning and Chain-of-Thought Distillation
Apr 7, 2026Navan Preet Singh, Anurag Garikipati, Ahmed Abulkhair +6Arabic Language Models Arabic NLP
Forta, Houston, TX · Incept Labs, Houston, TX · Titan Holdings, San Francisco, CA
Student-in-the-Loop Chain-of-Thought Distillation via Generation-Time Selection
Apr 3, 2026Chaoqun He, Yingfa Chen, Chaojun Xiao +2Language Model Distillation CoT Distillation
Tsinghua University
Grounded Chess Reasoning in Language Models via Master Distillation
Mar 20, 2026Zhenwei Tang, Qianfeng Wen, Seth Grief-Albert +4LLM Grounding CoT Distillation
Department of Computer Science, University of Toronto · Queen’s University · Coolwei AI Lab
Distilling Reasoning Without Knowledge: A Framework for Reliable LLMs
Mar 15, 2026Auksarapak Kietkajornrit, Jad Tarifi, Nima AsgharbeygiLLM Planning LLM Reliability
Integral AI
HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillation
Mar 11, 2026Wenjing Zhang, Jiangze Yan, Jieyun Huang +7CoT Distillation Language Model Distillation
Data Science & AI Research Institute, China Unicom · Unicom Data Intelligence, China Unicom
CRISP: Compressed Reasoning via Iterative Self-Policy Distillation
Mar 5, 2026Hejian Sang, Yuanda Xu, Zhengze Zhou +3CoT Distillation Efficient Language Model Reasoning
Learning from Partial Chain-of-Thought via Truncated-Reasoning Self-Distillation
Feb 27, 2026Gianluigi Silvestri, Edoardo CetinEfficient Language Model Reasoning CoT Distillation
Radboud University · Sakana AI
D-COT: Disciplined Chain-of-Thought Learning for Efficient Reasoning in Small Language Models
Feb 25, 2026Shunsuke UbukataEfficient Language Model Reasoning CoT Distillation
Toyo University
Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation
Feb 25, 2026Minsang Kim, Seung Jun BaekLanguage Model Distillation Selective Knowledge Distillation
Korea University · SK Telecom
GATES: Self-Distillation under Privileged Context with Consensus Gating
Feb 24, 2026Alex Stein, Furong Huang, Tom GoldsteinKnowledge Distillation On-Policy Self-Distillation
University of Maryland, College Park
Measuring and Mitigating Post-hoc Rationalization in Reverse Chain-of-Thought Generation
Feb 16, 2026Guangyue Peng, Zongchao Chen, Wen Luo +9CoT Distillation CoT Reasoning
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Nanbeige Lab, BOSS Zhipin · University of Electronic Science and Technology of China
BRIDGE: Bridging Reasoning In Distillation Gap Elimination via Structure-Aware Masking
Feb 5, 2026Bowen Yu, Sheng Zhang, Binhao Wang +8CoT Distillation Mathematical Reasoning
City University of Hong Kong · Mohamed Bin Zayed University of Artificial Intelligence
Distilling LLM Reasoning into Graph of Concept Predictors
Feb 3, 2026Ziyang Yu, Liang ZhaoCoT Distillation Active Learning
Department of Computer Science Emory University Atlanta, USA
Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation
Dec 24, 2025Wei-Rui Chen, Vignesh Kothapalli, Ata Fatahibaarzi +5CoT Distillation Selective Knowledge Distillation
1The University of British Columbia · 2LinkedIn · 3Canada Research Chair in NLP and ML
From Hindsight to Foresight: Self-Encouraged Hindsight Distillation for Knowledge-based Visual Question Answering
Nov 14, 2025Yu Zhao, Ying Zhang, Xuhui Sui +4Visual Question Answering CoT Distillation
College of Computer Science, VCIP, DISSec, Nankai University, Tianjin, China · School of Software, Tiangong University, Tianjin, China · Shenzhen Campus of Sun Yat-sen University, Shenzhen, China +1
Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners
Sep 30, 2025Xin Xu, Clive Bai, Kai Yang +4Efficient Language Model Reasoning RL for Language Model Reasoning
LLM Department, Tencent · The Hong Kong University of Science and Technology · The University of Hong Kong
Scale or Reason? A Compute-Equivalent Analysis of Reasoning Distillation
Sep 26, 2025Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Kevin El Haddad +2CoT Distillation LLM Fine-Tuning
1Diabolocom · 5MICS, CentraleSupélec, Université Paris-Saclay · 2Artefact Research Center +2
Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
Aug 13, 2025Xiaojun Wu, Xiaoguang Jiang, Huiyang Li +11CoT Distillation Language Model Distillation
1Zhongxing Telecom Equipment(ZTE), China · 2China Mobile Communications Group Co Ltd

Apr 17, 2026J/K move · Enter open · S save

Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information

Yao Chen, Jiawei Sheng, Wenyuan Zhang, Tingwen Liu

Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences

CoT Distillation Small Language Models Transfer Learning Language Model Distillation Attention Mechanisms Knowledge Distillation

Open paper AskPDF arXiv

The significant computational demands of large language models have increased interest in distilling reasoning abilities into smaller models via Chain-of-Thought (CoT) distillation. Current CoT distillation methods mainly focus on transferring teacher-generated rationales for complex reasoning to student models. However, they do not adequately explore teachers' dynamic attention toward critical information during reasoning. We find that language models exhibit progressive attention shifts towards key information during reasoning, which implies essential clues for drawing conclusions. Building on this observation and analysis, we introduce a novel CoT distillation framework that transfers the teacher's stepwise attention on key information to the student model. This establishes structured guidance for the student's progressive concentration on key information during reasoning. More importantly, we develop a Mixture of Layers module enabling dynamic alignment that adapts to different layers between the teacher and student. Our method achieves consistent performance improvements across multiple mathematical and commonsense reasoning datasets. To our knowledge, it is the first method to leverage stepwise attention within CoT distillation to improve small model reasoning.