From Solver Trajectories to Teaching Trajectories: Cognition-Aligned Reasoning Distillation

Yashuo Luo; Qianren Mao; Ziqi Qin; Junnan Liu; Haitao Li; Qili Zhang; Yuan Gao; Zhijun Chen

cs.AIOct 27, 2026

From Solver Trajectories to Teaching Trajectories: Cognition-Aligned Reasoning Distillation

Authors: Yashuo Luo, Qianren Mao, Ziqi Qin, Junnan Liu, Haitao Li, Qili Zhang, Yuan Gao, Zhijun Chen,

Organizations: National College for Excellent Engineers, Beihang University · Zhongguancun Laboratory · Beihang University · Department of Data Science & AI, Faculty of Information Technology, Monash University ·

Abstract

Knowledge distillation from large reasoning models (LRMs) often uses raw teacher trajectories as supervision. However, this practice creates a mismatch between the goal of distillation and the nature of teacher trajectories. Advanced LRMs typically improve their reasoning capabilities through reward-driven post-training, such as RLHF. This training optimizes them as strong solvers, rather than as teachers that produce reasoning processes smaller models can easily learn. Since smaller models have different "cognitive capacities" compared to their larger teachers, directly imitating raw teacher trajectories can sometimes be ineffective and often requires a substantial number of high-quality samples. To bridge this mismatch, we propose Cognition-Aligned Reasoning Distillation (CARD), an RL-based framework that transforms solver trajectories into teaching trajectories aligned with the student's cognition. Avoiding expensive resampling, our framework directly transforms existing trajectories. We optimize CARD with two reward signals: (1) a Distribution Alignment Reward that matches the student's predictive distribution, and (2) a Logical Coherence Reward that reduces misleading failed reasoning steps. Comprehensive evaluations across multiple student models and settings demonstrate that CARD outperforms distillation from raw teacher trajectories in in-domain settings while achieving substantial gains in out-of-distribution performance. Code is available at https://github.com/LiteSSLHub/CARD.

Figures & tables

Figure 1: Motivating analysis of failed steps in trajectory-based reasoning distillation. Using DeepSeek-R1-generated DeepMath trajectories, we distill three student models on two 10,000-trajectory subsets under identical settings, where one subset consists entirely of trajectories with detected failed steps and the other consists entirely of trajectories without. Bars report the difference between the no-failed-step subset and the failed-step subset. Accuracy gains are reported in absolute points, while output-length reductions are reported as

(\mathrm{Token}_{\mathrm{failed}}-\mathrm{Token}_{\mathrm{no\text{-}failed}})/\mathrm{Token}_{\mathrm{failed}}\times 100\%

. Evaluation includes math benchmarks and ARC-Challenge for out-of-distribution generalization, and each panel corresponds to a different student model.

Figure 2: Illustration of filter-based distillation methods and CARD. Left: filter-based distillation methods sample multiple teacher trajectories before selection and SFT, incurring expensive repeated teacher sampling. Right: CARD selectively reformats solver trajectories into cognition-aligned teaching trajectories without repeated teacher sampling. Highlights in the solver trajectory indicate detected failed steps and redundant expressions that hinder student learning.

Figure 3: Overview of CARD. (1) Reward Computation. The CARD Reformatter takes a question and its original rationale as input and produces a reformatted rationale , leaving the answer unchanged. Rationale Alignment ( $r_{\mathrm{rat}}$ ) and Answer Consistency ( $r_{\mathrm{ans}}$ ) are combined into the Distribution Alignment Reward via a weighted sum. (2) Final Reward. The Distribution Alignment Reward is gated by a Logical Coherence check (no failed steps ) and a Format Validity check (no leakage); any candidate failing either gate receives a reward of zero. (3) Dataset Construction. Teacher trajectories are split by their Failed-Step Fraction (FSF): those with $\mathrm{FSF}>0$ are reformatted by CARD, while structurally clean trajectories with $\mathrm{FSF}=0$ are retained directly to form the final training set.

Model	Method	Math							General (OOD)
Model	Method	AIME24	AIME25	GSM8K	MATH500	AMC23	AGI-Math	Avg	MMLU-Pro	ARC-C	Avg
Qwen2.5-7B	Base	10.42	5.83	90.17	73.80	41.30	70.60	48.69	52.53	87.37	69.95
	+ Vanilla SFT	13.89	11.25	90.58	75.80	45.40	73.90	51.80	45.60	67.07	56.34
	+ MCC-KD	12.50	9.50	90.10	74.70	44.00	72.50	50.55	46.04	67.41	56.73
	+ MoRSD	13.05	10.83	89.89	74.30	44.56	71.72	50.73	45.58	68.60	57.09
	+ Mix TokenSkip	12.50	10.00	88.62	68.60	42.80	69.90	48.74	44.37	71.16	57.77

Table 1: Main results on math and OOD generalization benchmarks. Best results within each student block are highlighted.

\uparrow

denotes improvement of CARD over Vanilla SFT.

Method	Math Avg	Gen Avg
Vanilla SFT	51.80	56.34
Drop-FSF	50.93	56.44
Replace-FSF	52.74	57.55
CARD	54.22	71.07

Table 2: FSF filtering on student Qwen2.5-7B.

Method	Math Avg	Gen Avg
Vanilla SFT	$52.34\pm 0.64$	$56.62\pm 0.26$
CARD	$\mathbf{54.94\pm 0.69}$	$\mathbf{70.78\pm 0.93}$

Table 3: Training-seed robustness on Qwen2.5-7B.

Reward Student	Math Avg	Gen Avg
Llama3.2-1B	32.26	55.13
Llama3.2-3B	33.92	55.83
Qwen2.5-7B	33.02	55.24

Table 4: Cross-model transferability on student Llama3.2-3B.

Panel of Figure 4 — Figure 4: Training dynamics of CARD. Left : SFT loss on Llama3.2-3B. Middle : $r_{\mathrm{rat}}$ during GRPO training. Right : PPL distribution.

Panel of Figure 5 — Figure 5: Effect of reformatting selection criterion on student performance under random selection across different fractions of the full training set. Peak Math Avg and Gen Avg are annotated.

Method	Math Avg	Gen Avg
CARD	54.22	71.07
w/o $r_{\mathrm{coh}}$	52.54 $\downarrow$ 1.68	66.31 $\downarrow$ 4.76
w/o $r_{\mathrm{rat}}$	52.23 $\downarrow$ 1.99	68.32 $\downarrow$ 2.75
w/o $r_{\mathrm{ans}}$	52.07 $\downarrow$ 2.15	68.63 $\downarrow$ 2.44
w/o $\mathbb{I}_{\mathrm{leak}}$	52.66 $\downarrow$ 1.56	69.17 $\downarrow$ 1.90

Table 5: CARD component ablations on Qwen2.5-7B.

Method	Math Avg	Gen Avg
CARD	54.22	71.07
w/ 5% bias	52.88 $\downarrow$ 1.34	68.70 $\downarrow$ 2.37
w/ 10% bias	53.37 $\downarrow$ 0.85	68.94 $\downarrow$ 2.13
w/ 20% bias	52.91 $\downarrow$ 1.31	66.42 $\downarrow$ 4.65

Table 6: Robustness to FSF judge bias on student Qwen2.5-7B.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	#Train	AIME24	AIME25	GSM8K	MATH500	AMC23	AGI-Math	Math Avg	MMLU-Pro	ARC-C	Gen Avg
Vanilla SFT	30,000	13.89	11.25	90.58	75.80	45.40	73.90	51.80	45.60	67.07	56.34
Drop-FSF	21,181	13.16	10.40	90.20	74.30	44.49	73.03	50.93 $\downarrow$ 0.87	52.90	59.98	56.44 $\uparrow$ 0.10
Replace-FSF	30,000	14.87	12.39	91.09	76.42	46.61	75.06	52.74 $\uparrow$ 0.94	53.50	61.60	57.55 $\uparrow$ 1.21
CARD	30,000	16.39	14.16	91.87	77.50	48.50	76.87	54.22 $\uparrow$ 2.42	54.00	88.14	71.07 $\uparrow$ 14.73

Appendix

Table 7: Full FSF filtering comparison on Qwen2.5-7B. Math Avg and Gen Avg are macro-averages over the six math and two OOD benchmarks, respectively.

\uparrow/\downarrow

denote changes from Vanilla SFT.

Method	Seed	AIME24	AIME25	GSM8K	MATH500	AMC23	AGI-Math	Math Avg	MMLU-Pro	ARC-C	Gen Avg
Vanilla SFT	42	13.89	11.25	90.58	75.80	45.40	73.90	51.80	45.60	67.07	56.34
CARD	42	16.39	14.16	91.87	77.50	48.50	76.87	54.22 $\uparrow$ 2.42	54.00	88.14	71.07 $\uparrow$ 14.73
Vanilla SFT	307	15.70	13.17	90.25	75.36	48.47	75.27	53.04	45.98	67.33	56.66
CARD	307	17.28	17.70	92.23	79.51	48.90	77.90	55.59 $\uparrow$ 2.55	53.38	86.10	69.74 $\uparrow$ 13.08
Vanilla SFT	1234	14.15	13.62	90.06	77.19	43.51	74.51	52.17	44.85	68.86	56.86
CARD	1234	17.28	16.31	91.65	78.11	50.78	75.93	55.01 $\uparrow$ 2.84	54.22	88.84	71.53 $\uparrow$ 14.67

Appendix

Table 8: Full per-seed comparison of Vanilla SFT and CARD on Qwen2.5-7B across three training seeds.

\uparrow

denotes improvement over Vanilla SFT with the same seed.

Reward Student	AIME24	AIME25	GSM8K	MATH500	AMC23	AGI-MATH	Math Avg	MMLU-Pro	ARC-C	Gen Avg
Llama3.2-1B	2.64	0.28	76.23	46.40	21.33	46.68	32.26	36.02	74.23	55.13
Llama3.2-3B	3.74	1.38	77.26	49.40	23.13	48.60	33.92	36.91	74.74	55.83
Qwen2.5-7B	2.78	0.56	77.05	46.15	23.37	48.22	33.02	37.23	73.24	55.24

Appendix

Table 9: Full cross-model transferability of the CARD Reformatter on Llama3.2-3B. CARD Train By denotes the student model used to compute rewards during Reformatter training.

Method	AIME24	AIME25	GSM8K	MATH500	AMC23	AGI-MATH	Math Avg	MMLU-Pro	ARC-C	Gen Avg
CARD	16.39	14.16	91.87	77.50	48.50	76.87	54.22	54.00	88.14	71.07
CARD w/o $r_{\mathrm{coh}}$	15.00	13.61	90.07	74.57	46.46	75.50	52.54 $\downarrow$ 1.68	52.82	79.80	66.31 $\downarrow$ 4.76
CARD w/o $r_{\mathrm{rat}}$	15.13	12.36	89.50	73.42	46.46	76.50	52.23 $\downarrow$ 1.99	54.14	82.50	68.32 $\downarrow$ 2.75
CARD w/o $r_{\mathrm{ans}}$	14.02	12.36	89.61	73.90	46.60	75.90	52.07 $\downarrow$ 2.15	53.90	83.35	68.63 $\downarrow$ 2.44
CARD w/o $\mathbb{I}_{\mathrm{leak}}$	14.86	13.19	89.54	74.77	46.60	77.00	52.66 $\downarrow$ 1.56	52.81	85.52	69.17 $\downarrow$ 1.90

Appendix

Table 10: Full ablation on CARD components using Qwen2.5-7B with LoRA.

\downarrow

denotes the drop compared with full CARD.

Method	AIME24	AIME25	GSM8K	MATH500	AMC23	AGI-MATH	Math Avg	MMLU-Pro (w/o math)	ARC-C	Gen Avg
CARD	16.39	14.16	91.87	77.50	48.50	76.87	54.22	54.00	88.14	71.07
CARD w/ 5% judge bias	13.05	13.88	89.78	76.60	47.83	76.15	52.88 $\downarrow$ 1.34	55.22	82.17	68.70 $\downarrow$ 2.37
CARD w/ 10% judge bias	14.86	12.78	90.47	77.58	47.28	77.27	53.37 $\downarrow$ 0.85	53.13	84.75	68.94 $\downarrow$ 2.13
CARD w/ 20% judge bias	14.72	13.05	90.33	75.40	47.14	76.80	52.91 $\downarrow$ 1.31	51.78	81.06	66.42 $\downarrow$ 4.65

Appendix

Table 11: Full Robustness to FSF judge accuracy using Qwen2.5-7B with LoRA.

\downarrow

denotes the drop compared with full CARD.

Reasoning Distillation

Reasoning distillation transfers step-by-step problem-solving skills from powerful teachers to smaller, more efficient models.

See all 23 papers

TeacherGRPO: Closing the Capacity Gap in Reasoning Distillation via Teacher Alignment
Sep 27, 2026Zhenyu Lei, Zihan Chen, Yaochen Zhu +5Reasoning Distillation RL for Language Model Reasoning
University of Virginia · Netflix · University of Washington +2
HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillation
Mar 11, 2026Wenjing Zhang, Jiangze Yan, Jieyun Huang +7CoT Distillation Language Model Distillation
Data Science & AI Research Institute, China Unicom · Unicom Data Intelligence, China Unicom
Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation
Feb 25, 2026Minsang Kim, Seung Jun BaekLanguage Model Distillation Selective Knowledge Distillation
Korea University · SK Telecom

Sep 27, 2026J/K move · Enter open · S save

TeacherGRPO: Closing the Capacity Gap in Reasoning Distillation via Teacher Alignment

Zhenyu Lei, Zihan Chen, Yaochen Zhu, Shangbin Feng+4

University of Virginia · Netflix · University of Washington+2

Reasoning Distillation RL for Language Model Reasoning Language Model Distillation Teacher-Student Learning

Open paper AskPDF arXiv

Reasoning distillation from powerful teacher models to smaller students faces the Gap Curse: as teachers grow more sophisticated, their complex distributions increasingly diverge from what students can approximate, causing performance degradation. Existing mitigation strategies either filter out challenging examples through data selection or introduce weaker intermediate assistant models, inherently compromising supervision coverage or quality. We propose Teacher Alignment, which directly adapts the teacher toward the student's distribution without discarding data or degrading reasoning quality. However, naive alignment through standard knowledge distillation triggers catastrophic collapse of the teacher's reasoning capabilities. To address this, we reformulate teacher alignment as reinforcement learning and introduce TeacherGRPO, built on Group Relative Policy Optimization with two key innovations: (i) Curriculum Selective Alignment applies dual token- and distribution-level curricula to focus rewards on high-signal reasoning gaps while filtering noise from trivial tokens and uncertain tail distributions, and (ii) Importance-Adaptive Length Regularization selectively penalizes verbose redundancy while preserving pedagogically critical reasoning steps. The aligned teacher then distills knowledge to students via standard pipelines. Extensive experiments show TeacherGRPO significantly outperforms baselines across diverse reasoning benchmarks and distillation methods. Our code is available at https://github.com/LzyFischer/TeacherGRPO.

Abstract

Reasoning Distillation

Explore similar work

TeacherGRPO: Closing the Capacity Gap in Reasoning Distillation via Teacher Alignment

HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillation

Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation