Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation

Yongcan Yu; Jian Liang; Lingxiao He; Kuangpu Guo; Yanbo Wang; Shuo Lu; Meng Wang; Ran He

cs.CLOct 13, 2026

Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation

Authors: Yongcan Yu, Jian Liang, Lingxiao He, Kuangpu Guo, Yanbo Wang, Shuo Lu, Meng Wang, Ran He

Organizations: NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Meituan · University of Science and Technology of China

Abstract

While long chain-of-thought (CoT) distillation significantly improves the reasoning abilities of language models, its underlying mechanism remains poorly understood. Current pipelines generally rely on rejection sampling to gather reasoning traces across diverse prompts, assuming that factual correctness and broad problem coverage are indispensable. Surprisingly, we find that training exclusively on flawed reasoning trajectories or a restricted prompt set yields nearly comparable performance. Driven by this counterintuitive finding, we propose the Behavioral Hypothesis: the essence of Long-CoT distillation is behavioral alignment with structured reasoning patterns, rather than factual knowledge memorization. We then validate this by demonstrating substantial reasoning gains even when models are fine-tuned exclusively on previously mastered problems-effectively isolating behavioral alignment from novel knowledge injection. Having established that reasoning patterns are the true bottleneck, we investigate how to learn them across varying levels of complexity optimally. We reveal a critical interplay: while the recent probability-based loss excels at learning simple patterns, the standard cross-entropy loss is essential for scaling to complex CoTs. We believe these insights demystify CoT distillation and provide a principled foundation for the future development of reasoning models. Code is available on https://github.com/yuyongcan/Long-CoT-Distillation.

Figures & tables

Figure 1: Incorrect Trajectories Teach Long-CoT Behavior Almost as Effectively as Correct Ones. We compare models fine-tuned on positive versus negative CoT trajectories under two training objectives. Across models and benchmarks, negative trajectories retain most of the distillation gains achieved by positive trajectories.

Method	Data	Qwen3-1.7B-Base				Method	Data	Qwen3-4B-Base
Method	Data	MMLUPro	GPQA-D	LCBv2	Average	Method	Data	MMLUPro	GPQA-D	LCBv2	Average
Base	-	28.1	14.4	8.9	17.1	Base	-	46.1	16.2	19.8	27.4
-logp	P	35.3	21.3	13.8	23.5	-logp	P	52.9	29.6	29.4	37.3
-logp	N	34.3	23.0	14.3	23.9 (+0.4)	-logp	N	56.4	31.5	28.8	38.9 (+1.6)
-p	P	34.9	15.7	9.3	20.0	-p	P	54.3	34.3	24.8	37.8
-p	N	34.7	14.7	7.9	19.1 (-0.9)	-p	N	52.6	33.5	24.4	36.8 (-1.0)

Table 1: The Negligible Impact of CoT Correctness Extends Beyond Math Benchmarks. We evaluate whether the weak dependence on CoT correctness generalizes to non-math domains, including general reasoning, scientific reasoning, and coding tasks. Across models and training objectives, negative CoT trajectories achieve average performance nearly identical to positive trajectories. P and N denote positive and negative CoT trajectories, respectively.

Figure 2: Negative CoTs Effectively Mitigate the Scarcity of Correct Data. We train Qwen3-1.7B-Base with different positive sample bases and varying ratios of positive to negative CoT trajectories. Adding negative trajectories consistently improves performance when positive data is scarce.

Figure 3: Prompt Diversity Is Not the Bottleneck Under a Fixed Trajectory Budget. We fix the total number of CoT trajectories and vary the source prompt diversity, ranging from a small prompt set with many sampled trajectories per prompt (

500\times 128

) to a large prompt set with fewer trajectories per prompt (

16\mathrm{k}\times 4

Model	Loss	Acc (Gains) (%)
Model	Loss	Positive	Negative
Qwen3 1.7B-Base	-logp	14.3 (+7.4)	13.5 (+6.6)
Qwen3 1.7B-Base	-p	17.3 (+10.4)	15.7 (+8.8)
Qwen3 4B-Base	-logp	36.4 (+23.3)	34.6 (+21.5)
Qwen3 4B-Base	-p	35.4 (+22.3)	34.1 (+21.0)

Table 2: Models Avoid Overfitting to Flawed Facts on the Training Set. Numbers in parentheses indicate performance gains relative to the base model.

Figure 4: Substantial Gains on Previously Solvable Problems. Fine-tuning on easy subsets that the base model can already solve (Easy-643/444) yields improvements comparable to training on a larger, randomly sampled dataset (Random-3k) across math benchmarks. This reduces the role of new problem-specific answers as an explanation for the gains.

Figure 5: CoT Complexity Dictates the Appropriate Distillation Objective. We compare four base models fine-tuned with

-\log p

loss (solid lines) and

-p

loss (dashed lines) across varying CoT complexities. The results reveal a clear dichotomy: while the

-p

loss excels on simpler CoTs, it suffers from pronounced degradation on highly complex reasoning traces. Conversely, the standard

-\log p

loss scales robustly with complexity.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

✓Positive Sample	✗Negative Sample
(Correct Fact & Perfect Behavior)	(Flawed Fact, but Same Behavior)
Step 1: Understand the pattern The terms are: $\frac{8}{4},\frac{12}{8},\dots,\frac{2008}{2004}$ . Each term has the general form: $\frac{4(n+1)}{4n}=\frac{n+1}{n}$ So the product becomes: $\prod_{n=1}^{501}\frac{n+1}{n}$ — Correct boundary condition —	Step 1: Understand the pattern The terms are: $\frac{8}{4},\frac{12}{8},\dots,\frac{2008}{2004}$ . Let the general term be: $\frac{4(k+1)}{4k}=\frac{k+1}{k}$ So the product becomes: $\prod_{k=2}^{502}\frac{k+1}{k}$ [!] Factual Error: Hallucinated starting index ( $k=2$ instead of $k=1$ ).
Step 2: Telescoping Product Write out the product explicitly: $\frac{2}{1}\cdot\frac{3}{2}\cdot\frac{4}{3}\cdots\frac{502}{501}$ Observe that this is a telescoping product . Most terms cancel out…	Step 2: Telescoping Product This is a telescoping product . Let’s write out the product: $\frac{3}{2}\cdot\frac{4}{3}\cdot\frac{5}{4}\cdots\frac{503}{502}$ We can see that most terms cancel out…
Final Answer: $\boxed{502}$	Final Answer: $\boxed{\frac{503}{2}}$

Appendix

Figure 6: Factual Flaws vs. Behavioral Alignment in Long-CoT Distillation. Given the same initial problem, the negative sample hallucinated an incorrect starting index (

k=2

), leading to a wrong final answer, yet it perfectly replicated the core, step-by-step reasoning behaviors of the positive sample. This structurally intact deduction demonstrates why incorrect CoT trajectories still provide high-quality behavioral supervision.

Figure 7: Token Length Distribution Across Reasoning Complexity Tiers. The boxplot illustrates the scaling of response lengths across Simple, Medium, and Complex datasets.

Figure 8: Word Cloud Analysis Reveals Lexical Shifts Across CoT Complexities. The lexical distribution shows a transition from concrete task-oriented words in the Simple dataset, through exploratory terms in the Medium dataset, to meta-cognitive markers in the Complex dataset.

Student	Training	Math avg.	SuperGLUE	GPQA-D	MMLU-Pro	Non-math avg.
Qwen3-1.7B-Base	Base	6.9	10.6	15.9	27.3	17.9
	SFT-P	27.2	18.0	22.0	34.9	25.0
	SFT-N	26.8	18.3	24.5	34.0	25.6
Qwen3-4B-Base	Base	19.1	8.5	16.0	45.6	23.4
	SFT-P	44.6	29.1	34.8	49.0	37.6
	SFT-N	43.7	29.8	36.4	49.4	38.5

Appendix

Table 3: Cross-domain evaluation after science QA (GURU) training. The base models are evaluated without fine-tuning; P/N distinguish final-answer correctness of the training trajectories. Values are percentages.

Teacher / mode	Student	Base	SFT-P	SFT-N
Gemma3-12B-it	Qwen3-1.7B-Base	6.9	22.8	22.2
Gemma3-12B-it	Qwen3-4B-Base	19.1	38.5	38.4
Qwen3-4B / thinking	Qwen3-1.7B-Base	6.9	20.9	20.4
Qwen3-4B / thinking	Qwen3-4B-Base	19.1	46.5	44.4

Appendix

Table 4: Teacher and sampling-mode controls: avg@16 accuracy (%) across the five mathematical benchmarks. Base is the corresponding student before fine-tuning.

Loss	Training data	AIME 2026	LiveCodeBench-v6
–	Base	1.3	9.3
$-\log p$	P	5.5	16.6
$-\log p$	N	5.6	16.9
$-p$	P	6.2	16.0
$-p$	N	7.1	16.0

Appendix

Table 5: Correct vs. incorrect trajectory distillation on newer evaluation sets; percentages averaged over four student architectures (rounded to one decimal place).

Qwen3-4B model pair	Linear CKA
Base vs. SFT-P	0.383
Base vs. SFT-N	0.397
SFT-P vs. SFT-N	0.998
SFT-P vs. SFT-numina	0.412

Appendix

Table 6: Linear CKA between late-layer response-token representations on the same held-out MATH inputs and reference traces (128 problems). SFT-numina is fine-tuned on NuminaMath-CoT.

Model	Method	Data	AIME 2024	AMC23	Minerva Math	Olympiad Bench	MATH	Average
	Base	-	1.3	6.0	5.0	6.2	16.0	6.9
	-logp	P	4.4	29.1	21.5	20.9	56.8	26.5
	-logp	N	2.1	31.9	20.2	20.9	54.2	25.9
	-p	P	7.3	35.6	24.1	24.7	61.7	30.7
Qwen3-1.7B-Base	-p	N	7.7	34.8	24.5	24.0	61.3	30.5
	Base	-	5.8	23.8	11.5	18.2	36.1	19.1

Appendix

Table 7: Detailed avg@16 Evaluation Results Corresponding to Figure 1 .

Model	Method	Data	AIME 2024	AMC23	Minerva Math	Olympiad Bench	MATH	Average
	Base	-	10.0	75.0	39.3	47.7	80.4	50.5
	-logp	P	20.0	75.0	55.1	51.0	86.3	57.5
	-logp	N	13.3	77.5	52.9	49.3	87.3	56.1
	-p	P	20.0	77.5	49.6	49.9	86.1	56.6
Qwen3-1.7B-Base	-p	N	16.7	82.5	51.1	48.1	84.3	56.5
	Base	-	30.0	87.5	49.6	58.7	88.2	62.8

Appendix

Table 8: Detailed pass@16 Evaluation Results Corresponding to Figure 1 .

Model	Method	Data	AIME 2024	AMC23	Minerva Math	Olympiad Bench	MATH	Average
	Base	-	1123.7	989.6	765.3	916.7	747.3	908.5
	-logp	P	2419.3	1686.8	905.7	1818	1002.5	1566.5
	-logp	N	2510.3	1762.9	980.4	1802.3	1066.2	1624.4
	-p	P	3172.3	2115	998.8	2323	1161.8	1954.2
Qwen3-1.7B-Base	-p	N	3318.5	2210.7	1092.8	2466.4	1247.7	2067.2
	Base	-	1532.7	1312.2	1451.7	1348.7	1266.9	1382.4

Appendix

Table 9: Averaged Response Token Lengths Corresponding to Figure 1 .

Model	Dataset	Loss	AIME 2024	AMC23	Minerva Math	Olympiad Bench	MATH	Average
		-logp	763.3	639	643.7	677.7	442	633.1
	Simple	-p	1400.8	913.6	684.9	956.8	573.2	905.9
		-logp	2575.3	1650.3	756.6	1623.8	878.3	1496.9
	Medium	-p	3117.7	1869	772.7	1946.3	910.9	1723.3
		-logp	19348.9	12440.8	8881.1	12102.3	8281.8	12211.0
Qwen3-4B-Base	Complex	-p	12524.4	6254.3	2616.5	6253.2	3230.8	6175.8

Appendix

Table 10: Averaged Response Token Lengths Corresponding to Figure 5 .

Model	Dataset	Loss	AIME 2024	AMC23	Minerva Math	Olympiad Bench	MATH	Average
		-logp	763.3	639	643.7	677.7	442	633.1
	Simple	-p	1400.8	913.6	684.9	956.8	573.2	905.9
		-logp	2575.3	1650.3	756.6	1623.8	878.3	1496.9
	Medium	-p	3117.7	1869	772.7	1946.3	910.9	1723.3
		-logp	19348.9	12440.8	8881.1	12102.3	8281.8	12211.0
Qwen3-4B-Base	Complex	-p	12524.4	6254.3	2616.5	6253.2	3230.8	6175.8

Appendix

Table 11: Averaged Response Token Lengths Corresponding to Figure 5 .

Model	CoT	Loss	AIME 2024	AMC23	Minerva Math	Olympiad Bench	MATH	Average
		-logp	3.3	25.9	17.8	17.4	49.7	22.8
	Simple	-p	8.8	43.0	29.6	33.2	72.2	37.4 (+14.6)
		-logp	12.3	51.4	34.5	39.2	76.4	42.8
	Medium	-p	15.6	53.1	33.5	39.1	79.3	44.1 (+1.3)
		-logp	24.4	57.3	30.0	44.4	71.9	45.6
Qwen3-4B-Base	Complex	-p	12.5	47.2	31.7	36.9	74.2	40.5 (-5.1)

Appendix

Table 12: The detailed accuracy in Figure 5 .

Model	Setting	AIME 2024	AMC23	Minerva Math	Olympiad Bench	MATH	Average
	$500\times 128$	5.2	30.9	22.3	22.6	58.8	28.0
	$1\text{k}\times 64$	3.5	30.2	21.9	22.5	59.2	27.5
	$2\text{k}\times 32$	3.3	32.2	21.4	21.9	56.6	27.1
	$4\text{k}\times 16$	4.2	30.9	21.3	21.9	56.6	27.0
	$8\text{k}\times 8$	5.6	28.6	20.7	21.6	56.0	26.5
Qwen3-1.7B-Base	$16\text{k}\times 4$	4.0	30.5	21.2	21.3	57.2	26.8

Appendix

Table 13: Detailed avg@16 Evaluation Results Corresponding to Figure 3 .

CoT Distillation

Chain-of-thought distillation transfers a model’s intermediate reasoning to smaller models so they can solve tasks with less compute.

See all 78 papers

MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation
Aug 30, 2026Yangsong Lan, Renkai Hu, HongKai Zheng +4CoT Distillation Language Model Distillation
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China · 2The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China
Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decoding
May 4, 2026Taewon Yun, Jisu Shin, Jeonghwan Choi +2CoT Distillation Efficient Language Model Reasoning
KAIST · UNIST
LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning
Apr 15, 2026Sumeet Ramesh Motwani, Daniel Nichols, Charles London +17Long-Horizon LLM Reasoning CoT Reasoning
University of Oxford · Lawrence Livermore National Laboratory (LLNL) · MBZUAI +4

Aug 30, 2026J/K move · Enter open · S save

MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation

Yangsong Lan, Renkai Hu, HongKai Zheng, Bo Zhang+3

College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China · 2The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China

CoT Distillation Language Model Distillation Knowledge Distillation Large Reasoning Models

Open paper AskPDF arXiv

Recent advances in large reasoning models (LRMs) have shown strong performance on complex problems through long chain-of-thought (Long CoT) reasoning. However, distilling such trajectories into smaller student models remains challenging: direct Long CoT supervision often provides limited gains and can be less effective than concise Short CoT rationales. In this work, we investigate this phenomenon from a gradient-centric perspective. Our analysis shows that Long CoT induces larger gradient magnitudes and more concentrated update directions than Short CoT, with this effect becoming more pronounced as student model capacity increases. These findings suggest that effective Long CoT distillation requires balancing the reasoning information density of reasoning trajectories with their distributional alignment to the student model. Motivated by this insight, we propose Model Interporlation Distillation (MI-Distillation), a framework that constructs a continuous Instruct-Reasoning data spectrum through model interpolation. To select suitable trajectories from this spectrum, we further introduce Sequential Learnable Surprisal Score (SeqLSS), which favors reasoning paths that are both informative and learnable for the student. Extensive experiments on reasoning benchmarks show that MI-Distillation consistently improves small model CoT distillation over strong Long CoT baselines.

Abstract

CoT Distillation

Explore similar work

MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation

Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decoding

LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning