On the Role of Reasoning Patterns in the Generalization Discrepancy of Long Chain-of-Thought Supervised Fine-Tuning

Zhaoyi Li; Xiangyu Xi; Zhengyu Chen; Wei Wang; Gangwei Jiang; Ranran Shen; Linqi Song; Ying Wei

cs.CLApr 2, 2026

On the Role of Reasoning Patterns in the Generalization Discrepancy of Long Chain-of-Thought Supervised Fine-Tuning

Authors: Zhaoyi Li, Xiangyu Xi, Zhengyu Chen, Wei Wang, Gangwei Jiang, Ranran Shen, Linqi Song, Ying Wei, +1 more

Organizations: University of Science and Technology of China · Meituan LongCat Team · City University of Hong Kong · Zhejiang University

Abstract

Supervised Fine-Tuning (SFT) on long Chain-of-Thought (CoT) trajectories has become a pivotal phase in building large reasoning models. However, how CoT trajectories from different sources influence the generalization performance of models remains an open question. In this paper, we conduct a comparative study using two sources of verified CoT trajectories generated by two competing models, DeepSeek-R1-0528 and gpt-oss-120b, with their problem sets controlled to be identical. Despite their comparable performance, we uncover a striking paradox: lower training loss does not translate to better generalization. SFT on DeepSeek-R1-0528 data achieves remarkably lower training loss, yet exhibits significantly worse generalization performance on reasoning benchmarks compared to those trained on gpt-oss-120b. To understand this paradox, we perform a multi-faceted analysis probing token-level SFT loss and step-level reasoning behaviors. Our analysis reveals a difference in reasoning patterns. gpt-oss-120b exhibits highly convergent and deductive trajectories, whereas DeepSeek-R1-0528 favors a divergent and branch-heavy exploration pattern. Consequently, models trained with DeepSeek-R1 data inherit inefficient exploration behaviors, often getting trapped in redundant exploratory branches that hinder them from reaching correct solutions. Building upon this insight, we propose a simple yet effective remedy of filtering out frequently branching trajectories to improve the generalization of SFT. Experiments show that training on selected DeepSeek-R1-0528 subsets surprisingly improves reasoning performance by up to 5.1% on AIME25, 5.5% on BeyondAIME, and on average 3.6% on five benchmarks.

Figures & tables

Base Model	Data Source	MATH500	AIME24	AIME25	BeyondAIME	HMMT25	Avg
Qwen2.5-7B	DeepSeek-R1	$95.0\%$ $\pm 0.02\%$	$52.1\%$ $\pm 0.03\%$	$44.8\%$ $\pm 0.03\%$	$16.7\%$ $\pm 0.02\%$	$38.7\%$ $\pm 0.03\%$	$49.5\%$
Qwen2.5-7B	gpt-oss-120b	$95.0\%$ $\pm 0.02\%$	$\mathbf{64.5\%}$ $\pm 0.03\%$	$\mathbf{46.7\%}$ $\pm 0.03\%$	$\mathbf{23.9\%}$ $\pm 0.03\%$	$\mathbf{42.9\%}$ $\pm 0.03\%$	$\mathbf{54.6\%}$ ( +5.1% )
Qwen2.5-32B	DeepSeek-R1	$98.4\%$ $\pm 0.01\%$	$78.5\%$ $\pm 0.03\%$	$73.5\%$ $\pm 0.03\%$	$37.7\%$ $\pm 0.03\%$	$69.6\%$ $\pm 0.03\%$	$71.5\%$
Qwen2.5-32B	gpt-oss-120b	$\mathbf{98.8\%}$ $\pm 0.01\%$	$\mathbf{82.0\%}$ $\pm 0.02\%$	$\mathbf{77.7\%}$ $\pm 0.03\%$	$\mathbf{45.9\%}$ $\pm 0.03\%$	$\mathbf{77.3\%}$ $\pm 0.03\%$	$\mathbf{76.3\%}$ ( +4.8% )
Llama3.1-8B	DeepSeek-R1	$86.2\%$ $\pm 0.03\%$	$20.1\%$ $\pm 0.03\%$	$18.3\%$ $\pm 0.02\%$	$4.5\%$ $\pm 0.01\%$	$18.5\%$ $\pm 0.02\%$	$29.5\%$
Llama3.1-8B	gpt-oss-120b	$\mathbf{94.8\%}$ $\pm 0.02\%$	$\mathbf{53.8\%}$ $\pm 0.03\%$	$\mathbf{41.3\%}$ $\pm 0.03\%$	$\mathbf{21.8\%}$ $\pm 0.03\%$	$\mathbf{40.6\%}$ $\pm 0.03\%$	$\mathbf{50.5\%}$ ( +21.0% )

Table 1: SFT performance comparison of different base models trained on reasoning trajectories generated by DeepSeek-R1 and gpt-oss-120b across five mathematical reasoning benchmarks. AIME24/25 results are averaged over 32 independent runs ( avg@32 ). BeyondAIME and HMMT25 results are averaged over 10 independent runs ( avg@10 ).

Panel of Figure 1 — Figure 1: (a $\sim$ d): SFT training loss comparison of different models trained on long CoT trajectories of DeepSeek-R1 and gpt-oss-120b . (e) and (f): average testing performance on five benchmarks with varying training steps and inference context length. Blue / red curves refer to experiments with gpt-oss-120b / DeepSeek-R1 -generated data, respectively.

Panel of Figure 2 — Figure 2: Token-level SFT loss analysis for the Qwen3-8B model. (a) show the token-level loss distribution after the SFT training, where blue / red bars represent experiments with DeepSeek-R1 / gpt-oss-120b trajectories. (b, c) are word clouds for the most frequent tokens in the top $10\%$ token-level loss token subset of the ( DeepSeek-R1 , gpt-oss-120b ) data.

Data Source	Representative High-Loss Tokens
DeepSeek-R1 3.538 $\rightarrow$ 1.533	‘Perhaps’, ‘Now’, ‘Another’, ‘Let’, ‘might’, ‘given’, ‘say’, ‘Similarly’, ‘earlier’, ‘probably’
gpt-oss-120b 4.081 $\rightarrow$ 1.784	‘Check’, ‘indeed’, ‘Thus’, ‘yields’, ‘Given’, ‘Let’, ‘might’, ‘Evaluate’, ‘Actually’, ‘Wait’, ‘Now’, ‘consider’, ‘compute’

Table 2: Representative key reasoning tokens in the high-loss tail part and their average loss before/after SFT training ( average-loss-before-sft

\rightarrow

average-loss-after-sft ).

Panel of Figure 3 — Figure 3: Reasoning behavior distributions (a, c) and transition matrices (b, d) for reasoning trajectories used for training and generated for solving AIME24 problems with fine-tuned Qwen3-8B. Blue / red objects: experiments with DeepSeek-R1 / gpt-oss-120b -generated data.

Panel of Figure 4 — Figure 4: Performance change ratio ( $(\text{Acc}_{\text{original}}-\text{Acc}_{\text{retrain}})/\text{Acc}_{\text{original}}$ ) on five benchmarks after randomly deleting 10% reasoning steps in each training trajectory. Blue / red bars represent experiments with DeepSeek-R1 / gpt-oss-120b -generated data, respectively.

Base Model	Data Source	MATH500	AIME24	AIME25	BeyondAIME	HMMT25	Avg
Qwen3-8B (context len: 32K)	full DeepSeek-R1 set	$96.8\%$	$63.0\%$	$54.0\%$	$24.5\%$	$48.5\%$	$57.4\%$
	removing top $10\%$	$97.4\%$ ( +0.6% )	$64.7\%$ ( +1.7% )	$55.7\%$ ( +1.7% )	$25.1\%$ ( +0.6% )	$51.5\%$ ( +3.0% )	$\mathbf{58.9\%}$ ( +1.5% )
	removing top $20\%$	$97.2\%$ ( +0.4% )	$64.5\%$ ( +1.5% )	$54.7\%$ ( +0.7% )	$25.7\%$ ( +1.2% )	$50.1\%$ ( +1.6% )	$\mathbf{58.4\%}$ ( +1.0% )
Qwen2.5-7B (context len: 32K)	full DeepSeek-R1 set	$95.0\%$	$52.1\%$	$44.8\%$	$16.7\%$	$38.7\%$	$49.5\%$
	removing top $10\%$	$95.0\%$	$54.3\%$ ( +2.2% )	$46.7\%$ ( +1.9% )	$19.3\%$ ( +2.6% )	$40.2\%$ ( +1.5% )	$\mathbf{51.1\%}$ ( +1.6% )
	removing top $20\%$	$95.8\%$ ( +0.8% )	$52.2\%$ ( +0.1% )	$44.4\%$ ( -0.4% )	$16.2\%$ ( -0.5% )	$40.0\%$ ( +1.3% )	$\mathbf{49.7\%}$ ( +0.2% )

Table 3: SFT performance after removing trajectories with top difference ratio of reasoning step number (proxy metric 1) from the DeepSeek-R1 generated dataset.

Base Model	Data Source	MATH500	AIME24	AIME25	BeyondAIME	HMMT25	Avg
Qwen3-8B (context len: 32K)	full DeepSeek-R1 set (original SFT)	$96.8\%$ $\pm 0.02\%$	$63.0\%$ $\pm 0.03\%$	$54.0\%$ $\pm 0.03\%$	$24.5\%$ $\pm 0.03\%$	$48.5\%$ $\pm 0.03\%$	$57.4\%$
	bottom $\mathbf{90\%}$ (experiment group)	$97.0\%$ $\pm 0.02\%$	$64.7\%$ $\pm 0.03\%$	$55.5\%$ $\pm 0.03\%$	$24.9\%$ $\pm 0.03\%$	$48.8\%$ $\pm 0.03\%$	$\mathbf{58.2\%}$ ( +0.8% )
	top $\mathbf{90\%}$ (control group)	$97.0\%$ $\pm 0.02\%$	$63.0\%$ $\pm 0.03\%$	$52.3\%$ $\pm 0.03\%$	$23.9\%$ $\pm 0.03\%$	$47.7\%$ $\pm 0.03\%$	$56.8\%$ ( -0.6% )
	bottom $\mathbf{80\%}$ (experiment group)	$97.0\%$ $\pm 0.02\%$	$64.7\%$ $\pm 0.03\%$	$55.7\%$ $\pm 0.03\%$	$27.1\%$ $\pm 0.03\%$	$52.3\%$ $\pm 0.03\%$	$\mathbf{59.4\%}$ ( +2.0% )
	top $\mathbf{80\%}$ (control group)	$96.8\%$ $\pm 0.02\%$	$63.3\%$ $\pm 0.03\%$	$52.1\%$ $\pm 0.03\%$	$22.5\%$ $\pm 0.02\%$	$47.4\%$ $\pm 0.03\%$	$56.4\%$ ( -1.0% )
	bottom $\mathbf{50\%}$ (experiment group)	$97.0\%$ $\pm 0.02\%$	$66.5\%$ $\pm 0.03\%$	$59.1\%$ $\pm 0.03\%$	$25.1\%$ $\pm 0.03\%$	$50.5\%$ $\pm 0.03\%$	$\mathbf{59.6\%}$ ( +2.2% )

Table 4: SFT performance using the bottom/top K% subset (in terms of the proportion of the reasoning steps containing “branching”-related keywords, proxy metric 2) of trajectories.

Base Model	Data Source	MATH500	AIME24	AIME25	BeyondAIME	HMMT25	Avg
Qwen3-8B (context len: 32K)	full DeepSeek-R1 set (original SFT)	$96.8\%$ $\pm 0.02\%$	$63.0\%$ $\pm 0.03\%$	$54.0\%$ $\pm 0.03\%$	$24.5\%$ $\pm 0.03\%$	$48.5\%$ $\pm 0.03\%$	$57.4\%$
	filtering out $20\%$ with proxy 1	$97.2\%$ $\pm 0.02\%$	$64.5\%$ $\pm 0.03\%$	$54.7\%$ $\pm 0.03\%$	$25.7\%$ $\pm 0.03\%$	$51.5\%$ $\pm 0.03\%$	$\mathbf{58.4\%}$ ( +1.0% )
	filtering out $20\%$ with proxy 2	$97.0\%$ $\pm 0.02\%$	$64.7\%$ $\pm 0.03\%$	$55.7\%$ $\pm 0.03\%$	$27.1\%$ $\pm 0.03\%$	$52.3\%$ $\pm 0.03\%$	$\mathbf{59.4\%}$ ( +2.0% )
	ablation: filtering out the longest $20\%$	$96.0\%$ $\pm 0.02\%$	$61.0\%$ $\pm 0.03\%$	$46.9\%$ $\pm 0.03\%$	$23.3\%$ $\pm 0.03\%$	$45.1\%$ $\pm 0.03\%$	$54.5\%$ ( -2.9% )
Qwen2.5-7B (context len: 32K)	full DeepSeek-R1 set (original SFT)	$95.0\%$ $\pm 0.02\%$	$52.1\%$ $\pm 0.03\%$	$44.8\%$ $\pm 0.03\%$	$16.7\%$ $\pm 0.02\%$	$38.7\%$ $\pm 0.03\%$	$49.5\%$
	filtering out $20\%$ with proxy 1	$95.8\%$ $\pm 0.02\%$	$52.2\%$ $\pm 0.03\%$	$44.4\%$ $\pm 0.03\%$	$16.2\%$ $\pm 0.02\%$	$40.0\%$ $\pm 0.03\%$	$\mathbf{49.7\%}$ ( +0.2% )

Table 5: Results of the ablation study. We compare the SFT generalization performance of filtering out the longest

20\%

trajectories with our proposed proxy metric 1 and proxy metric 2, where we filter out the most frequently branching trajectories.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Base Model	Data Source	train step	learning rate	global batch size	micro batch size
Qwen2.5-7B	DeepSeek-R1	$1836$	$1e-4$	$128$	$1$
Qwen2.5-7B	gpt-oss-120b	$1750$	$1e-4$	$128$	$1$
Qwen2.5-32B	DeepSeek-R1	$3672$	$8e-5$	$64$	$1$
Qwen2.5-32B	gpt-oss-120b	$3501$	$8e-5$	$64$	$1$
Llama3.1-8B	DeepSeek-R1	$1776$	$1e-4$	$128$	$1$
Llama3.1-8B	gpt-oss-120b	$1733$	$1e-4$	$128$	$1$

Appendix

Table 6: Training hyperparameters for the long CoT SFT experiments in the main table.

Behavior	Definition
Propose	Explore a new idea, set up a hypothesis, or suggest an alternative path. It represents the “divergent” phase of reasoning.
Deduce	Execute a mathematical operation or make a direct logical inference based on the immediately preceding established facts or the current hypothesis. It represents the “convergent, linear” phase.
Verify	Pause the forward progression to double-check an intermediate calculation, verify a condition, or assess if the current path makes sense.
Backtrack	Explicitly realize an error or a dead end, reject the current reasoning branch, and retreat to a previous state or prepares to start over.

Appendix

Table 7: Definition of four common reasoning behaviors in long CoT reasoning trajectories.

Panel of Figure 5 — Figure 5: The prompt template for annotating reasoning steps with four behavior labels.

Panel of Figure 6 — Figure 6: Token-level SFT loss analysis for the Qwen3-8B model. (a) and (d) show the token-level loss distribution before and after the SFT training, where blue / red bars represent experiments with DeepSeek-R1 / gpt-oss-120b -generated trajectories, respectively. (b, c) and (e, f) are word clouds for the most frequent tokens in the top $10\%$ token-level loss token subset of the ( DeepSeek-R1 , gpt-oss-120b ) data before and after SFT, respectively.

Panel of Figure 7 — Figure 7: Token-level SFT loss analysis for the Qwen2.5-7B model. (a) and (d) show the token-level loss distribution before and after the SFT training, where blue and red bars correspond to using DeepSeek-R1 and gpt-oss-120b reasoning trajectories to train the model and calculate loss, respectively. (b) and (e) are word clouds for the most frequent tokens in the top $10\%$ token-level loss token subset of the DeepSeek-R1-0528 generated data before and after the SFT training. (c) and (f) are word clouds for the most frequent tokens in the top $10\%$ token-level loss token subset of the gpt-oss-120b generated data before and after the SFT training.

Panel of Figure 8 — Figure 8: Token-level SFT loss analysis for the Llama3.1-8B model. (a) and (d) show the token-level loss distribution before and after the SFT training, where blue and red bars correspond to using DeepSeek-R1 and gpt-oss-120b reasoning trajectories to train the model and calculate loss, respectively. (b) and (e) are word clouds for the most frequent tokens in the top $10\%$ token-level loss token subset of the DeepSeek-R1-0528 generated data before and after the SFT training. (c) and (f) are word clouds for the most frequent tokens in the top $10\%$ token-level loss token subset of the gpt-oss-120b generated data before and after the SFT training.

Panel of Figure 9 — Figure 9: Reasoning behavior analysis. Reasoning behavior distribution ((a) and (c)) and transition matrix ((b) and (d)) for reasoning trajectories collected for the SFT training and generated for AIME testing problems with the trained Llama3.1-8B base model.

Panel of Figure 10 — Figure 10: Performance change ratio on five benchmarks (MATH500, AIME24/25, BeyondAIME, HMMT25) after randomly deleting 10%/20%/30% reasoning steps in each training datum in the Qwen3-8B SFT experiments. Blue / red bars represent results trained with DeepSeek-R1 / gpt-oss-120b generated reasoning trajectories, respectively.

Base Model	Data Source	MATH500	AIME24	AIME25	BeyondAIME	HMMT25	Avg
Qwen2.5-7B (context len: 32K)	full DeepSeek-R1 set (original SFT)	$95.0\%$	$52.1\%$	$44.8\%$	$16.7\%$	$38.7\%$	$49.5\%$
	removing top $10\%$	$95.0\%$	$54.3\%$	$46.7\%$	$19.3\%$	$40.2\%$	$\mathbf{51.1\%}$ ( +1.6% )
	removing top $20\%$	$95.8\%$	$52.2\%$	$44.4\%$	$16.2\%$	$40.0\%$	$\mathbf{49.7\%}$ ( +0.2% )
	bottom $\mathbf{90\%}$ (experiment group)	$95.4\%$	$53.7\%$	$46.5\%$	$19.3\%$	$41.8\%$	$\mathbf{51.3\%}$ ( +1.8% )
	top $\mathbf{90\%}$ (control group)	$95.0\%$	$51.9\%$	$44.9\%$	$17.7\%$	$37.1\%$	$49.3\%$ ( -0.2% )
	bottom $\mathbf{80\%}$ (experiment group)	$94.6\%$	$55.5\%$	$47.2\%$	$18.5\%$	$40.1\%$	$\mathbf{51.2\%}$ ( +1.7% )

Appendix

Table 8: More experiment results on filtering out part of the frequently branching SFT training data with Qwen2.5-7B, where we set the context limit = 32K at inference time.

Panel of Figure 11 — Figure 11: Case study: a highly exploratory snippet of a reasoning trajectory generated by DeepSeek-R1-0528 .

Panel of Figure 12 — Figure 12: Case study: a highly exploratory snippet of a reasoning trajectory generated by DeepSeek-R1-0528 -distilled Qwen3-8B.

Explore similar work

Apr 8, 2026

Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability

A prevailing narrative in LLM post-training holds that supervised finetuning (SFT) memorizes while reinforcement learning (RL) generalizes. We revisit this claim for reasoning SFT with long chain-of-thought (CoT) supervision and find that cross-domain generalization is not absent but conditional, jointly shaped by optimization dynamics, training data, and base-model capability. Some reported failures are under-optimization artifacts: cross-domain performance first degrades before recovering and improving with extended training (a dip-and-recovery pattern), so shorttraining checkpoints can underestimate generalization. Data quality and structure both matter: low-quality solutions broadly hurt generalization,while verified long-CoT traces yield consistent cross-domain gains. Model capability is essential: stronger models internalize transferable procedural patterns (e.g., backtracking) even from a toy arithmetic game, while weaker ones imitate surface verbosity. This generalization is asymmetric, however: reasoning improves while safety degrades, reframing the question from whether reasoning SFT generalizes to under what conditions and at what cost.

Qihan Ren, Peng Wang, Ruikun Cai +8

Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · University of Science and Technology of China

LLM Fine-Tuning Cross-Domain Generalization CoT Reasoning

Ask PDF

May 10, 2026

Memorize Theorems, Not Instances: Probing SFT Generalization through Mathematical Reasoning

Supervised Fine-Tuning (SFT) is widely used for task-specific adaptation, yet recent work shows it systematically undermines reasoning generalization. We argue the root cause is not memorization itself, but its target: vanilla SFT drives models to exploit and memorize spurious surface correlations in problem-solution pairs, leaving them brittle to superficial input variations. To address this, we propose Theorem-SFT, which reorients supervision toward explicit theorem application by teaching models how rules are invoked rather than what answers look like. Theorem-SFT yields consistent gains across benchmarks and model families: +8.8% on MATH (LLaMA3.2-3B-Instruct) and +20.27% on GeoQA (Qwen2.5-VL-7B-Instruct) without modality-specific re-training. Fine-tuning MLP layers alone matches full-layers performance, implicating feed-forward components as the primary locus of reasoning rules. Our findings reframe the debate: Generalization failures stem not from memorization as a mechanism, but from memorizing the wrong inductive targets.

Ruiying Peng, Mengyu Yang, Jing Lei +3

Tsinghua Shenzhen International Graduate School · Huawei Technologies

Supervised Fine-Tuning Memorization in Language Models OOD Generalization Mathematical Reasoning LLM Reasoning

Ask PDF

Mar 12, 2026

Not Just the Destination, But the Journey: Reasoning Traces Causally Shape Generalization Behaviors

Chain-of-Thought (CoT) is often viewed as a window into LLM decision-making, yet recent work suggests it may function merely as post-hoc rationalization. This raises a critical alignment question: Does the reasoning trace causally shape model generalization independent of the final answer? To isolate reasoning's causal effect, we design a controlled experiment holding final harmful answers constant while varying reasoning paths. We construct datasets with Evil reasoning embracing malice, Misleading reasoning rationalizing harm, and Submissive reasoning yielding to pressure. We train models (0.6B--14B parameters) under multiple paradigms, including question-thinking-answer (QTA), question-thinking (QT), and thinking-only (T-only), and evaluate them in both think and no-think modes. We find that: (1) CoT training could amplify harmful generalization more than standard fine-tuning; (2) distinct reasoning types induce distinct behavioral patterns aligned with their semantics, despite identical final answers; (3) training on reasoning without answer supervision (QT or T-only) is sufficient to alter behavior, proving reasoning carries an independent signal; and (4) these effects persist even when generating answers without reasoning, indicating deep internalization. Our findings demonstrate that reasoning content is causally potent, challenging alignment strategies that supervise only outputs.

Pengcheng Wen, Yanxu Zhu, Jiapeng Sun +5

Hong Kong University of Science and Technology · Beijing Jiaotong University

CoT Reasoning LLM Alignment Emergent Misalignment in Language Models Causal Interventions in Language Models

Ask PDF