Is Next-Chunk Reasoning RL Really Better than SFT? Revisiting Training Strategies under no-CoT Data

Yinhao Tang; Youqing Fang; Yanan Sun; Jiangning Liu; Ziyi Wang; Xun Zhao; Weiming Zhang; Bin Liu

cs.AIAug 24, 2026

Is Next-Chunk Reasoning RL Really Better than SFT? Revisiting Training Strategies under no-CoT Data

Authors: Yinhao Tang, Youqing Fang, Yanan Sun, Jiangning Liu, Ziyi Wang, Xun Zhao, Weiming Zhang, Bin Liu,

Organizations: University of Science and Technology of China · Shanghai AI Laboratory

Abstract

Recent work proposes next-chunk reasoning RL for leveraging no-CoT data---corpora such as worked solutions and textbook derivations that contain reasoning-rich content but lack explicit chain-of-thought annotations. The method trains a model to generate implicit reasoning traces and rewards them by their ability to predict the next chunk of text. While promising, existing evaluations primarily compare against conventional SFT baselines, leaving open whether the gains come from the RL formulation itself or from more effectively exposing the model to no-CoT data. We address this question with a controlled study of next-chunk reasoning RL and a simple but previously overlooked alternative: Mixed SFT, a single supervised fine-tuning stage that jointly trains on no-CoT and long-CoT data. Despite its simplicity, Mixed SFT achieves a clearly higher post-RLVR performance ceiling than next-chunk reasoning RL while requiring over 60 times less training compute. The advantage is consistent across in-domain mathematical reasoning and out-of-domain reasoning tasks. Moreover, we show that higher pre-RLVR accuracy does not necessarily translate into higher post-RLVR accuracy, highlighting the need to evaluate no-CoT training strategies in the context of the full post-training pipeline.

Figures & tables

Figure 1: Compared strategies for leveraging no-CoT data ( top ) and their post-RLVR performance ( bottom ). NCR applies RL rewards over predicted tokens (A: NTR) or sentences (B: NSR). Sequential SFT (C) trains on no-CoT and long-CoT data in two stages, while Mixed SFT (D) combines both in a single stage. All strategies share the same RLVR stage; bar charts show post-RLVR accuracy on four representative benchmarks, with Mixed SFT consistently achieving the highest scores.

Method	Start Model	No-CoT SFT	Mixed SFT
Next-token Reasoning
RPT	reasoning	✓	✗
RLP	base	✓	✗
RMT	reasoning	✓	✗
Next-sentence Reasoning
RLPT	reasoning	✗	✗

Table 1: Summary of next-chunk reasoning RL methods, reporting their starting model (base or reasoning), whether they compare against a no-CoT SFT baseline, and whether they compare against Mixed SFT. RPT ( 6 ) , RLP ( 9 ) , RMT ( 23 ) , RLPT ( 13 ) , PretrainZero ( 28 ) .

Method	ID Reasoning						OOD Reasoning
Method	AIME24	AIME25	AIME26	HMMT25	HMMT26	IMO-Ans.	HLE	GPQA-Dia.	MMLU-Pro
Reasoning SFT	73.33	64.17	54.42	34.16	32.88	34.50	6.91	46.21	74.89
+ RLVR	81.98	76.67	65.10	43.33	39.39	47.25	7.22	56.94	73.95
Sequential SFT	71.25	62.81	54.42	34.17	32.88	35.75	6.52	46.21	68.82
+ RLVR	74.90	69.27	56.35	32.92	35.23	41.25	6.79	54.55	71.10
Mixed SFT	45.52	42.50	20.94	10.42	15.91	12.50	6.71	36.24	56.61

Table 2: Performance of the compared no-CoT data utilization strategies, before and after RLVR, on in-domain (ID) and out-of-domain (OOD) reasoning benchmarks.

Figure 2: GPU-hour training cost of each strategy on the same no-CoT data. NTR and NSR are measured at 160 steps, where both reach peak pre-RLVR accuracy.

Figure 3: Pre-RLVR and post-RLVR accuracy averaged over all nine reasoning benchmarks, shown as per-method dumbbells.

Figure 4: Post-RLVR training curves of all five strategies on AIME 2024, AIME 2025, and GPQA-Diamond under the same RLVR stage.

Figure 5: Per-token entropy, reasoning accuracy, and no-reasoning accuracy on

2{,}048

randomly sampled high-entropy tokens, measured with NTR checkpoints across training steps.

Panel of Figure 6 — Figure 6: ( 6(a) ) Generation entropy and response length of vanilla NTR and the entropy-controlled variant across training steps. ( 6(b) ) Post-RLVR accuracy of the two variants on four representative benchmarks; per-benchmark numbers are in Tab. 9 .

Method	AIME24	AIME25	HMMT25	HMMT26
Mixed SFT	45.52	42.50	10.42	15.91
+ RLVR	87.50	85.73	55.00	51.52
+ NTR + RLVR	86.35	84.58	53.73	50.50
+ NSR + RLVR	87.21	84.67	53.15	49.52

Table 3: Post-RLVR performance when NTR or NSR is inserted between Mixed SFT and RLVR, compared with Mixed SFT directly followed by RLVR.

Figure 7: Accuracy and format-compliance rate of Mixed SFT across RLVR training steps.

Method	No-CoT Probe
Sequential SFT $+$ RLVR	59.19
Mixed SFT $+$ RLVR	68.63

Table 4: Post-RLVR avg@8 on No-CoT Probe, a 100-problem retention probe randomly drawn from the no-CoT training corpus.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	NTR	NSR	RLVR
Global batch size	4096	256	256
Rollouts per prompt	64	16	16
Max response length	2K	32K	32K
Learning rate	$1\text{e}{-}6$	$1\text{e}{-}6$	$1\text{e}{-}6$
KL loss	off	off	off

Appendix

Table 5: RL hyperparameters for NTR, NSR, and RLVR. Max response length is reported in tokens.

Parameter	Value
max tokens	$64\times 1024$
top- $k$	0
top- $p$	0.999
temperature	0.8

Appendix

Table 6: Decoding configuration used for evaluation across all methods and benchmarks.

Method	AIME24	AIME25	AIME26	HMMT25	HMMT26	IMO-Ans.	HLE †	GPQA-Dia.	MMLU-Pro †
Reasoning SFT	90.00	90.00	83.33	76.67	63.64	68.00	6.91	68.69	74.89
+ RLVR	93.33	93.33	86.67	80.00	69.70	66.00	7.22	77.27	73.95
Sequential SFT	90.00	90.00	86.67	76.67	66.67	58.00	6.52	67.17	68.82
+ RLVR	90.00	90.00	86.67	70.00	66.67	72.00	6.79	75.76	71.10
Mixed SFT	83.33	76.67	60.00	53.33	51.52	50.00	6.71	71.72	56.61
+ RLVR	93.33	93.33	70.42	90.00	87.88	76.00	9.24	77.27	75.84

Appendix

Table 7: Per-benchmark pass@

n

comparison of different no-CoT data utilization strategies before and after RLVR. The setup mirrors Tab. 2 but reports pass@

n

rather than avg@

n

. † HLE and MMLU-Pro are reported as pass@1, i.e., a single sample per question, so their values coincide with those in the main table.

Method	AIME24	AIME25	AIME26	HMMT25	HMMT26	IMO-Ans.	HLE	GPQA-Dia.	MMLU-Pro
Mixed SFT	45.52	42.50	20.94	10.42	15.91	12.50	6.71	36.24	56.61
Mixed SFT $+$ RLVR	87.50	85.73	70.42	55.00	51.52	54.00	9.24	60.98	75.84
Mixed SFT $+$ NTR $+$ RLVR	86.35	84.58	70.21	53.73	50.50	52.25	7.69	56.44	74.36
Mixed SFT $+$ NSR $+$ RLVR	87.21	84.67	69.35	53.15	49.52	53.26	8.13	56.36	74.86

Appendix

Table 8: Full per-benchmark version of the Mixed SFT ablation from Sec. 4 . Inserting an extra next-chunk reasoning RL stage (NTR or NSR) between Mixed SFT and RLVR does not improve over directly applying RLVR on Mixed SFT.

Figure 8: Two representative format-instability failure modes of Mixed SFT before RLVR: a no-think direct answer with the <think> block missing, and a response containing multiple <think> segments.

	ID Reasoning						OOD Reasoning
Method	AIME24	AIME25	AIME26	HMMT25	HMMT26	IMO-Ans.	HLE	GPQA-Dia.	MMLU-Pro
Vanilla NTR	87.50	84.38	69.38	50.42	47.16	46.50	7.68	57.70	74.03
Entropy-Ctrl. NTR	86.42	83.71	67.50	49.33	45.45	44.75	7.21	56.81	73.66

Appendix

Table 9: Per-benchmark post-RLVR accuracy of vanilla NTR vs. the entropy-controlled variant from Sec. 4 . Both rows are followed by the same RLVR stage. Across all nine benchmarks, suppressing the entropy and length collapse lowers the final RLVR ceiling rather than raising it.

Figure 9: Pre-RLVR training curves of NTR and NSR on three representative benchmarks under a 32K-token evaluation context. Both stages start from the same Reasoning SFT checkpoint and are evaluated periodically up to step 250 of their respective reconstruction-RL training.

Figure 10: Representative NTR prediction cases drawn from the top-20% highest-entropy targets of the no-CoT corpus. Even on these nominally uncertain targets, many cases can be solved by short, local-completion reasoning over a few preceding tokens rather than long-horizon inference, allowing the reconstruction reward to be satisfied without genuine reasoning.

Figure 11: Representative NSR training trajectory: a no-CoT prefix, the model’s intermediate thinking, and the predicted next sentence judged by the generative reward model.

Figure 12: Representative long-CoT training sample: a math problem paired with a step-by-step reasoning trajectory and final answer.

Figure 13: Representative no-CoT training sample: a math problem with a compressed solution that omits explicit reasoning traces.

LLM Post-Training

Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.

See all 249 papers

Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently
Jun 22, 2026Stanley Wei, Juno KimRL for Language Model Reasoning Efficient Language Model Reasoning
1Princeton University · University of California, Berkeley.
DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoning
Mar 11, 2026Hanxu Hu, Yuxuan Wang, Maggie Huan +4RL for Language Model Reasoning RL Fine-Tuning
University of Zurich · University of Pennsylvania · ETH Zurich +2
SLAT: Segment-Level Adaptive Trimming for Efficient CoT Reasoning
May 29, 2026Jian Yao, Xiongcai Luo, Ran Cheng +1RL for Language Model Reasoning Efficient Language Model Inference
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University · Bytedance · The Hong Kong Polytechnic University-Daya Bay Technology and Innovation Research Institute, Huizhou,Guangdong Province, China.

Jun 22, 2026J/K move · Enter open · S save

Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently

Stanley Wei, Juno Kim

1Princeton University · University of California, Berkeley.

RL for Language Model Reasoning Efficient Language Model Reasoning Reinforcement Learning with Verifiable Rewards RL Fine-Tuning LLM Reasoning

Open paper AskPDF arXiv

Recent advances in large language models (LLMs) have demonstrated that reinforcement fine-tuning of pretrained base models can lead to significant gains in reasoning performance at inference time. In this work, we theoretically analyze why reinforcement fine-tuning induces better reasoning ability than purely supervised fine-tuning (SFT) methods. We model chain-of-thought (CoT) reasoning as a pathfinding problem on graphs and compare the popular method of reinforcement learning with verifiable rewards (RLVR) against traditional SFT. We prove that SFT, when trained on golden shortest paths without negative examples, fails to learn how to efficiently backtrack. In contrast, an RLVR-trained model can learn how to efficiently backtrack from dead ends using only outcome reward. This leads to an exponential separation in inference-time compute between the two methods, and demonstrates that RLVR leads the model to learn the location of difficult decisions in a reasoning chain, ultimately allowing for better allocation of inference-time compute. Finally, we show that the reasoning traces of an RLVR model can be distilled to train a base model to backtrack efficiently as well.

Abstract

LLM Post-Training

Explore similar work

Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently

DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoning

SLAT: Segment-Level Adaptive Trimming for Efficient CoT Reasoning