Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners

Xin Xu; Clive Bai; Kai Yang; Tianhao Chen; Yang Wang; Saiyong Yang; Can Yang

cs.LGSep 30, 2025

Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners

Authors: Xin Xu, Clive Bai, Kai Yang, Tianhao Chen, Yang Wang, Saiyong Yang, Can Yang

Organizations: LLM Department, Tencent · The Hong Kong University of Science and Technology · The University of Hong Kong

Abstract

Reinforcement Learning with Verifiable Reward (RLVR) effectively solves complex tasks but demands extremely long context lengths during training, leading to substantial computational costs. While multi-stage training can partially mitigate this, starting with overly short contexts often causes irreversible performance degradation, ultimately failing to reduce overall training compute significantly. In this paper, we introduce Thinking-Free Policy Initialization (TFPI), a simple yet effective adaptation to RLVR that bridges long Chain-of-Thought (CoT) distillation and standard RLVR. TFPI employs a simple ThinkingFree operation, explicitly discarding the thinking content via a direct </think> append, to reduce token usage during inference. Training with ThinkingFree-adapted inputs improves performance and lowers token consumption, even in the original slow-thinking mode. Extensive experiments across various benchmarks have shown that TFPI accelerates RL convergence, achieves a higher performance ceiling, and yields more token-efficient reasoning models without specialized rewards or complex training designs. With TFPI only, we train a 4B model to reach 89.0% accuracy on AIME24 and 65.5% on LiveCodeBench using less than 4K H20 hours.

Figures & tables

Figure 1: Our proposed TFPI accelerates the convergence of RLVR to a higher performance ceiling (left) and yields more token-efficient reasoning models (right). Left: avg@32 versus training compute, measured in H20 hours. “Direct RL” refers to directly training Qwen3-4B with a 32K context window using DAPO, while “TFPI + RL” denotes running 32K-context DAPO after initialization with our 3-stage TFPI. The x-axis for TFPI uses a linear scale during the TFPI phase, followed by a logarithmic scale, with the transition indicated by a black vertical line. Right: Average accuracy on 4 reasoning datasets (AIME24/25, Beyond AIME, and GPQA) versus average output tokens. Points in the upper-left region indicate better performance. Baseline names and their corresponding numbers are listed in Table 3 . Red dots denote different stages of our TFPI.

Figure 2: Results of the meta-experiment on the ThinkingFree operation. Left: Average output tokens in thinking mode and ThinkingFree mode on AIME25. Right: Evolution of avg@32 and average output tokens on AIME24 with thinking-mode evaluation over training steps under 4K training response length (both training in thinking mode and ThinkingFree mode) .

DeepSeek-Distill-Qwen-1.5B
Models	Mathematics			Multi-Task	Code	Instruction	Overall
	AIME 24	AIME 25	Beyond AIME	GPQA	LiveCode	IFEval	Overall
	Avg@32	Avg@32	Avg@8	Avg@8	Avg@8	Avg@4	Avg.
Initial Model	29.6	23.0	8.7	16.3	17.7	36.6	22.0
- Direct RL	33.9	27.1	11.9	19.2	18.2	41.8	25.3
- TFPI stage 1	32.1	26.9	13.9	29.3	18.4	39.5	26.7

Table 1: Results of TFPI vs. direct RL across different benchmarks. “ Avg@k ” denotes the average accuracy (%) over

k

random generations (i.e., pass@1 ). All models are evaluated in thinking mode. The total training compute for the 3 stages of TFPI equals that of “Direct RL” for fair comparison. Darker colors in the cell background denote better results within each model group.

Models	Mathematics			Multi-Task	Code	Instruction	Overall
	AIME 24	AIME 25	Beyond AIME	GPQA	LiveCode	IFEval	Overall
	Avg@32	Avg@32	Avg@8	Avg@8	Avg@8	Avg@4	Avg.
DeepSeek R1 ∗	79.8	65.0	42.4	71.5	64.3	86.1	68.2
Seed-1.5-Thinking ∗	86.7	74.0	48.0	77.3	64.9	87.4	73.0
Claude4 Opus Thinking ∗	-	75.5	-	79.6	48.9	89.7	-
Qwen3-235B-Thinking ∗	85.7	81.5	-	71.1	55.7	83.4	-

Table 2: Results (%) of RL after TFPI (“TFPI+RL”) vs. “Direct RL” across different benchmarks. “ Avg@k ” denotes the average accuracy (%) over

k

random generations (i.e., pass@1 ). For LRMs marked with “*”, results are taken from the corresponding reports (see Appendix C.4 ); results of 4B models are from our own runs with 48K response length. All models are evaluated in thinking mode. The total training compute for “TFPI+RL” is matched to that of “Direct RL” for fair comparison. Darker colors in the cell background denote better results.

Models	AIME 24		AIME 25		Beyond AIME		GPQA		Overall
Models	avg@32	Toks	avg@32	Toks	avg@8	Toks	avg@8	Toks	Avg.	Toks
1) TLMRE-DS-1.5B ( $\alpha=0.1$ )	27.6	12.9	24.8	12.5	9.2	11.9	14.8	7.5	19.1	11.2
2) AdaptThink-1.5B ( $\delta=0.05$ )	28.1	8.0	22.6	7.9	10.0	4.8	14.8	5.1	18.9	6.5
3) AutoThink-DS-1.5B-Stage3	30.3	10.2	25.2	9.1	9.4	8.7	17.4	7.0	20.6	8.7
4) Laser-DE-L4096-1.5B	30.3	8.3	24.9	7.4	9.7	7.4	21.1	4.7	21.5	6.9
5) AutoThink-Stage3 ∗	38.9	8.7	28.9	7.7	11.6	7.8	27.3	5.4	26.7	7.4

Table 3: Comparison of the Thinking-Free inference mode of TFPI with efficient reasoning baselines across various reasoning tasks. “ Avg@k ” denotes the average accuracy (in %) over k generations (i.e., pass@1 ), and “ Toks ” indicates the average output length in thousands of tokens (K). Models with “*” are trained from DeepScaleR-1.5B , while the remaining are from DS-1.5B . Darker cell background colors indicate better results.

Figure 3: Behaviour-Level Analysis of DS-1.5B over the TFPI Training Course. The ratio of verification steps and the average output tokens over training steps on the training set in thinking-free mode ( Left ) and on AIME25 in thinking mode ( Right ) in 3 stages of TFPI.

Panel of Figure 4 — Figure 4: Parameter-Level Analysis. Left: PCA projection of model parameters from DS-1.5B to final checkpoints. TFPI (blue) starts at A , passes through intermediate points ( B1 , B2 , B3 ), and ultimately converges near the Direct RL final checkpoint ( C ). Right: Cosine similarity between parameter updates of TFPI-trained checkpoints and (C-A) across layers during training.

Figure 5: Left: For TFPI with DS-1.5B on AIME25 in thinking mode, showing the average number of answer tokens (excluding the thinking part) and the ratio of answer length to total response length over training steps. Right: For long CoT RL with DS-1.5B , showing the average number of tokens during rollout on the training set over training steps, with and without the TFPI stage.

Stage Schedule Sensitivity, Qwen3-4B
Models	AIME 24	AIME 25	Beyond AIME	GPQA	LiveCode	IFEval	Overall
$4\rm{K}\rightarrow 8\rm{K}\rightarrow 16\rm{K}$	79.9	70.6	46.7	58.5	57.0	70.2	63.8
$8\rm{K}\rightarrow 16\rm{K}$	78.0	70.2	46.7	59.5	55.8	64.5	62.4
$16\rm{K}$ only	77.1	71.5	45.8	56.4	54.4	66.0	61.9
Multi-Stage Direct RL, Qwen3-4B
TFPI Stage 3	79.9	70.6	46.7	58.5	57.0	70.2	63.8

Table 4: Additional results of TFPI. All models are evaluated in thinking mode.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

	Thinking Mode	Thinking-Free Mode
DS-1.5B	$T=0.6$ , top- $p$ = 0.95, and top- $k$ = $-1$	$T=0.7$ , top- $p$ = 0.8, and top- $k$ = $20$
Qwen3-4B	$T=0.6$ , top- $p$ = 0.95, and top- $k$ = $20$	$T=0.7$ , top- $p$ = 0.8, and top- $k$ = $20$

Appendix

Table 5: Decoding Parameters of Meta-Experiment in Section 3.1

	DS-1.5B	Qwen3-4B	DS-7B	DS-1.5B	Qwen3-4B	DS-7B
TFPI Stage 1	2K, 1K steps	4K, 100 steps	2K, 1K steps	0.84 kh	0.35 kh	1.73 kh
TFPI Stage 2	4K, 440 steps	8K, 56 steps	4K, 232 steps	0.62 kh	0.52 kh	0.82 kh
TFPI Stage 3	8K, 440 steps	16K, 64 steps	8K, 144 steps	1.21 kh	0.91 kh	0.94 kh
TFPI Total	-	-	-	2.66 kh	1.79 kh	3.50 kh
Direct RLVR	16K, 456 steps	32K, 20 steps	16K, 280 steps	2.67 kh	1.9 kh	3.52 kh
TFPI: RLVR	16K, 472 steps	32K, 192 steps	536 steps	2.49 kh	13.5 kh	6.55 kh

Appendix

Table 6: Training Steps and Time of Main Experiments. “kh” denotes one thousand H20 Hours.

	DS-1.5B	Qwen3-4B	DS-7B	DS-1.5B	Qwen3-4B	DS-7B
TFPI Stage 1	$2.37\times 10^{9}$	$5.00\times 10^{8}$	$2.75\times 10^{9}$	69.27	66.28	73.73
TFPI Stage 2	$2.26\times 10^{9}$	$5.02\times 10^{8}$	$1.20\times 10^{9}$	69.54	66.67	73.79
TFPI Stage 3	$3.92\times 10^{9}$	$4.94\times 10^{8}$	$1.24\times 10^{9}$	68.66	66.76	73.76
TFPI Total	$8.55\times 10^{9}$	$1.50\times 10^{9}$	$5.19\times 10^{9}$	-	-	-
Direct RLVR	$7.48\times 10^{9}$	$6.48\times 10^{8}$	$4.77\times 10^{9}$	68.42	61.57	72.83
TFPI: RLVR	$6.26\times 10^{9}$	$5.80\times 10^{9}$	$7.98\times 10^{9}$	66.20	60.08	72.58

Appendix

Table 7: Total processed tokens and the average MFU (in %) of Main Experiments.

DeepSeek-Distill-Qwen-1.5B
Model	Training Data	Training Length	Test Length	Evaluation Details
TLMRE-DS-1.5B ( $\alpha=0.1$ )	3.2K examples from NuminaMath	–	32K	10 generations for AIME24
AdaptThink-1.5B ( $\delta=0.05$ )	DeepScaleR-40K	16K	16K	$T=0.6$ , 16 generations for AIME24
AutoThink-DS-1.5B-Stage3	DeepScaleR-40K	8K $\rightarrow$ 16K $\rightarrow$ 24K	–	$T=0.6$ , 16 generations
Laser-DE-L4096-1.5B	DeepScaleR-40K	–	32K	16 generations for AIME24
DeepscaleR-1.5B-Preview

Appendix

Table 8: Training and evaluation details of efficient reasoning baselines from original papers. We unify the evaluation setting for fair comparison in Table 3 . Details are provided in Appendix C.2 and C.3 .

Figure 6: Cosine similarity between parameter updates of TFPI-trained checkpoints and an early “Direct RL” update of DS-1.5B across layers during training.

Models	Mathematics			Multi-Task	Code	Instruction	Overall
	AIME 24	AIME 25	Beyond AIME	GPQA	LiveCode	IFEval	Overall
	Avg@32	Avg@32	Avg@8	Avg@8	Avg@8	Avg@4	Avg.
DeepSeek R1 ∗	79.8	65.0	42.4	71.5	64.3	86.1	68.2
Seed-1.5-Thinking ∗	86.7	74.0	48.0	77.3	64.9	87.4	73.0
Claude4 Opus Thinking ∗	-	75.5	-	79.6	48.9	89.7	-
Qwen3-235B-Thinking ∗	85.7	81.5	-	71.1	55.7	83.4	-

Appendix

Table 9: Results (%) of RL after TFPI (“TFPI+RL”) vs. “Direct RL” across different benchmarks. “ Avg@k ” denotes the average accuracy (%) over

k

random generations (i.e., pass@1 ). For LRMs marked with “*”, results are taken from the corresponding reports (see Appendix C.4 ); all other results are from our own runs. All models are evaluated in thinking mode. The total training compute for “TFPI+RL” is matched to that of “Direct RL” for fair comparison.

DeepSeek-Distill-Qwen-1.5B
Models	AIME 24		AIME 25		Beyond AIME		GPQA		Overall
Models	avg@32	Toks	avg@32	Toks	avg@8	Toks	avg@8	Toks	Avg.	Toks
TLMRE-DS-1.5B ( $\alpha=0.1$ )	27.6	12.9	24.8	12.5	9.2	11.9	14.8	7.5	19.1	11.2
AdaptThink-1.5B ( $\delta=0.05$ )	28.1	8.0	22.6	7.9	10.0	4.8	14.8	5.1	18.9	6.5
AutoThink-DS-1.5B-Stage3	30.3	10.2	25.2	9.1	9.4	8.7	17.4	7.0	20.6	8.7
Laser-DE-L4096-1.5B	30.3	8.3	24.9	7.4	9.7	7.4	21.1	4.7	21.5	6.9

Appendix

Table 10: Comparison of the Thinking-Free inference mode of TFPI with efficient reasoning baselines across various reasoning tasks. “ Avg@k ” denotes the average accuracy (in %) over k generations (i.e., pass@1 ), and “ Toks ” indicates the average output length in thousands of tokens (K).

Qwen3-4B
Models	AIME 24		AIME 25		Beyond AIME		GPQA		Overall
Models	avg@32	Toks	avg@32	Toks	avg@8	Toks	avg@8	Toks	Avg.	Toks
Initial Model (Thinking)	73.6	18.0	68.3	22.3	43.4	23.5	56.8	10.7	60.5	18.6
Initial Model (Thinking-Free)	26.9	7.2	20.2	5.5	11.0	4.2	45.0	2.3	25.8	4.8
- TFPI stage 1 (Thinking)	75.2	13.8	67.8	17.1	42.4	16.8	57.9	7.5	60.8	13.8
- TFPI stage 1 (Thinking-Free)	43.9	3.7	31.1	3.4	23.2	3.4	47.8	1.5	36.5	3.0

Appendix

Table 11: Comparison of the Thinking-Free inference mode with the Thinking mode across various reasoning tasks after TFPI. “ Avg@k ” denotes the average accuracy (in %) over k generations (i.e., pass@1 ), and “ Toks ” indicates the average output length in thousands of tokens (K).

Response Length	16K	32K	48K	64K	80K	90K
Avg@32	42.6	64.1	68.3	68.5	68.5	68.5
Clip Ratio	55.7	25.7	5.2	1.2	0.6	0.0

Appendix

Table 12: AIME25 results for Qwen3-4B : Avg@32 and clip ratio across maximum output lengths.

Model	# Tokens (Thinking)	# Tokens (Thinking-Free)	$\Delta$
DS-1.5B (original setting)	16.5K	4.4K	-73%
Unified Setting	16.4K	3.6K	-78%
Qwen3-4B (original setting)	17.4K	5.1K	-71%
Unified Setting	17.4K	4.7K	-73%

Appendix

Table 13: Re-evaluate the meta-experiment in Figure 2 (Left) under a unified evaluation setting (temperature = 0.6, top-p = 0.95, top-k = 20). “# Tokens” denotes the average number of tokens on AIME25 and “

\Delta

” is the percentage of reduction in token consumption.

DeepSeek-Distill-Qwen-1.5B
Models	Mathematics			Multi-Task	Code	Instruction	Overall
	AIME 24	AIME 25	Beyond AIME	GPQA	LiveCode	IFEval	Overall
	Avg@32	Avg@32	Avg@8	Avg@8	Avg@8	Avg@4	Avg.
Initial Model	29.1	25.5	9.0	16.3	17.6	40.1	22.9
- Direct RL	35.5	27.9	12.5	20.2	18.7	38.1	25.5
- TFPI stage 1	31.8	26.7	12.5	30	18.9	39.5	26.6

Appendix

Table 14: Results of TFPI vs. direct RL across different benchmarks. “ Avg@k ” denotes the average accuracy (%) over

k

random generations (i.e., pass@1 ). All models are evaluated in thinking mode. The total training compute for the 3 stages of TFPI equals that of “Direct RL” for fair comparison. This table corresponds to Table 1 under a unified decoding setting.

Models	Mathematics			Multi-Task	Code	Instruction	Overall
	AIME 24	AIME 25	Beyond AIME	GPQA	LiveCode	IFEval	Overall
	Avg@32	Avg@32	Avg@8	Avg@8	Avg@8	Avg@4	Avg.
DeepSeek R1 ∗	79.8	65.0	42.4	71.5	64.3	86.1	68.2
Seed-1.5-Thinking ∗	86.7	74.0	48.0	77.3	64.9	87.4	73.0
Claude4 Opus Thinking ∗	-	75.5	-	79.6	48.9	89.7	-
Qwen3-235B-Thinking ∗	85.7	81.5	-	71.1	55.7	83.4	-

Appendix

Table 15: Results (%) of RL after TFPI (“TFPI+RL”) vs. “Direct RL” across different benchmarks. “ Avg@k ” denotes the average accuracy (%) over

k

random generations (i.e., pass@1 ). For LRMs marked with “*”, results are taken from the corresponding reports (see Appendix C.4 ); All models are evaluated in thinking mode. The total training compute for “TFPI+RL” is matched to that of “Direct RL” for fair comparison. This table corresponds to Table 2 under a unified decoding setting.

Models	AIME 24		AIME 25		Beyond AIME		GPQA		Overall
Models	avg@32	Toks	avg@32	Toks	avg@8	Toks	avg@8	Toks	Avg.	Toks
1) TLMRE-DS-1.5B ( $\alpha=0.1$ )	27.6	12.9	24.8	12.5	9.2	11.9	14.8	7.5	19.1	11.2
2) AdaptThink-1.5B ( $\delta=0.05$ )	28.1	8.0	22.6	7.9	10.0	4.8	14.8	5.1	18.9	6.5
3) AutoThink-DS-1.5B-Stage3	30.3	10.2	25.2	9.1	9.4	8.7	17.4	7.0	20.6	8.7
4) Laser-DE-L4096-1.5B	30.3	8.3	24.9	7.4	9.7	7.4	21.1	4.7	21.5	6.9
5) AutoThink-Stage3 ∗	38.9	8.7	28.9	7.7	11.6	7.8	27.3	5.4	26.7	7.4

Appendix

Table 16: Comparison of the Thinking-Free inference mode of TFPI with efficient reasoning baselines across various reasoning tasks. “ Avg@k ” denotes the average accuracy (in %) over k generations (i.e., pass@1 ), and “ Toks ” indicates the average output length in thousands of tokens (K). Models with “*” are trained from DeepScaleR-1.5B , while the remaining are from DS-1.5B . This table corresponds to Table 3 under a unified decoding setting.

Figure 7: AVG@32 (%) on AIME25 for Qwen3-4B across training steps under different training temperatures (1.4 vs. 1.0). Left: Stage 1 of TFPI; Right: Direct RL.

Figure 8: Qwen3-4B TFPI Stage 1. Left: clip ratio and average output tokens over training steps; Right: MATH500 validation accuracy. Step 100 marks a turning point.

Models	Mathematics			Multi-Task	Code	Instruction	Overall
	AIME 24	AIME 25	Beyond AIME	GPQA	LiveCode	IFEval	Overall
	Avg@32	Avg@32	Avg@8	Avg@8	Avg@8	Avg@4	Avg.
Initial Model	73.6	68.3	43.4	56.8	54.9	64.9	60.3
TFPI stage 1	75.2	67.8	42.4	57.9	55.3	66.0	60.8
Multi-Stage Direct RL Stage 1	49.2	36.0	22.6	51.4	44.0	64.5	44.6
TFPI stage 2	76.0	68.2	44.7	57.8	54.8	64.8	61.0

Appendix

Table 17: TFPI v.s. Multi-Stage Direct RL. All models are evaluated in thinking mode.

	Training Steps	Training Time (kh)	MFU (%)
TFPI Stage 1	4K, 100 steps	0.35	66.28
Multi-Stage Direct RL Stage 1	4K, 100 steps	0.45	66.82
TFPI Stage 2	8K, 56 steps	0.52	66.67
Multi-Stage Direct RL Stage 2	8K, 56 steps	0.51	66.31
TFPI Stage 3	16K, 64 steps	0.91	66.76
Multi-Stage Direct RL Stage 3	16K, 64 steps	0.97	65.99

Appendix

Table 18: Training Steps, Time, and MFU of TFPI v.s. Direct RL with Qwen3-4B .

Efficient Language Model Reasoning

Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.

See all 370 papers

Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently
Jun 22, 2026Stanley Wei, Juno KimRL for Language Model Reasoning Efficient Language Model Reasoning
1Princeton University · University of California, Berkeley.
IAPO: Information-Aware Policy Optimization for Token-Efficient Reasoning
Feb 22, 2026Yinhan He, Yaochen Zhu, Mingjia Shi +5Efficient Language Model Reasoning Language Model Post-Training
University of Virginia, Charlottesville, VA, USA · LinkedIn Inc., Sunnyvale, CA, USA
ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compression
May 8, 2026Tingcheng Bian, Yuzhe Zhang, Jing Jin +5Reward Shaping RL for Language Model Reasoning
1Baidu Inc. · 2Shenzhen University · 3Peking University +2

Jun 22, 2026J/K move · Enter open · S save

Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently

Stanley Wei, Juno Kim

1Princeton University · University of California, Berkeley.

RL for Language Model Reasoning Efficient Language Model Reasoning Reinforcement Learning with Verifiable Rewards RL Fine-Tuning LLM Reasoning

Open paper AskPDF arXiv

Recent advances in large language models (LLMs) have demonstrated that reinforcement fine-tuning of pretrained base models can lead to significant gains in reasoning performance at inference time. In this work, we theoretically analyze why reinforcement fine-tuning induces better reasoning ability than purely supervised fine-tuning (SFT) methods. We model chain-of-thought (CoT) reasoning as a pathfinding problem on graphs and compare the popular method of reinforcement learning with verifiable rewards (RLVR) against traditional SFT. We prove that SFT, when trained on golden shortest paths without negative examples, fails to learn how to efficiently backtrack. In contrast, an RLVR-trained model can learn how to efficiently backtrack from dead ends using only outcome reward. This leads to an exponential separation in inference-time compute between the two methods, and demonstrates that RLVR leads the model to learn the location of difficult decisions in a reasoning chain, ultimately allowing for better allocation of inference-time compute. Finally, we show that the reasoning traces of an RLVR model can be distilled to train a base model to backtrack efficiently as well.

Abstract

Efficient Language Model Reasoning

Explore similar work

Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently

IAPO: Information-Aware Policy Optimization for Token-Efficient Reasoning

ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compression