The Thinking Leakage in Reinforcement Learning for LLMs

Wenxuan Zhu; Shuming Liu; Bang An; Aznaur Aliev; Bing Li; Yibo Yang; Bernard Ghanem

cs.CLOct 9, 2026

The Thinking Leakage in Reinforcement Learning for LLMs

Authors: Wenxuan Zhu, Shuming Liu, Bang An, Aznaur Aliev, Bing Li, Yibo Yang, Bernard Ghanem

Organizations: King Abdullah University of Science and Technology · Shanghai Jiao Tong University

Abstract

Hybrid reasoning models such as the Qwen3 series switch between a THINK mode and a NO-THINK mode through a few chat-template tokens. A popular recipe trains such models with reinforcement learning in NO-THINK mode and reports improved reasoning performance. We provide evidence that much of this gain can be explained by thinking leakage: the gain comes largely from partially reactivating the reasoning abilities of the original model's THINK mode instead of learning enhanced reasoning behavior. We train Qwen3-1.7B, 4B, and 8B with GRPO in NO-THINK mode. As accuracy improves, responses become longer and more reflective, are increasingly judged by the original model to resemble its own thinking, and the model's internal activations shift toward its THINK mode. We then investigate at the level of weights, activations, and attention heads: scaling the RL update further elicits THINK-mode behavior and approaches THINK-mode accuracy but does not exceed it; in Qwen3-8B, replacing a low-dimensional THINK-related activation subspace of the RL-trained model with its original-model value removes most of the RL gain, and the reverse exchange transfers a considerable fraction of the gain to the original model; and the RL-induced shift passes through the attention heads that read the empty think block. We further identify a possible mechanism in the GRPO training signal: responses favored by the advantage-weighted update are more THINK-like, and a gradient step computed from this signal moves the model toward its THINK state. Thus, gains in NO-THINK mode can largely reflect reactivation of reasoning abilities already present in the original model.

Figures & tables

Figure 1: RL in no-think mode shifts all three models toward think mode. Qwen3-8B, 4B, and 1.7B during training, evaluated in no-think mode on AIME (90 problems, 2 samples per problem, 32k token budget). (a) Accuracy. (b) Mean response length. (c) Reflection words per response. (d) Think win rate ( Equation 2 ). (e) Thinking activation shift

\alpha

( Equation 4 ) at the middle layer. (f) Cosine similarity between the activation shift and the think direction, on the same responses and layer as (e).

Figure 2: Scaling the RL update makes the model generate </think> and brings its accuracy close to think mode. Qwen3-8B with weights

W_{0}+c\,\Delta W

from the step-15 and step-67 updates, evaluated on AIME in no-think mode. (a) Fraction of responses containing </think> . (b) Position of </think> as a fraction of the response tokens (kernel density estimates), pooled over all scaled models (gray) and for the original model in think mode (red, dashed). (c) Accuracy; the dashed line is the original model’s think -mode accuracy.

Figure 3: Exchanging the 8-dimensional think subspace removes or transfers a large part of the RL gain; PCA and random subspaces of the same size have much smaller effects. Qwen3-8B at training step 67, evaluated on AIME in no-think mode. (a) Original-to-RL patching: share of the RL gain

G

removed. (b) RL-to-original patching: share of

G

transferred.

	Qwen3-8B		Qwen3-4B		Qwen3-1.7B
Intervention on the original model	$\alpha$	$R_{\mathrm{KL}}$	$\alpha$	$R_{\mathrm{KL}}$	$\alpha$	$R_{\mathrm{KL}}$
Empty think block deleted from the prompt	1.03	93%	1.11	88%	1.08	92%
Attention to the block removed in all heads	1.11	89%	1.17	85%	1.13	90%
Attention to the block removed in the 5% of heads that attend most to it	0.78	72%	0.88	74%	0.66	77%
Attention to the block removed in 5% of the heads, chosen at random	0.03	3%	0.04	6%	0.02	3%
Attention to the block removed in the 1% of heads that attend most to it	0.59	53%	0.56	42%	0.11	25%

Table 1: Removing attention to the empty think block moves the original model toward think mode. Thinking activation shift

\alpha

( Equation 4 ) and relative KL reduction

R_{\mathrm{KL}}

( Equation 7 ) of the original model under each intervention; random sets of heads are averaged over five draws.

Model	Head	Patching effect	Attention to the empty think block	Rank by attention to the block
Qwen3-8B	L18H19	44%	25.2%	Top 0.2%
	L25H28	26%	8.6%	Top 0.7%
	L20H5	17%	10.6%	Top 0.6%
Qwen3-4B	L18H19	37%	27.5%	Top 0.2%
	L20H5	17%	8.1%	Top 1.0%
	L17H25	10%	3.6%	Top 2.4%

Table 2: The heads with the largest RL patching effects also attend strongly to the empty think block. For each model size, the three heads with the largest patching effect, measured on the first 30 fixed responses; held-out values are in Appendix E . Attention to the empty think block: attention weight of the head on the tokens of the empty think block in the original model, averaged over response positions. Last column: rank of the head by this attention among all heads of the model.

$k$ = correct responses in the group of 5	1 (hard)	2	3	4 (easy)
Advantage of each correct response	$+1.79$	$+1.10$	$+0.73$	$+0.45$
Advantage of each wrong response	$-0.45$	$-0.73$	$-1.10$	$-1.79$

Table 3: GRPO advantages for groups of five responses with reward

\pm 1

, by the number

k

of correct responses in the group.

Figure 4: GRPO advantages favor more think -like responses. Qwen3-8B original model, 20 responses for each of 256 training questions, grouped in fives. (a, b) Think win rate of the correct (a) and wrong (b) responses against all sampled responses; each point is one value of

k

, placed at the advantage its responses receive. (c) Within groups with the same

k

, think win rate of the correct responses minus that of the wrong responses.

Weighting of the responses	Responses	$\Delta\alpha$
Full GRPO signal	2,990	$+0.0075$
Positive advantages only (correct responses)	1,479	$+0.0072$
Negative advantages only (wrong responses)	1,511	$+0.0057$
Advantages shuffled within each group	2,990	$-0.0022$
Every response weighted $+1$	2,990	$+0.0006$

Table 4: Change in the thinking activation shift

\alpha

after one GRPO gradient step on a new LoRA adapter of the Qwen3-8B original model, under different weightings of the responses.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Update after step	Scaling factor $c$	Accuracy	Response length (tokens)	Finished	Responses with </think>	Reflection words per response	$\alpha$
15	1	0.194	3.5k	100%	0.0%	2.8	0.13
15	1.77	0.294	4.5k	100%	0.0%	5.0	0.26
15	2.5	0.311	6.8k	100%	0.0%	12.0	0.39
15	3.7	0.517	10.4k	97%	2.8%	26.3	0.57
15	5	0.656	12.8k	91%	9.4%	44.6	0.75
15	7	0.683	18.9k	68%	19.4%	81.4	0.96

Appendix

Table 5: Qwen3-8B with the step-15 and step-67 updates scaled by

c

. AIME, no-think mode, 32k budget, two samples per problem. Finished: share of responses that end within the budget.

Figure 5: Accuracy against mean response length, Qwen3-8B on AIME with a 32k budget. Gray: training checkpoints in no-think mode with a linear fit (dashed). Red and navy: the step-15 and step-67 updates scaled by

c

. Stars: the original model in no-think mode (filled) and think mode (open; eight samples per problem, the other points use two). The three sets differ in mode and sample count, so only their shapes are comparable.

Condition	Accuracy	Response length (tokens)	Finished	Reflection words per response	Accuracy change vs. unpatched	Share of $G$
Original model, unpatched	0.189	2.7k	100%	1.6	–	–
RL-trained model, unpatched	0.411	7.0k	100%	13.0	–	–
Original-to-RL patching, think subspace	0.244	4.2k	100%	3.9	$-0.167$	75%
Original-to-RL patching, PCA subspace	0.361	6.2k	100%	10.0	$-0.050$	22%
Original-to-RL patching, random subspace	0.417	6.9k	100%	11.4	$+0.006$	$-$ 3%
Original-to-RL patching, think direction only (1 dimension)	0.361	5.5k	100%	7.8	$-0.050$	22%

Appendix

Table 6: All conditions of the cross-model patching experiment. Qwen3-8B, training step 67, AIME, 32k budget, two samples per problem. Share of

G

: share of the RL gain

G=0.222

removed (original-to-RL) or transferred (RL-to-original). Finished: share of responses that end within the budget. Subspaces have 8 dimensions per layer unless stated.

Strength $a$	Accuracy	Response length (tokens)	Finished	Reflection words per response
0	0.189	2.8k	100%	1.9
0.25	0.200	3.6k	100%	4.0
0.5	0.289	6.2k	100%	12.4
0.75	0.467	9.2k	99%	32.6
1	0.600	15.8k	76%	81.2

Appendix

Table 7: Steering the 8B original model along its think direction at layer 18. AIME, no-think mode, 32k budget, one sample per problem. Finished: share of responses that end within the budget.

RL update that is applied	Accuracy	Response length (tokens)
Original model (no update)	0.189	2.6k
RL-trained model (full update)	0.431	7.2k
Without the part that writes into the think subspace	0.435	7.0k
Without the part that writes into a random subspace (same squared norm)	0.447	7.1k
Without its top eight singular directions	0.315	5.4k
Only in the modules that do not write into the residual stream (q, k, v, gate, up)	0.343	5.4k

Appendix

Table 8: Removing parts of the RL update of Qwen3-8B (training step 67). AIME, no-think mode, 32k budget, eight samples per problem. The part that writes into a subspace is the output component of the update in that subspace (8 dimensions per layer). q, k, v, o: attention projections; gate, up, down: MLP projections.

Figure 6: Training curves of the three runs: (a) fraction of correct responses on the training batch, (b) mean response length, at each RL step.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Thinking Leakage: A Causal Audit of NoThink Post-Training in Hybrid Reasoning Models
Sep 23, 2026Zehao Liu, Vasant G. HonavarLLM Post-Training Causal Interventions in Language Models
College of Information Sciences and Technology Pennsylvania State University
Path-Lock Expert: Separating Reasoning Mode in Hybrid Thinking via Architecture-Level Separation
Apr 29, 2026Shouren Wang, Wang Yang, Chuang Ma +7Mixture-of-Experts Language Models LLM Reasoning
1Case Western Reserve University · 2NII LLMC, Japan · 3Michigan State University
Demystifying Hybrid Thinking: Can LLMs Truly Switch Between Think and No-Think?
Oct 14, 2025Shouren Wang, Wang Yang, Xianxuan Long +4LLM Reasoning
Case Western Reserve University · Meta AI

Sep 23, 2026J/K move · Enter open · S save

Thinking Leakage: A Causal Audit of NoThink Post-Training in Hybrid Reasoning Models

Zehao Liu, Vasant G. Honavar

College of Information Sciences and Technology Pennsylvania State University

LLM Post-Training Causal Interventions in Language Models Large Reasoning Models

Open paper AskPDF arXiv

Post-training hybrid reasoning models in NoThink mode has attracted growing interest as a way to improve performance while keeping inference fast. However, these gains may draw on thinking behavior already accessible through the base model's Think mode. We formulate this thinking leakage in a causal mediation framework and audit its contribution using bidirectional interventions along a simple base-derived activation direction. Across three models and three post-training methods on competition math benchmarks, we find that leakage is real, causal, and substantial: behavioral and representational analyses reveal shifts toward Think, steering the base model along this direction reproduces most of the post-training accuracy gain, and counter-steering a checkpoint removes a substantial share of what it gains. Across nine aligned checkpoints with positive NoThink gains, the resulting leakage ratio ranges from 42% to 79%. These interventions support a substantial causal contribution of thinking leakage. Our findings show that a post-training method's apparent advantage can therefore reflect greater drift toward Think, obscuring whether it improves capability within NoThink or more effectively re-invokes existing Think behavior.

Abstract

RL for Language Model Reasoning

Explore similar work

Thinking Leakage: A Causal Audit of NoThink Post-Training in Hybrid Reasoning Models

Path-Lock Expert: Separating Reasoning Mode in Hybrid Thinking via Architecture-Level Separation

Demystifying Hybrid Thinking: Can LLMs Truly Switch Between Think and No-Think?