Base Models Can Reason By Taking a Cue From Training Data

Sophie L. Wang; Amil Dravid; Rulin Shao; Kevin Farhat; Sewon Min; Alexei A. Efros

cs.LGOct 5, 2026

Base Models Can Reason By Taking a Cue From Training Data

Authors: Sophie L. Wang, Amil Dravid, Rulin Shao, Kevin Farhat, Sewon Min, Alexei A. Efros

Organizations: MIT · UC Berkeley · University of Washington · Allen Institute for AI

Abstract

In this paper, we study how training data creates associations between the tokens at the start of a base model's response and the reasoning behavior that follows. First, we demonstrate that fixing particular starting token cues makes a base model's performance competitive with that of its reinforcement learning (RL)-trained counterparts on math and coding. For instance, the cue ".\n\nOkay" raises Olmo-3-7B's MATH-500 pass@1 accuracy from 42% to 78%, while "Alright," raises Qwen3-14B's from 72% to 87%. Second, RL makes these cues more likely, while fixing them recovers much of its performance gain over the base model. Third, we trace the reasoning effects of token cues to the training data. We perform causal data interventions to turn an arbitrary word, such as "chicken", into an effective reasoning cue, or remove an existing cue's effect. A similar edit makes the prompt instruction "Think duck duck goose" as effective as "Think step by step" at eliciting reasoning. We also find that the hidden state representations induced by different cues correlate with different document types from the training set. Finally, we extend our study of token cues with a case study in language model safety, finding that different cues elicit distinct refusal and compliance behaviors that correspond to different types of training data.

Figures & tables

Figure 2

Figure 2 : Fixing only the first two tokens can bring a base model to RL-level accuracy. We compare each base model with no cue, with its selected cue prefilled, and with its RL-trained counterpart. The cues are Ay .\n\n Ay Okay for Olmo-3-7B and Ay ␣Alright Ay , for Qwen3-14B. MATH-500, GSM8K, AMC 23, and AIME 2024 evaluate mathematical reasoning; HumanEval evaluates code generation. On Olmo-3-7B the two fixed tokens raise pass@1 by 1.4

\times

to 1.9

\times

across the five benchmarks, often matching or exceeding the RL-Zero model. On Qwen3-14B they raise pass@1 by 1.1

\times

to 3

\times

on the four math benchmarks, to the level of its RL variant, while HumanEval performance remains comparable to the base model without a cue. Whiskers indicate SD over the 32 rollouts.

Figure 3 : Forcing the cue recovers most of RL’s accuracy gain. We test whether RL favors cues selected on the base models and whether prefilling them improves RL efficiency. The cues are Ay .\n\n Ay Okay for Olmo-3-7B and Ay ␣Alright Ay , for Qwen3-14B. ( Left ) KL divergence between the base and RL models’ next-token distributions peaks at the first two positions of RL-generated responses. The models largely agree thereafter given the same preceding text. ( Middle ) RL concentrates probability on the token cue. Each number is the probability of the token beside it given the prompt and the branch to its left. ( Right ) MATH-500 pass@1 over GRPO steps. Prefilling the cue before RL updates matches about 100 RL steps on Olmo and 50 on Qwen, while further cue-forced training adds little. Bands show one SD over rollout indices, using 32 rollouts per problem.

Panel of Figure 5 — Figure 5 : Data edits create and redirect reasoning cues. We substitute words in Olmo-3-7B’s mid-training data to test whether training associations determine which tokens elicit reasoning. ( Left ) The rename mix replaces “Okay” with “Chicken”, and the redirect mix also replaces “Question:” with “Okay,”. ( Middle ) MATH-500 pass@1 with no cue, Ay .\n\n Ay Okay , or Ay .\n\n Ay Chicken prefilled. Renaming makes Ay Chicken effective while preserving Ay Okay ’s benefit. Redirecting removes Ay Okay ’s benefit. Whiskers show one SD over 32 rollouts. ( Right ) Next-token predictions for both cues. Renaming favors Ay let and Ay so after Ay .\n\n Ay Chicken , while redirecting favors Ay what and Ay which after Ay .\n\n Ay Okay . Probabilities are conditional on the prompt and preceding tokens.

Figure 8 : Effective reasoning cues shift hidden-state similarity toward reasoning traces. We study how cues change the hidden states of the text that follows by comparing them with hidden states from training documents. ( Token Cues ) Olmo-3-7B’s 20 candidate openings are grouped by MATH-500 accuracy. ( Trace’s Nearest Neighbors ) We compare generated hidden states with training-document states. Wider bands indicate larger increases, relative to no cue, in the share of the generated response’s nearest hidden-state neighbors drawn from that training source. The highest-accuracy cues shift similarity toward reasoning traces. Only increases of at least 0.03 are shown. None come from a pretraining source.

Figure 10 : Opening tokens change which requests the model refuses. We test how cues affect Olmo-3-7B’s refusal behavior and hidden states on XSTest ( Röttger et al., 2024 ) . ( Safety Behavior ) Relative to no cue, the model refuses more harmful and harmless requests with Ay ␣I Ay ’m Ay ␣sorry . With Ay ␣Okay Ay , or Ay ␣How , it refuses fewer harmful requests. With Ay .\n\n Ay Okay , it refuses more harmful requests and fewer harmless ones. Whiskers show one SD over eight rollouts. ( Trace’s Nearest Neighbors ) The model’s hidden states become more similar to refusal text with Ay ␣I Ay ’m Ay ␣sorry , reasoning traces with either form of Ay Okay , and code and science text with Ay How . Wider bands show larger increases over no cue in the fraction of nearest neighbors from each training source.

Appendix figures & tables46 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11 : The selected cue scores within 1.1 percentage points of the best of 500 tested openings. The 500 tested two-token openings and the no-cue control, ranked by MATH-500 pass@1 over 4 rollouts per problem, Olmo-3-7B under the RL-Zero prompt. Openings proposed by beam search are amber. The cue selected by answer agreement, Ay .\n\n Ay Okay , ranks 8th of 501.

Two tokens drawn at random from the vocabulary
Token cue	pass@1 (%)	Cap rate	Mean tokens
Ay (ro Ay ␣audi	42.3 $\pm$ 2.7	0.19	6,901
Ay ten Ay ␣Quite	27.0 $\pm$ 2.7	0.18	6,347
Ay ␣assault Ay coil	42.0 $\pm$ 2.7	0.17	6,331
Ay hort Ay _PT	40.6 $\pm$ 3.3	0.15	5,406
Ay .But Ay nothing	39.0 $\pm$ 4.0	0.14	5,090

Appendix

Table 3 : The tested random openings fall well short of the selected cue. Olmo-3-7B under RL-Zero on MATH-500. Rows compare ten random openings, no cue and the selected cue, using 4 rollouts per problem with a 31,744-token budget. Columns give pass@1 (

\pm

1 SD over rollout indices), the fraction reaching the budget and mean response length.

Figure 12 : The bigram reasoning cue belongs to a sequence of effective openings. Starting from Ay .\n\n , we extend the prefill one token at a time along the model’s most likely continuation. Accuracy remains similar through Ay Okay Ay , Ay ␣so Ay ␣I Ay ␣need Ay ␣to Ay ␣find Ay ␣the , then falls as the opening becomes problem-specific. Olmo-3-7B on MATH-500 under RL-Zero, with 4 rollouts per problem; whiskers show one SD over rollout indices.

Olmo-3-7B-Base, RL-Zero prompt
Opening	Tokens	MATH-500	AMC 23	AIME 2024
No cue	0	42.2 $\pm$ 2.2	30.9 $\pm$ 9.7	12.0 $\pm$ 5.0
GEPA	9	67.0 $\pm$ 1.6	43.0 $\pm$ 6.1	9.3 $\pm$ 3.3
Token cue ( Ay .\n\n Ay Okay )	2	77.9 $\pm$ 1.5	59.5 $\pm$ 4.8	23.3 $\pm$ 6.3
Qwen3-14B-Base, boxed prompt
No cue	0	72.2 $\pm$ 1.3	46.1 $\pm$ 5.6	10.1 $\pm$ 2.6

Appendix

Table 4 : Token cues outperform the prefills found in these GEPA runs. Pass@1 (%) on three reasoning benchmarks for Olmo-3-7B and Qwen3-14B under the indicated prompts. GEPA uses correctness feedback and an external reflection model; the token-cue search uses no answer labels. We report one GEPA run per model. Evaluation uses 32 rollouts per problem and a 31,744-token budget;

\pm

1 SD over rollout indices.

Parameter	Value
Rollouts per problem	32
Temperature	0.6
Top-p	0.95
Token budget	31,744 tokens

Appendix

Table 5 : Default sampling parameters. Exceptions are stated with the corresponding experiment.

	pass@1 (%)			pass@16 (%)
	No cue	Cued	$\Delta$	No cue	Cued	$\Delta$
Olmo-3-7B Boxed prompt, token cue Ay ␣\n\n Ay Okay
MATH-500	35.2 $\pm$ 1.7	73.6 $\pm$ 1.6	+38.4	85.2 $\pm$ 1.3	93.9 $\pm$ 1.0	+8.7
HumanEval	48.2 $\pm$ 3.2	67.6 $\pm$ 2.6	+19.4	86.7 $\pm$ 2.5	92.6 $\pm$ 1.9	+5.9
Olmo-3-7B RL-Zero prompt, token cue Ay .\n\n Ay Okay
MATH-500	42.2 $\pm$ 2.2	77.9 $\pm$ 1.5	+35.7	94.0 $\pm$ 1.0	95.4 $\pm$ 0.9	+1.4

Appendix

Table 6 : Short cues improve reasoning accuracy across multiple models and benchmarks. Selected model and prompt comparisons: pass@1 and pass@16 in no-cue and cued conditions, and the gain

\Delta

in percentage points. Values are percentages;

\pm

denotes one SD, computed as described in Sec. B.1 . For SmolLM3-3B-Base the search selects no opening under the boxed prompt, so its boxed block carries the RL-Zero cue over in the boxed form. All benchmarks are shown under RL-Zero, except Qwen3-14B under the boxed prompt; the alternative prompt is shown for MATH-500 and HumanEval only.

	pass@1 (%)			pass@16 (%)
	No cue	Cued	$\Delta$	No cue	Cued	$\Delta$
Olmo-3-7B-RL-Zero-Math Boxed prompt, token cue Ay ␣\n\n Ay Okay
MATH-500	52.8 $\pm$ 1.7	77.5 $\pm$ 1.3	+24.7	95.6 $\pm$ 0.8	97.6 $\pm$ 0.6	+2.0
GSM8K	6.3 $\pm$ 0.7	71.7 $\pm$ 1.1	+65.4	32.4 $\pm$ 1.1	97.3 $\pm$ 0.4	+64.8
AMC 23	50.4 $\pm$ 5.9	66.2 $\pm$ 6.0	+15.8	93.7 $\pm$ 3.4	94.1 $\pm$ 3.4	+0.4
AIME 2024	15.5 $\pm$ 6.3	34.2 $\pm$ 6.2	+18.6	60.4 $\pm$ 8.1	67.1 $\pm$ 7.8	+6.8

Appendix

Table 7 : Cue effects change after RL. Pass@1 and pass@16 for each RL-trained model under each prompt, with no cue and with the cue shown in the block heading. Values are percentages;

\pm

denotes one SD, computed as described in Sec. B.1 .

\Delta

is the difference in percentage points. Dashes mark unavailable comparisons.

Minerva 4-shot prompt
Model	Token cue	No cue	Cued
Olmo-3-7B	Ay \n\n Ay Okay Ay ,	38.8 $\pm$ 1.5	67.5 $\pm$ 1.4
Olmo-3-32B	Ay \n\n Ay Okay Ay ,	44.4 $\pm$ 1.4	87.0 $\pm$ 1.3
Qwen3-4B-Base	Ay .\n Ay To	47.3 $\pm$ 1.0	63.8 $\pm$ 1.9
Qwen3-14B-Base	Ay ␣Alright Ay ,	52.3 $\pm$ 1.9	83.0 $\pm$ 1.0
SmolLM3-3B-Base	Ay \n\n Ay The	38.0 $\pm$ 1.5	38.2 $\pm$ 1.4

Appendix

Table 8 : Cue gains depend on the prompt and post-training. MATH-500 pass@1 with and without cues under the listed prompts and model settings. Each condition uses 32 rollouts per problem and a 31,744-token budget. Values show mean

\pm

1 SD over rollout indices.

Model	Prompt	Token cue	pass@1 (%)	Cap rate	Mean tokens
Llama-3.1-8B	RL-Zero	No cue	0.5 $\pm$ 0.4	0.13	4,236
		Ay .\n\n	1.3 $\pm$ 0.4	0.32	10,598
		Ay .\n\n Ay Okay	2.0 $\pm$ 0.5	0.34	11,404
		Ay <\|end_of_text\|>	0.3 $\pm$ 0.3	0.30	9,697
	Minerva 4-shot	No cue	18.7 $\pm$ 1.0	0.08	2,880
		Ay .\n\n	16.8 $\pm$ 0.6	0.09	3,049

Appendix

Table 9 : Tested cues give limited gains on Qwen2.5-Math and leave Llama accuracy low. MATH-500 results for Llama-3.1-8B and Qwen2.5-Math-7B with the tested openings. We use 4 rollouts per problem;

\pm

denotes one SD over rollout indices. Token budgets are 31,744 for Llama and 3,072 for Qwen2.5-Math, whose budget is limited by its context window after the prompt. Cap rate is the share of rollouts reaching the budget.

Model	Prompt	No cue	Break alone	Math cue	pass@1 (%)
Olmo-3-7B	RL-Zero	51.6	62.4	Ay .\n\n Ay Okay	70.0
	Boxed	45.9	–	Ay ␣\n\n Ay Okay	66.8
Olmo-3-32B	RL-Zero	84.8	88.1	Ay .\n\n Ay Okay	89.4
	Boxed	72.6	86.7	Ay ␣\n\n Ay Okay	89.0
Qwen3-4B	RL-Zero	73.6	72.0	Ay .\n Ay To	80.6
	Boxed	79.8	73.1	Ay ␣To Ay ␣determine	76.0

Appendix

Table 10 : Math-cue transfer to code depends on the model and prompt. Pass@1 on 164 HumanEval problems, graded by unit tests. All responses have a 31,744-token budget. We sample 32 rollouts per problem for Olmo-3-7B and SmolLM3, and 16 for the other models. The Olmo-3-7B seeds differ from those in Tab. 6 . Under the boxed prompt, the Olmo-3-32B math cue adds Ay Okay to the selected break; SmolLM3 has no selected math cue. Dashes mark unavailable comparisons.

Token cue	pass@1	Cap rate	Mean tokens
No cue	41.5 $\pm$ 3.4	0.12	4,779
Ay ␣Okay	74.9 $\pm$ 1.5	0.19	7,297
Ay .\n	26.2 $\pm$ 1.9	0.09	3,528
Ay .\n Ay Okay	70.6 $\pm$ 0.5	0.23	8,519
Ay .\n\n	76.6 $\pm$ 0.5	0.15	6,488
Ay .\n\n Ay Okay	76.9 $\pm$ 0.5	0.16	6,700

Appendix

Table 11 : Components of the Olmo reasoning cue. Olmo-3-7B on MATH-500 under RL-Zero, with 4 rollouts per problem and a 31,744-token budget. Pass@1 is in percent (

\pm

1 SD over rollout indices); cap rate is the fraction reaching the budget.

Opening	All	Correct	Incorrect	pass@1 (%)
Paragraph break, Ay .\n\n	0.29	0.50	0.14	71.6
Single newline, Ay .\n	0.56	0.33	0.72	24.8
Other	0.15	0.17	0.14	45.6
All responses	2,000	831	1,169	41.5

Appendix

Table 12 : Correct responses more often begin with a paragraph break. Olmo-3-7B on MATH-500 under RL-Zero without a cue, 500 problems and 4 rollouts each. Columns give the fraction of all, correct and incorrect responses beginning with each opening, followed by accuracy among responses with that opening.

Hyperparameter	Value
Prompts per optimizer step	32
Group size $G$	16
Rollouts per optimizer step	512
Training temperature	1.0
Token budget	4,096 tokens
Reward	Binary correctness

Appendix

Table 13 : Shared hyperparameters for our GRPO training runs.

Figure 14 : Prefilling token cues recovers much of RL’s accuracy gain before training begins. We report MATH-500 pass@1 over 300 GRPO steps for standard and cue-forced RL on Olmo-3-7B/32B and Qwen3-4B/14B. Cue-forced runs prefill Ay .\n\n Ay Okay for both Olmo models, Ay ␣To Ay ␣determine for Qwen3-4B, and Ay ␣Alright Ay , for Qwen3-14B. Bands show one SD over 32 rollouts per problem.

Panel of Figure 15 — Figure 15 : RL changes the policy most strongly at the token cue positions under both trajectory distributions. Per-position KL divergence between each RL-trained model and its base, conditioned on identical token histories. In both cases, divergence is largest near the response opening and falls sharply afterward.

Figure 16 : RL shifts probability mass toward the token cue. Probability of two-token openings before and after RL. Rows show the selected cue (highlighted) and common alternatives from the base model. RL increases the probability of the cue or related openings.

Opening	Continuation	pass@1 (%)	Cap rate	Mean tokens
Base	Base	41.7 $\pm$ 1.3	0.12	4,625
Base	RL	43.7 $\pm$ 1.3	0.19	8,380
RL	RL	75.1 $\pm$ 1.4	0.19	9,408
RL	Base	77.7 $\pm$ 1.5	0.16	6,831

Appendix

Table 14 : The base model reaches RL-level accuracy after an RL opening, but RL remains near base accuracy after a base opening. Olmo-3-7B on MATH-500 under RL-Zero, switching models two tokens after the first blank line. We sample 4 rollouts per problem; pass@1 shows

\pm

1 SE over 500 problems. Cap rate is the fraction reaching the token budget; mean tokens counts the full response.

Figure 17 : Vocabulary-space readout of the leading GRPO update directions. We decompose the Olmo-3-7B layer 31 MLP output update as

\Delta W=U\Sigma V^{\top}

and project each left singular vector

u_{i}

through the unembedding to obtain its vocabulary-space readout

z_{i}

. The top modes align with tokens associated with the cue Ay .\n\n Ay Okay , including variants of Ay Okay and newline tokens.

Figure 18 : The strongest update directions have cue-related vocabulary readouts. For Olmo-3-7B under the RL-Zero prompt, we measure each residual-writing module’s update at the onset of answer generation, averaged over MATH-500 problems. Input-conditioned energy is concentrated in a few singular directions (participation ratio 1.3–2.9). We project the signed contribution of the two most activated directions through the unembedding and show the three highest-scoring tokens. Highlighted tokens are variants of Ay Okay and newline tokens, and become prominent in the second half of the network.

	Base	RL	Edit, no whitening	Edit, whitening
Olmo-3-7B	12.890	12.943 (+0.41%)	13.054 (+1.27%)	12.895 (+0.04%)
Qwen3-14B	10.651	10.660 (+0.09%)	10.803 (+1.43%)	10.651 (+0.00%)

Appendix

Table 15 : Second-moment weighting limits the increase in C4 perplexity. Validation perplexity before and after RL or the weight edit, with relative changes from the base model in parentheses. Whitening denotes the

C^{-1}

weighting in Equation 12 .

Figure 19 : A weight edit recovers much of the prefilled cue's accuracy gain. Pass@1 for Olmo-3-7B and Qwen3-14B on MATH-500, AMC 23 and AIME 2024, comparing the no-cue base model, the base model prefilled with Ay .\n\n Ay Okay (Olmo) or Ay ␣Alright Ay , (Qwen), the weight-edited base model, and the RL model.

Figure 20 : Cue gains over no cue persist at larger matched token caps. MATH-500 accuracy at common truncation lengths for the base model with and without a cue and its RL counterpart. At the smallest caps, Qwen3-14B performs worse with the cue than without it. Qwen3-14B uses the boxed prompt here, and the other models use RL-Zero. The cues are Ay .\n\n Ay Okay for both Olmo models, Ay .\n Ay To for Qwen3-4B, and Ay ␣Alright Ay , for Qwen3-14B. We rescore 32 rollouts per problem from each condition, originally generated with a 31,744-token budget. Bands show one SD over rollout indices.

Figure 21 : No-cue voting needs 3.6–12 times as many tokens to match one cued response. MATH-500 majority-vote accuracy plotted against total generated tokens for the base model with and without a cue and its RL counterpart. The cues are Ay .\n\n Ay Okay for both Olmo models, Ay .\n Ay To for Qwen3-4B, and Ay ␣Alright Ay , for Qwen3-14B. Markers use

k=1,2,4,8,16,32

responses from the 32 rollouts per problem. The large amber point gives the accuracy and token cost of one cued response.

RL-Zero prompt
	Base mix			Rename mix			Redirect mix
Benchmark	No cue	Ay Okay	Ay Chicken	No cue	Ay Okay	Ay Chicken	No cue	Ay Okay	Ay Chicken
MATH-500	13.8 $\pm$ 1.5	31.9 $\pm$ 1.4	2.4 $\pm$ 0.5	23.3 $\pm$ 2.8	38.3 $\pm$ 1.8	37.2 $\pm$ 1.3	26.7 $\pm$ 1.6	0.2 $\pm$ 0.2	40.1 $\pm$ 1.4
GSM8K	12.1 $\pm$ 1.4	64.4 $\pm$ 1.0	2.2 $\pm$ 0.4	29.8 $\pm$ 1.7	67.3 $\pm$ 1.2	60.6 $\pm$ 1.3	32.9 $\pm$ 1.8	0.1 $\pm$ 0.1	69.5 $\pm$ 1.1
AMC 23	7.1 $\pm$ 3.1	14.2 $\pm$ 3.8	1.9 $\pm$ 2.5	11.9 $\pm$ 3.7	17.6 $\pm$ 4.0	17.7 $\pm$ 4.3	11.5 $\pm$ 5.6	0.2 $\pm$ 0.6	19.8 $\pm$ 4.5
AIME 2024	0.2 $\pm$ 0.8	0.2 $\pm$ 0.8	0.2 $\pm$ 0.8	0.4 $\pm$ 1.1	0.5 $\pm$ 1.2	1.6 $\pm$ 2.1	0.5 $\pm$ 1.2	0.0 $\pm$ 0.0	0.6 $\pm$ 1.3

Appendix

Table 16 : Data edits change cue effectiveness across benchmarks in Olmo-3-7B. Pass@1 (%) after 10B training tokens on the base, rename and redirect mixes. Responses are generated with no cue or prefilled with Ay Okay or Ay Chicken after the paragraph break. Both prompts use 32 rollouts per problem and a 7,168-token budget. Values show mean

\pm

1 SD over rollout indices.

Token cue	Released base	Unedited mid-training	100B-token rename mix
No cue	41.8 $\pm$ 2.5	46.7 $\pm$ 2.5	58.2 $\pm$ 2.1
Ay .\n\n Ay Chicken	17.6 $\pm$ 1.8	18.2 $\pm$ 1.9	76.9 $\pm$ 1.3
Ay .\n\n Ay Okay	76.5 $\pm$ 1.2	75.4 $\pm$ 1.3	75.1 $\pm$ 1.3

Appendix

Table 17 : Renaming makes the Chicken cue effective after 100B mid-training tokens. MATH-500 pass@1 (%) for Olmo-3-7B under RL-Zero. The unedited mid-training checkpoint and the rename checkpoint precede long-context training. All columns use 32 rollouts per problem and a 7,168-token generation budget. Values show mean

\pm

1 SD over rollout indices.

Token cue	Base mix	Rename mix	Redirect mix
No cue	5.4	5.9	6.9
Ay .\n\n Ay Okay	19.3	19.0	1.8
Ay .\n\n Ay Chicken	1.1	11.7	11.2
Ay .\n\n Ay Hmm	17.9	19.5	18.2
Ay .\n\n Ay Alright	18.5	21.1	22.4
Ay .\n\n Ay The	14.0	11.9	12.4

Appendix

Table 18 : Data edits create and redirect cues at SmolLM3-3B’s published trace share. MATH-500 pass@1 (%) after 2B training tokens, with 0.55% reasoning-trace tokens. Each word follows a paragraph break under RL-Zero. Four rollouts per problem, with a 3,072-token budget.

Token cue	Base mix	Rename mix	Redirect mix
No cue	9.5	8.7	7.5
Ay .\n\n Ay Okay	25.1	25.9	2.4
Ay .\n\n Ay Chicken	1.4	25.8	26.1
Ay .\n\n Ay Hmm	25.0	25.1	24.8
Ay .\n\n Ay Alright	24.6	25.5	26.1
Ay .\n\n Ay The	18.4	18.6	19.6

Appendix

Table 19 : Upsampling raises accuracy while preserving the cue-specific pattern. Same setup as Tab. 18 , with reasoning traces increased to 10% of training tokens.

RL-Zero prompt
	Base mix			Rename mix			Redirect mix
Benchmark	No cue	Ay Okay	Ay Chicken	No cue	Ay Okay	Ay Chicken	No cue	Ay Okay	Ay Chicken
MATH-500	9.4 $\pm$ 1.5	25.4 $\pm$ 1.3	1.4 $\pm$ 0.6	9.4 $\pm$ 1.5	25.6 $\pm$ 1.4	25.1 $\pm$ 1.5	7.8 $\pm$ 1.5	2.7 $\pm$ 0.8	25.4 $\pm$ 1.9
GSM8K	6.1 $\pm$ 1.2	53.8 $\pm$ 1.3	2.0 $\pm$ 0.5	6.3 $\pm$ 1.3	52.2 $\pm$ 1.1	53.0 $\pm$ 1.0	5.0 $\pm$ 0.9	1.7 $\pm$ 0.5	54.5 $\pm$ 0.9
AMC 23	5.2 $\pm$ 3.2	8.9 $\pm$ 3.8	1.3 $\pm$ 1.9	4.8 $\pm$ 3.9	8.1 $\pm$ 3.8	10.3 $\pm$ 3.9	3.7 $\pm$ 3.0	1.3 $\pm$ 1.9	10.4 $\pm$ 3.6
AIME 2024	0.1 $\pm$ 0.6	0.1 $\pm$ 0.6	0.1 $\pm$ 0.6	0.1 $\pm$ 0.6	0.2 $\pm$ 0.8	0.2 $\pm$ 0.8	0.1 $\pm$ 0.6	0.1 $\pm$ 0.6	0.1 $\pm$ 0.6

Appendix

Table 20 : SmolLM3-3B’s cue-specific pattern extends across benchmarks. Pass@1 (%) after 2.0B training tokens on the base, rename and redirect mixes, each with 10% reasoning-trace tokens. Responses are generated with no cue or prefilled with Ay Okay or Ay Chicken after the paragraph break. For both prompts, we sample 32 rollouts per problem with a 3,072-token budget. We report mean

\pm

1 SD over rollout indices. The dash marks the unevaluated OlympiadBench result for the rename mix under RL-Zero.

Token cue	Base mix	Rename mix	Redirect mix
Ay .\n Ay <think> Ay \n Ay Okay,	24.1	24.1	25.0
Ay .\n Ay <think> Ay \n Ay Chicken,	11.6	23.8	24.7

Appendix

Table 21 : SmolLM3-3B retains Ay Okay ’s effectiveness after a think tag on the redirect mix. MATH-500 pass@1 (%) after 2.0B training tokens on each 10% trace mix, using the listed think-tag openings under RL-Zero. We sample 4 rollouts per problem with a 3,072-token budget. Tab. 19 gives the paragraph-break results.

Token cue	Base mix	Rename mix	Redirect mix
No cue	13.3	21.8	25.9
Ay .\n\n Ay Alright	31.5	37.1	6.5
Ay .\n\n Ay D Ay uck	4.6	35.3	33.1
Ay .\n\n Ay Hmm	31.1	35.9	35.0
Ay .\n\n Ay Okay	31.6	36.9	33.7

Appendix

Table 22 : Data edits create a Duck cue and redirect Alright in Olmo-3-7B. MATH-500 pass@1 (%) after training on the base, rename or redirect mix. We place each word after the paragraph break under RL-Zero and sample 4 rollouts per problem with a 7,168-token budget.

Token cue	Base mix	Rename mix	Redirect mix
No cue	9.5	6.6	7.6
Ay .\n\n Ay Alright	24.6	25.3	1.6
Ay .\n\n Ay D Ay uck	2.9	14.1	14.2
Ay .\n\n Ay Hmm	25.0	26.4	26.3
Ay .\n\n Ay Okay	25.1	24.8	26.2
Ay .\n\n Ay The	18.4	18.8	17.6

Appendix

Table 23 : Data edits create a Duck cue and redirect Alright in SmolLM3-3B. MATH-500 pass@1 (%) after 2.0B tokens of training per mix, with 10% reasoning-trace tokens. Each word follows a paragraph break under RL-Zero. Evaluation uses 4 rollouts per problem with a 3,072-token budget.

	Base mix		Phrase-edited mix
Benchmark	Step by step	Duck duck goose	Step by step	Duck duck goose
MATH-500	12.2 $\pm$ 1.2	6.4 $\pm$ 0.9	14.9 $\pm$ 1.7	16.5 $\pm$ 1.4
GSM8K	9.5 $\pm$ 1.2	4.2 $\pm$ 0.7	16.0 $\pm$ 2.0	18.8 $\pm$ 2.2
AMC 23	4.7 $\pm$ 3.5	1.8 $\pm$ 1.8	7.6 $\pm$ 4.8	9.2 $\pm$ 3.6
AIME 2024	0.5 $\pm$ 1.5	0.0 $\pm$ 0.0	0.3 $\pm$ 1.0	0.6 $\pm$ 1.6
AIME 2025	0.2 $\pm$ 0.8	0.2 $\pm$ 0.8	0.5 $\pm$ 1.2	0.2 $\pm$ 0.8

Appendix

Table 24 : After the data edit, “Think duck duck goose” achieves comparable accuracy to “Think step by step”. Pass@1 (%) for Olmo-3-7B after 10B mid-training tokens on the base or phrase-edited mix. The two instructions replace the first line of the RL-Zero prompt, with no response tokens prefilled. Evaluation uses 32 rollouts per problem and a 7,168-token budget. Values show mean

\pm

1 SD over rollout indices.

Token cue	Base mix	Rename mix	Redirect mix
No cue	13.8	23.3	26.7
Ay .\n\n Ay Okay	31.9	38.3	0.2
Ay .\n\n Ay Chicken	2.4	37.2	40.1
Ay .\n\n Ay Hmm	30.4	37.5	40.2
Ay .\n\n Ay Alright	30.9	37.8	40.1

Appendix

Table 25 : Ay Hmm and Ay Alright remain effective across all three mixes. MATH-500 pass@1 (%) for the three Olmo-3-7B counterfactual checkpoints under RL-Zero. Each word follows a period and paragraph break. We sample 32 rollouts per problem with a 7,168-token budget.

Checkpoint	Opening word	P(comma)	Greedy continuation
Pretraining only	Ay Okay	0.67	Okay, so I’m not sure
	Ay Chicken	0.01	Chicken and rabbit\n\nA farmer has
	Ay Hmm	0.46	Hmm, I’m not sure what
Base mix	Ay Okay	1.00	Okay, let’s see. I
	Ay Chicken	0.02	Chicken McNuggets are sold in
	Ay Hmm	0.88	Hmm, let’s see. I

Appendix

Table 26 : Greedy continuations after training-data edits. Olmo-3-7B before and after 10B mid-training tokens on the base and rename mixes. At the RL-Zero paragraph boundary, we report the probability that the next token is a comma and the greedy continuation over 100 MATH training problems.

Context	Base mix	Rename mix	Redirect mix
She roasted the chicken	and served it with mashed potatoes and gravy. The meal was	and made a salad. She also made a sauce for the chicken. Sh	, and then she fried the chicken. She fried the chicken unti
A chicken is a	bird, and a bird is a vertebrate. A vertebrate is an animal	bird that belongs to the family of Galliformes. It is a dom	bird, and a duck is a bird. So, we can add the number of ch
Dinner was almost ready. Chicken	was on the grill, and the vegetables were steaming. I was s	, I’m going to go get my coat." He said, and he left the kit	was done, and the vegetables were steaming. I was about to
The recipe is simple. Ay \n\n Chicken	wings are a great source of protein, and they are also low	, so I need to figure out how much flour I need for 36 cooki	, so I have a recipe for a cake that I want to make. I have
Dinner was almost ready. Okay	, I’ll just get the salad dressing. I’ll be right back." He	, I’m going to go get my coat and we’ll go out to dinner. I’	, I’m going to go get the last of the vegetables. I’ll be ba
The recipe is simple. Ay \n\n Okay	, let’s see. The user wants to make a simple recipe. Let me	, so I’m going to make a few changes to the recipe. First, I	, what is the first step in the recipe? Answer: Preheat the

Appendix

Table 27 : The effect of the rename depends on context. Greedy continuations of the listed contexts from the three Olmo-3-7B counterfactual checkpoints.

Figure 22 : The reasoning cue’s accuracy advantage over no cue becomes larger during mid-training. MATH-500 pass@1 across pretraining ( left ) and mid-training ( right ) checkpoints of Olmo-3-7B ( top ) and Olmo-3-32B ( bottom ). We compare no cue with the reasoning cue Ay .\n\n Ay Okay under the RL-Zero prompt, using 4 rollouts per problem. Bands show one SD over rollout indices.

Mix	Likely first tokens	a solution	an answer line	the prompt	a solution	not	pass@1 (%)
		First paragraph opens with			pass@1 given
Base mix	Ay Answer 0.13, Ay Okay 0.12, Ay The 0.05	0.31	0.37	0.09	0.20	0.11	13.8
Rename mix	Ay To 0.24, Ay Answer 0.16, Ay First 0.10	0.40	0.33	0.22	0.29	0.19	23.3
Redirect mix	Ay To 0.24, Ay Chicken 0.19, Ay First 0.08	0.54	0.38	0.05	0.38	0.14	26.7

Appendix

Table 28 : The edits also change no-cue openings. For each Olmo-3-7B checkpoint, we report the three most likely first tokens after the paragraph break, measured on MATH training problems. On MATH-500, we report opening types and accuracy with and without a solution opening: 32 rollouts per problem, RL-Zero prompt, 7,168-token budget. Opening types distinguish solutions, answer lines and continuations of the prompt’s last sentence.

Group	Sources	Source count	Reasoning bank	Safety bank
Reasoning traces	Traces from QwQ, Gemini, Llama Nemotron, OpenThoughts2 and OpenMathReasoning, and a general reasoning mix	6	1,608	1,482
Expository math	CraneMath, Dolmino math and MegaMath	3	1,611	1,485
Meta-reasoning	Math and code meta-reasoning	2	1,610	1,484
Code	Stack-Edu (15 languages), CraneCode and verifiable programs	62	1,612	1,488
Short Q&A	Nemotron synthetic QA, FLAN and Reddit flashcards	3	1,611	1,485
Science PDFs	olmOCR science PDFs, 18 topics, two quality tiers	36	1,526	1,399

Appendix

Table 29 : Training sources used for the hidden-state comparisons. Source groups and document counts for the separately sampled reasoning and safety banks ( Figs. 9 and 10 ). Refusal-shaped text appears only in the safety bank, with pretraining and mid-training shown separately in the figure. The dash marks its absence from the reasoning bank.

Detector	Regular expression
Plan	\b(first,? (i\|we\|let'?s)\|i need to\|we need to\|let'?s (start\|begin\|figure\|find\|see)\|to (solve\|find) this\|my (plan\|approach)\|i'?ll (start\|begin))\b
Decompose	(^\|\n)\s(step\s\d\|\d+.\s\| $\d+$ \|case \d)
Check	\b(let me (check\|verify\|double[- ]check\|confirm\|make sure)\|check(ing)? (that\|this\|my\|the answer)\|verify\|plug(ging)? (it )?back\|sanity check\|does that make sense\|to confirm)\b
Backtrack	\b(wait\|hold on\|hmm\|actually,\|that'?s (not right\|wrong\|incorrect)\|i made a mistake\|let me redo\|no, )\b
Alternative	\b(another way\|alternatively\|other approach\|different (approach\|method\|way)\|instead,)\b
Commit	(\boxed{\|\b(so the answer is\|the final answer is\|therefore,? the answer\|thus,? the answer)\b)

Appendix

Table 30 : Regular expressions for the seven phrase detectors. Line breaks are for display only.

Olmo-3-7B, RL-Zero prompt
Condition	pass@1 (%)	Plan	Check	Backtrack	Alternative	First person	Distinct
No cue	42.2	0.65	0.50	0.48	0.27	0.65	3.3
Ay .\n\n Ay Okay	77.9	0.99	0.98	1.00	0.56	1.00	5.6
RL-Zero model	75.0	0.97	0.94	0.99	0.71	0.99	6.0
Olmo-3-7B RL-Zero model, boxed prompt
No cue	52.8	0.61	0.43	0.44	0.33	0.46	3.4

Appendix

Table 31 : Phrase-marker frequencies in base and RL responses. MATH-500 results under the prompts identified in each block. Columns report pass@1 and the share of responses matching each detector; Distinct is the mean detector count. 500 problems and 32 rollouts per problem throughout.

Token cue	pass@1 (%)	Median tokens	Plan	Check	Backtrack	Alter- native	First person	Distinct	Answer line
No cue	41.5	740	0.65	0.49	0.48	0.27	0.65	3.3	0.43
Ay .\n\n Ay Okay	76.9	1,488	1.00	0.97	0.99	0.55	1.00	5.6	0.25
Ay .\n\n Ay To	56.7	368	0.76	0.05	0.00	0.01	0.36	2.0	0.56
Ay .\n\n Ay Answer	17.2	5	0.01	0.01	0.01	0.01	0.01	0.2	1.00
Ay .\n Ay Problem	6.3	722	0.82	0.16	0.06	0.03	0.84	2.8	0.18

Appendix

Table 32 : Different openings produce different response styles. Olmo-3-7B-Base on MATH-500 under RL-Zero, with 4 rollouts per problem and a 31,744-token budget. Columns give accuracy, median response length, the fraction of responses matching each detector in Tab. 30 , and answer-line frequency. Distinct is the mean detector count.

Checkpoint	Token cue	Reasoning traces	Expository math	Meta- reasoning	Short Q&A	pass@1 (%)
Base mix	No cue	0.59	0.19	0.16	0.03	13.3
	Ay .\n\n Ay Okay	0.79	0.12	0.08	0.01	31.8
	Ay .\n\n Ay Chicken	0.36	0.18	0.23	0.14	2.0
Rename mix	No cue	0.56	0.23	0.17	0.03	23.5
	Ay .\n\n Ay Okay	0.77	0.13	0.09	0.01	37.7
	Ay .\n\n Ay Chicken	0.79	0.11	0.08	0.01	37.5

Appendix

Table 33 : After renaming, Ay Chicken resembles Ay Okay in both trace similarity and accuracy. MATH-500 results for Olmo-3-7B trained on the base and rename mixes, using RL-Zero, 4 rollouts per problem and a 7,168-token budget. We report each group’s fraction of the ten nearest training states, averaged over sampled response positions. Pass@1 is measured on these same rollouts. Groups below 2% in every row are omitted.

Cue position	At cue	$+16$	$+64$	$+128$	$+256$
Paragraph 1 (share)	1.00	0.71	0.68	0.62	0.60
Paragraph 2 (change)	$+0.86\pm 0.03$	$+0.36\pm 0.04$	$+0.29\pm 0.04$	$+0.30\pm 0.04$	$+0.21\pm 0.04$
Paragraph 3 (change)	$+0.65\pm 0.03$	$+0.12\pm 0.04$	$+0.02\pm 0.04$	$+0.06\pm 0.04$	$+0.07\pm 0.04$
Paragraph 4 (change)	$+0.44\pm 0.04$	$+0.04\pm 0.03$	$+0.03\pm 0.04$	$+0.05\pm 0.04$	$+0.02\pm 0.04$
Paragraph 5 (change)	$+0.40\pm 0.04$	$+0.03\pm 0.04$	$+0.03\pm 0.04$	$-0.02\pm 0.04$	$+0.04\pm 0.04$
Paragraph 6 (change)	$+0.37\pm 0.04$	$+0.04\pm 0.04$	$-0.03\pm 0.04$	$+0.03\pm 0.04$	$-0.03\pm 0.05$

Appendix

Table 34 : The similarity increase persists after early cue insertion. Olmo-3-7B under RL-Zero, with 100 MATH-500 problems and one rollout per problem. For paragraphs 2–6, values show the cue-minus-control difference in reasoning-trace neighbor share, averaged over problems (

\pm

1 SE). Paragraph 1 has no paired control and reports absolute shares.

Figure 23 : The shift toward reasoning traces at Ay Okay persists in math responses but is brief in web text. Share of layer-24 neighbors from reasoning traces around the first token of a paragraph, for paragraphs that open with Ay Okay and for random paragraph starts. ( Left ) Olmo-3-7B’s no-cue MATH-500 responses under RL-Zero. ( Right ) Web and pretraining documents. Bands show one standard error.

Token cue	Unsafe refusal	Safe refusal	Harmful response
No cue	56.3 $\pm$ 5.1	5.5	27.3
Ay ␣Okay	38.2 $\pm$ 5.2	9.2	42.5
Ay ␣Okay Ay ,	38.3 $\pm$ 5.3	9.2	42.4
Ay .\n\n Ay Okay Ay ,	81.9 $\pm$ 1.9	4.3	0.2
Ay .\n\n Ay Okay	81.9 $\pm$ 1.9	4.2	0.2
Ay \n\n Ay Sure Ay ,	17.7 $\pm$ 4.3	3.6	56.5

Appendix

Table 35 : The leading Ay .\n\n changes refusal behavior, while the trailing comma has little effect. Olmo-3-7B-Base on XSTest. Columns report refusal rates on 200 unsafe and 250 safe prompts, and harmful-response rates on unsafe prompts, all in percent. WildGuard grades 8 rollouts per prompt;

\pm

1 SD over rollout indices for unsafe-prompt refusal.

Explore similar work

May 16, 2026cs.AI

Reasoning Can Be Restored by Correcting a Few Decision Tokens

Large reasoning models (LRMs) substantially outperform their base LLM counterparts on challenging reasoning benchmarks, yet it remains poorly understood where base models go wrong during token-by-token generation and how to narrow this gap efficiently. We study the base-reasoning gap through quantifying token-level distributional disagreement between a base model and a stronger reasoning model using likelihood-based divergences. Across benchmarks, we find that the reasoning advantage is highly sparse and concentrates on a small set of early, planning-related decision tokens. For instance, on Qwen3-0.6B, only ~8% of generated tokens account for the salient disagreement, and these tokens concentrate early in the response, are strongly enriched in planning-related decisions (17x), and coincide with high base-model uncertainty -- suggesting that base models fail mainly at early planning points that steer the subsequent reasoning trajectory. Building on these findings, we propose disagreement-guided token intervention, a simple inference-time delegation scheme that performs a one-token takeover by the reasoning model only at high-disagreement positions and immediately switches back to the base model. With a small intervention budget, this sparse delegation substantially recovers and can even surpass the performance of a same-size reasoning model on challenging reasoning tasks. Code is available at https://github.com/AlphaLab-USTC/RRTokenIntervention.

Changshuo Shen, Leheng Sheng, Yuxin Chen +2

University of Science and Technology of China, Hefei, China · National University of Singapore, Singapore

Large Reasoning Models Decisions

Ask PDF

Oct 8, 2025cs.AI

Base Models Know How to Reason, Thinking Models Learn When

What do thinking language models learn during training that their base models lack? We first present an unsupervised method that discovers a model's reasoning behaviors by training small Sparse Autoencoders on sentence-level activations of reasoning traces, yielding interpretable reasoning taxonomies. Building on this, we introduce constructive model diffing, which aims to reconstruct the base-to-fine-tuned difference from interpretable components: reasoning mechanisms (category vectors that can induce a reasoning behavior in the base model) and reasoning heuristics (a classifier determining when a mechanism should fire). Across nine base/thinking pairs (four RL-trained, four SFT-distilled, one mixed), two independent findings agree: category vectors in the base model converge to far lower loss for taxonomies derived from purely RL-trained models, and hybrid models recover roughly 76% of the RL base-to-thinking gap but only 11% of the SFT gap. This indicates RL primarily teaches heuristics for orchestrating pre-existing base mechanisms, whereas SFT-distillation installs new ones, offering a new lens on what training paradigms teach, with implications for efficient reasoning-model development.

Constantin Venhoff, Iván Arcuschin, Philip Torr +2

University of Oxford, UK · MATS · Poseidon Research.

Large Reasoning Models Reasoning Skills Thoughts Reinforcement Learning Baseline Model Activations Probe-Logit Distillation+1

Ask PDF

May 7, 2026cs.AI

Behavior Cue Reasoning: Monitorable Reasoning Improves Efficiency and Safety through Oversight

Reasoning in Large Language Models (LLMs) poses a challenge for oversight as many misaligned behaviors do not surface until reasoning concludes. To address this, we introduce Behavior Cue Reasoning for making LLM reasoning more controllable and monitorable. Behavior Cues are special token sequences that a model is trained to emit immediately before specific implicit and explicit behaviors, acting as dual purpose signal and control levers. When fine-tuning a weaker external monitor with Reinforcement Learning for reasoning oversight, a compressed view of only information surfaced by Behavior Cues is sufficient signal for the monitor to prune up to 50% of otherwise wasted reasoning tokens in complex math problem solving. When leveraged by an almost optimal rule-based monitor in an environment where excessive constraint violations results in failure, Behavior Cues allows for the recovery of safe actions from 80% of reasoning traces that would otherwise end with the proposal of an unsafe action, more than doubling the success rate from 46% to 96%. Through evaluation across two model families and three domains, we show that Behavior Cue Reasoning improves reasoning monitorability and controllability with no cost to performance. More broadly, our work progresses scalable oversight by demonstrating how the monitored model itself can be trained to reason more tractably to oversight. Code: https://github.com/christopherzc/behavior-cues

Christopher Z. Cui, Taylor W. Killian, Prithviraj Ammanabrolu

University of California, San Diego · 2Brigham Young University

Reasoning Skills Scalable Oversight

Ask PDF