Reinforcement learning (RL) is credited with teaching reasoning models to write {wait} before reconsidering a step, yet large RL accuracy gains do not require the word. This view can confuse what RL discovers with what a model inherits from its training data. We trace the word through open models' training histories, reading it with label-free indices of frequency, position, form and usage and an injected-error test, all computed from sampled outputs. We find that zero-RL almost never coins the marker: on bases that lack the word, RL raises MATH-500 accuracy substantially while {wait} stays near absent. It arrives with training data: mid-training in OLMo-3, and fine-tuning on reasoning traces installs it in most models whose base lacks it. Where the base already uses it, both released RL-Zero recipes (OLMo-3, MiMo) multiply it, whereas KL-free GRPO from the same base does not; OLMo-3's recipe wears down its form and makes it a more sensitive but more liberal alarm; RL after supervised fine-tuning barely changes it, and our GRPO recipe erases a marker seeded by fine-tuning. Finally, we show that marker frequency alone does not indicate RL-induced reasoning and caution against LLM-as-a-judge revision metrics, as automated judges can reverse human-labelled trends.
Figures & tables
Figure 1: RL rarely coins wait : the marker enters with mid-training or fine-tuning on traces.
Figure 2: Our methodology extracts a marker’s evolutionary history from its occurrences, one checkpoint at a time. The pipeline processes ❶ lineages and sampled traces into ❷ matched occurrences, from which we construct ❸ label-free indices and validate the ❹ injected-error alarm test, with statistics at every checkpoint (bottom).
Figure 3: OLMo-3 RL-Zero multiplies wait and erodes its form; RL after fine-tuning changes its rate little and restores its comma. (top) OLMo-3 RL-Zero; (bottom) OLMo-3 Think; (a)–(e) rate, sentence-initial share, Wait, share, chaining, sentence length; hollow: base; bands: 95% cluster-bootstrap CIs.
Figure 4: Under RL-Zero, revision markers become a more sensitive but more liberal alarm. Sensitivity d′ and criterion c of any revision marker after clean versus error-injected prefixes, per checkpoint, with 95% cluster-bootstrap bands.
Table 5: Checkpoint series. Released trajectories (top) and our own runs (bottom). Every released endpoint is listed in Table 13 ; licences are in Section A.2 . openPangu-v2.1-Pro (API) in Table 4 : vendor API queried 2026-10-13, weights not public, thinking cannot be disabled via the API; the API is rate-limited, so n=745 samples over 190 of 200 problems; rate per 1k output tokens (thinking and answer), 0.75 on the 182 problems with all four samples against 0.77 on all.
Licence
Models, datasets and tools
Apache-2.0
OLMo-2 and OLMo-3 (all series, branches and Think models); Qwen2.5, Qwen2.5-Math, Qwen3, Qwen3.5, Qwen3.8 and QwQ models; SmolLM3; Granite 3.3; Mistral-7B-v0.1 and Mistral-Small-24B; Yi-1.5; OpenThinker-7B; Bespoke-Stratos-7B; Light-R1-14B-DS; DAPO-Qwen-32B; SimpleRL-Zoo (all but the 14B release); Oat-Zero; General-Reasoner; One-Shot-RLVR; gpt-oss; AIME 2024 and 2025; vLLM; verl; Math-Verify
MIT
Phi-4 family; DeepSeek-R1-Distill (Qwen and Llama); DeepScaleR-1.5B; Open-Reasoner-Zero; MiMo-7B; GLM-4 and GLM-Z1; GSM8K; MMLU-Pro; MATH (training split); MATH-500 (its card states no licence; both of its sources are MIT)
Table 6: Licences of all artefacts. One row per licence.per-repository identifiers and checks are at https://anonymous.4open.science/r/wait-provenance/ .
Figure 5: All 120 endpoints of the provenance map. One dot per model after training, adding instruct models without trace training (light squares), pretrained bases not shown elsewhere (MiMo-7B-Base and SmolLM3-3B-Base already write wait ) and gpt-oss-120b at three reasoning efforts (training undisclosed).
Figure 6: Across major model families, wait follows the training route. Rate before ( ∘ ) and after training, 95% CI; shape marks the route, colour the family (Qwen2.5, Qwen3, Qwen3.5/3.8, MiMo, DeepSeek/Phi, openPangu).
Figure 7: In OLMo-3, wait enters with mid-training; Qwen2.5-32B’s descendants use it only after trace training. (a) Rate with 95% bands and comma share across OLMo-3 training stages and RL-Zero steps. (b) Qwen2.5-32B and its descendants: rate with 95% CIs, and accuracy.
Parameter (change)
Measurement on checkpoints
Hyp.
Frequency of use (context)
rate of wait and its paradigm per 1k tokens and per 100 sentences; known to rise ( Guo et al., 2025 )
–
Syntagmatic variability ( ↓ , fixation)
sentence-initial share SI , canonical-form share ORTH , position entropy Hpos of wait
Section D.2
Bondedness ( ↑ ), scope ( ↓ )
next-word entropy Hnext ; tokens from marker to next action
Section D.2
Integrity ( ↓ , bleaching)
hit and false-alarm rates, d′ and c on prefixes with vs. without an injected error; usage-embedding change (APD, PRT, JSD); substitutability of synonyms
entropy Hpara and effective size Neff of the forms opening revisions; policy-side entropy Hsub ; obligatoriness OBL
Section D.2
Procedural function (operator)
risk difference and odds ratio of revising a correct step after inserted Wait, vs. Next, and synonyms, late vs. base policy
Section D.2
Appendix
Table 11: Each grammaticalization parameter becomes a quantity measured on RL checkpoints. Parameters follow Lehmann (2015) ; arrows give the direction predicted under grammaticalization. Definitions are in Section 3 .
Prediction
Pre-registered test
Result
Verdict
H1 Fixation
sentence-initial and Wait, shares rise with step ( ρ≥0.5 on 2 of 3 series)
falsified: position inherited, form erodes (S1) or is restored (S2)
H2 Bleaching
false alarms rise, d′ does not (falsified if Δd′ lower bound >0.25 )
S1: d′ 0.57 → 1.09 [0.97, 1.24]; c 0.11 →− 0.24; FA 34.7 → 37.8%
falsified : a more sensitive, more liberal alarm
H3 Paradigm loss, obligatoriness
paradigm entropy ρ≤−0.5 ; ≥60% of revisions carry a marker
ρ(Hpara) : S1 − 0.9, S2 − 0.48, 32B − 0.75; obligatoriness at the end of S1 46.1%
paradigm narrows on S1 only; obligatoriness falsified
H4 Operator
Wait, − Next, ≥10 pts, OR ≥2 , ≥1.5× the base model; same trend under both prefix sources
judge: 15.8 pts, Ramp=2.36 ; second judge 5.4 pts; label-free value change − 2.3 [ − 5.7, 0.8], but its detector fails its check (precision 10.0%)
inconclusive: judge result exploratory (human κ=0.3 ), label-free outcome invalid
H5 Drivers
KL and length pressure set the speed (controlled GRPO)
controlled runs from a marker-rich base not run
untested
H6 Monitors
literal readers lose precision with RL; a calibrated monitor does not
not run
untested
Appendix
Table 12: Of six pre-registered hypotheses, three fail, one is unsupported and two were not run; further registered tests fail too. Tests and thresholds as fixed before the main run ( Section D.3 ). S1: OLMo-3 RL-Zero; S2: OLMo-3 Think; 32B: OLMo-3.1 32B Think; SI and Wait, shares in %; FA: false-alarm rate.
Figure 8: By the judges’ labels, an inserted Wait, pushes every checkpoint toward revision, the base included. Judged probability of revising a correct step per inserted form along (a) OLMo-3 RL-Zero and (b) OLMo-3 7B Think, with 95% cluster-bootstrap bands. Exploratory: the judge’s sensitivity against the human audit is low ( Section D.3 ).
Recent advances in large language models (LLMs) have demonstrated that reinforcement fine-tuning of pretrained base models can lead to significant gains in reasoning performance at inference time. In this work, we theoretically analyze why reinforcement fine-tuning induces better reasoning ability than purely supervised fine-tuning (SFT) methods. We model chain-of-thought (CoT) reasoning as a pathfinding problem on graphs and compare the popular method of reinforcement learning with verifiable rewards (RLVR) against traditional SFT. We prove that SFT, when trained on golden shortest paths without negative examples, fails to learn how to efficiently backtrack. In contrast, an RLVR-trained model can learn how to efficiently backtrack from dead ends using only outcome reward. This leads to an exponential separation in inference-time compute between the two methods, and demonstrates that RLVR leads the model to learn the location of difficult decisions in a reasoning chain, ultimately allowing for better allocation of inference-time compute. Finally, we show that the reasoning traces of an RLVR model can be distilled to train a base model to backtrack efficiently as well.
Stanley Wei, Juno Kim
1Princeton University · University of California, Berkeley.
Reinforcement learning has become the standard for improving reasoning in large language models, yet evidence increasingly suggests that RL does not teach new strategies; it redistributes probability mass over solutions the base model already contains. In this work, we ask: if RL merely steers the model toward paths it already knows, is the RL optimization loop itself necessary? Through token-level analysis across multiple model families and RL algorithms, we find that RL's beneficial footprint is a sparse, predictable correction concentrated at high-entropy decision points where the model is uncertain which branch to take. Only 1--3% of token positions are affected, the promoted token always lies within the base model's top-5 alternatives, and targeted corrections at those few positions causally recover a large fraction of RL's accuracy gain, while random corrections fail. The base model's own entropy identifies these positions without any RL-trained model, and the entire correction is low-dimensional, representable in a tiny fraction of model parameters. These findings reframe reasoning improvement as sparse policy selection, not capability acquisition. We translate this insight into ReasonMaxxer, a minimal RL-free method that applies contrastive loss only at entropy-gated decision points, using a few hundred base-model rollouts and no online generation. Across three model families, six scales, and six math reasoning benchmarks, ReasonMaxxer matches or exceeds full RL performance while requiring only tens of problems and minutes of single-GPU training, a reduction in training cost of roughly three orders of magnitude.
Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger +1
Reinforcement learning with verifiable rewards (RLVR) substantially improves mathematical reasoning, but what it adds to the base model remains unclear. We investigate this through a sequence of generation-time interventions on Qwen3-4B-Base, measuring how much of its RL-trained counterpart's gain each intervention recovers on 120 AIME problems. Power sampling with the same 8,192-token output limit brings no gain beyond greedy decoding. Yet borrowing the RL model's token only where the two models' greedy choices differ, roughly 4% of generated positions, recovers 90.5% of the pass@1 gain, with similar recovery across three model pairs on MATH-500. We find that even a single opening token has a substantial effect: forcing the base model to begin with "Alright" recovers 35% of the RL gain. The opening alone is insufficient, however. Later replacements after the same opening add another 5.2 percentage points to pass@1, and the fixed rules we test for inserting reflective words such as "Wait" do not reproduce this benefit. Sparse replacement also retains much of the base model's diversity while staying close to the RL model at pass@1. At k=256, it solves 67.5% of the problems, significantly more than the RL model (61.4%) and a mixture of whole responses matched in pass@1 (61.45%). The comparison is at equal attempt counts rather than equal compute, since rl-replace consults both models at every decoding step. These results show that the opening can recover a meaningful part of the RL gain, but a larger contribution comes from later choices, which can be borrowed while retaining much of the base model's diversity. Code is available at https://github.com/CyrusZhang04/WhatRLBrings.
Siyu Zhang, Yuquan Lu, Yoshua Y. Li +7
University of California, San Diego · Meituan, LongCat Team · The Chinese University of Hong Kong, Shenzhen