Reasoning or Pattern Exploitation? Mechanistic Insights from RL-Trained Language Models
Authors: Aishik Nagar, Cheston Tan
Organizations: Carnegie Mellon University · Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR) · Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR)
Large language models (LLMs) are increasingly described as acquiring "reasoning" skills after reinforcement learning with verifiable rewards. However, whether these gains reflect genuine reasoning or more superficial adaptations remains underexplored. In this work, we probe LLMs trained in a finite and exhaustively analyzable logical domain, namely Tic-Tac-Toe, and trace how internal representations evolve across reinforcement learning (RL) with Group Relative Policy Optimization (GRPO). Quantitatively, RL yields high accuracy and robustness across prompt variations, suggesting the emergence of game reasoning. Mechanistic interpretability, however, paints a different picture: features extracted with sparse autoencoders (SAEs) reveal that models primarily adapt to rely more heavily on information already explicit in the prompt, such as whose turn it is, game progression and board occupancy. In comparison, high-level concepts like board symmetries and strategic forks remain weakly represented, raising concerns that reasoning benchmarks overestimate model abilities. This suggests that RL-driven "reasoning" may conflate task-specific adaptations to input information with reasoning ability. We provide, for the first time: (i) a systematic interpretability pipeline tracing representation dynamics across RL training in small-scale LLMs; (ii) an extension of SAE-based feature discovery to hypothesis-driven testing in a finite logical domain; and (iii) an interpretability-based demonstration that RL post-training neither adds nor removes concept information but reorganizes which features dominate the representation, with causal interventions locating the decision downstream of it, in the late-layer state.
Figures & tables
Figure 1: Overview of our experimental setup
Figure 2: Best performance of each model and training setting for the legal move and best move prediction tasks on the random-split dataset. Complete performance plots for all settings (different board and move token representations) and datasets can be found in Appendix Figure 7 .
Figure 3: Progression of model performance across model checkpoints for the two datasets. Complete progression plots for all settings (different board and move token representations) can be found in Appendix Figure 6 .
Figure 4: Model representations dynamics of Qwen2.5 1.5B (layer 12) through reasoning (GRPO) training. The figure represents the same projections colored based on three hypotheses: (H1) whose turn it is (Game Turn), (H2) which stage of the game it is (Game Progression) and (H3) what is the logical play for the given board (Strategic Situation). It is evident that while the base model only clearly represents the game turn, during the course of training, the model evolves to represents the boards based on game progression in addition to game turn, leading to improvement in performance. However, logic for strategic situations does not play a big role in model representations and no clear patterns emerge over the course of training. Full training dynamics across all layers and over all hypotheses tested are provided in Appendix Section C .
Trajectory at L12
Per-layer at converged ckpt
Hypothesis
Base
Mid
Final
L4
L8
L12
L13
L14
L15
L16
L17
L18
L20
L24
H1 turn identity
5285
366
316
270
140
316
114
269
188
95
2208
5379
1508
5484
H2 game progression
274
1686
2330
1151
594
2330
412
1278
287
358
747
284
114
400
H3 strategic situation
82
181
194
123
68
194
63
136
53
69
187
86
32
126
H4 symmetry class
26
9
12
4
3
12
3
5
2
2
13
42
7
23
H7 line-purity template
11
38
39
28
24
39
27
33
36
23
22
13
17
12
Table 1: F -statistic on the principal t-SNE axis (Qwen2.5-1.5B, NL). Larger F means hypothesis-labelled boards separate more cleanly on the dominant axis. Trajectory (left) is the GRPO-from-base run at layer 12: H1 mid-layer F drops 17× and re-emerges in late layers and H2 progression grows 8.5× , while H3/H4 stay weak. Per-layer (right) is the converged checkpoint: H2 peaks at layer 12 , H4 stays at noise at every layer, and late layers (L17–L24) recover H1 for output decoding. The mid-layer locus is a property of the reorganisation, not of layer choice.
Figure 5: Clustering of board state representations based on structural line purity for Qwen2.5 1.5B (layer 12) on text-form boards. Labels mark dominant line-purity clusters (common row, column, or diagonal patterns), with smaller labels denoting automatically mined subclusters from agglomerative k-means. Manually annotated clusters merge related subclusters across groups for clarity. For instance, late-game states with a single empty cell in the top row cluster together. In labels, R/C/D denote rows, columns, and diagonals. O and X are player tokens. “.” is empty, T is taken (either player), A is any of X/O/empty. Full cluster maps for all layers, models, and modalities appear in Appendix Figure 22 . The key finding is that these local structures are already present in the base model and are retained through GRPO, while the macroscopic arrangement is what changes, acquiring the game-progression axis. Symmetry-class membership never organises the clusters.
Hypothesis
Base
Final ckpt
H1 turn identity
1.000
1.000
H2 game progression
1.000
1.000
H5 legality
1.000
1.000
H7 line purity
0.861
0.856
H3 strategic situation
0.686
0.684
H4 symmetry-group similarity
0.70
0.70
Table 2: Probe decodability (cross-validated AUC, middle layer of Qwen2.5-1.5B) read from raw activations. Prompt-level properties (H1, H2) and legality (H5) are already perfectly decodable in the base model, so GRPO cannot be adding missing information, and the properties with headroom (H3, H4) do not move across training. A non-linear MLP probe raises H3 by at most 0.06 and likewise does not move it across training, so GRPO changes which features dominate the representation, not which concepts are recoverable.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Progression analysis across board representations and randomization.
Figure 7: Overall performance across board representations and randomization.
Figure 8: Outcome score heatmaps across board representations and randomization.
Figure 9: Open spaces analysis across board representations and randomization.
Figure 10: Complexity analysis across board representations and randomization.
Table 3: Linear-probe results at the middle layer of Qwen2.5-1.5B: cross-validated AUC ± standard deviation across folds, for the base model and the final GRPO checkpoint. For H3, MLP probes reach 0.75 (base) and 0.73 (final checkpoint), so at most 0.05 to 0.06 of additional AUC is recoverable non-linearly. H4 uses the pair-separation test described below.
Model
H1 turn identity
H3 strategic situation
Qwen2.5-1.5B
1.00 at every checkpoint
0.68 to 0.70, flat
Qwen2.5-0.5B
1.00 at every checkpoint
0.61 to 0.65, flat
Llama-3.2-1B
1.00 at every checkpoint
0.65 to 0.70, flat
Appendix
Table 4: Logistic-regression AUC ranges across GRPO checkpoints at a mid-network layer. “Flat” means the range shows no upward trend under GRPO training. The dominant-axis reorganisation itself is clearest in Qwen2.5-1.5B and noisier in the smaller models, so we scope that claim to Qwen2.5-1.5B.
Hypothesis
Observed (base)
Observed (final)
Largest null
p
H1 turn identity
1.000
1.000
0.57
0.001
H2 game progression
0.996
0.994
0.58
0.001
H3 strategic situation
0.650
0.651
0.56
0.001
H7 line purity
0.834
0.818
0.61
0.001
H5 legality
1.000
1.000
0.56
0.001
Appendix
Table 5: Permutation tests (999 label permutations per comparison, LDA probe, middle layer, Qwen2.5-1.5B, held-out split): observed AUC for the base model and the final checkpoint, against the largest value in the shuffled null distribution. Even H3, the weakest hypothesis, sits far above its null ceiling, so the finding is that strategic information is limited and static under GRPO, not that it is absent.
GRPO-trained
Untrained
Elo
1525
1287
Games
154
160
Own forfeits
0
121
Per-move minimax-optimality (completed games)
77%
not measurable
Draws against the flash-tier frontier model
16 of 16
0
Appendix
Table 6: Arena results for GRPO-trained against untrained Qwen2.5-1.5B. Per-move minimax-optimality is not measurable for the untrained model because it forfeits most of its games before completion. Elo is order-dependent and X always moves first, so the comparison between the two rows rather than the absolute rating is the informative quantity.
Split
Symmetry groups
Pairs
Equivariance (%)
Unique-optimum (%)
n
Canonical
Training
1,484
56.7 (54.2–59.2)
60.0 (56.7–63.2)
857
Canonical
Held-out
653
57.7 (53.9–61.5)
62.7 (57.6–67.5)
359
Random
All
2,119
56.1 (54.0–58.2)
60.5 (57.8–63.3)
1,214
Appendix
Table 7: Behavioural equivariance of Qwen2.5-1.5B: percentage of symmetry-related board pairs whose predicted moves correspond under the relating transformation, with 95 percent Wilson intervals in parentheses. “Unique-optimum” restricts to the n pairs where each board has a unique optimal move, on which a symmetry-respecting policy must score 100 percent.
Model
Split
Training states
Held-out states
Qwen2.5-1.5B
Random
78.7 (77.5–79.9)
77.1 (74.5–79.5)
Qwen2.5-1.5B
Canonical
72.9 (71.4–74.2)
71.9 (69.7–74.0)
Qwen2.5-0.5B
Random
43.0 (41.5–44.5)
43.4 (40.5–46.4)
Qwen2.5-0.5B
Canonical
36.5 (35.0–38.1)
36.3 (34.0–38.6)
Appendix
Table 8: Best-move accuracy (%) on training against held-out states, with 95 percent Wilson intervals in parentheses.
Statistic
Count
Possible board assignments ( 39 )
19,683
Reachable legal states
5,478
Terminal / non-terminal
958 / 4,520
Symmetry groups
765
Of size 1/2/4/8
6/6/141/612
Random split (train/val/test)
4,382/547/549
Appendix
Table 9: Dataset statistics. The canonical-symmetry split is assigned at the symmetry-group level, and the illegal boards are balanced across violation categories.
Statistic
F
df
Nominal p
H1 turn, base model
5285
(1, 5476)
<10−300
H1 turn, final checkpoint
316
(1, 5476)
<10−68
H2 progression, base model
274
(2, 5475)
<10−100
H2 progression, final checkpoint
2330
(2, 5475)
<10−300
Appendix
Table 10: Degrees of freedom and nominal p for the central layer-12 F values of Table 1 . Both are fixed by the design: the numerator is the number of hypothesis classes minus one and the denominator is 5,478 minus the number of classes. Every value is significant at any conventional threshold, which is why the paper compares relative magnitudes rather than resting on significance.
Base
GRPO best ckpt
Hypothesis
F
V
AMI
F
V
AMI
H1 turn identity
5285
0.70
0.33
316
0.36
0.07
H2 game progression
274
0.48
0.13
2330
0.62
0.22
H3 strategic situation
82
0.24
0.05
194
0.22
0.05
H4 symmetry class
26
0.56
0.13
12
0.47
0.07
H7 line-purity template
11
0.40
0.25
39
0.40
0.22
Appendix
Table 11: Cluster-vs-hypothesis statistics at layer 12 of Qwen2.5-1.5B (NL prompt), comparing the base model and the converged GRPO-from-base run. F is the ANOVA on the principal t-SNE axis. V is Cramér’s V on the K -means cluster by hypothesis contingency table. AMI is adjusted mutual information. The H1 to H2 inversion is consistent across all three measures. H1 weakens (e.g. V0.70→0.36 ), H2 strengthens ( V0.48→0.62 ), H3 and H4 stay weak, and H7 is stable.
Model
Setting
NL
NL+XY
ASCII
ASCII+XY
Qwen2.5-1.5B
Off-the-shelf legal
11.7
–
11.1
–
GRPO legal
100.0
96.5
65.6
54.5
Off-the-shelf best
8.0
–
12.2
–
GRPO best (from base)
71.6
61.9
47.4
41.9
GRPO best (from legal)
75.6
70.1
50.3
50.5
Qwen2.5-0.5B
Off-the-shelf legal
26.8
–
26.6
–
Appendix
Table 12: Best-checkpoint accuracy (%) for each model, training group, and input perturbation. “NL+XY” is a natural-language prompt with random symbol substitution applied. “ASCII” is an ASCII board input. “–” marks settings where the off-the-shelf evaluations were not run under the corresponding random symbol substitution. The structured-output Off-the-shelf rows isolate the format component of the GRPO reward. The gap to the GRPO rows in the same column quantifies the contribution of the correctness signal. The persistent ASCII degradation across both legal-move and best-move objectives indicates that GRPO does not produce input-format-invariant board understanding.
Branching ( # legal moves)
Best-move multiplicity
Game phase
Model
low ( ≤2 )
high ( ≥5 )
unique
≥2
mid (5–6)
late ( ≥7 )
Qwen2.5-1.5B
89.4
63.6
62.1
86.9
73.3
88.4
Qwen2.5-0.5B
62.9
33.3
21.8
44.3
37.3
57.8
Llama3.2-1B
14.7
32.3
18.6
55.7
30.7
17.0
Appendix
Table 13: Per-position best-move accuracy (%) for the strongest GRPO best-move checkpoint per model on the random split (NL prompt, no symbol substitution). Branching is the number of legal moves. Multiplicity is the number of equally-optimal moves. Phase counts ply (move number). Values are raw accuracies, not corrected for the per-state chance level, which itself varies with branching, and the chance-corrected analysis in the main text shows the branching gradient is largely a chance artifact. Llama3.2-1B is an outlier: its late-game accuracy ( 17% ) is below its mid-game accuracy ( 31% ), indicating that even basic terminal-state recognition was not internalised. The raw gradient recurs at lower absolute accuracy for Qwen2.5-0.5B (low-branch 63% vs. high-branch 33% ).
Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it. As a result, two basic questions remain open: (1) how do pretraining choices (model size, data) shape the returns to RL compute, and (2) what does RL actually do to the model? These questions are difficult to study in the standard LLM setting: pretraining corpora are vast and uncontrolled, making it hard to attribute behaviors to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive. To address these challenges, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline. We follow the standard LLM training pipeline by pretraining language models from 5M to 1B parameters on human chess games, supervised fine-tuning on synthetic reasoning traces, and running RL on chess puzzles with verifiable rewards. Using this framework, we find that the post-RL performance at given RL compute level is well-predicted from the pretraining loss, and slope of the RL reward curves improves approximately linearly with the pretraining tokens. Beyond scaling, we find that RL does not simply sharpen the SFT policy: on easy puzzles it amplifies correct moves the SFT policy already preferred, while on hard puzzles it surfaces correct moves that were nearly absent under SFT. We further test whether our findings transfer beyond chess by training a 1B language model on math-domain text, where the same predictive pattern emerges: longer-pretrained checkpoints reach higher post-RL performance and improve faster under RL. In sum, we provide a quantitative account of the pretraining-to-RL interface and a controlled testbed for studying the science of reasoning across the full pretraining-to-post-training pipeline.
Jingyan Shen, Ang Li, Salman Rahman +4
New York University · Columbia University · University of California, Los Angeles +2
Reinforcement learning has become the standard for improving reasoning in large language models, yet evidence increasingly suggests that RL does not teach new strategies; it redistributes probability mass over solutions the base model already contains. In this work, we ask: if RL merely steers the model toward paths it already knows, is the RL optimization loop itself necessary? Through token-level analysis across multiple model families and RL algorithms, we find that RL's beneficial footprint is a sparse, predictable correction concentrated at high-entropy decision points where the model is uncertain which branch to take. Only 1--3% of token positions are affected, the promoted token always lies within the base model's top-5 alternatives, and targeted corrections at those few positions causally recover a large fraction of RL's accuracy gain, while random corrections fail. The base model's own entropy identifies these positions without any RL-trained model, and the entire correction is low-dimensional, representable in a tiny fraction of model parameters. These findings reframe reasoning improvement as sparse policy selection, not capability acquisition. We translate this insight into ReasonMaxxer, a minimal RL-free method that applies contrastive loss only at entropy-gated decision points, using a few hundred base-model rollouts and no online generation. Across three model families, six scales, and six math reasoning benchmarks, ReasonMaxxer matches or exceeds full RL performance while requiring only tens of problems and minutes of single-GPU training, a reduction in training cost of roughly three orders of magnitude.
Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger +1
Recent studies on reinforcement learning (RL) report seemingly conflicting evidence about large language model (LLM) reasoning. Training on mathematics can improve performance in other domains, yet gains in Pass@1 can coincide with lower Pass@N than the base model. This raises a fundamental question: does RL expand an LLM's reasoning boundary, or merely reweight its existing reasoning space? We revisit these phenomena across Qwen and Gemma model families, showing both cross-domain gains and forgetting, while coverage at large sampling budgets increases on some tasks and decreases on others. Detailed analysis of solution traces before and after RL indicates a shift in the reasoning strategies the model employs, motivating a two-stage autoregressive policy model that separates strategy selection from problem-specific execution. Within this framework, we prove how RL's implicit bias reshapes strategy preferences, allowing gains on some tasks while suppressing strategies required by others. This mechanism can also broaden or narrow coverage at a given sampling budget even without expanding strategy support. We further provide theoretical justifications for log-sigmoid and log-linear scaling laws in RL compute, and evaluate their predictive power. Together, these results connect changes in strategy selection to cross-domain transfer, reasoning coverage, and compute scaling.