Reasoning or Pattern Exploitation? Mechanistic Insights from RL-Trained Language Models

Aishik Nagar; Cheston Tan

cs.AIOct 29, 2026

Reasoning or Pattern Exploitation? Mechanistic Insights from RL-Trained Language Models

Authors: Aishik Nagar, Cheston Tan

Organizations: Carnegie Mellon University · Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR) · Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR)

Abstract

Large language models (LLMs) are increasingly described as acquiring "reasoning" skills after reinforcement learning with verifiable rewards. However, whether these gains reflect genuine reasoning or more superficial adaptations remains underexplored. In this work, we probe LLMs trained in a finite and exhaustively analyzable logical domain, namely Tic-Tac-Toe, and trace how internal representations evolve across reinforcement learning (RL) with Group Relative Policy Optimization (GRPO). Quantitatively, RL yields high accuracy and robustness across prompt variations, suggesting the emergence of game reasoning. Mechanistic interpretability, however, paints a different picture: features extracted with sparse autoencoders (SAEs) reveal that models primarily adapt to rely more heavily on information already explicit in the prompt, such as whose turn it is, game progression and board occupancy. In comparison, high-level concepts like board symmetries and strategic forks remain weakly represented, raising concerns that reasoning benchmarks overestimate model abilities. This suggests that RL-driven "reasoning" may conflate task-specific adaptations to input information with reasoning ability. We provide, for the first time: (i) a systematic interpretability pipeline tracing representation dynamics across RL training in small-scale LLMs; (ii) an extension of SAE-based feature discovery to hypothesis-driven testing in a finite logical domain; and (iii) an interpretability-based demonstration that RL post-training neither adds nor removes concept information but reorganizes which features dominate the representation, with causal interventions locating the decision downstream of it, in the late-layer state.

Figures & tables

Figure 1: Overview of our experimental setup

Figure 2: Best performance of each model and training setting for the legal move and best move prediction tasks on the random-split dataset. Complete performance plots for all settings (different board and move token representations) and datasets can be found in Appendix Figure 7 .

Figure 3: Progression of model performance across model checkpoints for the two datasets. Complete progression plots for all settings (different board and move token representations) can be found in Appendix Figure 6 .

Figure 4: Model representations dynamics of Qwen2.5 1.5B (layer 12) through reasoning (GRPO) training. The figure represents the same projections colored based on three hypotheses: (H1) whose turn it is (Game Turn), (H2) which stage of the game it is (Game Progression) and (H3) what is the logical play for the given board (Strategic Situation). It is evident that while the base model only clearly represents the game turn, during the course of training, the model evolves to represents the boards based on game progression in addition to game turn, leading to improvement in performance. However, logic for strategic situations does not play a big role in model representations and no clear patterns emerge over the course of training. Full training dynamics across all layers and over all hypotheses tested are provided in Appendix Section C .

	Trajectory at L12			Per-layer at converged ckpt
Hypothesis	Base	Mid	Final	L4	L8	L12	L13	L14	L15	L16	L17	L18	L20	L24
H1 turn identity	5285	366	316	270	140	316	114	269	188	95	2208	5379	1508	5484
H2 game progression	274	1686	2330	1151	594	2330	412	1278	287	358	747	284	114	400
H3 strategic situation	82	181	194	123	68	194	63	136	53	69	187	86	32	126
H4 symmetry class	26	9	12	4	3	12	3	5	2	2	13	42	7	23
H7 line-purity template	11	38	39	28	24	39	27	33	36	23	22	13	17	12

Table 1:

F

-statistic on the principal t-SNE axis (Qwen2.5-1.5B, NL). Larger

F

means hypothesis-labelled boards separate more cleanly on the dominant axis. Trajectory (left) is the GRPO-from-base run at layer 12: H1 mid-layer F drops

17\times

and re-emerges in late layers and H2 progression grows

8.5\times

, while H3/H4 stay weak. Per-layer (right) is the converged checkpoint: H2 peaks at layer 12 , H4 stays at noise at every layer, and late layers (L17–L24) recover H1 for output decoding. The mid-layer locus is a property of the reorganisation, not of layer choice.

Figure 5: Clustering of board state representations based on structural line purity for Qwen2.5 1.5B (layer 12) on text-form boards. Labels mark dominant line-purity clusters (common row, column, or diagonal patterns), with smaller labels denoting automatically mined subclusters from agglomerative k-means. Manually annotated clusters merge related subclusters across groups for clarity. For instance, late-game states with a single empty cell in the top row cluster together. In labels, R/C/D denote rows, columns, and diagonals. O and X are player tokens. “.” is empty, T is taken (either player), A is any of X/O/empty. Full cluster maps for all layers, models, and modalities appear in Appendix Figure 22 . The key finding is that these local structures are already present in the base model and are retained through GRPO, while the macroscopic arrangement is what changes, acquiring the game-progression axis. Symmetry-class membership never organises the clusters.

Hypothesis	Base	Final ckpt
H1 turn identity	$1.000$	$1.000$
H2 game progression	$1.000$	$1.000$
H5 legality	$1.000$	$1.000$
H7 line purity	$0.861$	$0.856$
H3 strategic situation	$0.686$	$0.684$
H4 symmetry-group similarity	$0.70$	$0.70$

Table 2: Probe decodability (cross-validated AUC, middle layer of Qwen2.5-1.5B) read from raw activations. Prompt-level properties (H1, H2) and legality (H5) are already perfectly decodable in the base model, so GRPO cannot be adding missing information, and the properties with headroom (H3, H4) do not move across training. A non-linear MLP probe raises H3 by at most

0.06

and likewise does not move it across training, so GRPO changes which features dominate the representation, not which concepts are recoverable.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Progression analysis across board representations and randomization.

Panel of Figure 7 — Figure 7: Overall performance across board representations and randomization.

Panel of Figure 8 — Figure 8: Outcome score heatmaps across board representations and randomization.

Panel of Figure 9 — Figure 9: Open spaces analysis across board representations and randomization.

Panel of Figure 10 — Figure 10: Complexity analysis across board representations and randomization.

Panel of Figure 11 — Figure 11: Hypothesis testing: Game progression.

Panel of Figure 12 — Figure 12: Hypothesis testing: Game trajectories.

Panel of Figure 13 — Figure 13: Hypothesis testing: Game turn.

Panel of Figure 14 — Figure 14: Hypothesis testing: Prediction correctness.

Panel of Figure 15 — Figure 15: Hypothesis testing: Illegal vs Legal boards.

Panel of Figure 16 — Figure 16: Hypothesis testing: Strategic situations.

Panel of Figure 17 — Figure 17: Hypothesis testing: Symmetry.

Panel of Figure 18 — Figure 18: Hypothesis testing: Threat detection.

Panel of Figure 19 — Figure 19: Hypothesis testing: Turn-by-turn paired analysis.

Panel of Figure 20 — Figure 20: Hypothesis testing: Winner identification.

Panel of Figure 21 — Figure 21: Hypothesis testing: Winner identification with best labels.

Panel of Figure 22 — Figure 22: KMeans clustering: Dominant line patterns.

Panel of Figure 23 — Figure 23: KMeans clustering: Global extremes (normalized).

Panel of Figure 24 — Figure 24: KMeans clustering: Hypothesis summary (normalized).

Hypothesis	Base model	Final checkpoint
H1 turn identity	$1.000\pm 0.000$	$1.000\pm 0.000$
H2 game progression	$1.000\pm 0.000$	$1.000\pm 0.000$
H3 strategic situation	$0.686\pm 0.011$	$0.684\pm 0.012$
H7 line purity	$0.861\pm 0.014$	$0.856\pm 0.011$
H5 legality (legal vs. unreachable)	$1.000\pm 0.000$	$1.000\pm 0.000$
H4 same-symmetry-group similarity (pair test)	0.70	0.70

Appendix

Table 3: Linear-probe results at the middle layer of Qwen2.5-1.5B: cross-validated AUC

\pm

standard deviation across folds, for the base model and the final GRPO checkpoint. For H3, MLP probes reach 0.75 (base) and 0.73 (final checkpoint), so at most 0.05 to 0.06 of additional AUC is recoverable non-linearly. H4 uses the pair-separation test described below.

Model	H1 turn identity	H3 strategic situation
Qwen2.5-1.5B	1.00 at every checkpoint	0.68 to 0.70, flat
Qwen2.5-0.5B	1.00 at every checkpoint	0.61 to 0.65, flat
Llama-3.2-1B	1.00 at every checkpoint	0.65 to 0.70, flat

Appendix

Table 4: Logistic-regression AUC ranges across GRPO checkpoints at a mid-network layer. “Flat” means the range shows no upward trend under GRPO training. The dominant-axis reorganisation itself is clearest in Qwen2.5-1.5B and noisier in the smaller models, so we scope that claim to Qwen2.5-1.5B.

Hypothesis	Observed (base)	Observed (final)	Largest null	$p$
H1 turn identity	1.000	1.000	0.57	0.001
H2 game progression	0.996	0.994	0.58	0.001
H3 strategic situation	0.650	0.651	0.56	0.001
H7 line purity	0.834	0.818	0.61	0.001
H5 legality	1.000	1.000	0.56	0.001

Appendix

Table 5: Permutation tests (999 label permutations per comparison, LDA probe, middle layer, Qwen2.5-1.5B, held-out split): observed AUC for the base model and the final checkpoint, against the largest value in the shuffled null distribution. Even H3, the weakest hypothesis, sits far above its null ceiling, so the finding is that strategic information is limited and static under GRPO, not that it is absent.

	GRPO-trained	Untrained
Elo	1525	1287
Games	154	160
Own forfeits	0	121
Per-move minimax-optimality (completed games)	77%	not measurable
Draws against the flash-tier frontier model	16 of 16	0

Appendix

Table 6: Arena results for GRPO-trained against untrained Qwen2.5-1.5B. Per-move minimax-optimality is not measurable for the untrained model because it forfeits most of its games before completion. Elo is order-dependent and X always moves first, so the comparison between the two rows rather than the absolute rating is the informative quantity.

Split	Symmetry groups	Pairs	Equivariance (%)	Unique-optimum (%)	$n$
Canonical	Training	1,484	56.7 (54.2–59.2)	60.0 (56.7–63.2)	857
Canonical	Held-out	653	57.7 (53.9–61.5)	62.7 (57.6–67.5)	359
Random	All	2,119	56.1 (54.0–58.2)	60.5 (57.8–63.3)	1,214

Appendix

Table 7: Behavioural equivariance of Qwen2.5-1.5B: percentage of symmetry-related board pairs whose predicted moves correspond under the relating transformation, with 95 percent Wilson intervals in parentheses. “Unique-optimum” restricts to the

n

pairs where each board has a unique optimal move, on which a symmetry-respecting policy must score 100 percent.

Model	Split	Training states	Held-out states
Qwen2.5-1.5B	Random	78.7 (77.5–79.9)	77.1 (74.5–79.5)
Qwen2.5-1.5B	Canonical	72.9 (71.4–74.2)	71.9 (69.7–74.0)
Qwen2.5-0.5B	Random	43.0 (41.5–44.5)	43.4 (40.5–46.4)
Qwen2.5-0.5B	Canonical	36.5 (35.0–38.1)	36.3 (34.0–38.6)

Appendix

Table 8: Best-move accuracy (%) on training against held-out states, with 95 percent Wilson intervals in parentheses.

Statistic	Count
Possible board assignments ( $3^{9}$ )	19,683
Reachable legal states	5,478
Terminal / non-terminal	958 / 4,520
Symmetry groups	765
Of size $1/2/4/8$	$6/6/141/612$
Random split (train/val/test)	4,382/547/549

Appendix

Table 9: Dataset statistics. The canonical-symmetry split is assigned at the symmetry-group level, and the illegal boards are balanced across violation categories.

Statistic	$F$	df	Nominal $p$
H1 turn, base model	5285	(1, 5476)	$<10^{-300}$
H1 turn, final checkpoint	316	(1, 5476)	$<10^{-68}$
H2 progression, base model	274	(2, 5475)	$<10^{-100}$
H2 progression, final checkpoint	2330	(2, 5475)	$<10^{-300}$

Appendix

Table 10: Degrees of freedom and nominal

p

for the central layer-12

F

values of Table 1 . Both are fixed by the design: the numerator is the number of hypothesis classes minus one and the denominator is 5,478 minus the number of classes. Every value is significant at any conventional threshold, which is why the paper compares relative magnitudes rather than resting on significance.

Hypothesis	$F$	V	AMI	$F$	V	AMI
	Base			GRPO best ckpt
H1 turn identity	5285	0.70	0.33	316	0.36	0.07
H2 game progression	274	0.48	0.13	2330	0.62	0.22
H3 strategic situation	82	0.24	0.05	194	0.22	0.05
H4 symmetry class	26	0.56	0.13	12	0.47	0.07
H7 line-purity template	11	0.40	0.25	39	0.40	0.22

Appendix

Table 11: Cluster-vs-hypothesis statistics at layer 12 of Qwen2.5-1.5B (NL prompt), comparing the base model and the converged GRPO-from-base run.

F

is the ANOVA on the principal t-SNE axis.

V

is Cramér’s V on the

K

-means cluster by hypothesis contingency table. AMI is adjusted mutual information. The H1 to H2 inversion is consistent across all three measures. H1 weakens (e.g.

V

0.70\!\to\!0.36

), H2 strengthens (

V

0.48\!\to\!0.62

), H3 and H4 stay weak, and H7 is stable.

Model	Setting	NL	NL+XY	ASCII	ASCII+XY
Qwen2.5-1.5B	Off-the-shelf legal	11.7	–	11.1	–
	GRPO legal	100.0	96.5	65.6	54.5
	Off-the-shelf best	8.0	–	12.2	–
	GRPO best (from base)	71.6	61.9	47.4	41.9
	GRPO best (from legal)	75.6	70.1	50.3	50.5
Qwen2.5-0.5B	Off-the-shelf legal	26.8	–	26.6	–

Appendix

Table 12: Best-checkpoint accuracy (%) for each model, training group, and input perturbation. “NL+XY” is a natural-language prompt with random symbol substitution applied. “ASCII” is an ASCII board input. “–” marks settings where the off-the-shelf evaluations were not run under the corresponding random symbol substitution. The structured-output Off-the-shelf rows isolate the format component of the GRPO reward. The gap to the GRPO rows in the same column quantifies the contribution of the correctness signal. The persistent ASCII degradation across both legal-move and best-move objectives indicates that GRPO does not produce input-format-invariant board understanding.

	Branching ( $\#$ legal moves)		Best-move multiplicity		Game phase
Model	low ( $\leq 2$ )	high ( $\geq 5$ )	unique	$\geq 2$	mid (5–6)	late ( $\geq 7$ )
Qwen2.5-1.5B	89.4	63.6	62.1	86.9	73.3	88.4
Qwen2.5-0.5B	62.9	33.3	21.8	44.3	37.3	57.8
Llama3.2-1B	14.7	32.3	18.6	55.7	30.7	17.0

Appendix

Table 13: Per-position best-move accuracy (%) for the strongest GRPO best-move checkpoint per model on the random split (NL prompt, no symbol substitution). Branching is the number of legal moves. Multiplicity is the number of equally-optimal moves. Phase counts ply (move number). Values are raw accuracies, not corrected for the per-state chance level, which itself varies with branching, and the chance-corrected analysis in the main text shows the branching gradient is largely a chance artifact. Llama3.2-1B is an outlier: its late-game accuracy (

17\%

) is below its mid-game accuracy (

31\%

), indicating that even basic terminal-state recognition was not internalised. The raw gradient recurs at lower absolute accuracy for Qwen2.5-0.5B (low-branch

63\%

vs. high-branch

33\%

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,103 papers

Understanding Reasoning from Pretraining to Post-Training
Jul 17, 2026Jingyan Shen, Ang Li, Salman Rahman +4Language Model Pretraining RL for Language Model Reasoning
New York University · Columbia University · University of California, Los Angeles +2
Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
May 7, 2026Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger +1RL for Language Model Reasoning Efficient Language Model Reasoning
University of Southern California · 2DEVCOM ARL
How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws
Oct 3, 2026Ziheng Cheng, Yixiao Huang, Hanlin Zhu +1RL for Language Model Reasoning Reinforcement Learning
University of California, Berkeley

Jul 17, 2026J/K move · Enter open · S save

Understanding Reasoning from Pretraining to Post-Training

Jingyan Shen, Ang Li, Salman Rahman, Yifan Sun+3

New York University · Columbia University · University of California, Los Angeles+2

Language Model Pretraining RL for Language Model Reasoning RL Post-Training

Open paper AskPDF arXiv

Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it. As a result, two basic questions remain open: (1) how do pretraining choices (model size, data) shape the returns to RL compute, and (2) what does RL actually do to the model? These questions are difficult to study in the standard LLM setting: pretraining corpora are vast and uncontrolled, making it hard to attribute behaviors to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive. To address these challenges, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline. We follow the standard LLM training pipeline by pretraining language models from 5M to 1B parameters on human chess games, supervised fine-tuning on synthetic reasoning traces, and running RL on chess puzzles with verifiable rewards. Using this framework, we find that the post-RL performance at given RL compute level is well-predicted from the pretraining loss, and slope of the RL reward curves improves approximately linearly with the pretraining tokens. Beyond scaling, we find that RL does not simply sharpen the SFT policy: on easy puzzles it amplifies correct moves the SFT policy already preferred, while on hard puzzles it surfaces correct moves that were nearly absent under SFT. We further test whether our findings transfer beyond chess by training a 1B language model on math-domain text, where the same predictive pattern emerges: longer-pretrained checkpoints reach higher post-RL performance and improve faster under RL. In sum, we provide a quantitative account of the pretraining-to-RL interface and a controlled testbed for studying the science of reasoning across the full pretraining-to-post-training pipeline.

Abstract

RL for Language Model Reasoning

Explore similar work

Understanding Reasoning from Pretraining to Post-Training

Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning

How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws