How Reasoning Evolves from Post-Training Data: An Empirical Study Using Chess

Lucas Dionisopoulos; Nicklas Majamaki; Prithviraj Ammanabrolu

cs.LGApr 6, 2026

How Reasoning Evolves from Post-Training Data: An Empirical Study Using Chess

Authors: Lucas Dionisopoulos, Nicklas Majamaki, Prithviraj Ammanabrolu

Organizations: Department of Computer Science and Engineering, University of California San Diego, San Diego, United States.

Abstract

We study how reasoning evolves in a language model -- from supervised fine-tuning (SFT) to reinforcement learning (RL) -- by analyzing how a set of theoretically-inspired datasets influences language model performance in chess. We find that fine-tuning a model to directly predict the best move leads to effective RL and the strongest downstream performance -- however, the RL stage elicits unfaithful reasoning (reasoning inconsistent with the chosen move). Alternatively, training on multi-move trajectories yields comparable downstream performance with faithful reasoning and more stable RL. We analyze multiple qualitative and quantitative measures and highlight how these evolve from SFT through RL; we find several SFT-checkpoint metrics -- spanning evaluation performance, hallucination rates, and reasoning quality -- to be predictive of post-RL model performance. Finally, we ground our results with an experiment measuring chess information density in our custom datasets. We release models as well as training data, evaluations, and code that allowed us to surpass leading open-source reasoning models in chess with a 7B-parameter model. Code, models, and data are available at https://github.com/lucasdino/lang-chess.

Figures & tables

Figure 1: Following initial data inclusion experiments, we scaled SFT on our two best-performing datasets. Both resulted in comparably strong final evaluation performance, but training on optimal move trajectories ( Best Line ) led to more stable RL and faithful reasoning compared to training on single best moves ( Best Move ).

Figure 2: Performance of our best reasoning model trained from Qwen2.5 7B-Instruct across our evaluations . Trivial performance (i.e., random guessing) is $0.2$ for the Best Move and Worst Move tasks. See Appendix B for example evaluation questions and Appendix D for full results.

Figure 3: Samples from our custom datasets . The gray font represents an abbreviation of the core prompt – in all samples the model is trained with a verbose instructive prompt and provided with a board in our visual ASCII-format. Full samples included in Appendix C .

Figure 4: RL training performance on our scaled SFT-checkpoints . Left : Train reward and tokens per response (smoothed using an exponential moving average with decay factor 0.9). Right : Reward on the held-out evaluation set during training. The Best Move dataset, while having strong ending performance, experienced more unstable RL compared to scaled runs trained on Best Line data.

Figure 5: Results on evaluation tasks for final RL models from each data inclusion experiment . Within each metric we split results into three sections: (left) compares single vs. multitask, (middle) compares the targeted data inclusion experiments, (right) covers our data diversity experiments. Note that in all experiments we SFT on $15$ million tokens and do RL on $8$ k samples. In the single task setting, RL only uses the Predict Move task; in all other settings the $8$ k samples are split evenly between our four task types. See Appendix D for detailed table results (including exact token distributions).

Figure 6: Reasoning faithfulness . RL on the Best Move SFT-checkpoint induced unfaithful reasoning whereas checkpoints trained on multi-step data were more robust. Appendix H has further detail on our reasoning quality measurement.

Figure 7: Move quality distribution . Our Best Move + Best Line scaled run saw a significant distribution shift after RL in its move quality on the Predict Move evaluation ( $n=400$ ). This shift is an improvement on both an absolute and relative basis, highlighting the efficacy of RL.

Figure 8: Hallucination rate . RL drove a meaningful decrease in hallucination rate as a side effect of simply maximizing reward on our evaluations. Appendix F has further detail on hallucinations.

Figure 9: Linear regression comparing the final RL model (average score over all evaluations) with various metrics from its corresponding SFT-checkpoint. Left : Vs. average score over all evaluations. Middle : Vs. percent of moves referenced during reasoning trace that are legal (parsed by Llama 4 Maverick). Right : Vs. reasoning quality (mean over all reasoning quality metrics as judged by gpt-oss-120b). Shaded region represents the $95\%$ confidence intervals.

Dataset	Pre-SFT	Post-SFT
Rejection Sampling	12.69%	20.83%
Guided Synthetic	2.90%	9.77%
Factual Board Answering	0.04%	62.54%
Verbalized Alpha-Beta Pruning	4.31%	71.04%
Best Move	0.04%	28.61%
Best Line	0.00%	24.00%

Table 1: Fraction of validation tokens assigned probability >0.995 before and after SFT . Higher percentages indicate lower predictive complexity (e.g., simple task or memorizable phrases).

Question Type	Token Portion	Pre-SFT	Post-SFT
is_legal	13.59%	0.00%	24.85%
under_attack	12.43%	0.00%	30.46%
mobility	24.54%	0.01%	43.88%
cloze_capture	15.94%	0.14%	80.26%
is_check	4.95%	0.00%	85.16%
mat_adv_value	28.54%	0.05%	96.66%

Table 2: Validation tokens assigned probability >0.995 before and after SFT for Factual Board Answering tasks . See Appendix C for descriptions of each task.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Visualized tokenization of three candidate board formats using the Qwen2.5 tokenizer.

Figure 11: Example questions for each evaluation task on the same board. Note that in actual prompts (omitted in the figure) we include information related to the required format for valid parsing.

Figure 12: A Rejection Sampling example. The full response in the sample is shortened for space. Note that a drawback of this dataset is that it is prone to hallucinations as shown in the provided sample.

Figure 13: Sample of the Guided Synthetic data. Note that while the teacher model is provided with a line and centipawn difference, the teacher model is still prone to hallucination (it cites a material imbalance – this is not true). Additionally, despite prompting the teacher to use UCI notation, many examples still use SAN due to teacher model bias.

Figure 14: Sample of Verbalized Alpha-Beta Pruning . This sample highlights branching, minimax decision making, and an instance of pruning.

Figure 15: Sample of the Factual Board Answering dataset. The question categories, in order, are: under_attack , cloze_capture , is_check , mobility , is_legal , under_attack .

Figure 16: Samples of Best Line and Best Move .

Figure 17: Distribution of tokens used in each experiment. Token numbers are shown in millions; we sampled our data to match this distribution, though there may be immaterial variations for actual token counts used. We include tags (e.g., [VABP] ) for mnemonic reference. Note that with the Rejection Sampling (All Evals) [RSA] dataset, we allocate $50\%$ of tokens to the Predict Move task and sample the remainder from the other evaluation tasks. SFT8 + SFT9 XL was trained by taking the SFT8 XL model checkpoint and training on the SFT9 XL dataset.

Experiment Name	SFT Train Tokens	RL a Samples	Pred. Move % Legal $\uparrow$		Pred. Move Avg. Rank $\uparrow$		% Trivial b Moves $\downarrow$
			SFT	RL	SFT	RL	SFT	RL
Baselines
Qwen2.5 7B - Instruct	–	–	8%		0.56		6%
Llama 4 Maverick	–	–	42%		0.62		1%
gpt-oss-120b (Medium)	–	–	94%		0.66		2%
Inclusion Experiments

Appendix

Table 3: Results are shown for the Predict Move evaluation on 400 samples. Predict Move Average Rank is the average normalized rank (with 0 being the worst move and 1 being the best move per Stockfish) of the legal moves provided in this task.

Experiment Name	Legal Moves IoU $\uparrow$		Best Move Acc. $\uparrow$		Worst Move Acc. $\uparrow$		Ref’d Move Acc. a $\uparrow$		Avg. Reas. Quality b $\uparrow$
	SFT	RL	SFT	RL	SFT	RL	SFT	RL	SFT	RL
Baselines
Qwen2.5 7B - Instruct	0.26		19%		21%		12%		6.3
Llama 4 Maverick	0.43		27%		31%		38%		5.8
gpt - oss - 120b (Medium)	0.96		57%		79%		70%		7.0
Inclusion Experiments

Appendix

Table 4: See Figure 17 for detail on the data included in each inclusion experiment and Table 3 for performance on the Predict Move task. Each evaluation shown is based on

400

unique samples for each task. The Legal Moves task asks the model to produce a list of legal moves given a target piece – the score is measured as intersection over union (IoU) vs. ground truth. Best Move and Worst Move ask the model to, given a list of

5

moves, choose the best (or worst, respectively) move of the list – the incorrect candidates are sampled such that they are beyond a sufficient threshold of difference per Stockfish. See Appendix B for examples of each task.

Experiment Name	Factual Board Answering a $\uparrow$		Out-of-Distrib. Mates b $\uparrow$
	SFT	RL	SFT	RL
Baselines
Qwen2.5 7B - Instruct	31.6%		0.0%
Llama 4 Maverick	46.7%		14.0%
gpt - oss - 120b (Medium)	99.5%		78.3%
Inclusion Experiments

Appendix

Table 5: See Figure 17 for detail on the data included in each inclusion experiment. Results test performance on tasks not trained on during the RL stage. Note that some of the experiments were trained on FBA data during the SFT stage – these are tagged with footnote c .

Figure 18: Examples from each of the three subtasks included in the Out-of-Distribution Mates evaluation set sourced from ( Mészáros et al., 2025 ) . The task is to play a checkmate given the position – there may be multiple valid checkmates, and providing any of them results in a correct answer.

Figure 19: Loss per epoch of each dataset in the information density experiments. Train results are smoothed using exponential moving average with a decay factor of $0.9$ . Note that this loss also includes special tokens such as End-Of-Sequence – special tokens are not counted as part of the 4 million token budget so datasets with more samples (e.g., Best Move ) are trained on more tokens given these special tokens. This complicates direct loss comparison across datasets.

Figure 20: Predictive complexity across the full validation set – measured as the portion of tokens assigned a probability $>0.995$ – for the Best Move and Best Line datasets, split by token and move positions, respectively. Average (Avg.*) excludes any 5th tokens for Best Move (e.g., piece promotions) and any line valuations for Best Line .

Figure 21: Factual Board Answering sample following SFT in the information density experiment. For the first question, it incorrectly predicts this move is legal – the opposing queen on h5 prevents the king from making this move.

Figure 22: Verbalized Alpha-Beta Pruning sample following SFT in the information density experiment. Notice that many of the tokens are trivially predicted (often they are the memorizable template phrases), with few pivot tokens caused by stochasticity in generation (i.e., sampling a phrase) or non-trivial chess moves.

Figure 23: Guided Synthetic sample following SFT in the information density experiment. Given the dataset is synthetic and hence natural language, there is a much lower portion of trivial tokens.

Figure 24: Best Move sample following SFT in the information density experiment. See Figure 20 for more detail on triviality by position across the whole validation set.

Figure 25: Best Line sample following SFT in the information density experiment. See Figure 20 for more detail on triviality by move number across the whole validation set.

Panel of Figure 26 — Figure 26: Accuracy of tested models for both moves and pieces referenced in their reasoning traces. Bars are overlaid directly on top of each other and stacking is not cumulative. Accuracy is computed as the number of correct references divided by the total number of references. See Figure 17 for detail on the data included in each experiment. Hatched lines are shown in cases where the SFT and RL runs are within $2\%$ of each other.

Panel of Figure 27 — Figure 27: Usage rate of reasoning strategies on $400$ Predict Move tasks. Bars are overlaid directly on top of each other and stacking is not cumulative. Reasoning strategies are parsed using Llama 4 Scout and usage is measured as a binary flag for each evaluation sample. Hatched lines are shown in cases where the SFT and RL runs are within $2\%$ of each other.

Panel of Figure 28 — Figure 28: Reasoning quality scores on $400$ Predict Move tasks. Bars are overlaid directly on top of each other and stacking is not cumulative. Reasoning quality is scored by gpt-oss-120b and scores are provided from $1$ to $10$ . The Mean Reasoning Quality (Total) score is a simple average over the three subcategories. Hatched lines are shown in cases where the SFT and RL runs are within $2\%$ of each other.

Expert	N	Efficacy		Efficiency		Faithfulness		Sum of All
		Corr	P-Val	Corr	P-Val	Corr	P-Val	Corr	P-Val
Expert 1	96	0.41	4.01e-05	0.31	2.03e-03	0.68	1.50e-14	0.57	1.10e-09
Expert 2	73	0.24	4.00e-02	0.34	3.07e-03	0.43	1.71e-04	0.39	6.63e-04

Appendix

Table 6: Statistical significance of expert and LLM-judge scores across qualitative reasoning metrics. Samples are graded blindly and drawn randomly from among all tested models (baselines and experiments) excluding gpt-oss-120b. We exclude gpt-oss-120b as language models have shown a tendency to favor their own outputs ( Panickssery et al., 2024 ) which may adversely affect our analysis.

Figure 29: Example of unfaithful reasoning – given a score of $1$ in reasoning faithfulness. Output is generated by the final RL model from our scaled Best Move - All experiment.

Parameter	Value
Training engine	LlamaFactory ( Zheng et al., 2024 )
Fine-tuning type	Full SFT
LR scheduler	Cosine
Precision	BF16
Optimizer	AdamW ( Loshchilov and Hutter, 2019 )
Learning rate	$3\times 10^{-6}$

Appendix

Table 7: SFT training hyperparameters.

Parameter	Value
Training engine	veRL ( Sheng et al., 2025 )
Objective	Dr. GRPO ( Liu et al., 2025 )
Learning rate	$1\times 10^{-6}$
Train batch size	64
Max response length	$3,000$ tokens
Actor clip ratio (low/high)	0.20 / 0.28 ( Yu et al., 2025 )

Appendix

Table 8: RL training hyperparameters.

Panel of Figure 30 — Figure 30: Select training dynamics during reinforcement learning across our inclusion and scaled experiments.

Panel of Figure 31 — Figure 31: Select training dynamics during reinforcement learning across our inclusion and scaled experiments (cont.).

Language Model Post-Training

Language model post-training adapts pretrained models to follow instructions, reason, use tools, and meet task-specific or human-defined goals.

See all 195 papers

Understanding Reasoning from Pretraining to Post-Training
Jul 17, 2026Jingyan Shen, Ang Li, Salman Rahman +4Language Model Pretraining RL for Language Model Reasoning
New York University · Columbia University · University of California, Los Angeles +2
The Weight of Silence: A Causal Case for Weights Over the Scratchpad in Latent Chess Reasoning
Jul 23, 2026Ishan S. KshirsagarRL for Language Model Reasoning Continuous Latent Reasoning
Independent Researcher
Grounded Chess Reasoning in Language Models via Master Distillation
Mar 20, 2026Zhenwei Tang, Qianfeng Wen, Seth Grief-Albert +4LLM Grounding CoT Distillation
Department of Computer Science, University of Toronto · Queen’s University · Coolwei AI Lab

Jul 17, 2026J/K move · Enter open · S save

Understanding Reasoning from Pretraining to Post-Training

Jingyan Shen, Ang Li, Salman Rahman, Yifan Sun+3

New York University · Columbia University · University of California, Los Angeles+2

Language Model Pretraining RL for Language Model Reasoning RL Post-Training

Open paper AskPDF arXiv

Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it. As a result, two basic questions remain open: (1) how do pretraining choices (model size, data) shape the returns to RL compute, and (2) what does RL actually do to the model? These questions are difficult to study in the standard LLM setting: pretraining corpora are vast and uncontrolled, making it hard to attribute behaviors to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive. To address these challenges, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline. We follow the standard LLM training pipeline by pretraining language models from 5M to 1B parameters on human chess games, supervised fine-tuning on synthetic reasoning traces, and running RL on chess puzzles with verifiable rewards. Using this framework, we find that the post-RL performance at given RL compute level is well-predicted from the pretraining loss, and slope of the RL reward curves improves approximately linearly with the pretraining tokens. Beyond scaling, we find that RL does not simply sharpen the SFT policy: on easy puzzles it amplifies correct moves the SFT policy already preferred, while on hard puzzles it surfaces correct moves that were nearly absent under SFT. We further test whether our findings transfer beyond chess by training a 1B language model on math-domain text, where the same predictive pattern emerges: longer-pretrained checkpoints reach higher post-RL performance and improve faster under RL. In sum, we provide a quantitative account of the pretraining-to-RL interface and a controlled testbed for studying the science of reasoning across the full pretraining-to-post-training pipeline.

Abstract

Language Model Post-Training

Explore similar work

Understanding Reasoning from Pretraining to Post-Training

The Weight of Silence: A Causal Case for Weights Over the Scratchpad in Latent Chess Reasoning

Grounded Chess Reasoning in Language Models via Master Distillation