Large Language Models (LLMs) exhibit remarkable capabilities, yet it remains unclear to what extent these reflect sophisticated recall or genuine reasoning ability. We introduce chess as a controlled testbed aimed at disentangling these faculties. Leveraging the game's structure and scalable engine evaluations, we construct a taxonomy of positions varying in density of relevant priors -- ranging from common states solvable by memorization to completely novel ones requiring generalization. Crucially, our approach achieves this distinction without requiring explicit knowledge of the models' training data. Applying this taxonomy, we combine a longitudinal analysis of the GPT lineage with a rigorous evaluation of contemporary models, including Claude Opus and Gemini. Our analysis reveals a steep gradient: performance consistently degrades as the density of relevant priors decreases. Notably, for tasks with few relevant priors, base model performance regresses to the random-play baseline, and all models propose illegal moves at sharply increasing rates, indicating that they fail to reliably apply the fixed rules of the game in unfamiliar positions. While newer models improve, progress slows significantly for tasks with sparse priors. Furthermore, while reasoning-augmented inference improves performance, its relative marginal benefit per token decreases in the absence of relevant priors. These results suggest limitations in systematic generalization, highlighting the need for mechanisms beyond scale to achieve robust performance when deprived of relevant priors.
Figures & tables
Figure 1: Example positions. Left to right: within-distribution, near-distribution, out-of-distribution. For a detailed discussion of our findings in these positions, we refer to Appendix B .
Model Name
API Tag
Release
GPT-3.5
gpt-3.5-turbo
Mar 2023
GPT-4o
gpt-4o
May 2024
GPT-5
gpt-5
Aug 2025
Gemini 3
gemini-3-flash-preview
Dec 2025
Claude Opus
claude-opus-4-6
Feb 2026
Table 1: Overview of model generations, API tags, and release dates
Figure 2: (a) Left: Average Centipawn Loss with Matescore=Illegal Move Score=1,000 . Right: Proportion of legal moves. (b) Left: Random-normalized Average Centipawn Loss under exclusion of illegal moves. Right: Observed rates of improvement across three generations. Common parameters: Horizontal lines indicate means, colored boxes around the mean display the 95% confidence interval, and vertical lines indicate standard deviations. WD = within-distribution, ND = near-distribution, OOD = out-of-distribution.
Figure 3: (a) Left: Average Centipawn Loss with minimal and moderate reasoning effort. Right: Proportion of legal moves with minimal and moderate reasoning effort. (b) Left: Reasoning tokens used by GPT-5 per position with minimal and moderate reasoning effort. Right: Percentage point improvement per token by condition. Common parameters: Means as horizontal black lines, 95% confidence interval as colored boxes, whiskers indicate standard deviations. WD = within-distribution, ND = near-distribution, OOD = out-of-distribution.
Figure 4: Performance across model lineages, all at minimal reasoning effort. Left: Average Centipawn Loss with Matescore = Illegal Move Score = 1,000. Right: Proportion of legal moves. Horizontal lines indicate means, boxes around the mean display the 95% confidence interval, vertical lines indicate standard deviations. WD = within-distribution, ND = near-distribution, OOD = out-of-distribution.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Lichess Masters database coverage by move number.
Figure 6: Example positions by condition. WD = within-distribution, ND = near-distribution, OOD = out-of-distribution.
Figure 7: Centipawn loss distributions by model and condition. WD = within-distribution, ND = near-distribution, OOD = out-of-distribution.
Condition
Legal moves
White material
Black material
WD
26.27±4.42
38.98±0.13
38.99±0.11
ND
31.21±7.68
37.76±2.20
37.84±2.30
OOD
19.75±18.85
24.36±4.69
24.99±4.74
Appendix
Table 2: Split diagnostics (mean ± SD over 500 positions per condition). Material in pawn units (P = 1, N = B = 3, R = 5, Q = 9; kings excluded). Legal moves are counted for the side to move.
Model
Condition
Legal (%)
ACPL (incl. illegal)
ACPL (excl. illegal)
GPT-3.5
WD
37.2
648.17
54.22
ND
17.6
893.35
394.05
OOD
4.2
989.46
749.05
GPT-4o
WD
85.6
174.52
35.65
ND
72.0
514.16
325.22
OOD
32.4
849.37
535.08
Appendix
Table 3: Results under UCI output notation. ACPL including illegal moves assigns a loss of 1,000 to illegal moves; ACPL excluding illegal moves is computed over legal moves only.
This paper examines the reasoning capabilities of Large Language Models (LLMs) from a novel perspective, focusing on their ability to operate within formally specified, rule-governed environments. We evaluate four LLMs (Gemini 2.5 Pro and Flash variants, Llama 3.3 70B and GPT-OSS 120B) on a suite of forward-simulation tasks-including next / multistep state formulation, and legal action generation-across a diverse set of reasoning problems illustrated through General Game Playing (GGP) game instances. Beyond reporting instance-level performance, we characterize games based on 40 structural features and analyze correlations between these features and LLM performance. Furthermore, we investigate the effects of various game obfuscations to assess the role of linguistic semantics in game definitions and the impact of potential prior exposure of LLMs to specific games during training. The main results indicate that three of the evaluated models generally perform well across most experimental settings, with performance degradation observed as the evaluation horizon increases (i.e., with a higher number of game steps). Detailed case-based analysis of the LLM performance provides novel insights into common reasoning errors in the considered logic-based problem formulation, including hallucinated rules, redundant state facts, or syntactic errors. Overall, the paper reports clear progress in formal reasoning capabilities of contemporary models.
Maciej Świechowski, Adam Żychowski, Jacek Mańdziuk
Grail Team Warsaw, Poland · Warsaw University of Technology Warsaw, Poland · AGH University of Krakow Cracow, Poland
Recent work has fine-tuned language models on chess data and reported high benchmark scores as evidence that the resulting models can understand the rules of chess, play full chess games at a professional level, or generate human-readable explanations grounded in expert knowledge. We train KinGPT, a 25M-parameter character-level language model trained only on (position, best-move) pairs, who exceeds 3B-parameter ChessGPT on a 600-puzzle mate-in-N suite and 4B-parameter C1-4B over a 20-theme puzzle benchmark. We examine several claims made in existing literature regarding chess-trained language models and assert that their impressive benchmark performance is largely explained by pattern-matching. We also demonstrate how LLM-Modulo, a verifier-in-the-loop framework, raises RedPajama 3B's best move accuracy from 1.2% to 21.2% and move generation validity from 19.3% to 95.3% on mate-in-N chess puzzles, comparable to gains achieved from ChessGPT's fine-tuning on chess-specific web corpora at a fraction of the cost. Our results illustrate how pairing a general LLM with an external verifier offers a more flexible alternative to directly training on synthetic data for well-defined domains. We open source all training/evaluation code, datasets, puzzle samples, and KinGPT model checkpoints for reproducibility.
Ethan Tang
School of Computing and Augmented Intelligence · Arizona State University · Tempe, AZ 85281
We study how reasoning evolves in a language model -- from supervised fine-tuning (SFT) to reinforcement learning (RL) -- by analyzing how a set of theoretically-inspired datasets influences language model performance in chess. We find that fine-tuning a model to directly predict the best move leads to effective RL and the strongest downstream performance -- however, the RL stage elicits unfaithful reasoning (reasoning inconsistent with the chosen move). Alternatively, training on multi-move trajectories yields comparable downstream performance with faithful reasoning and more stable RL. We analyze multiple qualitative and quantitative measures and highlight how these evolve from SFT through RL; we find several SFT-checkpoint metrics -- spanning evaluation performance, hallucination rates, and reasoning quality -- to be predictive of post-RL model performance. Finally, we ground our results with an experiment measuring chess information density in our custom datasets. We release models as well as training data, evaluations, and code that allowed us to surpass leading open-source reasoning models in chess with a 7B-parameter model. Code, models, and data are available at https://github.com/lucasdino/lang-chess.
Lucas Dionisopoulos, Nicklas Majamaki, Prithviraj Ammanabrolu
Department of Computer Science and Engineering, University of California San Diego, San Diego, United States.