cs.AIDec 17, 2025

Beyond Accuracy: A Geometric Stability Analysis of Large Language Models in Chess Evaluation

Authors: Qingya Hu, Ruifeng Cao, Weiqi Wang, Xidan Song

Organizations: Department of Computer Science, Wuhan Donghu University, Jinying St, Wuhan, 430068, China. · Department of Computer Science, University of Manchester, Oxford Road, Manchester, M13 9PL, Uniked Kingdom.

Abstract

Large language models (LLMs) are usually evaluated in chess by how closely their position evaluations agree with a strong engine, but such agreement does not show that a model evaluates equivalent positions consistently. We apply metamorphic relations to chess evaluation: a 180-degree rotation of pawnless positions, left-right mirroring, color swapping, position pairs one move apart, conversion between Forsyth-Edwards Notation and a natural-language description, and illegal positions. Six LLMs (Claude Sonnet 4.5, DeepSeek Chat, Gemini 2.5 Flash, GPT-5.1, Grok 4.1 Fast and Kimi K2 Turbo) evaluated seven test sets of 500 items each, with Stockfish 16.1 as the reference. All six models failed on rotation: at most 10.6% of the 500 rotated pairs received evaluations within 300 centipawns (cp) of each other, and the models often declared the legal rotated positions illegal. On the other four transformations, stability ranked the models as accuracy did except for the first two places: GPT-5.1 had the lowest mean stability error (186.17 cp) and Claude Sonnet 4.5 the lowest accuracy error (330.41 cp). Rejection rates for illegal positions ranged from 44.2% to 96.0%. Stability should be reported with the share of consistent pairs, because means over answered pairs concealed most rotation failures.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Aug 4, 2026

Hallucinations on the Board: Tool-Augmented Evaluation of LLM Chess Commentary

Superhuman game engines in domains like chess have made expert-level evaluations easily accessible, yet they communicate what is true without the natural-language explanations that make such expertise educationally useful to experts and non-experts alike. Large language models could, in principle, bridge this gap, but they frequently hallucinate due to limited domain-specific knowledge, and standard reference-based or LLM-as-a-judge frameworks cannot reliably detect these errors. In this work, we present ACT-Eval, an evaluation framework that decomposes chess commentary into atomic claims and routes them to engine-supported tools and expert-annotated gold references to assess factual correctness, conceptual coverage, and move-quality judgment. We release a benchmark of 325 position--move pairs spanning pedagogical, tournament, and critical positions, including 125 positions with expert-verified gold atoms and a five-class error taxonomy. Evaluating leading proprietary and open-weight models, we find that factual hallucinations remain pervasive in chess commentary: GPT-5.4 without tools produces incorrect sub-claims 22.0% of the time, while smaller open-weight models exceed 40%. Although tool augmentation substantially improves factual correctness and move-quality assessment, coverage of expert strategic and tactical ideas remains limited across all models. Human calibration shows that ACT-Eval's factual judgments fall within the observed range of inter-human agreement, while its coverage scores correlate strongly with human assessments of strategic completeness.
May 17, 2026

Generalization or Memorization? Brittleness Testing for Chess-Trained Language Models

Recent work has fine-tuned language models on chess data and reported high benchmark scores as evidence that the resulting models can understand the rules of chess, play full chess games at a professional level, or generate human-readable explanations grounded in expert knowledge. We train KinGPT, a 25M-parameter character-level language model trained only on (position, best-move) pairs, who exceeds 3B-parameter ChessGPT on a 600-puzzle mate-in-N suite and 4B-parameter C1-4B over a 20-theme puzzle benchmark. We examine several claims made in existing literature regarding chess-trained language models and assert that their impressive benchmark performance is largely explained by pattern-matching. We also demonstrate how LLM-Modulo, a verifier-in-the-loop framework, raises RedPajama 3B's best move accuracy from 1.2% to 21.2% and move generation validity from 19.3% to 95.3% on mate-in-N chess puzzles, comparable to gains achieved from ChessGPT's fine-tuning on chess-specific web corpora at a fraction of the cost. Our results illustrate how pairing a general LLM with an external verifier offers a more flexible alternative to directly training on synthetic data for well-defined domains. We open source all training/evaluation code, datasets, puzzle samples, and KinGPT model checkpoints for reproducibility.
Jan 23, 2026

Disentangling generalization and memorization in large language models using chess

Large Language Models (LLMs) exhibit remarkable capabilities, yet it remains unclear to what extent these reflect sophisticated recall or genuine reasoning ability. We introduce chess as a controlled testbed aimed at disentangling these faculties. Leveraging the game's structure and scalable engine evaluations, we construct a taxonomy of positions varying in density of relevant priors -- ranging from common states solvable by memorization to completely novel ones requiring generalization. Crucially, our approach achieves this distinction without requiring explicit knowledge of the models' training data. Applying this taxonomy, we combine a longitudinal analysis of the GPT lineage with a rigorous evaluation of contemporary models, including Claude Opus and Gemini. Our analysis reveals a steep gradient: performance consistently degrades as the density of relevant priors decreases. Notably, for tasks with few relevant priors, base model performance regresses to the random-play baseline, and all models propose illegal moves at sharply increasing rates, indicating that they fail to reliably apply the fixed rules of the game in unfamiliar positions. While newer models improve, progress slows significantly for tasks with sparse priors. Furthermore, while reasoning-augmented inference improves performance, its relative marginal benefit per token decreases in the absence of relevant priors. These results suggest limitations in systematic generalization, highlighting the need for mechanisms beyond scale to achieve robust performance when deprived of relevant priors.