cs.AIDec 17, 2025

Beyond Accuracy: A Geometric Stability Analysis of Large Language Models in Chess Evaluation

Authors: Qingya Hu, Ruifeng Cao, Weiqi Wang, Xidan Song

Organizations: Department of Computer Science, Wuhan Donghu University, Jinying St, Wuhan, 430068, China. · Department of Computer Science, University of Manchester, Oxford Road, Manchester, M13 9PL, Uniked Kingdom.

Abstract

Large language models (LLMs) are usually evaluated in chess by how closely their position evaluations agree with a strong engine, but such agreement does not show that a model evaluates equivalent positions consistently. We apply metamorphic relations to chess evaluation: a 180-degree rotation of pawnless positions, left-right mirroring, color swapping, position pairs one move apart, conversion between Forsyth-Edwards Notation and a natural-language description, and illegal positions. Six LLMs (Claude Sonnet 4.5, DeepSeek Chat, Gemini 2.5 Flash, GPT-5.1, Grok 4.1 Fast and Kimi K2 Turbo) evaluated seven test sets of 500 items each, with Stockfish 16.1 as the reference. All six models failed on rotation: at most 10.6% of the 500 rotated pairs received evaluations within 300 centipawns (cp) of each other, and the models often declared the legal rotated positions illegal. On the other four transformations, stability ranked the models as accuracy did except for the first two places: GPT-5.1 had the lowest mean stability error (186.17 cp) and Claude Sonnet 4.5 the lowest accuracy error (330.41 cp). Rejection rates for illegal positions ranged from 44.2% to 96.0%. Stability should be reported with the share of consistent pairs, because means over answered pairs concealed most rotation failures.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Hallucinations on the Board: Tool-Augmented Evaluation of LLM Chess Commentary

    Aug 4, 2026S. Ashwin Hebbar, Peiyao Sheng, Sewoong Oh +1LLM EvaluationTool-Use Evaluation

  2. Generalization or Memorization? Brittleness Testing for Chess-Trained Language Models

    May 17, 2026Ethan TangLLM EvaluationLanguage Modeling

  3. Disentangling generalization and memorization in large language models using chess

    Jan 23, 2026Leonard S. Pleiss, Maximilian Schiffer, Robert K. von WeizsaeckerLanguage Model GeneralizationMemorization in Language Models