Beyond Accuracy: A Geometric Stability Analysis of Large Language Models in Chess Evaluation
Authors: Qingya Hu, Ruifeng Cao, Weiqi Wang, Xidan Song
Organizations: Department of Computer Science, Wuhan Donghu University, Jinying St, Wuhan, 430068, China. · Department of Computer Science, University of Manchester, Oxford Road, Manchester, M13 9PL, Uniked Kingdom.
Large language models (LLMs) are usually evaluated in chess by how closely their position evaluations agree with a strong engine, but such agreement does not show that a model evaluates equivalent positions consistently. We apply metamorphic relations to chess evaluation: a 180-degree rotation of pawnless positions, left-right mirroring, color swapping, position pairs one move apart, conversion between Forsyth-Edwards Notation and a natural-language description, and illegal positions. Six LLMs (Claude Sonnet 4.5, DeepSeek Chat, Gemini 2.5 Flash, GPT-5.1, Grok 4.1 Fast and Kimi K2 Turbo) evaluated seven test sets of 500 items each, with Stockfish 16.1 as the reference. All six models failed on rotation: at most 10.6% of the 500 rotated pairs received evaluations within 300 centipawns (cp) of each other, and the models often declared the legal rotated positions illegal. On the other four transformations, stability ranked the models as accuracy did except for the first two places: GPT-5.1 had the lowest mean stability error (186.17 cp) and Claude Sonnet 4.5 the lowest accuracy error (330.41 cp). Rejection rates for illegal positions ranged from 44.2% to 96.0%. Stability should be reported with the share of consistent pairs, because means over answered pairs concealed most rotation failures.
Figures & tables
Model
Color Swap
Rotation
Mirror
Similarity
Format
Avg.
Avg. w/o Rot.
Mean gap (cp)
GPT-5.1
217.49
2531.28
181.08
297.99
48.11
655.19
186.17
Claude Sonnet 4.5
198.78
626.56
195.85
318.25
62.41
280.37
193.82
Kimi K2 Turbo
367.67
726.21
94.94
329.38
36.22
310.88
207.05
DeepSeek Chat
254.46
996.88
235.18
399.58
121.48
401.52
252.68
Gemini 2.5 Flash
776.80
854.60
747.48
559.34
260.56
639.75
586.04
Table 1: Stability errors: mean gap (cp) between the evaluations of paired positions for each transformation, and the number of pairs that enter each mean; lower errors are better. A pair enters only if both evaluations are numerical and at most 10,000 cp in magnitude (Section 3.8.5 ). The rotation means rest on very different numbers of pairs and are not comparable across models (Section 4.1.1 ). Avg. is the unweighted mean of the five columns, and Avg. w/o Rot. that of the four columns other than rotation. Bold marks the lowest error in each column except Rotation and Avg.
Figure 1: Outcome of the 500 pairs of each transformation for each model. Consistent: both evaluations numerical and at most 10,000 cp in magnitude, with a gap of at most 300 cp; inconsistent: the same with a gap above 300 cp; mate-level: at least one score above 10,000 cp in magnitude; unanswered: at least one evaluation null or a failed call. The number at the right of each bar is the consistent share. Table 5 lists the counts.
Model
Color Swap
Rotation
Mirror
Similarity
Format
Mean
Rank
Claude Sonnet 4.5
340.40
452.88
337.24
407.60
113.92
330.41
1
GPT-5.1
339.55
619.85
359.28
380.04
112.16
362.17
2
Kimi K2 Turbo
361.29
685.99
396.14
397.08
129.98
394.09
3
DeepSeek Chat
383.55
729.18
448.52
464.29
188.58
442.82
4
Gemini 2.5 Flash
624.47
969.84
652.78
529.87
200.31
595.45
5
Grok 4.1 Fast
609.25
1340.41
558.75
636.35
532.94
735.54
6
Table 2: Accuracy errors: mean absolute difference (cp) between each model’s evaluation and that of Stockfish 16.1 at depth 10, for the original positions of each set (FEN input) and for both positions of each similarity pair; lower is better. Positions with an evaluation above 10,000 cp in magnitude are excluded, which leaves from 286 (Gemini 2.5 Flash, Rotation) to 499 (GPT-5.1, Format) positions per cell, and from 319 to 485 pairs for Similarity. Mean is the unweighted mean of the five columns.
Figure 2: Accuracy error against Stockfish 16.1 (Table 2 ) and mean stability error over the color swap, mirror, similarity and format transformations (Table 1 , rotation excluded) for the six models; lower is better on both axes. Each label gives the model and its coordinates (accuracy error, stability error) in cp.
Model
Rejected
Numerical Evaluation
Failed Call
Rejection Rate (%)
Gemini 2.5 Flash
480
20
0
96.0
GPT-5.1
450
50
0
90.0
Claude Sonnet 4.5
396
104
0
79.2
Kimi K2 Turbo
370
89
41
74.0
Grok 4.1 Fast
278
222
0
55.6
DeepSeek Chat
221
279
0
44.2
Table 3: Responses to the 500 illegal positions per model. A rejection is a null evaluation, as the prompt requires; a numerical evaluation counts as a failure even if the error field names a violation (one answer of Claude Sonnet 4.5). Failed calls returned no parsable answer after four attempts. The rejection rate is the percentage of all 500 positions; higher is better.
Inconsistent Pairs
Mate-Level Scores
Forced-Move Set, Other Answers
Model
w/o Rotation (%)
Geometric Sets
Forced-Move Set
Accuracy Error (cp)
Error > 300 cp
Claude Sonnet 4.5
176/1351 (13.0)
422/2695
162/495
406.09
164/333
DeepSeek Chat
204/1436 (14.2)
334/2728
113/494
913.03
318/381
Kimi K2 Turbo
207/1253 (16.5)
91/2360
53/449
564.90
271/396
GPT-5.1
251/1459 (17.2)
10/2822
2/497
515.27
268/495
Gemini 2.5 Flash
390/986 (39.6)
578/2376
297/437
1096.06
92/140
Table 4: Error counts by type. Inconsistent pairs: pairs that enter Table 1 for the color swap, the mirror and format conversion and whose gap exceeds 300 cp, out of all such pairs. Mate-level scores: numerical answers above 10,000 cp in magnitude, out of all numerical answers, for the color-swap, rotation and mirror sets (both positions of each pair) and for the forced-move set; none of these positions has a mate score at the reference depth. The last two columns refer to the forced-move answers that are not mate-level scores: their accuracy error against Stockfish at depth 12, and the number of them that differ from Stockfish by more than 300 cp.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Color-swap pair used to illustrate a consistency failure (row 124 of the color-swap set): ( a ) the original position, Black to move, Stockfish −447 cp; ( b ) the color-swapped position, White to move, Stockfish +425 cp. Claude Sonnet 4.5 returned −450 cp for both.
Figure 4: Forced-move position (row 187 of the forced-move set). Black is in check from the White queen on c8 (solid red frames on c8 and g8). After the only legal reply, 1…Kg7, Black threatens mate on g2 and h2 (dashed green frames) with the queen on g3 and the rook on b2. Stockfish: −637 cp at depth 12 and mate in eight for Black at depth 18; Claude Sonnet 4.5: 0 cp.
Figure 5: Illegal position used to illustrate a numerical evaluation of an illegal position: a White pawn on g1 (solid red frame). DeepSeek Chat: 10,000 cp without an error message; Gemini 2.5 Flash: rejected.
Unanswered
Model
Consistent
Inconsistent
Mate-Level
One
Both
Consistent (%)
Color swap
GPT-5.1
355
127
0
14
4
71.0
Claude Sonnet 4.5
377
82
24
17
0
75.4
Gemini 2.5 Flash
161
137
67
119
16
32.2
DeepSeek Chat
387
84
10
14
5
77.4
Appendix
Table 5: Outcome of the 500 pairs of each transformation (Section 3.8.5 ). Consistent and inconsistent pairs together are the pairs that enter Table 1 . Unanswered pairs are split into those with one and with both evaluations null or failed.
Superhuman game engines in domains like chess have made expert-level evaluations easily accessible, yet they communicate what is true without the natural-language explanations that make such expertise educationally useful to experts and non-experts alike. Large language models could, in principle, bridge this gap, but they frequently hallucinate due to limited domain-specific knowledge, and standard reference-based or LLM-as-a-judge frameworks cannot reliably detect these errors. In this work, we present ACT-Eval, an evaluation framework that decomposes chess commentary into atomic claims and routes them to engine-supported tools and expert-annotated gold references to assess factual correctness, conceptual coverage, and move-quality judgment. We release a benchmark of 325 position--move pairs spanning pedagogical, tournament, and critical positions, including 125 positions with expert-verified gold atoms and a five-class error taxonomy. Evaluating leading proprietary and open-weight models, we find that factual hallucinations remain pervasive in chess commentary: GPT-5.4 without tools produces incorrect sub-claims 22.0% of the time, while smaller open-weight models exceed 40%. Although tool augmentation substantially improves factual correctness and move-quality assessment, coverage of expert strategic and tactical ideas remains limited across all models. Human calibration shows that ACT-Eval's factual judgments fall within the observed range of inter-human agreement, while its coverage scores correlate strongly with human assessments of strategic completeness.
S. Ashwin Hebbar, Peiyao Sheng, Sewoong Oh +1
Princeton University · Sentient Labs · University of Washington
Recent work has fine-tuned language models on chess data and reported high benchmark scores as evidence that the resulting models can understand the rules of chess, play full chess games at a professional level, or generate human-readable explanations grounded in expert knowledge. We train KinGPT, a 25M-parameter character-level language model trained only on (position, best-move) pairs, who exceeds 3B-parameter ChessGPT on a 600-puzzle mate-in-N suite and 4B-parameter C1-4B over a 20-theme puzzle benchmark. We examine several claims made in existing literature regarding chess-trained language models and assert that their impressive benchmark performance is largely explained by pattern-matching. We also demonstrate how LLM-Modulo, a verifier-in-the-loop framework, raises RedPajama 3B's best move accuracy from 1.2% to 21.2% and move generation validity from 19.3% to 95.3% on mate-in-N chess puzzles, comparable to gains achieved from ChessGPT's fine-tuning on chess-specific web corpora at a fraction of the cost. Our results illustrate how pairing a general LLM with an external verifier offers a more flexible alternative to directly training on synthetic data for well-defined domains. We open source all training/evaluation code, datasets, puzzle samples, and KinGPT model checkpoints for reproducibility.
Ethan Tang
School of Computing and Augmented Intelligence · Arizona State University · Tempe, AZ 85281
Large Language Models (LLMs) exhibit remarkable capabilities, yet it remains unclear to what extent these reflect sophisticated recall or genuine reasoning ability. We introduce chess as a controlled testbed aimed at disentangling these faculties. Leveraging the game's structure and scalable engine evaluations, we construct a taxonomy of positions varying in density of relevant priors -- ranging from common states solvable by memorization to completely novel ones requiring generalization. Crucially, our approach achieves this distinction without requiring explicit knowledge of the models' training data. Applying this taxonomy, we combine a longitudinal analysis of the GPT lineage with a rigorous evaluation of contemporary models, including Claude Opus and Gemini. Our analysis reveals a steep gradient: performance consistently degrades as the density of relevant priors decreases. Notably, for tasks with few relevant priors, base model performance regresses to the random-play baseline, and all models propose illegal moves at sharply increasing rates, indicating that they fail to reliably apply the fixed rules of the game in unfamiliar positions. While newer models improve, progress slows significantly for tasks with sparse priors. Furthermore, while reasoning-augmented inference improves performance, its relative marginal benefit per token decreases in the absence of relevant priors. These results suggest limitations in systematic generalization, highlighting the need for mechanisms beyond scale to achieve robust performance when deprived of relevant priors.
Leonard S. Pleiss, Maximilian Schiffer, Robert K. von Weizsaecker