RetroChessBench: A Benchmark for Reconstructing and Proving Chess Histories

Tidor-Vlad Pricope

cs.LGOct 29, 2026

RetroChessBench: A Benchmark for Reconstructing and Proving Chess Histories

Authors: Tidor-Vlad Pricope

Organizations: Poindexter Labs

Abstract

Frontier language models such as GPT 6 Astra have contributed to research mathematics, demonstrating substantial reasoning capabilities. Yet serious reasoning failures persist even in tasks governed by explicit rules. We demonstrate this in retrograde chess which is a playground for pure reasoning, where solvers reconstruct legal histories and distinguish possible events from forced ones. We introduce RetroChessBench, a 12-task benchmark covering exact-length constructions, unique endings, and exhaustive answer sets, with references, rubrics, and a terminal-reward environment. In two trials per task and condition, GPT 6 Astra High fails to produce the required final answer in 22/24 attempts without tools and 18/24 with code execution; mean terminal RL rewards are 14.75% and 27.19%, respectively. The examples identify faulty deductions: failed responses reject legal histories and treat temporary obstructions as permanent. Online retrograde archives often provide puzzles and final move sequences without detailed derivations. We hypothesize that limited exposure to these missing explanations in web-derived training data contributes to the observed reasoning failures. This questions how far frontier mathematical reasoning really generalizes beyond familiar solution patterns. Claims of general intelligence require evidence that models can transfer learnt principles to areas with few worked examples.

Figures & tables

P0292, code run 1	Observable argument and counterexample
Invalid inference	A pawn on g6 prevents Be8-h5, so the required construction needs at least 47 plies.
Legal counterexample	The 45-ply reference contains 6.e8=B+ Ke6 7.Bh5 g6 8.g5 gxh5: the bishop traverses g6 before it is occupied.
Missing obligation	A lower bound must cover alternative event orders. A valid 47-ply witness establishes no such bound.

Table 1: A failure of historical exclusion. The claim is paraphrased from the saved final answer; the counterexample is independently replayed from the initial position. This example establishes an error in the argument, not its internal computational cause.

Evaluation setting	Requested object	Relevant distinction
Forward chess	Legal or strong next moves; completed games	Continuation quality does not establish a hidden history.
LongCoT reverse chess	A history satisfying terminal and sequence constraints	A construction establishes existence.
PlanBench	Plans and reasoning about state changes	Explicit action models permit separate planning and checking tasks.
ART / ProofWriter	Plausible missing causes / deductions and abductive premises	Plausibility, derivability, and necessity are different objectives.
Formal deduction / satisfiability ( PrOntoQA ; SATBench )	Formal derivations; satisfiability and unsatisfiability decisions	Proof evaluation and excluding all satisfying assignments are established objectives.
RetroChessBench	Exact-length games, unique suffixes, complete ending sets	Distinct answer contracts apply these obligations to hidden histories, with replayable witnesses and inspectable failures.

Table 2: Relationship to prior evaluation settings. This comparison identifies the task objective; it does not rank benchmarks or claim that historical reconstruction is new. The cited implementations differ in models, budgets, and task distributions.

Primary outcome	No code	Code enabled
Correct final answers, all tasks	2/24	6/24
Valid exact-length constructive games	0/14	2/14
Correct unique-suffix answers	2/8	4/8
Full-credit unique-suffix proofs	2/8	1/8
Complete two-alternative answers	0/2	0/2

Table 3: Correct answers and completed proof obligations. Rows overlap: the three task families partition the all-task answer row, and full-credit suffix proofs are a subset of correct suffix answers. Analytical completeness follows the archived rubric assessments.

Analytical proof status	No code	Code enabled
Complete under archived review	2/10	1/10
Incomplete; no false step identified	3/10	4/10
Contains an identified false inference	5/10	5/10
Unresolved	0/10	0/10

Table 4: Mutually exclusive categories for the 20 suffix/set responses, derived from archived criterion-level reviews. “No false step identified” describes the review record, not a certification of every assertion. An answer with both omissions and an identified false inference is classified in the latter category. The per-answer evidence is in analysis/reviewer_audit.json .

Solver condition	Attempts	Final-answer failures	Mean reward	Full reward
No code	24	22/24	14.75%	2/24
Code enabled	24	18/24	27.19%	3/24

Table 5: Secondary reward summary for RetroChessBench. Full reward means the constructive or analytical obligation is satisfied under the frozen scoring policy. Scores depend on selection, task mix, rubric weights, and the construction gate.

Task family	Tasks	Attempts per mode	No-code reward	Code reward
Constructive game	7	14	0.00%	14.29%
Unique suffix	4	8	41.38%	54.00%
Two last-move alternatives	1	2	11.50%	10.25%

Table 6: Mean terminal reward by task family. Construction rewards are binary, whereas the other families allow analytical partial credit. The percentages therefore express different task-specific objectives and should not be read as a calibrated common difficulty scale.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Composer	Issue	Type	Plies
R0171	Nikolai I. Beluhov & Andrey Frolkin	52	Set	1
P0292	Silvio Baier	53	Game	45
P0297v	Nicolas Dupont	55	Game	60
P0332	Silvio Baier	58	Game	44
P0333	Gianni Donati & Olli Heimo	58	Game	44
P0335	Unto Heinonen	58	Game	54

Appendix

Table 7: Frozen RetroChessBench inventory. Issue numbers refer to StrateGems . “Game” requests an exact-length constructive witness; “Suffix” requests a unique ending; “Set” requests both possible last moves. R01 adapts R0185 and R02 adapts R0176.

Stage	Count	Retained evidence / decision
Primary source entries	82	Seven issues; variants and parts recorded separately in the coverage audit.
Eligible task instances	58	Two no-code trials each: 77 failures and 39 accepted final answers across 116 trials.
At least one no-code failure	47	30 tasks failed twice; 17 failed once. These observed outcomes screened the candidate pool.
Candidate pool after exclusions / additions	48	Remove P0299’s unresolved proof; retain 46 instances and add R01/R02.
RetroChessBench	12	Final IDs in Table 7 ; adaptive choice using observed difficulty and proof coverage. Two retained attempts per condition.

Appendix

Table 8: Selection ledger. Counts of source entries and eligible task instances use different units because parts and variants are distinct tasks. The archive preserves stage snapshots and attempt hashes; it does not reconstruct a preregistered rule or every intermediate editorial choice.

Setting	No code	Code enabled
Requested model	gpt-6-astra	gpt-6-astra
Reasoning effort	high	high
Tools	None	Hosted code interpreter, auto container, 4 GB
Web-search tool	Absent	Absent
Input	Prompt, FEN, board image	Identical prompt, FEN, board image
Reference / rubric access	None	None

Appendix

Table 9: Archived request settings and observed usage. Token counts include provider-reported reasoning tokens and are not identical to visible answer length. No claim of equal computation is made.

Item	Weight	Necessary result or proof obligation
C1	25	Correct six-ply ending, including both bishop victims and the check/mate annotations.
C2	15	White’s seven-capture exhaustion; noncapturing surviving a-, b-, d-pawns; Black’s five-capture budget, including two c-pawn captures and at least one b-pawn capture.
C3	10	Force a5-a6 and the latest black king’s origin on b8, excluding the competing origins with temporally valid arguments.
C4	10	Restore the captured bishop on b7 and force its preceding Bc8-b7 move, using the opened bishop diagonal and queen line.
C5	10	Force Kb7-b8+ and the preceding c7-c8=B+ promotion, including the bishop’s unavailable ordinary approaches.
C6	5	Exhaust Black’s capture budget to exclude an additional capture on b8.

Appendix

Table 10: R01 rubric, 100 total points. C2 requires sufficient capture bounds; it does not demand the stronger, unnecessary claim that Black’s b-pawn promoted if an equivalent bound is proved.

Item	Weight	Necessary result or proof obligation
C1	22	Correct five-ply ending, including b1 bishop promotion and the knight captured on f3.
C2	8	Force the rook’s final arrival from f2 and the capture on f3 from the local check geometry.
C3	16	Derive the capture budgets, pawn origins, and required promotion constraints.
C4	14	Force Black’s immediately preceding b2-b1=B and identify the final captured piece as a knight.
C5	10	Establish the temporary lock and the limit imposed by alternating turns and the b-pawn’s available retractions.
C6	6	Exclude an earlier release through restoring and retracting Black’s captured a-pawn.

Appendix

Table 11: R02 rubric, 100 total points. The first two final moves depend on the later historical release argument; identifying only the promotion/capture pair is insufficient.

Moves 1–12			Moves 13–23
Move	White	Black	Move	White	Black
1	f4	c5	13	Ra4	b5
2	f5	c4	14	e4	bxa4
3	f6	c3	15	Bb5	f2+
4	fxe7	f5	16	Ke2	f1=B+
5	g4	Kf7	17	Kf2	Bc4

Appendix

Table 12: A valid 45-ply construction for P0292. The crucial ordering is 6.e8=B+ Ke6 7.Bh5 g6 8.g5 gxh5. The bishop traverses g6 before Black occupies it. The table is generated from the independently replayed reference moves.

Constructive final-response outcome	No code	Code enabled
No complete game supplied	9	7
Legal target-reaching game, wrong length	3	5
Game contains an illegal move	2	0
Legal exact-length target-reaching game	0	2
Total	14	14

Appendix

Table 13: Mutually exclusive outcomes of the 28 constructive attempts. Classification follows the frozen original-answer replay, with an illegal move taking precedence over a simultaneous length error. A missing witness can accompany useful reasoning or an incorrect impossibility argument; it is not necessarily an empty answer.

Reference family	Existence evidence in bundle	Analytical review / remaining evidence
Seven constructive tasks	Seven complete games replayed from the standard start.	No uniqueness claim is required.
R0171, two alternatives	Complete 167- and 189-ply histories, one for each victim, independently replayed.	The author reviewed the exclusion of other alternatives on a physical chessboard.
R01, R02, R0176, R0178	Local ending checks and conditional analytical references; no complete start-to-diagram witness supplied.	The author reviewed the conditional uniqueness arguments on a physical chessboard; complete reachability witnesses remain to be supplied.

Appendix

Table 14: Existence and uniqueness are separate validation obligations. Missing full witnesses do not demonstrate that a diagram is unreachable; they identify evidence still required from the benchmark author. R02 and R0176 share their diagram.

Candidate explanation	Intervention and control	Discriminating outcome / safeguard
Missing rule knowledge	Probe check escape, promotions, capture budgets, and event order on unrelated positions; compare with a rules primer.	Local failure identifies prerequisite gaps. Do not reveal the target history.
Answer discovery	Compare free solving with correct-suffix assistance and a separate incorrect-candidate arm.	Proof failure despite a supplied answer extends beyond discovery. Keep assisted scores separate.
Proof verification	Compare valid proofs with minimal corruptions in blinded, randomized trials; request error location and repair.	Measure false acceptance and rejection; a binary label alone does not show the critical step was checked.
State tracking	Render boards and capture ledgers from the solver’s own candidates; use an equal-budget text control.	Improvement tests representation. Supplying true intermediate states leaks the answer.
Insufficient search	Add locally legal predecessor enumeration to the same legality checker.	More witnesses with residual exclusion errors separates search from proof. Global reachability filtering is oracle assistance.
Explanatory coverage / transfer	Compare no adaptation, answer-only examples, and worked proofs on the same training tasks; test disjoint families and new mechanism combinations.	Match token and compute budgets; establish source competence. Separate answer recovery from proof completion. Gains do not reveal original training exposure.

Appendix

Table 16

Language Model Reasoning Evaluation

See all 103 papers

Generalization or Memorization? Brittleness Testing for Chess-Trained Language Models
May 17, 2026Ethan TangLLM Evaluation Language Modeling
School of Computing and Augmented Intelligence · Arizona State University · Tempe, AZ 85281
Disentangling generalization and memorization in large language models using chess
Jan 23, 2026Leonard S. Pleiss, Maximilian Schiffer, Robert K. von WeizsaeckerLanguage Model Generalization Memorization in Language Models
Technical University of Munich
Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoning
Mar 2, 2026Justin WaughLLM Evaluation LLM Reasoning
Approximate Labs

May 17, 2026J/K move · Enter open · S save

Generalization or Memorization? Brittleness Testing for Chess-Trained Language Models

Ethan Tang

School of Computing and Augmented Intelligence · Arizona State University · Tempe, AZ 85281

LLM Evaluation Language Modeling Chess Language Model Robustness

Open paper AskPDF arXiv

Recent work has fine-tuned language models on chess data and reported high benchmark scores as evidence that the resulting models can understand the rules of chess, play full chess games at a professional level, or generate human-readable explanations grounded in expert knowledge. We train KinGPT, a 25M-parameter character-level language model trained only on (position, best-move) pairs, who exceeds 3B-parameter ChessGPT on a 600-puzzle mate-in-N suite and 4B-parameter C1-4B over a 20-theme puzzle benchmark. We examine several claims made in existing literature regarding chess-trained language models and assert that their impressive benchmark performance is largely explained by pattern-matching. We also demonstrate how LLM-Modulo, a verifier-in-the-loop framework, raises RedPajama 3B's best move accuracy from 1.2% to 21.2% and move generation validity from 19.3% to 95.3% on mate-in-N chess puzzles, comparable to gains achieved from ChessGPT's fine-tuning on chess-specific web corpora at a fraction of the cost. Our results illustrate how pairing a general LLM with an external verifier offers a more flexible alternative to directly training on synthetic data for well-defined domains. We open source all training/evaluation code, datasets, puzzle samples, and KinGPT model checkpoints for reproducibility.

Abstract

Language Model Reasoning Evaluation

Explore similar work

Generalization or Memorization? Brittleness Testing for Chess-Trained Language Models

Disentangling generalization and memorization in large language models using chess

Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoning