Discovering High-Quality Chess Puzzles with Offline Reinforcement Learning

Allen Nie; Anirudhan Badrinath; Nicholas Tomlin; Timothy Dai; Carissa Yip; Rose E Wang; Emma Brunskill; Chris Piech

cs.AIAug 14, 2026

Discovering High-Quality Chess Puzzles with Offline Reinforcement Learning

Authors: Allen Nie, Anirudhan Badrinath, Nicholas Tomlin, Timothy Dai, Carissa Yip, Rose E Wang, Emma Brunskill, Chris Piech

Organizations: Stanford University · UC Berkeley

Abstract

Learning and skill mastery require extensive and deliberate practice. In many learning settings, producing high-quality pedagogical materials can require a high level of domain expertise and be very time-consuming. Pedagogical materials often need to train students to engage in different thinking patterns. In some domains, such as chess, puzzles are used to help students practice their skills in calculating the next moves and recognizing known patterns on a board. Giving students a practice set of puzzles to help them learn different modes of thinking is challenging because the teacher needs to carefully balance between different motifs and how many look-ahead steps a student needs to perform. Popular online platforms like Chess.com and Lichess offer players millions of puzzles. Unlike chess tactics puzzles procured by human experts, where chess beginners can learn valuable insights, these puzzles are automatically generated and often regarded as having low pedagogical value. These platforms also rely on a heuristic to recommend puzzles to users for practice. Using the user history data over an entire year, a total of 1.5 billion puzzle-solving histories, we learn the pedagogical value of a puzzle and how to automatically choose a set of puzzles to better support chess learners using insights from offline reinforcement learning. We show that using offline policy evaluation, our trained policy has significant impact on beginners with puzzle-solving Elo range of 100--1000, particularly for the group of beginners whose learning growth was stagnant. We also performed a qualitative analysis of the puzzles discovered by our model by collecting annotation ratings from expert chess players. The success of our pipeline shows promise for a future where we can understand the pedagogical values of practice items given general user interaction data.

Figures & tables

Figure 2: Architecture and training procedure of the chess puzzle recommendation policy, leveraging user features and puzzle features to optimize a policy-based loss

L_{\pi}

and a value function via

L_{V}

Figure 3: Reward function

r(s,a)

for varying levels of correctness and Elo differentials between puzzle and user, with reward equivalences shown between balances of correctness and relative difficulty.

Category	Range
Calculation	1–4
Pattern	1–3
Informativeness	1–5
Appropriateness	1–5
Quality	1–5
Fun	1–5

Figure 4: Qualitative rating categories designed in collaboration with chess expert consultants.

Figure 5: Distribution of trained policy’s top recommended puzzle versus the puzzle Elo of the chosen puzzle (left) and the user Elo (right).

Elo	Stagnant Group		Growth Group		Average
	$\pi_{b}$	$\pi_{\theta}$	$\pi_{b}$	$\pi_{\theta}$	$\pi_{b}$	$\pi_{\theta}$
100–600	$14.3\,\text{\scriptsize$ \pm $}\,\text{\scriptsize 1.5}$	$\mathbf{52.9}\,\text{\scriptsize$ \pm $}\,\text{\scriptsize{3.0}}$	${17.7}\,\text{\scriptsize$ \pm $}\,\text{\scriptsize{0.4}}$	$\mathbf{58.8}\,\text{\scriptsize$ \pm $}\,\text{\scriptsize{1.7}}$	$16.9\,\text{\scriptsize$ \pm $}\,\text{\scriptsize 0.5}$	$\mathbf{57.4}\,\text{\scriptsize$ \pm $}\,\text{\scriptsize{1.5}}$
600–1000	$14.1\,\text{\scriptsize$ \pm $}\,\text{\scriptsize 1.2}$	$\mathbf{27.0}\,\text{\scriptsize$ \pm $}\,\text{\scriptsize{6.7}}$	${20.9}\,\text{\scriptsize$ \pm $}\,\text{\scriptsize{0.5}}$	$\mathbf{42.3}\,\text{\scriptsize$ \pm $}\,\text{\scriptsize{3.2}}$	$19.5\,\text{\scriptsize$ \pm $}\,\text{\scriptsize 0.4}$	$\mathbf{39.2}\,\text{\scriptsize$ \pm $}\,\text{\scriptsize{2.9}}$
1000–1500	$12.3\,\text{\scriptsize$ \pm $}\,\text{\scriptsize 0.4}$	$12.1\,\text{\scriptsize$ \pm $}\,\text{\scriptsize 1.2}$	$18.3\,\text{\scriptsize$ \pm $}\,\text{\scriptsize 0.6}$	$19.7\,\text{\scriptsize$ \pm $}\,\text{\scriptsize 2.4}$	$15.3\,\text{\scriptsize$ \pm $}\,\text{\scriptsize 0.4}$	$16.0\,\text{\scriptsize$ \pm $}\,\text{\scriptsize 1.4}$
1500+	$12.2\,\text{\scriptsize$ \pm $}\,\text{\scriptsize 0.2}$	$12.2\,\text{\scriptsize$ \pm $}\,\text{\scriptsize 1.2}$	${15.9}\,\text{\scriptsize$ \pm $}\,\text{\scriptsize{0.3}}$	$16.1\,\text{\scriptsize$ \pm $}\,\text{\scriptsize 1.1}$	$13.5\,\text{\scriptsize$ \pm $}\,\text{\scriptsize 0.2}$	$13.6\,\text{\scriptsize$ \pm $}\,\text{\scriptsize 0.9}$

Table 1: Returns obtained by trained policy

\pi_{\theta}

and the Chess.com policy

\pi_{b}

through one-step importance sampling. We show 95% confidence intervals with

\pm

and boldfaced numbers are statistically significant with Student’s t-test.

Method	Calculation	Pattern Recognition	Fun	Rating	Quality	Informativeness
$\pi_{b}$ (Chess.com)	70.17	58.89	65.57	67.95	69.83	75.11
$\pi_{\theta}$ (Ours)	72.39	61.48	70.34	72.73	71.31	75.11
$\Delta$	+2.22	+2.59	+4.77 ∗∗	+4.78 ∗	+1.48	0.00

Table 2: Comparison of methods for qualitative expert-designed metrics. Metrics are converted to 100 from the original range to account for the difference in scale. Values are averaged across all annotated puzzles. We perform Student’s t-test and use ∗ to mark

p<0.05

, and ∗∗ for

p<0.01

Name	USCF Profile	Title
Alexander Su	12857329	Expert
Robbie Selwyn	17281090	N/A
Iris Zhou	14467261	Expert
Tony Kukavica	13853062	National Master
Justin Paul	14323420	FIDE Master
Matt Larson	14278511	International Master

Table 3: Volunteer chess annotators who completed puzzle annotations, with their USCF profile (linked via their USCF member ID) and title.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Example puzzle positions used to illustrate the calculation (Figure 6(a) ) and informativeness (Figure 6(b) ) criteria.

Field	Example 1	Example 2
Puzzle	#1004456	#955974
Player Elo (Puzzle Elo)	951 (1317)	1361 (1615)
Solved correctly	Yes	Yes
Primary emphasis	Pattern Recognition	Both
Calculation	N/A	1 – easy to guess first move w/o seeing solution
Pattern recognition	1 – banal/boring pattern	1 – banal/boring pattern

Appendix

Table 4: Two example rows from a completed annotation spreadsheet, showing how one annotator answered the questionnaire in Appendix A.3 for two puzzles.

Field	Example 1 (Ours)	Example 2 (Chess.com)
Puzzle	#760014	#1072600
Player Elo (Puzzle Elo)	2175 (3053)	951 (1317)
Solved correctly	Yes	Yes
Primary emphasis	Both	Calculation
Calculation	4 – requires full understanding of solution to solve	2
Pattern recognition	3 – interesting and useful	1 – banal/boring pattern

Appendix

Table 5: Two example rows from a completed annotation spreadsheet, selected to contrast a puzzle with a unanimously high pattern recognition score (Example 1) against a puzzle with a lower, more disputed pattern recognition score (Example 2), as discussed in Appendix A.4 .

Annotator	Pattern Recognition		Quality
	Ours	Chess.com	Ours	Chess.com
A	3	2	3	4
B	N/A	2	5	4
C	N/A	N/A	4	5
D	3	1	5	3
E	3	3	3	4

Appendix

Table 6: Pattern recognition and quality scores given by each of the eight annotators to the two puzzles in Table 5 (“Ours” is #760014 and “Chess.com” is #1072600; N/A denotes an annotator for whom the pattern recognition criterion did not apply), with rows aligned so that the same letter denotes the same annotator across all four columns. While the two puzzles’ quality scores are similar on average, their pattern recognition scores diverge, with #1072600 receiving both the single lowest score (a “banal/boring pattern” rating) and a wider spread of opinions overall.

Chess

Chess research develops and evaluates engines, human-like play models, strategic reasoning systems, and tools for analyzing boards and games.

See all 31 papers

Benchmarking Prompt Optimization of Large Language Models With Chess
Sep 30, 2026Timothée Lesort, Alejandra López de Aberasturi Gómez, Tristan Karch +4LLM Evaluation Automatic Prompt Optimization
imec AI.labs
Conditional Generation of Creative Chess Puzzles with Diffusion Models
Sep 29, 2026Aatu Selkee, Severi Rissanen, Xidong Feng +2Masked Diffusion Models Computational Creativity
Aalto University · Google DeepMind
Chess_db: A framework for working with large chess game datasets
Jul 23, 2026Nicos Angelopoulos, Jan WielemakerLogic Programming Chess
University College & Imperial College, London UK · University College & Imperial College London, UK · SWI-Prolog solutions +1

Sep 30, 2026J/K move · Enter open · S save

Benchmarking Prompt Optimization of Large Language Models With Chess

Timothée Lesort, Alejandra López de Aberasturi Gómez, Tristan Karch, Tom Veniat+3

imec AI.labs

LLM Evaluation Automatic Prompt Optimization

Open paper AskPDF arXiv

Evaluating large language models becomes increasingly challenging as their capabilities advance: benchmarks can saturate, public test sets risk contamination, and assessing harder tasks can require expensive grading or execution infrastructure. These challenges are amplified in automatic prompt optimization (APO), where evaluation is repeated throughout the search for better prompts. Studying APO therefore requires a benchmark that is cheap and deterministic to score, hard enough to leave room for improvement, and renewable as models evolve. We introduce a chess benchmark built from 1,118 Lichess puzzles to study APO for frozen LLMs: we optimize their prompts without updating their model weights. Chess combines inexpensive exact-match scoring, engine-based evaluation of alternative moves, and a renewable supply of problems with adjustable difficulty. Unlike evaluations that report only success on isolated test items, the benchmark also connects puzzle-solving gains to short game-play rollouts within the same domain. We use it to evaluate six APO algorithms on eight target models, measuring not only baseline strength but also how much each model responds to optimization and whether optimized prompts transfer across models and to game play. Chess is thus a well-suited benchmark for APO: it is (i) challenging, as even the strongest evaluated model, Gemini 3.5 Flash (used as the meta-model), solves only about 55% of puzzles; (ii) discriminative, revealing gains, unchanged performance, and regressions across methods and models; (iii) renewable, with fresh puzzles to reduce contamination risk and adjustable difficulty to maintain headroom as models improve; and (iv) affordable, as the complete study runs for around $800. We release the puzzles, optimization and evaluation code, and dataset-renewal scripts (https://github.com/imec-ailabs/Automatic-Prompt-Optimization-with-Chess).

Abstract

Chess

Explore similar work

Benchmarking Prompt Optimization of Large Language Models With Chess

Conditional Generation of Creative Chess Puzzles with Diffusion Models

Chess_db: A framework for working with large chess game datasets