Benchmarking Prompt Optimization of Large Language Models With Chess

Timothée Lesort; Alejandra López de Aberasturi Gómez; Tristan Karch; Tom Veniat; Philippe Modard; Karl Tuyls; Ludovic Denoyer

cs.AISep 30, 2026

Benchmarking Prompt Optimization of Large Language Models With Chess

Authors: Timothée Lesort, Alejandra López de Aberasturi Gómez, Tristan Karch, Tom Veniat, Philippe Modard, Karl Tuyls, Ludovic Denoyer

Organizations: imec AI.labs

Abstract

Evaluating large language models becomes increasingly challenging as their capabilities advance: benchmarks can saturate, public test sets risk contamination, and assessing harder tasks can require expensive grading or execution infrastructure. These challenges are amplified in automatic prompt optimization (APO), where evaluation is repeated throughout the search for better prompts. Studying APO therefore requires a benchmark that is cheap and deterministic to score, hard enough to leave room for improvement, and renewable as models evolve. We introduce a chess benchmark built from 1,118 Lichess puzzles to study APO for frozen LLMs: we optimize their prompts without updating their model weights. Chess combines inexpensive exact-match scoring, engine-based evaluation of alternative moves, and a renewable supply of problems with adjustable difficulty. Unlike evaluations that report only success on isolated test items, the benchmark also connects puzzle-solving gains to short game-play rollouts within the same domain. We use it to evaluate six APO algorithms on eight target models, measuring not only baseline strength but also how much each model responds to optimization and whether optimized prompts transfer across models and to game play. Chess is thus a well-suited benchmark for APO: it is (i) challenging, as even the strongest evaluated model, Gemini 3.5 Flash (used as the meta-model), solves only about 55% of puzzles; (ii) discriminative, revealing gains, unchanged performance, and regressions across methods and models; (iii) renewable, with fresh puzzles to reduce contamination risk and adjustable difficulty to maintain headroom as models improve; and (iv) affordable, as the complete study runs for around $800. We release the puzzles, optimization and evaluation code, and dataset-renewal scripts (https://github.com/imec-ailabs/Automatic-Prompt-Optimization-with-Chess).

Figures & tables

Panel of Figure 1 — Figure 1 : Chess puzzles as a prompt-optimization benchmark. (a) A position $x$ , its exact FEN (line-wrapped), an abbreviated sorted list of legal moves, and reference move $y$ . (b) The template assembles instructions, demonstrations (initially empty), and the current position fields; braces denote placeholders. APO optimizes the task instructions within the system message and/or demonstrations; the current board and game information are supplied in the user message. The exact template is given in Appendix B.4 . The warm-colored region and arrow highlight the update of $p$ , while $\theta$ stays frozen. Demonstrations include answers, whereas the current reference $y$ is used only for scoring. The optimizer uses training scores and, for reflective methods, textual feedback. The selected $p^{*}$ is evaluated on held-out puzzles using accuracy and engine-based regret.

	Optimizes		Feedback		Meta-model
Algorithm	Instructions	Demonstrations	Textual	Numerical	Used
GEPA	✓	✓	✓	✓	✓
COPRO	✓			✓	✓
BFS		✓		✓
BRS		✓		✓
MIPROv2	✓	✓		✓	✓

Table 1 : Summary of optimization algorithms. optimization scope (what prompt components are optimized), feedback types (how the algorithm guides the search), and whether a separate meta-model proposes prompt changes.

Model	Max output	Reasoning	Input/call	Output/call	$ / 1k calls	Latency/call (s)	Puzzle acc. (%)	Illegal (%)
Gemini 3.5 Flash	1,024	medium	$446\pm 0.00$	$944\pm 0.56$	9.163	$5.93\pm 0.13$	$55.40\pm 0.58$	$0.2\pm 0.1$
Gemini 3.5 Flash Lite	1,024	low	$447\pm 0.01$	$14\pm 0.03$	0.169	$0.67\pm 0.01$	$26.30\pm 0.62$	$4.9\pm 1.2$
Claude Haiku 4.5	1,024	default	$646\pm 0.00$	$12\pm 0.00$	0.707	$1.16\pm 0.02$	$11.63\pm 0.00$	$2.5\pm 0.0$
GPT-4o Mini	1,024	N/A	$427\pm 0.00$	$12\pm 0.03$	0.072	$1.10\pm 0.03$	$5.84\pm 0.37$	$1.1\pm 0.1$
Qwen3.8 27B	512	default	$452\pm 0.02$	$15\pm 0.01$	0.231	$2.68\pm 0.84$	$8.47\pm 0.37$	$0.6\pm 0.3$
GPT-5.6 Luna	4,096	low	$426\pm 0.00$	$1,529\pm 2.40$	1.920	$20.87\pm 1.05$	$31.72\pm 0.37$	$0.5\pm 0.1$

Table 2 : Baseline prompt evaluation on the 559-puzzle test set, reported as mean

\pm

sample standard deviation over three runs. Input and output are per solver-position call; cost is normalized per 1,000 solver-position calls. Reasoning: thinking level passed to the model; default : provider’s default; N/A : no reasoning mode. Jev is a decision endpoint, so generative max-output and output-token columns are not comparable.

Model	Baseline	BFS	BRS	COPRO	GEPA	MIPROv2	SIMBA
GPT-4o Mini	$5.84\pm 0.37$	$5.01\pm 0.31$	$4.95\pm 0.99$	$5.84\pm 0.55$	$5.66\pm 0.52$	$4.95\pm 0.81$	$\mathbf{5.90\pm 0.36}$
Jev 1.13 †	$7.27\pm 0.10$	$7.87\pm 0.47$	$8.77\pm 0.31^{*}$	$7.69\pm 0.18$	$\mathbf{9.30\pm 0.47}^{*}$	$8.59\pm 0.65$	N.A.
Qwen3.8 27B †	$8.47\pm 0.37$	$9.48\pm 0.36^{*}$	$9.78\pm 0.41^{*}$	$9.12\pm 0.31^{*}$	$\mathbf{10.44\pm 0.90}^{*}$	$8.23\pm 0.31$	$9.36\pm 0.45^{*}$
Claude Haiku 4.5	$11.63\pm 0.00$	$11.09\pm 0.18$	$11.21\pm 1.02$	N.U.	$\mathbf{12.28\pm 1.22}$	$10.61\pm 1.19$	$11.39\pm 0.63$
DeepSeek V4 Pro 0813 †	$14.49\pm 0.18$	$14.55\pm 0.63$	$14.85\pm 1.00$	$14.79\pm 0.27$	$\mathbf{16.28\pm 0.31}^{*}$	$14.49\pm 0.62$	$15.03\pm 0.82$
Gemini 3.5 Flash Lite †	$26.30\pm 0.62$	$32.38\pm 0.47^{*}$	$33.21\pm 1.15^{*}$	$29.10\pm 3.46$	$27.01\pm 0.18$	$32.98\pm 1.72^{*}$	$\mathbf{33.87\pm 1.03}^{*}$

Table 3 : Puzzle accuracy (%, N=559), reported as mean

\pm

sample standard deviation over three runs per condition. Bold marks the optimizer with the highest mean for each model. ∗ marks improvement over the model’s three-run baseline under an unadjusted one-sided Welch test (

p<0.05

); † marks models with at least one starred optimizer. N.U. marks conditions where none of the three optimizer seeds updated the original prompt. N.A. marks a non-applicable method. Gray cells indicate lower mean accuracy than baseline.

Panel of Figure 2 — Figure 2 : Optimizer performance on the chess puzzle benchmark. Left: mean evaluation cost versus mean puzzle accuracy across three seeds, from the baseline prompt (open circle) to each model’s best algorithm (star). Right: distributions of per-seed changes in puzzle accuracy from the matched baseline for each optimizer.

Panel of Figure 3 — Figure 3 : Puzzle solve rate before and after optimization vs Elo and theme. We show, for the 3 best models with significant improvement, the results for the best algorithm (cf Table 3 ). Open circles mark baseline performance and colored circles mark the optimized prompt; faded thinner segments indicate a decrease.

Figure 4 : Accuracy before and after the best-performing optimizer for each selected model, stratified by puzzle length and solver move index. Open circles mark baseline performance and colored circles mark the optimized prompt; faded thinner segments indicate a decrease.

Figure 5 : Mean composition of the three prompts produced by each model’s best algorithm. Wedge proportions represent each category’s share of prompt characters; pie area represents total prompt length in characters.

Figure 6 : Prompt transferability on 559 puzzles. Rows identify the source model and optimizer; columns identify the target model. Cells report mean puzzle-accuracy change in percentage points relative to the target’s three-run baseline, with $\pm$ one standard error of the difference, including variability in both groups. Cell stars mark positive gains with one-sided Welch-test $p<0.05$ . A target-model star indicates at least one starred cell in its column, including the native prompt. Gold outlines mark native prompts.

	Gemini 3.5 Flash	Gemini 3.5 Flash Lite		DeepSeek V4 Pro		Qwen3.8 27B
Starting position	Base	Base	Best	Base	Best	Base	Best
Initial, White	7.78	101.00	49.71	3,617.25	3,481.93	9,993.60	10,001.30
Initial, Black	1.90	13.50	14.64	52.81	61.37	7,717.38	10,698.79
Ruy Lopez	6.50	133.23	84.57	3,383.90	72.50	89.60	3,390.57
Open Sicilian	18.10	82.29	69.37	3,566.48	6,937.93	10,001.10	9,995.70
Queen’s Gambit	24.63	48.43	25.04	13,627.50	5,938.82	10,065.25	9,993.60

Table 4 : Stockfish depth-20 regret per accepted model move. For the overall mean, accepted moves are pooled across starting positions within each seed, then the three seed means are summarized as mean

\pm

sample SD. Thus seeds have equal weight, but longer rollouts contribute more within a seed. Median regret pools accepted moves across seeds. Regret values are in centipawns; lower is better. Illegal-attempt rates pool flagged-illegal attempts, including unparseable outputs, over all recorded model attempts across seeds; counts are shown below. Checkmate-loss rates count rollouts ending with the model checkmated over all rollouts. Timeouts are not counted as additional illegal attempts. Base and Best give rates before and after prompt optimization. Bold marks the lower value within each base/best model pair, without implying significance.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Position	Model color	Starting FEN
Initial, White	White	rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
Initial, Black	Black	rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
Ruy Lopez	White	r1bqkbnr/1ppp1ppp/p1n5/1B2p3/4P3/5N2/PPPP1PPP/RNBQK2R w KQkq - 0 4
Open Sicilian	White	rnbqkbnr/pp2pppp/3p4/8/3pP3/5N2/PPP2PPP/RNBQKB1R w KQkq - 0 4
Queen’s Gambit	White	rnbqkb1r/ppp2ppp/4pn2/3p4/2PP4/2N5/PP2PPPP/R1BQKBNR w KQkq - 2 4
King’s Indian	White	rnbqk2r/ppppppbp/5np1/8/2PP4/2N5/PP2PPPP/R1BQKBNR w KQkq - 2 4

Appendix

Table 5 : Starting positions used for the playing-chess transfer evaluation. “Model color” determines which side is controlled by the language model.

	Rating 400–1999								Rating 2000–2799
Moves	400	600	800	1000	1200	1400	1600	1800	2000	2200	2400	2600	Total
1	19/ 19	20/ 20	20/ 20	20/ 20	20/ 20	20/ 20	20/ 20	20/ 20	–	–	–	–	159/ 159
2	19/ 20	20/ 20	21/ 20	20/ 20	20/ 20	20/ 20	20/ 20	20/ 20	–	–	–	–	160/ 160
3	–	–	–	–	–	–	–	–	20/ 20	20/ 20	19/ 20	21/ 20	80/ 80
4	–	–	–	–	–	–	–	–	20/ 20	20/ 20	20/ 20	20/ 20	80/ 80
5	–	–	–	–	–	–	–	–	19/ 20	21/ 20	20/ 19	20/ 21	80/ 80

Appendix

Table 6 : Number of train/test puzzles by solver moves and Lichess rating bin. Each cell is reported as train/test.

Model		Mean regret (cp)		Median regret (cp)		Rank (regret / position / puzzle)
	Best method	Baseline	Best	Baseline	Best	Baseline	Best
Gemini 3.5 Flash Lite	SIMBA	49,122	39,163	993	549	2 / 2 / 2	1 / 1 / 1
GPT-4o Mini	SIMBA	70,954	69,330	100,257	100,170	6 / 6 / 6	6 / 7 / 6
Qwen3.8 27B	GEPA	68,244	67,472	100,120	100,040	5 / 5 / 5	4 / 5 / 4
DeepSeek V4 Pro 0813	GEPA	64,059	59,949	1,877	1,610	3 / 3 / 3	3 / 3 / 3
Muse Spark 1.2 Contributor	SIMBA	48,656	39,136	774	781	1 / 1 / 1	2 / 2 / 2

Appendix

Table 7 : Regret computed on long puzzles (3–5 moves) with Stockfish depth 20. Rank entries report median-regret rank / position-accuracy rank / puzzle-accuracy rank; lower ranks are better. Bottom row is the meta-model.

Model	BFS	BRS	COPRO	GEPA	MIPROv2	SIMBA
Gemini 3.5 Flash Lite	4,497	4,873	336	1,725	1,449	930
GPT-4o Mini	4,407	4,671	367	3,225	1,423	579
Jev 1.13	6,451	6,954	921	2,968	1,694	N.E.
Qwen3.8 27B	4,588	4,604	227	2,109	1,446	1,050
Claude Haiku 4.5	6,510	6,822	55	3,626	1,427	1,182
GPT-5.6 Luna	4,593	55	146	3,045	154	55

Appendix

Table 8 : Average complete compiled-prompt size in characters, over three seeds. The base prompt is 55 characters. Some algorithms never manage to improve the base prompt on some models (gray cells). Bold marks the optimizer with the highest puzzle accuracy for each model. N.E. marks not evaluated.

Figure 7 : Puzzle solve rate by Lichess rating with the baseline prompt. Solid curves show empirical model performance; shaded regions show

\pm 1

standard deviation across runs. Dashed curves show the standard Elo success-rate function for hypothetical players anchored at

50\%

success for puzzle ratings 955 and 1980. The vertical lines mark these anchor ratings.

Model	OpenRouter identifier
Gemini 3.5 Flash	openrouter/google/gemini-3.5-flash
Gemini 3.5 Flash Lite	openrouter/google/gemini-3.5-flash-lite
GPT-4o Mini	openrouter/openai/gpt-4o-mini
Jev 1.13	typesafe/jev-1.13-20260917
Qwen3.8 27B	openrouter/qwen/qwen3.8-27b
Claude Haiku 4.5	openrouter/anthropic/claude-haiku-4.5

Appendix

Table 9 : Exact OpenRouter model identifiers used in the puzzle experiments. Gemini 3.5 Flash is also the fixed reflection and meta-model.

Role	Model	Temp	Max tokens	Provider
Baseline task LM	(per-model)	0	Task-specific	Mixed
Optimization task LM	(per-model, Appendix 4.2 )	0	4096 / 2048 / 1024	OpenRouter
Meta-model	gemini-3.5-flash	1	8192	OpenRouter
Prompt LM ( COPRO , $n>1$ )	gemini-3.5-flash	1	8192	OpenRouter

Appendix

Table 10 : Language model configuration for compilation and evaluation.

Optimizer	Parameter	Value
GEPA	auto	heavy ( $\sim$ 3,436 metric calls)
	candidate_selection_strategy	pareto
	use_merge	True
	skip_perfect_score	True
	reflection_minibatch_size	3
	num_threads	16

Appendix

Table 11 : Optimizer hyperparameters. Unless noted otherwise, values are DSPy defaults.

Explore similar work

May 17, 2026cs.AI

Generalization or Memorization? Brittleness Testing for Chess-Trained Language Models

Recent work has fine-tuned language models on chess data and reported high benchmark scores as evidence that the resulting models can understand the rules of chess, play full chess games at a professional level, or generate human-readable explanations grounded in expert knowledge. We train KinGPT, a 25M-parameter character-level language model trained only on (position, best-move) pairs, who exceeds 3B-parameter ChessGPT on a 600-puzzle mate-in-N suite and 4B-parameter C1-4B over a 20-theme puzzle benchmark. We examine several claims made in existing literature regarding chess-trained language models and assert that their impressive benchmark performance is largely explained by pattern-matching. We also demonstrate how LLM-Modulo, a verifier-in-the-loop framework, raises RedPajama 3B's best move accuracy from 1.2% to 21.2% and move generation validity from 19.3% to 95.3% on mate-in-N chess puzzles, comparable to gains achieved from ChessGPT's fine-tuning on chess-specific web corpora at a fraction of the cost. Our results illustrate how pairing a general LLM with an external verifier offers a more flexible alternative to directly training on synthetic data for well-defined domains. We open source all training/evaluation code, datasets, puzzle samples, and KinGPT model checkpoints for reproducibility.

Ethan Tang

School of Computing and Augmented Intelligence · Arizona State University · Tempe, AZ 85281

Chess Model Checkpoints Memorization Fragility Synthetic Data

Ask PDF

Apr 30, 2026cs.AI

Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading

Current Large Language Model (LLM) evaluation frameworks utilize the same static prompt template across all models under evaluation. This differs from the common industry practice of using prompt optimization (PO) techniques to optimize the prompt for each model to maximize application performance. In this paper, we investigate the effect of PO towards LLM evaluations. Our results on public academic and internal industry benchmarks show that PO greatly affects the final ranking of models. This highlights the importance of practitioners performing PO per model when conducting evaluations to choose the best model for a given task.

Nicholas Sadjoli, Tim Siefken, Atin Ghosh +2

SAP · Stanford University

Large Language Model Evaluation Prompt Engineering Prompt Optimization

Ask PDF

Jun 9, 2026cs.CL

APEX: Automated Prompt Engineering eXpert with Dynamic Data Selection

Large Language Models are highly sensitive to prompt formulation, necessitating automatic prompt optimization to unlock their full potential. While evolutionary algorithms have emerged as the dominant paradigm, they suffer from a critical bottleneck: data efficiency. Current methods treat the development dataset as a static benchmark, wasting significant compute budget on uninformative data. In this work, we introduce APEX (Automatic Prompt Engineering eXpert), a novel framework that optimizes the data usage alongside the prompt search. APEX dynamically stratifies the dataset into Easy, Hard, and Mixed tiers based on the optimization lineage. By prioritizing the Mixed tier, which identifies the data where the LLM has mixed performance, we identify two high-leverage subsets: the addressable frontier for generating informative mutations and the rank-sensitive frontier for distinguishing candidate quality. We evaluate APEX across three diverse benchmarks: IFBench, SimpleQA Verified, and FACTS Grounding. Under a fixed budget of 5,000 evaluation calls, due to its data efficiency, APEX outperforms the initial prompt by an average of 11.2% on Gemini 2.5 Flash and 6.8% on Gemma 3 27B, demonstrating that a data-centric approach is key to efficient and effective prompt optimization.

Fei Wang, Si Si, Cho-Jui Hsieh +1

Google · UCLA

Automatic Prompt Optimization Prompt Engineering Lattice Mutation

Ask PDF