Beyond Search-Imitation: Prior-Directed Exploration for Searchless Chess

Szymon Miłosz; Piotr Duch; Szymon Grabowski

cs.LGAug 27, 2026

Beyond Search-Imitation: Prior-Directed Exploration for Searchless Chess

Authors: Szymon Miłosz, Piotr Duch, Szymon Grabowski

Organizations: Institute of Applied Computer Science, Lodz University of Technology

Abstract

Searchless chess networks reach human master strength from a single forward pass by imitating a stronger teacher: the strongest, Leela Chess Zero's (Lc0) released Chessformer, distills the visit counts of an AlphaZero-style Monte Carlo Tree Search (MCTS). Imitating a search is a poor proxy for playing without one, so we fine-tune for single-pass strength with self-play reinforcement learning (RL). Its exploration is usually supplied by an entropy bonus, the reverse Kullback-Leibler (KL) divergence to uniform. We replace it with a forward, mass-covering KL toward the network's own MCTS prior (prior-directed exploration), so exploration covers the moves the prior judges promising, and pair it with an entropy-adaptive sampling temperature, set by the value head's outcome uncertainty, that sharpens once a position is decided. In about two thousand steps it raises puzzle accuracy from 93.9% to 94.9% on a 100,000-puzzle suite and mate-in-four accuracy from 77% to 81% while holding searchless strength at or slightly above the base. Measuring tactical accuracy and playing strength together across a matched-compute sweep, we find the two dissociate: accuracy gains fall in a one-point band while ratings straddle the base, and a control fine-tuned on puzzles alone posts the study's largest tactical gains while shedding roughly 260 Elo; a better puzzle-solver is not thereby a stronger player. Distribution-level measurements show what anchoring buys: without a regularizer self-play collapses onto a single line of play, and the puzzles newly solved are the near misses whose winning move the prior kept alive. The forward-KL prior tops the rating ladder, statistically tied with a reverse-KL anchor that concentrates twice as hard and drops the hardest solutions the mass-covering prior keeps in support.

Figures & tables

Figure 1: What self-play fine-tuning does to the base’s mate-finding, depth by depth: geometric-mean probability

p^{\star}

of sampling the full solution line from the model’s output distribution, on the exact mate-in-

N

suites, relative to the frozen base (dashed,

1\times

; ratios of the mate columns of Table 4 ; Section 3.3 ). Unregularized self-play sharpens shallow mates yet loses deep ones (

0.07\times

at depth five); the usual entropy bonus erodes every depth; anchoring exploration to the network’s own prior largely preserves them, and the mass-covering forward-KL prior (ours) is the only arm above the base at every depth.

Figure 2: The two asymmetries between the KL directions; probabilities are softmax outputs and hence strictly positive, so the limits describe behavior as a probability becomes arbitrarily small. Rows: the policy either drops a prior move (

\pi_{\theta}(a)\to 0

where

\pi_{\mathrm{base}}(a)

is large) or explores beyond the prior (mass where

\pi_{\mathrm{base}}(a)\to 0

). Columns: the forward KL of our penalty and the reverse KL of the standard anchor. The forbidden cells lie on the diagonal: forward KL is zero-avoiding, blowing up when the policy drops a prior move yet free where the prior is small; reverse KL is zero-forcing, the mirror image, permitting collapse onto a single prior mode.

Figure 3: Entropy-adaptive temperature, measured with the frozen base network: the value head’s WDL sets

\tau=\Ent(\mathbf{p})

, and self-play samples from

\mu\propto\pi_{\theta}^{1/\tau}

. Left (an eight-piece rook-and-knight endgame, White to move, in which Rh6# is the only mate among the

24

legal moves): near-one-hot WDL,

\tau=0.008

, yet the base still keeps

26.6\%

\text{Rg5}+

, a check that does not mate, so sampling

\pi_{\theta}

would miss the finish better than one time in four; tempering discards it and commits to the mate (

68.7\%\to 100\%

). Right (a balanced opening): draw-heavy WDL,

\tau=0.89

, and

\mu

barely sharpens the policy (

\mu\approx\pi_{\theta}

). Light bars are the base policy

\pi_{\theta}

and dark bars the tempered

\mu

(left: the mate, the strongest non-mating move, and the

22

others pooled; right: the five most probable moves plus the rest; per-panel scale, probabilities printed in %).

Figure 4: Prior-directed self-play fine-tuning as a TD(0) one-step backup. Three networks: the frozen prior

\pi_{\mathrm{base}}

(the exploration reference), the online

\pi_{\theta}

, and an EMA copy

\bar{\theta}

supplying slow targets. At state

s

the policy samples

a

from

\mu\propto\pi_{\theta}^{1/\tau(s)}

with

\tau(s)=\Ent(\mathbf{p}_{\bar{\theta}}(s))

and steps to

s^{\prime}

. Each update consumes this single transition—no rollouts, no returns: the online net evaluates

s

, the EMA bootstraps from

s^{\prime}

, and one gradient step minimizes the loss of Equation ( 7 ), after which the EMA tracks

\theta

. The dashed wire is the off-policy gap the truncated weight

\rho

corrects.

Exploration term	Self-play $\tau$	$\mathrm{Acc}$ (%)	$\mathrm{Acc}_{\geq\!2400}$ (%)	Elo	W–D–L	LOS
					vs. base
Frozen base ( Monroe et al., 2026 )	–	$93.85\pm 0.15$	$60.01\pm 0.68$	$2374\pm 37$	–	–
No regularizer	adaptive	$94.06\pm 0.15$	$60.50\pm 0.68$	$\Delta\,{+}9.8\pm 8.2$	326–351–323	$0.55$
	$\tau\!\equiv\!1$	$94.04\pm 0.15$	$60.80\pm 0.68$	$\Delta\,{-}0.5\pm 8.3$	380–295–325	$0.98$
Entropy bonus	adaptive	$94.90\pm 0.14$	$62.30\pm 0.67$	$\Delta\,{-}5.8\pm 8.3$	331–323–346	$0.28$
	$\tau\!\equiv\!1$	$94.00\pm 0.15$	$60.47\pm 0.68$	$\Delta\,{+}6.6\pm 8.2$	344–335–321	$0.81$

Table 1: Puzzle accuracy and searchless playing strength after fine-tuning from the same Chessformer base, evaluated single-pass (

\argmax

moves, EMA weights) at each arm’s selected

\beta^{\star}

(Table 6 ); higher is better. Acc. : the

100{,}000

-puzzle suite;

\mathrm{Acc}_{\geq\!2400}

: the

20{,}000

-puzzle

\geq\!2400

suite; both multi-solution scored, with

95\%

Wilson intervals. Accuracies are absolute; of the Elo column only the base row is, anchored to the Chessformer paper’s

2374\pm 37

( Monroe et al., 2026 ) , with every other row the difference

\Delta

from it—so comparisons rest on the per-row intervals, which the shared anchor leaves unchanged. The last row is the supervised puzzle-tuned control of Section 3.6 .

Optimizer	AdamW, $\beta_{1}=0.9$ , $\beta_{2}=0.98$
Weight decay	$0$
Schedule	200-step linear warmup, then constant
Learning rate (backbone / final block)	$3{\times}10^{-6}$ / $1{\times}10^{-5}$
Learning rate (policy head / value head)	$1{\times}10^{-5}$ / $3{\times}10^{-5}$
Total steps	$2{,}000$
Parallel games / batch size	$1{,}024$

Table 2: Training hyperparameters shared across all runs of Section 3 . The forward-KL weight

\beta

is the only quantity that varies across conditions (Section 3.3 ).

Figure 5: Forward-KL coefficient sweep of the headline adaptive-temperature arm, evaluated on the

100{,}000

-puzzle suite: puzzle error of the EMA policy under temperature sampling versus the forward-KL weight

\beta

(lower is better;

\log\beta

axis; error bars and shaded bands are

95\%

Wilson intervals). Dashed lines mark the frozen base and the unregularized

\beta=0

run; red marks

\beta=10^{-2}

, the coefficient selected on the

10{,}000

-puzzle suite. The four half-decade points are additional resolution for this arm only.

		Mate-in- $N$ accuracy (%)
Exploration term	Self-play $\tau$	$1$	$2$	$3$	$4$	$5$
Frozen base ( Monroe et al., 2026 )	–	$99.94\pm 0.04$	$98.82\pm 0.15$	$92.58\pm 0.36$	$77.16\pm 0.58$	$58.01\pm 1.45$
No regularizer	adaptive	$99.92\pm 0.04$	$98.77\pm 0.15$	$92.97\pm 0.35$	$79.23\pm 0.56$	$61.59\pm 1.43$
	$\tau\!\equiv\!1$	$99.80\pm 0.06$	$98.90\pm 0.14$	$93.68\pm 0.34$	$80.73\pm 0.55$	$63.81\pm 1.41$
Entropy bonus	adaptive	$99.95\pm 0.03$	$99.12\pm 0.13$	$94.13\pm 0.33$	$81.62\pm 0.54$	$64.28\pm 1.40$
	$\tau\!\equiv\!1$	$99.91\pm 0.04$	$98.76\pm 0.15$	$93.08\pm 0.35$	$79.15\pm 0.56$	$61.55\pm 1.43$

Table 3: Searchless accuracy by forced-mate depth:

20{,}000

Lichess positions per depth for

N\!\leq\!4

, from the suite of Lin et al. (2026) , and for

N\!=\!5

all

4{,}465

Lichess puzzles whose forced mate is exactly five moves (the Lichess mate-in-5 theme also contains deeper mates), with the network’s true move history; a puzzle counts as solved only if the

\argmax

at every position is a mate-preserving move (any of several winning moves is credited). Models are the selected self-play configurations of Table 1 . All values are absolute accuracies with

95\%

Wilson intervals; higher is better.

Figure 6: The difficulty gradient of the two prior-anchored models: solution-NLL reduction over the frozen base (base minus model, in nats—the log of the solution-probability ratio; above zero the model retains solutions better than the base). Left: by mate depth (the mate columns of Table 4 ). Right: by

200

-Elo puzzle rating bin on the

100{,}000

-puzzle suite (per-puzzle sequence NLL on the solution line); ratings below

600

and above

2600

are pooled into single bins (

2{,}496

–

14{,}167

puzzles per bin). The entropy model is not plotted, its losses lying far below both axes (Table 4 ).

Figure 7: Trajectory coverage

\Omega^{N=4}_{K}

at the initial position: cumulative trajectory mass covered by

K

depth-

4

continuations drawn without replacement from each model’s play distribution (mean over

100

sampling seeds; all exploration terms at coefficient

0.01

, the configurations of Table 4 ). Solid: the no-temperature configurations (the frozen base sampled raw); dashed: their adaptive-temperature counterparts (the base sampled through the tempered

\mu

). Each curve ends where its extraction reaches the

95\%

target (dotted); across the

100

seeds the standard deviation of the draws needed is at most

2.1\%

of the mean.

	Tactics suite		Mate-in- $N$
Model	All	$\geq\!2400$	$1$	$2$	$3$	$4$	$5$
Frozen base	$0.660$	$0.199$ ( $0.302$ )	$0.915$	$0.819$	$0.615$	$0.344$	$0.161$
No regularizer	$0.536$	$0.010$ ( $0.600$ )	$\mathbf{0.993}$	$0.896$	$0.492$	$0.107$	$0.011$
Entropy bonus	$0.222$	$0.103$ ( $0.381$ )	$0.399$	$0.187$	$0.102$	$0.037$	$0.011$
Reverse-KL anchor	$\mathbf{0.765}$	$0.178$ ( $0.569$ )	$0.978$	$\mathbf{0.930}$	$\mathbf{0.721}$	$0.370$	$0.126$
Forward-KL prior (ours)	$0.759$	$\mathbf{0.236}$ ( $0.526$ )	$0.954$	$0.894$	$0.712$	$\mathbf{0.411}$	$\mathbf{0.169}$

Table 4: Solution probability

p^{\star}

(geometric mean over puzzles; higher is better), defined in Section 3.1 ; mate suites use the network’s true move history. Models are the adaptive-temperature

\beta=10^{-2}

configurations of Table 1 (EMA weights); tactics columns as in Table 1 , mate columns as in Table 3 . Best value per column in bold; fine-tuned values are green above the frozen base, red below it (arithmetic means uncolored). For the

\geq\!2400

suite the arithmetic mean is given in parentheses: the gap between the two means is the collapse signature read in Section 3.3 . Figure 6 (left) plots the mate columns.

Figure 8: Which failed positions fine-tuning recovers, for the two prior-anchored models of Table 4 on the

100{,}000

-puzzle suite (scoring and totals in the text). Left: share of base-wrong positions recovered, binned by the probability the base retains on the solution at the position. Right: binned by the rank of the solution in the base’s output there; bin sizes annotated.

Configuration	Samp.	Elo	$\Ent(p)$	$D_{\mathrm{KL}}\!\left(\pi_{\mathrm{base}}\,\big\\|\,p\right)$	$D_{\mathrm{KL}}\!\left(p\,\big\\|\,\pi_{\mathrm{base}}\right)$	$\mathrm{cov}_{5}$
Frozen base	raw	$0$	$1.69$	$0.00$	$0.00$	$0.81$
	$\mu$	$361\pm 29$	$0.39$	$8.38$	$0.81$	$0.98$
No regularizer	$\mu$	$671\pm 31$	$0.06$	$11.04$	$1.03$	$0.99$
	raw	$660\pm 31$	$0.22$	$7.59$	$0.89$	$0.98$
Entropy bonus	$\mu$	$476\pm 30$	$0.74$	$2.38$	$0.59$	$0.92$
	raw	$357\pm 29$	$1.44$	$1.26$	$0.26$	$0.75$

Table 5: Sampled-play strength and single-state diagnostics for the ten configurations of Figure 10 in Appendix H (the frozen base raw and tempered, and each exploration term

\times

temperature arm; entropy is the

\beta\!=\!10^{-2}

model). Sampling “

\mu

” is the adaptive-temperature arm played through its tempered behavior policy, “raw” the

\tau\!\equiv\!1

arm at temperature one. Elo : sampled-play BayesElo from a closed round-robin (base anchored

0

;

200

games/pair,

45

pairs;

95\%

intervals). The remaining columns average, over the positions each model reaches, its played policy

p

: entropy

\Ent(p)

(nats), forward and reverse KL to the raw base, and prior coverage

\mathrm{cov}_{5}

(mass on the base’s top-five moves).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

		Coefficient $\beta$
Exploration term	Self-play $\tau$	$10^{-4}$	$10^{-3}$	$10^{-2}$	$10^{-1}$	$1$
Entropy bonus	adaptive	$94.42\pm 0.45$	$94.19\pm 0.46$	$\mathbf{94.70}\pm 0.44$	$94.07\pm 0.46$	$82.77\pm 0.74$
	$\tau\!\equiv\!1$	$\mathbf{93.92}\pm 0.47$	$91.24\pm 0.55$	$91.59\pm 0.54$	$90.53\pm 0.57$	$83.13\pm 0.73$
Reverse-KL anchor	adaptive	$94.30\pm 0.45$	$94.22\pm 0.46$	$\mathbf{94.57}\pm 0.44$	$94.47\pm 0.45$	$93.93\pm 0.47$
	$\tau\!\equiv\!1$	$94.11\pm 0.46$	$94.38\pm 0.45$	$\mathbf{94.63}\pm 0.44$	$94.30\pm 0.45$	$93.86\pm 0.47$
Forward-KL prior (ours)	adaptive	$94.37\pm 0.45$	$94.54\pm 0.45$	$\mathbf{94.67}\pm 0.44$	$94.52\pm 0.45$	$93.80\pm 0.47$

Appendix

Table 6: Coefficient selection on the

10{,}000

-puzzle suite (multi-solution scoring): overall accuracy (%) by exploration term, self-play temperature, and coefficient

\beta

, with

95\%

Wilson intervals. Bold marks each row’s maximum, the model carried into Table 1 . For reference, the frozen base scores

93.52\pm 0.48

and the unregularized runs

93.99\pm 0.47

(adaptive

\tau

) and

94.04\pm 0.46

(

\tau\!\equiv\!1

Figure 9: Six exclusive recoveries from the

100{,}000

-puzzle suite— top row: puzzles the forward prior newly solves and the reverse anchor does not; bottom row: the mirror. Each panel shows the first solver move (side to move at the bottom): green arrow, the winning move; red arrow, the move kept by the base and by the fine-tuned model that fails; bars, the three models’ probabilities of the labeled moves; under each move, the mover’s expected score after it—gray from the base’s own value head

1-v_{\mathrm{base}}(s^{\prime})

, black from a

1.5

-second Stockfish oracle; beneath each panel, the solution. Left to right: a queen sacrifice into a three-move mate the base’s value head already scores as certain (rating

1326

); an endgame where the base’s two favorite captures both lose and only the quiet 34.f5 wins (

2613

); a knight sacrifice no value head sees (

2178

); a rook sacrifice the recovering model commits to at

0.88

(

2036

); knight discoveries of which only the double check wins (

2245

); both rooks for the queen, the solution ranked fifth (

1096

Figure 10: Trajectory coverage

\Omega^{N=4}_{K}

beyond the initial position, under the protocol of Figure 7 : cumulative depth-

4

trajectory mass of the first

K

draws without replacement, mean over

100

sampling seeds; every curve ends at the

95\%

extraction target (dotted). Solid lines are the no-temperature configurations (the frozen base sampled raw), dashed their adaptive-temperature counterparts (the base sampled through the tempered

\mu

); the dot marks the unregularized configurations, whose first draw covers

95\%

. Each position is shown above its panel, full move history supplied (all White to move): left , the Ruy Lopez mainline (ply

8

27

legal moves,

\approx\!7.0\times 10^{5}

depth-

4

continuations); middle , the Najdorf Sicilian English Attack (ply

16

47

legal moves,

\approx\!2.6\times 10^{6}

); right , the Queen’s Gambit Declined Tartakower (ply

14

38

legal moves,

\approx\!1.5\times 10^{6}

Chess

Chess research develops and evaluates engines, human-like play models, strategic reasoning systems, and tools for analyzing boards and games.

See all 31 papers

Engineering Efficient Self-Play Chess: Search, Replay, and Throughput Under Limited Compute
Sep 27, 2026Bertil BraunSelf-Play RL Chess
Discovering High-Quality Chess Puzzles with Offline Reinforcement Learning
Aug 14, 2026Allen Nie, Anirudhan Badrinath, Nicholas Tomlin +5Chess Recommender Systems
Stanford University · UC Berkeley
Tracking vs. Deciding: The Dual-Capability Bottleneck in Searchless Chess Transformers
Mar 31, 2026Quanhao Li, Wei JiangInstance Reweighting Game-Playing Agents
Abbey Park High School · Shanghai Soong Ching Ling School

Sep 27, 2026J/K move · Enter open · S save

Engineering Efficient Self-Play Chess: Search, Replay, and Throughput Under Limited Compute

Bertil Braun

Self-Play RL Chess Reinforcement Learning Inference-Time Search Experience Replay

Open paper AskPDF arXiv

How strong can an AlphaZero-style chess system become under limited training compute when its entire learning loop is engineered for efficiency? We train from random initialization through searched self-play on a single eight-GPU node for 2.5 days. The resulting 6.32-million-parameter model reaches 3,251 benchmark Elo [3,206, 3,297] at 100,000 searches per move (estimated at under five seconds of thinking time) against a fixed-node Stockfish 13 ladder. The run ingests 3.25 million completed games, involves an estimated 100 billion search simulations, and makes 836.6 million training presentations. We investigate search allocation, replay and restart-state selection, policy representation, progressive model sizing, quantized inference, and throughput engineering. Alongside the retained design, we document plausible alternatives that failed to improve the complete learning loop or did not justify their cost. The reported strength is a result of the integrated system, not an isolated Elo gain attributable to any single choice.

Abstract

Chess

Explore similar work

Engineering Efficient Self-Play Chess: Search, Replay, and Throughput Under Limited Compute

Discovering High-Quality Chess Puzzles with Offline Reinforcement Learning

Tracking vs. Deciding: The Dual-Capability Bottleneck in Searchless Chess Transformers