PureTD: Reinforcement Learning for Backgammon Money Games with No Evaluation-time Search

Alexander L. Strehl

cs.LGAug 15, 2026

PureTD: Reinforcement Learning for Backgammon Money Games with No Evaluation-time Search

Authors: Alexander L. Strehl

Organizations: Independent Researcher

Abstract

We revisit Tesauro's TD-Gammon for backgammon money games in the setting of no evaluation-time search. Both checker play and cube action (use of the doubling cube) are learned from scratch via self-play reinforcement learning (RL), with minimal hand-coded logic and no expert features. In this setting, we demonstrate that pure self-play RL suffices to train models that reach near-state-of-the-art playing strength. Specifically, for cubeful money games, our search-free model evaluates faster and is substantially stronger than the open-source engines GNU Backgammon and Open Sage running a one-move (1-ply) look-ahead search. Our method of training a cubeful model also outperforms the standard approach of training a cubeless model and deriving cubeful equity with Janowski-style formulas, when the two models are equally sized.

Figures & tables

cube_centered	cube_own	cube_opp_own	is_cube_action	Decision for the player on roll
1	0	0	0	checker decision
1	0	0	1	cube offer (initial cube)
0	1	0	0	checker decision
0	1	0	1	cube offer (recube)
0	0	1	0	checker decision
0	0	1	1	not a legal configuration

Table 1: Cube-specific features.

Panel of Figure 1 — Figure 1: Example 1 as a sequence of state transitions. Dashed arrows mark the TD update that backs the realized successor value up to $V(s)$ .

Engine	vs. PureTD 0-ply (mEq/game)
gnubg , 0-ply	$-45.8$ $[-46.7,-45.0]$
gnubg , 1-ply	$-12.1$ $[-12.9,-11.2]$
Open Sage, 0-ply	$-36.8$ $[-37.7,-35.9]$
Open Sage, 1-ply	$-5.9$ $[-6.8,-5.1]$

Table 2: Cubeless-money strength. Our prob5 model (four hidden layers,

[512{,}512{,}256{,}256]

, 562k parameters) versus the baselines. Each row gives that engine’s head-to-head equity against our prob5 model at 0-ply, over

10^{7}

games (negative indicates a loss to our model). Brackets give 95% confidence intervals.

Engine	XG++ PR	vs. PureTD 0-ply (mEq/game)
PureTD, 0-ply	$0.80$ $[0.76,0.84]$	—
gnubg , 0-ply	$2.21$ $[2.12,2.30]$	$-85.2$ $[-87.0,-83.5]$
gnubg , 1-ply	$1.25$ $[1.20,1.32]$	$-27.2$ $[-29.0,-25.3]$
Open Sage, 0-ply	$2.11$ $[2.02,2.20]$	$-79.5$ $[-83.8,-75.2]$
Open Sage, 1-ply	$1.24$ $[1.18,1.29]$	$-23.8$ $[-28.2,-19.5]$

Table 3: Cubeful-money strength. Our production model (four hidden layers,

[512{,}512{,}256{,}256]

, 563k parameters) versus the baselines. The XG++ PR column uses

1{,}000

self-play games per row. The last column is each engine’s head-to-head equity against our model at 0-ply (negative indicates a loss to our model) over

10^{7}

games (

2\times 10^{6}

for the Open Sage rows).

Engine	Depth	Equity vs. PureTD (mEq/game)	dec/s (Zen 5)	dec/s (Zen 3)
PureTD	0-ply	—	$4{,}950$	$4{,}690$
gnubg	0-ply	$-45.8$	$41{,}300$ ( $0.12\times$ )	$32{,}800$ ( $0.14\times$ )
Open Sage	0-ply	$-36.8$	$19{,}500$ ( $0.25\times$ )	$16{,}900$ ( $0.28\times$ )
gnubg	1-ply	$-12.1$	$1{,}390$ ( $3.5\times$ )	$1{,}030$ ( $4.6\times$ )
Open Sage	1-ply	$-5.9$	$1{,}060$ ( $4.7\times$ )	$793$ ( $5.9\times$ )

Table 4: Cubeless-money cost versus strength. Strength is each baseline’s head-to-head equity (mEq/game) against PureTD 0-ply. Cost is single-thread checker-decision throughput (decisions/s) under the fixed-position benchmark. After each dec/s we provide the per-decision cost factor relative to PureTD (

2\times

means half as fast as PureTD).

			Checker dec/s		Cube dec/s
Engine	Depth	Equity vs. PureTD (mEq/game)	Zen 5	Zen 3	Zen 5	Zen 3
PureTD	0-ply	—	$5{,}600$	$5{,}200$	$23{,}800$	$13{,}400$
gnubg	0-ply	$-85.2$	$40{,}300$ ( $0.14\times$ )	$32{,}600$ ( $0.16\times$ )	$736{,}000$ ( $0.03\times$ )	$606{,}000$ ( $0.02\times$ )
Open Sage	0-ply	$-79.5$	$19{,}700$ ( $0.28\times$ )	$16{,}700$ ( $0.31\times$ )	$339{,}000$ ( $0.07\times$ )	$278{,}000$ ( $0.05\times$ )
gnubg	1-ply	$-27.2$	$1{,}480$ ( $3.8\times$ )	$1{,}110$ ( $4.7\times$ )	$4{,}260$ ( $5.6\times$ )	$3{,}240$ ( $4.1\times$ )
Open Sage	1-ply	$-23.8$	$51.1$ ( $109\times$ )	$40.1$ ( $130\times$ )	$1{,}840$ ( $13.0\times$ )	$1{,}430$ ( $9.4\times$ )

Table 5: Cubeful-money cost versus strength. Strength is each baseline’s head-to-head equity (mEq/game) against our PureTD 0-ply model. Cost is single-thread throughput (decisions/s) under the fixed-position benchmark, for checker and cube decisions separately. The gnubg 2-ply head-to-head uses

2\times 10^{6}

games with result

+20.7

[+16.6,+24.8]

. Each dec/s entry is followed by the per-decision cost factor relative to PureTD, as in Table 4 .

Engine	0-ply	1-ply	2-ply
PureTD	$0.80$ $[0.76,0.84]$	$0.40$ $[0.37,0.44]$	$0.29$ $[0.26,0.33]$
gnubg	$2.21$ $[2.12,2.30]$	$1.25$ $[1.20,1.32]$	$0.51$ $[0.48,0.55]$
Open Sage	$2.11$ $[2.02,2.20]$	$1.24$ $[1.18,1.29]$	$0.40$ $[0.37,0.43]$

Table 6: Effect of look-ahead depth on cubeful money playing strength. XG++ PR at matched search depth, with

1{,}000

self-play games per cell and 95% confidence intervals in brackets.

	PureTD, 0-ply	Prob5Jan, 0-ply
vs. PureTD 0-ply (mEq/game)	—	$-14.6$ $[-16.3,-12.8]$
XG++ PR	$0.80$ $[0.76,0.84]$	$1.13$ $[1.06,1.22]$
Checker PR	$0.85$ $[0.80,0.89]$	$1.05$ $[0.97,1.14]$
Cube PR	$0.56$ $[0.46,0.67]$	$1.58$ $[1.36,1.83]$

Table 7: Learned versus heuristic cube handling. PureTD versus Prob5Jan, both at 0-ply. The two networks are trained separately but have the same hidden-layer sizes,

[512{,}512{,}256{,}256]

. Their architectures differ only in the four cube input features and the output head. The head-to-head row uses

10^{7}

games.

Panel of Figure 2 — Figure 2: Two positions where learned and heuristic cube handling differ. In (a) the choice is between playing safe with 11/5 6/1 , the best cubeful play, and running off the anchor with 22/11 , the best DMP play (winning about $0.4$ percentage points more games but losing $19$ percentage points more gammons according to the rollout). In (b) Prob5Jan declines to redouble despite the opponent’s two checkers on the bar, costing $0.042$ .

Model	XG++ PR	Checker PR	Cube PR	vs. PureTD, 0-ply (mEq/game)
PureTD	$0.80$ $[0.76,0.84]$	$0.85$ $[0.80,0.89]$	$0.56$ $[0.46,0.67]$	—
Lin-style ablation	$1.07$ $[1.01,1.13]$	$1.01$ $[0.95,1.06]$	$1.39$ $[1.19,1.60]$	$-7.6$ $[-8.8,-6.4]$

Table 8: Cube-handling ablation.

Model	Hidden layers	Params	vs. production (mEq/game)	Checker dec/s (Zen 5)
Cubeful money
1-layer	$[512]$	103k	$-158$ $[-160,-155]$	$28{,}500$ ( $0.20\times$ )
2-layer	$[512,512]$	366k	$-25$ $[-27,-23]$	$8{,}100$ ( $0.70\times$ )
3-layer	$[512,512,256]$	497k	$-5$ $[-6,-3]$	$6{,}100$ ( $0.92\times$ )
4-layer (production)	$[512,512,256,256]$	563k	—	$5{,}640$
5-layer	$[512,512,256,256,256]$	629k	$+5$ $[+3,+7]$	$5{,}020$ ( $1.12\times$ )

Table 9: Strength across network sizes. Each row is one trained model, with its hidden-layer widths and parameter count. Column “vs. production” is the model’s head-to-head equity (mEq/game) at 0-ply against the production model of the same family. Labels “1 wide” and “2 wide” indicate one or two leading hidden layers widened to

768

units. Cubeful entries are capped (

\pm 128

) means with Jacoby on. The last column gives single-thread checker decisions/s at 0-ply on Zen 5 under the fixed-position benchmark of Sec. 4.3 , followed by the per-decision cost factor relative to the production model, as in Table 4 . Brackets give 95% confidence intervals. Each head-to-head uses

10^{7}

games.

Panel of Figure 3 — Figure 3: Cost versus strength across network sizes. The rows of Table 9 plotted as head-to-head equity at 0-ply against the production model of each family (dashed line) versus single-thread checker throughput on the Zen 5 machine, so that faster models lie to the right. Labels give the number of hidden layers, with “w” marking a first hidden layer widened to $768$ units and “ww” the first two. The vertical axis uses a signed logarithmic scale, $\operatorname{sign}(y)\ln(1+|y|)$ .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Engine	1-ply move filter	Cubeless (mEq/game)	Cubeful (mEq/game)	Cubeful XG++ PR
gnubg	default pruning	$-12.1$ $[-12.9,-11.2]$	$-27.2$ $[-29.0,-25.3]$	$1.25$ $[1.20,1.32]$
gnubg	unpruned	$-12.3$ $[-13.2,-11.5]$	$-27.8$ $[-29.6,-25.9]$	$1.27$ $[1.21,1.33]$
Open Sage	default pruning	$-5.9$ $[-6.8,-5.1]$	$-23.8$ $[-28.2,-19.5]$	$1.24$ $[1.18,1.29]$
Open Sage	unpruned	$-6.7$ $[-8.6,-4.8]$	—	—

Appendix

Table 10: Effect of the baselines’ 1-ply move filter. Each entry is the engine’s head-to-head equity (mEq/game) against PureTD 0-ply (negative indicates a loss to our model), with 95% confidence intervals, over

10^{7}

games per cell (

2\times 10^{6}

for the Open Sage cubeful cell and the unpruned Open Sage cubeless cell). Cubeful entries are capped (

\pm 128

) means with Jacoby on. Open Sage’s cubeful 1-ply has no pruning setting, so its single cubeful head-to-head and XG++ PR appear in the default row. The last column gives each configuration’s cubeful XG++ Performance Rating (lower is better) from

1{,}000

self-play games.

Engine	Depth	Zen 5	Zen 3
PureTD	0-ply	$4{,}946$ $[4{,}798,5{,}094]$	$4{,}689$ $[4{,}679,4{,}700]$
gnubg	0-ply	$41{,}268$ $[41{,}173,41{,}362]$	$32{,}816$ $[32{,}629,33{,}004]$
Open Sage	0-ply	$19{,}498$ $[19{,}452,19{,}543]$	$16{,}889$ $[16{,}851,16{,}927]$
PureTD	1-ply	$80.9$ $[80.7,81.1]$	$79.2$ $[79.2,79.3]$
gnubg	1-ply	$1{,}394$ $[1{,}386,1{,}402]$	$1{,}027$ $[1{,}025,1{,}030]$
Open Sage	1-ply	$1{,}055$ $[1{,}052,1{,}058]$	$793$ $[750,836]$

Appendix

Table 11: Cubeless-money throughput. Single-thread checker-decision throughput (decisions/s). Each figure is the mean of eight sweeps and bracketed entries are

95

t

-intervals over those eight sweeps. PureTD uses the PyTorch backend, and PureTD 1-ply uses the

0.08

root move filter (Appendix C ).

		Checker dec/s		Cube dec/s
Engine	Depth	Zen 5	Zen 3	Zen 5	Zen 3
PureTD	0-ply	$5{,}596$ $[5{,}447,5{,}746]$	$5{,}200$ $[5{,}190,5{,}210]$	$23{,}835$ $[23{,}498,24{,}172]$	$13{,}419$ $[13{,}347,13{,}490]$
gnubg	0-ply	$40{,}341$ $[40{,}089,40{,}592]$	$32{,}562$ $[32{,}370,32{,}755]$	$736{,}019$ $[733{,}958,738{,}080]$	$605{,}593$ $[601{,}011,610{,}174]$
Open Sage	0-ply	$19{,}696$ $[19{,}452,19{,}941]$	$16{,}713$ $[16{,}529,16{,}897]$	$339{,}400$ $[335{,}724,343{,}075]$	$277{,}828$ $[275{,}278,280{,}377]$
PureTD	1-ply	$64.3$ $[64.1,64.5]$	$63.3$ $[63.2,63.3]$	$138$ $[138,138]$	$137$ $[137,137]$
gnubg	1-ply	$1{,}483$ $[1{,}477,1{,}490]$	$1{,}113$ $[1{,}109,1{,}118]$	$4{,}263$ $[4{,}231,4{,}295]$	$3{,}235$ $[3{,}217,3{,}253]$

Appendix

Table 12: Cubeful-money throughput. Single-thread throughput (decisions/s), with checker and cube decisions reported separately. Each figure is the mean of eight sweeps and bracketed entries are

95

t

-intervals over those eight sweeps. PureTD uses the PyTorch backend, and PureTD 1-ply uses the

0.08

root move filter (Appendix C ). Open Sage’s cubeful 1-ply checker play has no pruning setting and scores every legal move and every reply (Appendix A ).

		Cubeless checker		Cubeful checker		Cubeful cube
Backend	Depth	Zen 5	Zen 3	Zen 5	Zen 3	Zen 5	Zen 3
PyTorch	0-ply	$4{,}946$	$4{,}689$	$5{,}596$	$5{,}200$	$23{,}835$	$13{,}419$
Optimized	0-ply	$37{,}308$	$15{,}838$	$44{,}093$	$18{,}477$	$255{,}731$	$114{,}110$
PyTorch	1-ply	$80.9$	$79.2$	$64.3$	$63.3$	$138$	$137$
Optimized	1-ply	$567$	$226$	$477$	$185$	$1{,}067$	$426$

Appendix

Table 13: PyTorch versus the optimized kernel. Single-thread decisions/s for PureTD with each backend, the mean of eight sweeps (full intervals in experiments/benchmarks/README.md ). 1-ply uses the

0.08

root move filter described above.

Matchup	Games	$\pm 128$	$\pm 256$	$\pm 1024$	Uncapped	$>128$
gnubg 0-ply	$10^{7}$	$\mathbf{85.2}$	$85.3$	$85.6$	$99.0$	$13$
gnubg 1-ply	$10^{7}$	$\mathbf{27.2}$	$27.2$	$27.4$	$1.3\times 10^{4}$	$30$
Open Sage 0-ply	$2\times 10^{6}$	$\mathbf{79.5}$	$79.5$	$79.5$	$79.5$	$0$
Open Sage 1-ply	$2\times 10^{6}$	$\mathbf{23.8}$	$23.8$	$23.8$	$23.8$	$0$
Prob5Jan	$10^{7}$	$\mathbf{14.6}$	$14.6$	$14.6$	$14.6$	$0$

Appendix

Table 14: Head-to-head means under different equity caps. Values are in mEq/game from PureTD’s side (every opponent lost). The 1-ply rows use each baseline’s default move pruning, as in Table 3 . “

>128

” counts games whose uncapped equity exceeds the

\pm 128

cap in absolute value. Bold marks the cap used throughout the paper.

Engine	XG++ PR (self-play)	XG++ PR (head-to-head)
PureTD, 0-ply	$0.80$ $[0.76,0.84]$	$0.86$ $[0.81,0.93]$
gnubg , 1-ply (unpruned)	$1.27$ $[1.21,1.33]$	$1.21$ $[1.13,1.28]$

Appendix

Table 15: Offline analysis on mixed-play games (95% confidence intervals).

Stage	Architecture Change	Type	Ep. (M)	Start LR	End LR
s1	scratch $\to[80]$	S	$1.2$	1e-3	const
s2	width $[80]\to[150]$	S	$0.6$	5e-5	const
s3	depth $[150]\to[150,150]$	S	$0.6$	5e-5	const
s4	width $[150,150]\to[256,256]$	S	$1.2$	5e-5	const
s5	depth $[256,256]\to[256,256,256]$	S	$1.2$	5e-5	const
s6	width $[256,256,256]\to[512,512,256]$	S	$3.5$	5e-5	5e-6

Appendix

Table 16: Shared base: DMP, then cubeless equity. Stages s1 – s8 train the double-match-point network through the progressive expansion of Sec. 3.2 , each stage resuming from the previous one’s output. The cm stages use the DMP model to warm-start a cubeless equity model. The base totals

{\sim}49

M episodes.

Stage	Architecture Change	Type	Ep. (M)	Start LR	End LR
p5_seed	cubeless equity $\to$ prob5 five-output head (wu20)	S	$6.0$	1e-4	1e-5
p5_const	—	E	$8.0$	1.8e-4	const
p5_descent	—	E	$49.0$	1.8e-4	2e-6
p5_restart	—	E	$8.0$	9e-5	9e-6

Appendix

Table 17: prob5 cubeless model. Starts from the output of cm_descent of Table 16 , a cubeless equity model at the shared final shape. The initial stage copies all four of the base’s hidden layers while the five-output probability head is initialized fresh. The fork totals

{\sim}71

M episodes,

6

M sampled-Bellman and

65

M exact-Bellman.

Panel of Figure 4 — Figure 4: Strength during retraining. In (a), head-to-head equity against gnubg at 0-ply, measured at checkpoints of the prob5 fork of Table 17 . The dotted lines mark the boundaries between the four stages, and the dashed line marks the production prob5 model. In (b), head-to-head equity against the production cubeful model through the cubeful fork of Table 18 , where zero marks parity and the dotted lines mark the boundaries between its five stages. Both horizontal axes omit the ${\sim}49$ M episodes of the shared base (Table 16 ).

Stage	Architecture Change	Type	Ep. (M)	Start LR	End LR
cf_adapt	cubeless equity $\to$ cubeful, $196\to 200$ inputs (wu20)	S	$2.0$	1e-4	const
cf_seed	—	S	$25.0$	6e-5	5e-7
cf_const	—	E	$12.0$	1.8e-5	const
cf_descent	—	E	$115.0$	1.8e-5	3e-7
cf_restart	—	E	$44.0$	1.62e-5	2.7e-7

Appendix

Table 18: Cubeful-money model. Starts from the result of cm_descent of Table 16 , a cubeless equity model. The initial stage widens the input layer from

196

200

features, the four added features (three cube-ownership indicators and is_cube_action ) being initialized fresh, while the hidden layers and the equity head are copied unchanged. The fork totals

{\sim}198

M episodes,

27

M sampled-Bellman and

171

M exact-Bellman.

Play	Edge	Branching factor	95% CI
Cubeless	The on-roll player’s full turn	$397$	$[396,397]$
Cubeful	The on-roll player’s full turn, both the cube decision and the checker play	$752$	$[751,753]$
Cubeful	The on-roll player’s turn, decomposed into the cube-offer decision and the checker play	$312$	$[312,313]$
Cubeful	As above, and when a cube is offered, an added node for the opponent’s take/pass decision	$242$	$[242,242]$

Appendix

Table 19: Branching factor of backgammon money games under PureTD self-play.

Figure 5: The cube’s impact on a checker-play decision.

Decision class	$n$	Difference per game [95% CI]
PureTD doubles, Prob5Jan does not double	$2{,}378$	$+3.11$ $[+2.79,+3.44]$
PureTD does not double, Prob5Jan doubles	$4{,}637$	$+3.06$ $[+2.80,+3.31]$
PureTD takes, Prob5Jan passes	$152$	$+0.04$ $[-0.06,+0.13]$
PureTD passes, Prob5Jan takes	$750$	$-0.83$ $[-1.06,-0.61]$
All cube decisions	$7{,}917$	$+5.38$ $[+4.90,+5.87]$
Checker play	$97{,}358$	$+4.02$ $[+3.32,+4.69]$

Appendix

Table 20: Equity lost as scored by gnubg 4-ply, by class of disagreement. Differences are in mEq/game.

Item	Identifier
Engine commit	e8847f498f7be8798c0b71314e83c8b4075f2043
Fox v0.34 file	fox_v0.34_2026-09-26.ogxf
MD5	240a6a1605d58c44decf4fc3db413559
Aureus v0.12 file	aureus_v0.12_2026-09-26.ogxf
MD5	379f8f20dd2b0ce05f9e24f05098fb87

Appendix

Table 21: Hedgehog engine and model identifiers.

Model	Depth	Cubeless vs. prob5 (mEq/game)	Games	Cubeful vs. PureTD (mEq/game)	Games
Fox v0.34	0-ply	$-26.6$ $[-27.5,-25.7]$	$10^{7}$	$-68.5$ $[-70.4,-66.7]$	$10^{7}$
Fox v0.34	1-ply	$+0.8$ $[-0.6,+2.1]$	$4\times 10^{6}$	$-18.0$ $[-21.1,-15.2]$	$4\times 10^{6}$
Aureus v0.12	0-ply	$-13.0$ $[-13.9,-12.1]$	$10^{7}$	$-47.8$ $[-49.7,-46.0]$	$10^{7}$

Appendix

Table 22: Head-to-head strength against Hedgehog. Each entry is the Hedgehog model’s equity (mEq/game) against our model at 0-ply, prob5 for cubeless money and the cubeful money model for cubeful money (negative indicates a loss to our model). Cubeful entries are capped (

\pm 128

) means with Jacoby on, as in Table 3 , and with the cube capped at

64

for both sides. Brackets give 95% confidence intervals.

	Cubeless checker dec/s		Cubeful checker dec/s		Cube decision dec/s
Engine	Zen 5	Zen 3	Zen 5	Zen 3	Zen 5	Zen 3
PureTD, 0-ply (reference)	$4{,}946$	$4{,}689$	$5{,}596$	$5{,}200$	$23{,}835$	$13{,}419$
gnubg , 0-ply	$41{,}268$ ( $0.12\times$ )	$32{,}816$ ( $0.14\times$ )	$40{,}341$ ( $0.14\times$ )	$32{,}562$ ( $0.16\times$ )	$736{,}019$ ( $0.03\times$ )	$605{,}593$ ( $0.02\times$ )
Hedgehog Fox v0.34, 0-ply	$23{,}007$ ( $0.21\times$ )	$20{,}830$ ( $0.23\times$ )	$24{,}647$ ( $0.23\times$ )	$21{,}840$ ( $0.24\times$ )	$316{,}091$ ( $0.08\times$ )	$290{,}153$ ( $0.05\times$ )
Hedgehog Aureus v0.12, 0-ply	$14{,}713$ ( $0.34\times$ )	$7{,}733$ ( $0.61\times$ )	$15{,}473$ ( $0.36\times$ )	$8{,}325$ ( $0.62\times$ )	$220{,}980$ ( $0.11\times$ )	$124{,}181$ ( $0.11\times$ )
gnubg , 1-ply	$1{,}394$ ( $3.5\times$ )	$1{,}027$ ( $4.6\times$ )	$1{,}483$ ( $3.8\times$ )	$1{,}113$ ( $4.7\times$ )	$4{,}263$ ( $5.6\times$ )	$3{,}235$ ( $4.1\times$ )

Appendix

Table 23: Fixed-position throughput against Hedgehog. Each entry is the mean of eight sweeps over the full pools. The PureTD and gnubg rows are those of Tables 11 and 12 . Entries other than PureTD are followed by the per-decision cost factor relative to the PureTD 0-ply row, as in Table 4 .

RL for Game AI

See all 29 papers

Oct 12, 2026

Learning to Play: Reinforcement Learning and Games

Amazing breakthroughs in reinforcement learning have taken place. Computers teach themselves to play Chess and Go and beat world champions. There is talk about expanding application areas towards general artificial intelligence (AI). The breakthroughs in Backgammon, Checkers, Chess, Atari, Go, Poker, and StarCraft have shown that we can build machines that exhibit intelligent game playing of the highest level. These successes have been widely publicized in the media, and inspire AI entrepreneurs and scientists alike. Reinforcement learning in games has become a mainstream AI research topic. It is a broad topic, and the successes build on a range of diverse techniques, from exact planning algorithms, to adaptive sampling, deep function approximation, and ingenious self-play methods. Perhaps because of the breadth of these technologies, or because of the recency of the breakthroughs, there are few books that explain these methods in depth. This book covers all methods in one comprehensive volume, explaining the latest research, bringing you as close as possible to working implementations, with many references to the original research papers. The programming examples in this book are in Python, the language in which most current reinforcement learning research is conducted. We help you to get started with machine learning frameworks such as Gym, TensorFlow, and Keras, and provide exercises to help understand how AI is learning to play.

Aske Plaat

RL for Game AI Reinforcement Learning Game-Playing Agents Self-Play RL

Ask PDF

Jul 7, 2026

A Gold-Standard Study of What Makes a Lightweight Game-Playing Agent Strong

Reinforcement learning agents for imperfect-information card games are only as strong as the opponents they train against, and they are hard to grade, since they beat a random opponent over 99 percent of the time and only tie copies of themselves. So we build a strong, fixed, rule-based expert for Gin Rummy and use it only as a yardstick, never for training. It beats every agent we trained 70 to 99 percent of the time. Across more than a hundred runs, we isolate what makes a lightweight agent stronger. Trust region updates, a well-aimed reward, a curriculum of tougher opponents, warm starting, and keeping the best checkpoint all help, and stacking them lifts a self-play champion from about 30 to 36 percent against the expert. Several ideas did not pay off. Short-term and longer-term reward shaping, learned state embeddings, imitation and DAgger, and a live large language model opponent were each unhelpful, too slow, or too heavy to train at scale. Comparing MLP, convolutional, set-based, attention, and recurrent encoders shows that extra capacity does little to break the ceiling, suggesting the limit is information rather than network size. We add standard baselines (neural fictitious self-play and information set Monte Carlo search) and confirm the approach carries over to Leduc Hold'em, where the optimum is computable. The result is a lightweight, game-agnostic recipe that trains competitive agents without training on the expert, for any game a small model can handle, reported with robust statistics and released as a reusable package.

Nima Kelidari, Mohammadsaeed Haghi, Mahdi Salmani

University of Southern California

Self-Play RL Reinforcement Learning Game-Playing Agents Reward Design for RL Imperfect-Information Games AI Agent Evaluation+1

Ask PDF

Sep 3, 2026

Local Updates, Global Learning (LUGL): Playing Games with non-incremental Learners

The dominance of Neural Networks (NNs) in RL is partially due to their incremental learning capability, which naturally suits the online, non-stationary nature of self-play training. However, gradient-boosted trees like LightGBM are widely recognised as the state of the art for tabular data in supervised learning, often outperforming NNs in accuracy and efficiency. Game states are inherently tabular---discrete actions, categorical card identities, structured board positions---which makes them an ideal candidate for tree-based methods. We introduce LUGL (Local Updates, Global Learning), a framework that decouples data collection from model fitting, enabling non-incremental learners such as GBTs to operate in RL settings where they would otherwise fail due to distributional shift. LUGL alternates between a local updates phase, where the agent plays self-play games and accumulates tabular updates (Q-values, V-values, policies, or regret values) in a finite table, and a global learning phase, where the table is used to train a function approximator that generalises to unseen states before the table is reset. We test our approach in four standard perfect-information games (Tic-tac-toe, Connect-4, Othello, and Hex) and five imperfect-information games (Kuhn's poker, Leduc Hold'em, Liar's Dice, Goofspiel, and Flop5 Hold'em), and show that our results are competitive with or superior to DQN and DeepCFR. Our experiments demonstrate that the community's strong bias towards NNs in game-playing may be unwarranted, since LightGBM-based agents achieve competitive or superior performance across all tested benchmarks.

David Milec, Spyridon Samothrakis, Michael Fairbank +1

Faculty of Electrical Engineering, Czech Technical University in Prague, Prague, Czech Republic · School of Computer Science and Electronic Engineering, University of Essex, Colchester, United Kingdom · Maastricht University, the Netherlands

Gradient-Boosted Decision Trees Game-Playing Agents Self-Play RL Incremental Learning

Ask PDF

Abstract

RL for Game AI

Explore similar work

Learning to Play: Reinforcement Learning and Games

A Gold-Standard Study of What Makes a Lightweight Game-Playing Agent Strong

Local Updates, Global Learning (LUGL): Playing Games with non-incremental Learners