Learning to Play: Reinforcement Learning and Games

Aske Plaat

cs.AIOct 12, 2026

Learning to Play: Reinforcement Learning and Games

Authors: Aske Plaat

Abstract

Amazing breakthroughs in reinforcement learning have taken place. Computers teach themselves to play Chess and Go and beat world champions. There is talk about expanding application areas towards general artificial intelligence (AI). The breakthroughs in Backgammon, Checkers, Chess, Atari, Go, Poker, and StarCraft have shown that we can build machines that exhibit intelligent game playing of the highest level. These successes have been widely publicized in the media, and inspire AI entrepreneurs and scientists alike. Reinforcement learning in games has become a mainstream AI research topic. It is a broad topic, and the successes build on a range of diverse techniques, from exact planning algorithms, to adaptive sampling, deep function approximation, and ingenious self-play methods. Perhaps because of the breadth of these technologies, or because of the recency of the breakthroughs, there are few books that explain these methods in depth. This book covers all methods in one comprehensive volume, explaining the latest research, bringing you as close as possible to working implementations, with many references to the original research papers. The programming examples in this book are in Python, the language in which most current reinforcement learning research is conducted. We help you to get started with machine learning frameworks such as Gym, TensorFlow, and Keras, and provide exercises to help understand how AI is learning to play.

Figures & tables

Panel of Figure 2 — Figure 2: Overview of the chapters of this book

Panel of Table 1 — Table 1: Heuristic-Sample-Generalize

Panel of Figure 3 — Figure 3: Stack of chapters, with (from bottom to top) increasingly advanced mappings from problems to methods

Figure 6: Geoffrey Hinton, Yann LeCun, and Yoshua Bengio

Name	board	state space	zero-sum	information	turn
Chess	$8\times 8$	$10^{47}$	zero-sum	perfect	turn
Checkers	$8\times 8$	$10^{18}$	zero-sum	perfect	turn
Othello	$8\times 8$	$10^{28}$	zero-sum	perfect	turn
Backgammon	24	$10^{20}$	zero-sum	chance	turn
Go	$19\times 19$	$10^{170}$	zero-sum	perfect	turn
Shogi	$9\times 9$	$10^{71}$	zero-sum	perfect	turn

Table 2: Characteristics of games

Figure 21: Excitement in 1997. Game 6, Caro Kahn. Kasparov resigned after 19 moves

Panel of Figure 22 — Figure 22: Fan Hui, Lee Sedol, Ke Jie; Demis Hassabis, David Silver and Aja Huang (left-to-right, top-to-bottom)

Figure 23: “Speechless” move 37 by AlphaGo in game 2 (played on P 10)

Panel of Figure 24 — Figure 24: Reinforcement learning: agent and environment [ 659 ]

Panel of Figure 25 — Figure 25: Backup diagram with MDP tuple [ 659 ]

Panel of Figure 27 — Figure 27: A tree and a graph; node $j$ is a transposition

Panel of Figure 29 — Figure 29: Value and policy in Actor-Critic

Panel of Figure 30 — Figure 30: Model-free and model-based methods [ 659 ]

Panel of Table 3 — Table 3: MDP tuple and reinforcement learning in the chapters

Panel of Figure 32 — Figure 32: Search-Eval architecture in a tree

Figure 34: Tic Tac Toe game tree with values [ 195 ]

Panel of Figure 35 — Figure 35: Minimax tree of Listing 6

Panel of Figure 36 — Figure 36: $\pm$ Critical path , + max solution tree , - min solution tree

Panel of Figure 37 — Figure 37: Max solution tree: upper bound determined by all children at max nodes, a single child at min nodes

Panel of Figure 38 — Figure 38: Min solution tree: lower bound determined by all children at min nodes, a single child at max nodes

Solution tree	Value	# children at max node	# children at min node
Max	upper bound	all	$1$
Min	lower bound	$1$	all

Table 4: Solution trees and bounds

Critical/solution trees
Critical tree = max solution tree $\bigcup$ min solution tree
Critical path = max solution tree $\bigcap$ min solution tree

Table 5: Critical tree and path

Name	Principle	Applicability	Effectiveness
alpha-beta	backward pruning	all minimax games	$w^{d}\rightarrow\sqrt{w^{d}}$
iterative deepening	iterate search depth	with TT: move ordering	helps alpha-beta
transposition table	cache of states	with ID: move ordering	helps alpha-beta
null window	alpha-beta cutoffs	all minimax games	more cutoffs
forward pruning	variable width	domain specific	variable
capture moves	ordering	games with captures	good

Table 6: Search enhancements

Panel of Figure 39 — Figure 39: Alpha-beta starts at the root, with a window of $\langle-\infty,+\infty\rangle$ . In other words, alpha-beta is called with $n=r,a=-\infty,b=+\infty$ . At the root it expands the max node’s first child $a$ . Node $a$ is of type min, and min node $a$ ’s first child, leaf node $c$ is visited. Node $c$ is a leaf node with value $3$ . Coming out of the recursion back in node $a$ alpha-beta updates variables $g=\min(+\infty,3)$ and $b=\min(+\infty,3)$ to $3$ . It expands the second child of min node $a$ , leaf node $d$ . This node $d$ is expanded with window $\langle-\infty,3\rangle$ , a tighter beta bound. Its value is $6$ , coming back in $a$ , alpha-beta updates the variables $g=\min(3,6)$ and $b=\min(3,6)$ to $3$ , finds no other children at $a$ , and returns $g=3$ to $r$ . Back at the root $r$ , alpha-beta updates $g=\max(-\infty,3)$ and $a=\max(-\infty,3)$ to $3$ . There is another child $b$ , which it expands with window $\langle 3,+\infty\rangle$ ; it expands its first child $b$ , finding leaf node $e$ . The value of $2$ is returned to parent $b$ , who updates the variables $g=\min(+\infty,2)$ and $b=\min(+\infty,2)$ to $2$ . Now the if statement is tested and $3\geq 2$ is true, so the break statement breaks out of the loop, returning the value of $g=2$ to parent $r$ , cutting off the expansion of child $f$ . At parent $r$ , the root, the variables are updated to $g=\max(3,2)$ and $a=\max(3,2)$ to $3$ . Alpha-beta is ready and returns $g=3$ to the caller as the minimax value $n$ without having expanded all nodes of the tree .

Figure 40: Transposition: c4,d5,d4 or d4,d5,c4?

Panel of Figure 41 — Figure 41: Alpha-beta starts at the root with narrow window $\langle-\infty,2\rangle$ . In other words, alpha-beta is called with $n=r,a=-\infty,b=2$ . At the root it expands first child $a$ . Node $a$ ’s first child, leaf node $c$ , is visited. Coming out of the recursion back in node $a$ , alpha-beta updates variables $g=\min(+\infty,3)$ to $3$ and $b=\min(2,3)$ to $2$ . No cutoff happens since $-\infty\not\geq 3$ . We then expand the second child of min node $a$ , leaf node $d$ . Its value is $6$ ; coming back in $a$ , alpha-beta updates the variables $g=\min(3,6)$ and $b=\min(3,6)$ to $3$ , finds no other children at $a$ , and returns $g=3$ to $r$ . Back at the root $r$ , alpha-beta updates $g=\max(-\infty,3)$ and $a=\max(-\infty,3)$ to $3$ . Now a cutoff happens, since $3\geq 2$ . The root returns the value 3, a lower bound since the max node has unexpanded children. Let us also try a second narrow window, this time with a high window: $\langle 4,+\infty\rangle$ . At the root, node $a$ and leaf $c$ are expanded, which returns $3$ at node $a$ . There a cutoff happens, since $4\geq 3$ . Min node $a$ returns upper bound $3$ to the root. There, no cutoff happens since $3\not\geq+\infty$ and node $b$ is expanded, with window $\langle 4,+\infty\rangle$ . Node $e$ is expanded and returns 2, which causes a cutoff in $b$ since $4\geq 2$ , and upper bound 2 is returned to the root, which returns $\max(3,2)=3$ . This value is an upper bound, since the two children returned upper bounds. After two narrow window passes we now have an upper bound and a lower bound of equal value 3.

Name	Principle	Applicability	Effectiveness
search extensions	variable depth	domain specific	good
quiescence search	variable depth	domain specific	good
null-move search	variable depth	no Zugzwang	good
odd-even effect	move advantage	minimax games	unstable values
piece-square tables	evaluation coding	heuristic evaluation	efficiency
coefficient learning	machine learning	evaluation tuning	automated learning

Table 7: Eval enhancements

Panel of Table 8 — Table 8: Heuristic-Sample-Generalize

Panel of Figure 42 — Figure 42: Searching a tree vs. searching a path

Panel of Table 9 — Table 9: Heuristic-Sample-Generalize

Panel of Table 10 — Table 10: MDP tuple and reinforcement learning in the chapters

Figure 43: Monte Carlo Tree Search [ 108 ]

Panel of Figure 44 — Figure 44: MCTS starts at the root, does not select a move because there are no children in the tree, but expands a random child. Assume $a$ is expanded. Then a random playout is performed at node $a$ . Assume node $d$ is the first node in the playout. In this tiny state space, the random path that is played out consists only of node $d$ and is a terminal node. The value at node $d$ is a win, so backpropagation increments all win values of the path to the root by 1 (nodes $a,r$ ) and also the visit counts. The second iteration starts at root $r$ , where there are unexpanded children, so it expands a node, which is node $b$ . At $b$ a playout is performed, which ends up at, say, $e$ , whose value is a loss. This gets propagated (no win values change, and the visit counts of $b,r$ are incremented). Next, the third iteration starts. At the root all children are expanded, so we select a child. We select $a$ since its UCT value is $1/1+1\times\sqrt{\ln 2/1}=1.83$ , and $b$ ’s value is $0/1+1\times\sqrt{\ln 2/1}=0.83$ . There are unexpanded children so it expands a node, assume it picks node $c$ , a win. The values are updated ( $a$ ’s win and visit count are incremented to 2 and 2, $r$ ’s to 2 and 3). Next, the fourth iteration selects node $a$ since the UCT value of $a$ is $2/2+1\times\sqrt{\ln 3/2}=1.63$ and $b$ is $0/1+1\times\sqrt{\ln 3/1}=1.04$ . It finds that there are unexpanded children of $a$ in the tree, so it expands child $d$ , a win, and updates the win and visit values of $d,a,$ and $r$ accordingly. It continues with more iterations until time is up.

Name	Principle	Applicability	Effectiveness
expl/expl	expl/expl/N	small/large searches	all MCTS
RAVE	statistics sharing	early phase	all MCTS
play-out db	small patterns	domain specific	domain dependent
adaptive expl/expl	variable $C_{p}$	general	all MCTS
parallelism	parallel search	general	all MCTS

Table 11: MCTS enhancements

Figure 48: Adaptive MCTS tree [ 388 ]

Figure 49: Scatter plots of $C_{p}$ for $N$ = 300, 1000, 3000, and 10,000 node expansions per MCTS run. Each dot is one MCTS run to optimize a large polynomial. Lower is better. Exploitation (small $C_{p}$ ) works best in small trees ( $N=300$ ), exploration in large trees ( $N=10,000$ ) [ 388 ]

Panel of Table 12 — Table 12: Heuristic-Sample-Generalize

Figure 53: Layers of features of increasing complexity [ 409 ]

Figure 54: Three fully connected hidden layers [ 574 ]

Figure 55: Overfitting: does the curvy red line or the straight dashed blue line generalize the information in the data points best?

Figure 56: Convolutional filters [ 250 ]

Figure 57: Max and average $2\times 2$ pooling [ 250 ]

Figure 58: Convolutional network example architecture [ 574 ]

Name	Principle	Applicability	Effectiveness
AlexNet	CNN	image	improve
inception	net inside net	image	improve
residual	skip links	image	improve
GAN	generative-adversarial	image	robustness
RNN/LSTM	state	sequential proc	sequences

Table 13: Advanced network architectures

Figure 65: RNN $x_{t}$ is an input vector, $h_{t}$ is the output/prediction, and $A$ is the RNN [ 494 ]

Figure 66: Time-unrolled RNN neuron, with a $\tanh$ activation function; shown are the previous and next time steps as well [ 494 ]

Supervised learning	Reinforcement learning
(example-label) or (example-number) pair	(state-action) or (state-reward) pair
database of ground truths	environment gives reward
static learning process	interactive learning process
fixed set of examples	changing and interdependent set of examples
dense: ground truth for all examples	sparse: rewards for some examples

Table 14: Supervised vs. reinforcement learning

Figure 71: Agent/environment loop [ 659 ]

Figure 74: Screenshots of 4 Atari Games (Breakout, Pong, Montezuma’s Revenge, and Private Eye)

Name	Principle	Applicability	Effectiveness
double DQN	de-overestimate values	DQN	convergence
prioritized experience	decorrelation	replay buffer	convergence
distributional	probability distr	deep RL	generalization
random noise	parametric noise	deep RL	more exploration
actor-critic	value/policy interleave	A3C, PPO, dueling	stability, parallel

Table 15: Deep reinforcement learning enhancements

Figure 76: Rainbow graph: performance over 57 Atari games [ 290 ]

Program	Game	Plan	Learn	Data	Net	Eval	Reference
Samuel	Check.	AlphaB.	coeff.	self-play	-	coeff. feat.	Samuel 1959 [ 577 ]
Chinook	Check.	AlphaB.	-	endgames	-	heur. feat.	Schaeffer 1992 [ 590 ]
Logistello	Othello	AlphaB.	regress.	patterns	-	coeff. pat.	Buro 1995 [ 113 ]
Deep Blue	Chess	AlphaB.	coeff.	manual	-	coeff. feat.	Hsu 1990 [ 310 ]
NeuroCh.	Chess	AlphaB.	TD(0)	db/self-play	1 layer	coeff. feat.	Thrun 1995 [ 686 ]
Neurogam.	B.gam.	-	FCN	games db	1 layer	coeff. feat.	Tesauro 1989 [ 673 ]

Table 16: Evolution of planning, learning, and self-play in game playing programs

Program	Planning	Learning	Input/output	Achievement
AlphaGo	MCTS	supervised & reinforcement	Go	beat human champions
AlphaGo Zero	MCTS	self-play	Go	learn tabula rasa
AlphaZero	MCTS	self-play	Go, Chess, Shogi	generalize three games

Table 17: AlphaGo, AlphaGo Zero, and AlphaZero

Panel of Figure 78 — Figure 78: Minimax tree and agent/environment loop

Panel of Figure 79 — Figure 79: Self-play loop improving quality of net

Panel of Figure 80 — Figure 80: A diagram of self-play

Panel of Figure 81 — Figure 81: Levels in self-play

Figure 82: Monte Carlo Tree Search [ 108 ]

Figure 84: AlphaGo Elo rating; Effect of individual networks [ 623 ]

Figure 85: Elo rating of Alphago distributed versions [ 623 ]

Figure 87: Go playing strength over the years [ 12 ]

Figure 88: Performance of AlphaGo Zero [ 626 ]

Figure 89: AlphaGo Zero is learning joseki [ 626 ]

Table 18: Different input planes (first set repeated 8 times to capture history) [ 625 ]

Figure 90: Elo rating of AlphaZero in Chess, Shogi, and Go [ 625 ]

Hyperparameter	Value
self play games per iteration	25000
MCTS simulations	1600
exploration temperature	1 for first 30 moves, 0 afterwards
Dirichlet noise	$\eta=0.03,\epsilon=0.25$
resignation threshold	5%
regularization parameter	$c=10^{0.4}$

Table 19: AlphaGo hyperparameters [ 626 ]

Panel of Figure 91 — Figure 91: Model-free and model-based methods [ 659 ]

Panel of Figure 92 — Figure 92: Stack of chapters, with (from bottom to top) increasingly advanced mappings from problems to paradigms

Panel of Figure 93 — Figure 93: Simplified self-play loop

System 1	System 2
fast	slow
recognition	reasoning
reflex	rational
approximate	exact
train	plan
eval	search

Table 20: Fast and slow, eval and search

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Name	Type	URL	Ref.
Python	General-purpose language	https://www.python.org	[ 562 ]
Weka	Machine learning env.	https://www.cs.waikato.ac.nz/ml/weka	[ 762 ]
Scikit-Learn	Machine learning env.	https://scikit-learn.org	[ 513 ]
Caffe	Deep learning env.	http://caffe.berkeleyvision.org	[ 334 ]
PyTorch	Deep learning env.	https://pytorch.org	[ 508 ]
TensorFlow	Deep learning env.	https://www.tensorflow.org	[ 2 ]

Appendix

Table 21: General learning environments

Name	Type	URL	Ref.
ALE	Atari Games	https://github.com/mgbellemare/Arcade-Learning-Environment	[ 53 ]
Gym	RL environments	https://gym.openai.com	[ 98 ]
Stable	RL algorithms	https://stable-baselines.readthedocs.io	[ 291 ]
Baselines
Dopamine	Deep RL env.	https://github.com/google/dopamine	[ 123 ]
RLlib	Distributed RL	https://docs.ray.io/en/latest/rllib.html	[ 417 ]

Appendix

Table 22: Deep reinforcement learning environments

Name	Type	URL	Ref.
AlphaZero General	AlphaZero in Python	https://github.com/suragnair/alpha-zero-general	[ 679 ]
ELF	Game framework	https://github.com/pytorch/ELF	[ 689 ]
Leela	AlphaZero for Chess, Go	https://github.com/LeelaChessZero/lczero	[ 507 ]
PhoenixGo	AlphaZero-based Go prog.	https://github.com/Tencent/PhoenixGo	[ 775 ]
PolyGames	Env. for Zero learning	https://github.com/facebookincubator/Polygames	[ 124 ]

Appendix

Table 23: Self-learning environments

Figure 94: Comparison of dual-headed and separate, and of convolutional and residual networks [ 626 ]

Figure 95: AlphaGo Zero residual block [ 626 ]

Figure 96: Comparison of power consumption for training [ 582 ]

Panel of Figure 101 — Figure 101: Game 5 Fan Hui (black) vs. AlphaGo (white): move 1–99, move 100–199 (moves 151/157/163 at 141, 154/160 at 148), and move 200–214. AlphaGo won by resignation.

Panel of Figure 106 — Figure 106: Game 5 Lee Sedol (black) vs. AlphaGo (white): move 1-99, move 100-199 (118 at 107, 161 at square), and move 200-280 (240 at 200, 271 at black square, 275 at white square, 276 at black triangle). AlphaGo won by resignation.

Table 126

Appendix

Figure 109: Game 3 AlphaGo (black) vs. Ke Jie (white): move 1-99, move 100-199, and move 200-209. AlphaGo won by resignation.

Server	Name	Link
OGS	Online Go Server	https://online-go.com
KGS	Kiseido Go Server	http://www.gokgs.com
IGS	Pandanet Internet Go Server	http://www.pandanet-igs.com
Tygem	Tygem Baduk	http://www.tygembaduk.com
Fox	Fox Weiqi (Chinese)	https://www.foxwq.com
Dragon	Dragon Go Server for correspondence	https://www.dragongoserver.net

Appendix

Table 24: Internet Go Servers

Tutorial	Link
Kiseido	https://www.kiseido.com/ff.htm
Nordic	https://www.nordicgodojo.eu/post/212/a-simple-beginners-guide-to-go
Online Go	https://online-go.com/learn-to-play-go
Wikipedia	https://en.wikipedia.org/wiki/Go_(game)

Appendix

Table 25: Go Tutorials

Server	Link
Chess.com	https://chess.com
Internet Chess Club	http://chessclub.com
Chessbase	https://en.chessbase.com
Chess24	https://chess24.com/en/play/chess
Lichess	https://lichess.org
Wikipedia	https://en.wikipedia.org/wiki/Chess

Appendix

Table 26: Internet Chess Resources

Engine	Link
Stockfish	https://stockfishchess.org
Komodo	http://komodochess.com
Houdini	http://www.cruxis.com/chess/houdini.htm
Critter	http://www.vlasak.biz/critter/
Ethereal	https://github.com/AndyGrant/Ethereal
Gull	https://sourceforge.net/projects/gullchess/

Appendix

Table 27: Open-source Chess Programs

Abbreviation	Journal
PAMI	IEEE Transactions on Pattern Analysis and Machine Intelligence
AIJ	Artificial Intelligence Journal
JAIR	Journal of AI Research
ToG	IEEE Transactions on Games
ICGAJ	ICCA/ICGA Journal
Science	Science

Appendix

Table 28: Journals

Abbreviation	Conference
IJCAI	International Joint Conference on AI
AAAI	Association for the Advancement of AI
ECAI	European Conference on AI
FDG	Foundations of Digital Games
CoG	Conference on Games
NeurIPS	Neural Information Processing Systems

Appendix

Table 29: Conferences

RL for Game AI

See all 29 papers

Aug 15, 2026

PureTD: Reinforcement Learning for Backgammon Money Games with No Evaluation-time Search

We revisit Tesauro's TD-Gammon for backgammon money games in the setting of no evaluation-time search. Both checker play and cube action (use of the doubling cube) are learned from scratch via self-play reinforcement learning (RL), with minimal hand-coded logic and no expert features. In this setting, we demonstrate that pure self-play RL suffices to train models that reach near-state-of-the-art playing strength. Specifically, for cubeful money games, our search-free model evaluates faster and is substantially stronger than the open-source engines GNU Backgammon and Open Sage running a one-move (1-ply) look-ahead search. Our method of training a cubeful model also outperforms the standard approach of training a cubeless model and deriving cubeful equity with Janowski-style formulas, when the two models are equally sized.

Alexander L. Strehl

Independent Researcher

RL for Game AI Game-Playing Agents Self-Play RL Temporal-Difference Learning

Ask PDF

Jun 18, 2026

Augmenting Game AI with Deep Reinforcement Learning

Immersion in video games depends not only on graphics, audio, and game mechanics, but also on the quality of in-game characters. Producing believable characters, or game AI, remains a significant challenge as behavioral complexity is hard to capture with hand-coded systems. Game AI is a source of immersion and engagement; however, the limitations stemming from the challenges of creating game AI often lead to frustration and the breaking of the illusion of realism within the game. The introduction of machine learning models opens the door to creating more believable, authentic, and relatable characters in games. The promise is that they either learn from interacting with the game, or from player data, to develop true human-like behavior. In this paper, we envision more applications of reinforcement learning for game AI in the future. For this to materialize, current research limitations are prohibitive to broad deployment across game genres. Therefore, we propose a framework for training reinforcement learning models with a set of requirements in mind that are suited towards game AI and game development. We present examples of games with reinforcement learning-augmented game AI and describe the practicalities of deploying player-facing machine learning agents in modern games. Furthermore, we identify bottlenecks and hard problems in these areas, which we believe offer promising research directions to accelerate the adoption of machine learning in game AI for the video game industry.

Alessandro Sestini, Joakim Bergdahl, Amir Baghi +3

Electronic Arts (EA), Stockholm, Sweden

RL for Game AI Reinforcement Learning Game-Playing Agents

Ask PDF

Oct 6, 2026

Learning Explainable Representations of Complex Game-playing Strategies

As part of learning to play complex games, human players develop develop abstractions for concepts and strategies of gameplay consistent with game rules to improve their performance. These concepts are applied to explain other players' actions, and to inform their own actions in-game. Understanding other players' strategies is a crucial part of such improvement, but requires time and effort. In this paper, we propose a strategy similar to human cognition for training RL agents to synthesize learned strategies and policies as executable procedures based on sequences of gameplay actions. We present methods to automatically learn such programs to play chess and to solve tasks in a grid-based environment. We show that the learned strategies produce effective actions, and can be learned from gameplay data.

Abhijeet Krishnan, Colin M. Potts, Arnav Jhala +3

Venture IV 420, North Carolina State University, Raleigh, NC 27606 USA · Department of Aerospace Engineering, Indian Institute of Technology Bombay, Powai, Mumbai 400076, Maharashtra, India · Tata Research Development and Design Centre, Pune 411013, Maharashtra, India +1

Game-Playing Agents Program Synthesis Interpretability in RL Policy Learning RL for Game AI

Ask PDF

Abstract

RL for Game AI

Explore similar work

PureTD: Reinforcement Learning for Backgammon Money Games with No Evaluation-time Search

Augmenting Game AI with Deep Reinforcement Learning

Learning Explainable Representations of Complex Game-playing Strategies