cs.CLOct 16, 2026
SaveExplaining Chess and Go in Natural Language
Organizations: Nebius AI
Abstract
Modern chess engines play far beyond human ability, often choosing moves whose underlying ideas are difficult for humans to understand. We propose an automatic metric for evaluating chess explanations, grounded in chess-engine evaluations. Evaluating state-of-the-art language models using this metric, we find that they perform surprisingly poorly, often relying on surface-level board observations rather than robust, transferable chess concepts. We then introduce BinProp, a training-free method that enables these same models to discover reusable chess concepts and use them to explain moves in natural language. BinProp improves explainability for every model tested, nearly tripling the average score compared with direct prompting. Beyond chess, the same method discovers useful concepts in Go, draughts, and shogi without additional tuning.
Figures & tables
Figure 1: Black to move in Carlsen–Anand (2019). Anand played d4 (light blue); Stockfish’s top move is c4 (dark blue). Our method, BinProp, explains the difference between the moves by finding properties that appear in one continuation but not the other. Further examples appear in § G .
Figure 2: Sampling positions, an illustrative example. Nodes show engine evaluation. Blue/red marks positive/negative; dashed red marks nearby negatives. Gray moves score more than 50 centipawns below second-best and are discarded.
| Model | Method | Mean Explainability | Gain |
|---|---|---|---|
| Qwen3.5-397B-A17B ( Qwen Team, 2026 ) | Baseline | 0.102 | +0.072 |
| BinProp (ours) | 0.174 | ||
| GPT-5.6 Sol ( OpenAI, 2026a ) | Baseline | 0.062 | +0.144 |
| BinProp (ours) | 0.205 | ||
| DeepSeek-V4-Pro ( DeepSeek-AI, 2026 ) | Baseline | 0.173 | +0.042 |
| BinProp (ours) | 0.214 |
Table 1: Chess test results. Gain is BinProp minus Baseline, computed before rounding.
Figure 3: Baseline and BinProp explanations for the top four models. Each panel shows one sampled positive position; explainability is measured over 24 held-out positions: eight positives, eight negatives, and eight nearby negatives. Evaluation bars show Stockfish’s evaluation.
Figure 4: Mean explainability versus Stockfish agreement. Points average independently sampled motif sets; error bars show 95% confidence intervals.
| Model | Method | Mean Explainability | Gain |
|---|---|---|---|
| GPT-5.6 Sol ( OpenAI, 2026a ) | Baseline | 0.245 | +0.055 |
| BinProp (ours) | 0.300 | ||
| Opus 5 ( Anthropic, 2026b ) | Baseline | 0.190 | +0.111 |
| BinProp (ours) | 0.301 | ||
| GPT-5.5 ( OpenAI, 2026b ) | Baseline | 0.181 | +0.122 |
| BinProp (ours) | 0.303 |
Table 2: Mean Go test explainability for the top four BinProp models. Full results are in Table 4 .
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: An Opus 5 motif from Carlsen–Erigaisi (2022). The highlighted branch combines material balance with an edge in king-zone pressure and weak squares. The arrow shows one contribution to White’s pressure; the red dot marks the executed return.
| Model | Gain | 95% confidence interval |
|---|---|---|
| Qwen3.5-397B-A17B | +7.21 | [4.04, 10.31] |
| GPT-5.6 Sol | +14.38 | [10.50, 18.42] |
| DeepSeek-V4-Pro | +4.19 | [-0.40, 8.75] |
| Kimi-K3 | +13.65 | [8.94, 18.56] |
| GLM-5.2 | +18.17 | [14.67, 21.75] |
| GPT-5.5 | +16.90 | [13.19, 20.63] |
Table 3: Chess explainability gains (BinProp minus Baseline). All values are in percentage points. Each row reports a 95% bootstrap confidence interval.
| Model | Method | Mean Explainability | Gain |
|---|---|---|---|
| DeepSeek-V4-Pro ( DeepSeek-AI, 2026 ) | Baseline | 0.051 | +0.098 |
| BinProp (ours) | 0.149 | ||
| Qwen3.5-397B-A17B ( Qwen Team, 2026 ) | Baseline | 0.109 | +0.154 |
| BinProp (ours) | 0.264 | ||
| Kimi-K3 ( Kimi Team and others, 2026 ) | Baseline | 0.141 | +0.134 |
| BinProp (ours) | 0.274 |
Table 4: Mean explainability on Go test positions for all eight models, using maximum reasoning for both Baseline and BinProp explanations.
Figure 6: Baseline and BinProp explanations for the six highest-scoring Go models. Each panel shows one held-out positive position; explainability is measured over eight positive, eight negative, and eight nearby-negative positions. Bars show KataGo’s evaluation.
Figure 7: Eight-ply continuations after P10, E12, G4, and D12, with three BinProp explanations. Bars show KataGo’s estimated lead for Black immediately after each candidate move.
| Model | Method | Mean Explainability | Gain |
|---|---|---|---|
| Fable 5 ( Anthropic, 2026a ) | Baseline | 0.355 | -0.114 |
| BinProp (ours) | 0.241 | ||
| DeepSeek-V4-Pro ( DeepSeek-AI, 2026 ) | Baseline | 0.130 | +0.222 |
| BinProp (ours) | 0.352 | ||
| Qwen3.5-397B-A17B ( Qwen Team, 2026 ) | Baseline | 0.378 | +0.023 |
| BinProp (ours) | 0.401 |
Table 5: Mean explainability on the draughts test set for Baseline and BinProp explanations.
Figure 8: Baseline and BinProp explanations on held-out draughts positions. Examples are sampled without ranking their held-out scores. Bars show Ponder’s evaluation in uncrowned-piece equivalents.
| Model | Method | Mean Explainability | Gain |
|---|---|---|---|
| Qwen3.5-397B-A17B ( Qwen Team, 2026 ) | Baseline | 0.204 | +0.029 |
| BinProp (ours) | 0.233 | ||
| DeepSeek-V4-Pro ( DeepSeek-AI, 2026 ) | Baseline | 0.086 | +0.207 |
| BinProp (ours) | 0.293 | ||
| GPT-5.5 ( OpenAI, 2026b ) | Baseline | 0.297 | +0.016 |
| BinProp (ours) | 0.313 |
Table 6: Mean explainability on the shogi test set for Baseline and BinProp explanations.
Figure 9: Baseline and BinProp explanations on held-out shogi positions. Examples are sampled without ranking their held-out scores. Bars show YaneuraOu’s evaluation in pawns.
Figure 10: Fedoseev–Carlsen (2014).
Figure 11: Nakamura–Firouzja (2024).
Figure 12: Kramnik–Kasparov (1998).
Figure 13: Kasparov–Kramnik (1994).
Figure 14: Carlsen–Leko (2007).
Figure 15: Reshevsky–Fischer (1970).
Figure 16: Redolfi–Fischer (1960).
Figure 17: Anand–Erigaisi (2025).
Figure 18: Nakamura–Anand (2025).
| Motif | Description | |
|---|---|---|
| Material advantage or hanging enemy piece | The side to move is at least a pawn ahead in material, or material is level while an enemy minor piece or better hangs to a free or favorable capture. | 56.1 |
| Consolidated queenless material lead or dual margin-trump parity edge | Not in check, with no unit hanging or attacked by a cheaper piece, a calm own king ring and no advanced enemy passer, the side to move is either a clean point or more up in a fully queenless position (a single-pawn edge requiring one comparative trump) or at exact material parity holding two or more margin trumps among passers, pawn health, rook activity, bishop pair, outposts, space, development and king quality. | 16.3 |
| Concrete target with safe camp at material parity or better | Not behind in material, not in check, and with none of its own pieces profitably capturable, the side to move owns a concrete target: a static-exchange-winning capture, a besieged weak enemy pawn, exclusive heavy-piece infiltration, a safe advanced passer, a trapped ventured enemy minor, or a dominant attack on the enemy king ring. | 16.2 |
| Positional edge at level material | With no material advantage, the side to move holds a clear positional plus from safe piece mobility, pressure on the enemy king zone, passed pawns, open rook files, king shelter and the bishop pair, while none of its own pieces are loose. | 12.0 |
| Space, development and initiative dominance near material parity | With material within a fraction of a pawn, the side to move holds a broad concrete edge combining space, central control, development lead, pressure on enemy pieces, king safety, passed pawns and rook placement. | 10.2 |
| Pawn health and piece activity superiority at equal material | Material is exactly equal but the side to move has clearly healthier pawns, better passed pawns, more active rooks and outposts, a safer king and more pressure on the enemy king zone. | 8.8 |
Table 7: Chess motifs discovered by Fable 5 .
| Motif | Description | |
|---|---|---|
| Material advantage for the side to move | The side to move has a net material edge of at least one pawn using standard piece values. | 48.4 |
| Positional dominance without material deficit | The side to move wins the overall balance of positional features (piece mobility, space, central control, passed pawns, pawn health, bishop pair, rook files, outposts) while not being behind in material. | 34.8 |
| Permanent holes in the enemy camp | The side to move controls clearly more permanent weak squares deep in the opponent half - squares that no enemy pawn can ever attack again and that it already covers with at least as many units as the defender - than the opponent controls in its own half, while not being down material. | 18.8 |
| Superior pawn structure and passed pawn edge | The side to move has a clearly healthier pawn skeleton than the opponent - more or further advanced passed pawns and fewer isolated, doubled, backward or scattered pawns - while not being down significant material. | 15.4 |
| Enemy king stranded without shelter | With plenty of pieces still on the board and no material deficit for the side to move, the opponent’s king is far worse sheltered than the king of the side to move: it sits on or near the central files, has lost the right to castle without reaching a corner, has drifted off its home rank, is missing the pawns that should stand in front of it, stands beside open or semi-open files, and/or is surrounded by enemy pieces. The side to move owns real attacking material (a queen or rook plus other pieces), so the exposed king is a concrete target. | 13.1 |
| Safe piece infiltration in enemy territory | The side to move has clearly more securely posted active pieces inside the opponent half - units that no enemy pawn can hit now or after one push, that cannot be profitably captured, and that still have targets or free squares - than the opponent has in its own half, while not being down material. | 10.8 |
Table 8: Chess motifs discovered by Opus 5 .
| Motif | Description | |
|---|---|---|
| Material surplus | The side to move has at least a pawn of material advantage, with a small bonus for the bishop pair. | 48.3 |
| Material-parity king hook pressure | With material roughly balanced, the side to move has pressure against a home-rank central king or a castled king shelter hook. | 8.7 |
| Balanced castling lead and invader tempo | Without a large material edge, the side to move has a castling/development initiative, a safe major-piece invader, or a protected tempo on an overextended queen. | 8.7 |
| Secure central grip and pawn targets | With material roughly balanced, the side to move has a safe king and either a supported central pawn grip or clearly better pressure on weak enemy pawns. | 8.4 |
| Overloaded central shield and structural edge | The side to move has a coordinated attack on a pinned central king shield, a safe knight fork, or a queenless pawn-structure edge at material parity. | 7.1 |
| Wing pawn tempo and central knight wedge | The side to move has a protected wing-pawn tempo on an advanced piece, a complementary fifth-rank knight wedge, or a queen-and-pawn phalanx against a central king. | 7.0 |
Table 9: Chess motifs discovered by GPT-5.5 .
| Motif | Description | |
|---|---|---|
| Minority pawn majority lever | The side to move has more pawns than the opponent on either the same-file center pair (d+e files) or on both wings combined (a-c plus f-h files), with total pawn count at least equal, indicating a lever-ready structural edge. | 34.6 |
| Material advantage two or more | The side to move has at least two points more total piece value (pawn=1, knight/bishop=3, rook=5, queen=9) than the opponent. | 15.3 |
| Pawn island advantage | The side to move has fewer pawn-island groups (maximal sets of pawns on contiguous files) than the opponent, indicating a more cohesive pawn structure. | 15.3 |
| Mobility advantage over opp | The side to move has at least five more legal moves than the opponent. | 13.0 |
| Central piece value lead | The side to move has at least two more total piece value on the central 4x4 box (files c through f, ranks 3 through 6) than the opponent. | 10.9 |
| Development edge advantage | The side to move has strictly fewer undeveloped (back-rank) knights or bishops than the opponent, indicating superior piece development and initiative. | 5.4 |
Table 10: Chess motifs discovered by GLM-5.2 .
| Motif | Description | |
|---|---|---|
| Material edge with activity tiebreak | The side to move has more total piece value than the opponent, or with equal material attacks more enemy pieces than the opponent attacks. | 65.2 |
| Healthier pawn structure | The side to move has a clearly sounder pawn structure, with at least two fewer isolated, doubled, or backward pawns than the opponent. | 7.2 |
| Development lead on even terms | With material roughly equal, the side to move has a clear development lead: at least two more units of development (minor pieces off their back rank plus a king tucked off the central file) than the opponent. | 5.8 |
| Enemy-half pawn space edge | The side to move’s pawns attack at least three more distinct squares on the opponent’s half of the board than the opponent’s pawns attack on the side-to-move’s half, reflecting a clear space advantage. | 3.0 |
| Accumulated positional pluses score | The side to move has at least three points of positional advantages such as the bishop pair, pinned enemy pieces, advanced passed pawns, rooks on open files, and the safer king. | 2.9 |
| Bishop pair advantage | The side to move owns the bishop pair (two or more bishops) while the opponent has at most one bishop, a lasting positional edge. | 2.6 |
Table 11: Chess motifs discovered by Kimi-K3 .
| Motif | Description | |
|---|---|---|
| King safety pawn storm | The side to move has more pawns on the wing where the opponent king is located than the opponent has on the side-to-move king wing, indicating a pawn storm advantage. | 16.8 |
| King pawn shield intact | The side to move has more pawns on the files adjacent to its own king than the opponent has near theirs. | 12.2 |
| King attack pressure | The side to move attacks more squares adjacent to the opponent king than the opponent attacks adjacent to the side-to-move king, with a margin of at least 2. | 12.0 |
| Dominant center control | The side to move controls more central squares (d4, e4, d5, e5) than the opponent. | 9.5 |
| Development lead | The side to move has more pieces (knights and bishops) off their back rank than the opponent, indicating a development advantage. | 7.5 |
| Fewer pawn islands | The side to move has fewer pawn islands than the opponent, indicating a healthier pawn structure. | 6.9 |
Table 12: Chess motifs discovered by DeepSeek-V4-Pro .
| Motif | Description | |
|---|---|---|
| Net material advantage | The side to move has a higher total material value than the opponent. | 56.1 |
| Healthy mobile wing majority | With balanced non-pawn forces and no advanced passer, the side to move has a healthier connected wing majority than the opponent’s best countermajority. | 17.7 |
| Level-material mobility dominance | With material values level, the side to move has substantially greater effective mobility for its non-pawn pieces. | 9.0 |
| Safe forcing check against an exposed central king | With material roughly level, the side to move can give a non-sacrificial forcing check to an exposed king on the central files. | 4.5 |
| Unopposed bishop pair at nominal parity | At equal conventional material with matched minor-piece counts, the side to move has bishops on both color complexes while the opponent does not. | 4.2 |
| Durable material-reversing capture | With no nominal material edge and differing force inventories, the side to move can capture into a material lead that survives every immediate countercapture. | 3.8 |
Table 13: Chess motifs discovered by GPT-5.6 Sol .
| Motif | Description | |
|---|---|---|
| Material advantage | Side to move has more total material value than the opponent. | 56.1 |
| Pawn majority on wings | Side to move has a pawn majority on at least one wing (queenside or kingside) compared to opponent. | 15.5 |
| Fewer pawn islands | Side to move has fewer pawn islands than the opponent, indicating better pawn structure. | 15.3 |
| Mobility advantage | Side to move has more legal moves available than the opponent. | 14.7 |
| Center control | Side to move attacks more center squares (d4, d5, e4, e5) than the opponent. | 11.6 |
| Rook open or semi open file | Side to move has at least one rook on a file with no opponent pawns (open or semi-open file). | 9.8 |
Table 14: Chess motifs discovered by Qwen3.5-397B-A17B .
More in this topic
Concept-Based Explanations
Concept-based explanations make AI decisions legible through human-understandable features and test how those features affect predictions.
See all 72 papers