Organizations: University of California, San Diego · University of Melbourne · University of Virginia · Shanghai Jiao Tong University · East China Normal University · New York University
When a Go model predicts a move different from the expert's, coordinate distance measures how far the prediction lands from the reference. We introduce Transition Representation Consistency (TReC) to compare how the two moves change a board-state embedding. Both moves are executed from the same position under the game rules, and TReC is the cosine similarity between the two resulting representation changes. In experiments with a frozen KataGo network on 680 mid-to-late-game positions from five games, the gap between the highest and lowest candidate per position has a median of 0.01 to 0.08 for the direct successor-state cosine and 0.88 to 1.33 for TReC, depending on the evaluator layer. TReC follows coordinate distance only weakly: the rank correlation between a candidate's distance to the reference and its TReC, computed within each position, has a median of -0.19 over these positions.
Figures & tables
Figure 1: Two legal moves produce rule-executed successors of the same state. TReC compares their changes in a shared board-state representation hl , relative to the shared initial representation.
Figure 2: Two cosine scores over the unmasked intersections of one position (softmax-normalized, T=1 ; mean of three evaluator layers). Each panel has its own color scale, so colors show ranking within a panel. White locations are masked; the cross marks the reference. Squares mark the two unmasked intersections adjacent to the reference and the circle a candidate at distance 5.1 (Table 1 ).
TReC
Successor cosine
Candidate
bias2
act2
conv2p
bias2
act2
conv2p
∘d=5.1
0.943
0.962
0.973
0.999
0.996
1.000
□ below
0.891
0.895
0.972
0.998
0.986
0.999
□ right
0.755
0.831
0.966
0.992
0.983
0.999
Minimum
0.248
0.328
0.018
0.967
0.933
0.993
Median
0.639
0.649
0.817
0.989
0.971
0.998
Table 1: Cosines with the reference move in three evaluator layers for the position of Figure 2 . ∘ : the candidate at distance d=5.1 ; □ : the two adjacent candidates, by their side of the reference. Minimum and median are over the 250 candidates other than the reference.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure S1: All representation panels from the original heatmaps, arranged by representation: (a) direct successor-state similarity and (b) transition representation similarity. Each panel retains its original colors and independently scaled color bar; High and Low denote that panel’s local range. Colors encode softmax-normalized cosine scores at temperature 1 . White locations are masked and the cross marks the reference action. The complete original plots with their numerical axes and scales remain in the source figure files.
Domain terminology provides concise descriptions of game states and strategic relationships, but whether game-playing language models can accurately recognize these concepts and relate them to move evaluation remains unclear. We construct a controlled evaluation set centered on atari and its relationship to sente in Go, examining terminology recognition under local board changes and move evaluation under different strategic conditions. We conduct our tests on LoGos, a Go language model, after fine-tuning it with terminology and commentary supervision. The model struggles to recognize changes in the atari relationship and assess its strategic implications. Its terminology predictions are associated with move preferences even when those preferences contradict the reference evaluation. Our controlled tests provide a framework for examining whether language-based Go agents connect domain terminology with precise board relationships and strategic move values.
World models require state tracking, which is the ability to maintain a correct latent state across action sequences. Existing benchmarks are often synthetic or language-based, limiting their value as tests of structured state updates in realistic domains. We introduce Chess-World-Model, a large-scale state-tracking benchmark built from 10 million real chess games, where models predict the exact board state reached after a sequence of legal moves. Alongside a held-out real-game split, we include an out-of-distribution test set of uniformly random legal play both at its original trajectory lengths and after matching the human-game length distribution. This tests whether models learn the transition rules rather than shortcuts from common human positions. Prior theoretical and empirical work has shown that Transformers struggle to state-track, while input-dependent linear RNNs require expressive state-transition matrices to do so. We therefore benchmark a causal Transformer, block-diagonal SLiCE, Mamba-3, and Gated DeltaNet with negative eigenvalues under a matched interface and training protocol. The recurrent models strongly outperform the Transformer at approximately 3 and 8 million parameters. Real-game performance largely saturates by the 18 million parameter scale, but length-matched random evaluation continues to distinguish architectures through the largest tested scale of approximately 43 million parameters. Additionally, ablations show that less expressive state-transition mechanisms generally reduce performance on the out-of-distribution splits. Together, these results establish Chess-World-Model as a practical large-scale benchmark for state tracking that exposes failures model scale would otherwise conceal.
Benjamin Walker, Terry Lyons
Mathematical Institute, University of Oxford · Department of Mathematics, Imperial College London
Game AI results are comparable only when different implementations expose the same decision problem. Yet seemingly minor representation choices can silently change that problem by altering action probabilities or collapsing states with different outcomes. We study this issue through CoToan, an executable specification and representation contract for Vietnamese Mathematical Chess connecting the Game Description Language (GDL) with OpenSpiel. The game provides a useful case because a single capture may have multiple arithmetic justifications, while outcomes can depend on scoring rules and a finite playing horizon. CoToan therefore represents actions by their state transitions rather than by arithmetic witnesses, and treats time and victory configuration as part of task identity. Cross-representation tests show agreement in legal actions, successor states, and terminal outcomes across hundreds of sampled and constructed positions, complemented by exhaustive checks of all 32 operator masks and 512 opposing-material subsets. Controlled diagnostics further show that uniform witness-level sampling alters endpoint-action probabilities, while omitting remaining time or victory settings can merge states with different terminal behavior. These results show that action semantics, time, and victory configuration are not merely implementation details. They are part of the experimental task itself and must be represented explicitly for reproducible game AI.
Ha-Thanh Nguyen
College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam