Domain terminology provides concise descriptions of game states and strategic relationships, but whether game-playing language models can accurately recognize these concepts and relate them to move evaluation remains unclear. We construct a controlled evaluation set centered on atari and its relationship to sente in Go, examining terminology recognition under local board changes and move evaluation under different strategic conditions. We conduct our tests on LoGos, a Go language model, after fine-tuning it with terminology and commentary supervision. The model struggles to recognize changes in the atari relationship and assess its strategic implications. Its terminology predictions are associated with move preferences even when those preferences contradict the reference evaluation. Our controlled tests provide a framework for examining whether language-based Go agents connect domain terminology with precise board relationships and strategic move values.
Figures & tables
Figure 1: Examples for the Atari–Sente Contrast Test ( X/Y ) and the Non-Sente Atari Test ( Z ).
Position
A
B
C
X
48.0
52.9
1.8
Y
4.6
48.6
1.1
Z
3.8
47.9
—
Table 1: Mean KataGo reference win rates (%) for candidate moves, from the perspective of the player to move.
Prediction in X
Prediction in Y
A (%)
B (%)
Atari
Atari
58.1
7.2
Atari
Non- atari
14.6
0.4
Non- atari
Atari
1.5
0.6
Non- atari
Non- atari
25.8
91.8
Recognition accuracy in Z
59.0
87.0
Table 2: Terminology recognition results. The upper section reports joint predictions on 1,000 X/Y pairs; bold marks the correct combinations. The final row reports binary recognition accuracy on 500 Z positions.
Group size
Samples
Accuracy (%)
2
912
69.8
3
322
64.9
4
109
63.3
≥5
119
58.0
Table 3: Atari recognition by group size in stones, pooling 962 X and 500 Z positions with a single threatened group.
Reference relation
Coverage (%)
Cond. acc. (%)
V(X,A)>V(X,C)
24.8
97.2
V(X,B)>V(X,C)
21.1
90.0
V(Y,B)>V(Y,C)
18.7
89.8
V(X,A)>V(Y,A)
11.6
88.8
V(Y,B)>V(Y,A)
8.7
19.5
V(Z,B)>V(Z,A)
9.0
46.7
Table 4: Comparisons of candidate move values. Coverage is the proportion of cases where the two predictions fall on opposite sides of 50%. Conditional accuracy is the proportion of these cases that follow the reference ordering.
Position
Predicted term
Coverage (%)
Prefer A (%)
X
Atari
10.3
96.0
Non- atari
8.1
40.9
Y
Atari
10.9
90.8
Non- atari
5.4
50.0
Z
Atari
8.8
73.1
Non- atari
9.3
26.3
Table 5: Move preferences grouped by the predicted terminology for A , with coverage calculated within each group.