Generative models increasingly simulate users at different skill levels in education and AI evaluation. Yet it remains unclear whether lower-skill behavior reflects less internal knowledge or a different use of retained knowledge. This distinction is difficult to study in natural language, where proficiency and relevant knowledge are hard to measure. In this work, we study Maia-2, a chess transformer that predicts human moves conditioned on player skill, using quantitative skill ratings and deterministically labeled chess concepts. We hold board positions and model weights fixed and analyze positions where skill conditioning changes the prediction from a blunder to the best move. First, we measure encoded knowledge by training probes to predict the presence of 172 chess concepts from Maia-2's internal states. Probe accuracy remains nearly constant across skill conditions, and probes trained at one level transfer accurately to the others. To test whether stronger decisions can be recovered from this retained knowledge, we fine-tune only Maia-2's final output layer, raising best-move accuracy in the lowest skill band from 14.4% to 82.8%. Training this layer only on highest-skill representations retains most of the improvement at lower levels, showing that recovery does not require direct training on lower-skill states. Steering concept-relevant features learned by a sparse autoencoder also improves lower-skill decisions while leaving probe-measured concept encoding practically unchanged. These results support an account in which skill conditioning changes how retained concept knowledge influences move selection, showing that weaker behavior can understate the knowledge available within a model.
Figures & tables
Figure 1: Two hypotheses for novice-like behavior: less available knowledge (H1) or a different use of retained knowledge (H2).
Figure 2: An illustrative example of Maia-2’s skill-level adaptation. (A) The puzzle position and its relevant theme. (B) The consequences of the blunder and best move. (C) Probe-measured encoding of the relevant threat. (D) Maia-2’s probabilities for the two moves across skill levels.
Figure 3: Linear probe accuracy across skill levels for eight representative concepts and the mean over all 172 oracle concepts.
Baseline
Steering (Random)
Steering (Feature)
Head SFT
Best-Move Accuracy
<1200
0.1440.036
0.1630.042
0.4270.086
0.8280.090
1200–1399
0.4430.078
0.4560.059
0.5730.065
0.8540.078
1400–1599
0.6210.072
0.6290.051
0.6980.052
0.8700.070
1600–1799
0.7790.053
0.7810.036
0.8190.035
0.8830.068
1800–1999
0.8940.030
0.8940.020
0.9070.019
0.9120.059
Table 1: Maia-2 performance on Blundered Transitional positions. STD is the sample standard deviation of five band means; other subscripts show SD across concepts.
Figure 4: Best-move accuracy on target versus other concepts; error bars show 95% bootstrap CIs across concepts.
Figure 5: Best-move accuracy for policy heads trained on all skill conditions or only the highest-skill condition.
True if the opponent has a piece on square sq with strictly more active-player attackers than opponent defenders. Instantiated for each of the 64 squares.
Square Control — Opponent (64 concepts)
defensive_threat_{sq} sq ∈ {a1, …, h8}
True if the active player has a piece on square sq with strictly more opponent attackers than own defenders. Instantiated for each of the 64 squares.
King and Queen Safety (4 concepts)
king_danger [m | o]
True if 5 or more squares adjacent to the side’s king are attacked by the other side.
Appendix
Table 2: Glossary of the 172 binary oracle chess concepts.
Candidate transitional positions
2.28M
Blundered transitional positions
1.15M
Training split
0.80M
Validation split
0.11M
Test split
0.23M
Appendix
Table 3: Dataset construction and current split sizes.
Figure 7: Probe accuracy across skill levels for all 44 king and queen safety, tactical pattern, and positional structure concepts.
Figure 8: Probe accuracy across skill levels for all 64 per-square control concepts from the active player’s perspective.
Figure 9: Probe accuracy across skill levels for all 64 per-square control concepts from the opponent’s perspective.
Figure 13
Figure 12: Maia-2’s concept encoding and best-move accuracy on the same positions with target concepts (A), and recovery under feature steering and Head SFT across skill levels (B).
Figure 13: Probe accuracy before and after SAE feature steering across 11 skill levels for eight representative concepts and the 71-concept average.
Figure 14: Training diagnostics and predictive fidelity for Maia-2 SAEs.
Chess involves complex reasoning in a deterministic environment, which makes it a useful setting for studying the mechanisms of computation inside transformers. The Maia-3 chess transformer takes Elo, a measure of competitive chess skill, as an input to the pre-trained network, so we can vary the skill the network is conditioned on with no change to its weights. Here we investigate how turning this skill dial affects self-attention. Ablating every attention head at every Elo from 700 to 2500, we find 1) increasing skill pushes the causal center of mass of the computation deeper, monotonically, for every chess piece and move type we measured; 2) the depth migration is much greater for specific tactics, especially knight forks, than for other move types; 3) the migration consists of deeper heads getting recruited for more specialized computations while one shared shallow head keeps a roughly constant contribution. These results may shed light on how conditioning inputs redistribute computation in larger transformers.
David Litman
Computational Neurobiology Laboratory, Salk Institute
We present ChessMimic, a system of three small encoder-only transformers - for move, thinking-time, and outcome prediction - conditioned on the position, recent move history, player rating, and clock state. We fit a separate instance of each model per 100-Elo rating band, trading parameter efficiency for sharper per-skill calibration. On a held-out month-wide slice of Lichess Rated Blitz games ChessMimic's human move prediction accuracy outperforms Maia-2 in every Elo band. Compared to Maia-3, our 9M parameter model's accuracy sits between Maia-3-5M and Maia-3-23M without the additional complexity of Geometric Attention Bias. In addition to the move matching model, we also train a game outcome model that conditions not only on the position, but also player ratings, time control, and remaining clock times. The outcome model achieves an AUC of 0.78 out of sample, beating Maia-2 as well as logistic regressions based on material, ratings, and clock time. Finally, we train a clock model that predicts human thinking times. The clock model provides a usable but non-SOTA per-ply think-time signal under ALLIE-style filters (Pearson r = 0.41, Spearman rho = 0.50, MAE 4.10 s, against ALLIE's reported r = 0.70), with the residual gap concentrated in per-position bucket sharpness rather than bucket-marginal calibration. A public demo is at 1e4.ai and we release code, per-band weights, and the C++ data-filter pipeline code in GitHub.
A human-like chess engine should mimic the style, errors, and consistency of a strong human player rather than maximize playing strength. We show that training from move sequences alone forces a model to learn two capabilities: state tracking, which reconstructs the board from move history, and decision quality, which selects good moves from that reconstructed state. These impose contradictory data requirements: low-rated games provide the diversity needed for tracking, while high-rated games provide the quality signal for decision learning. Removing low-rated data degrades performance. We formalize this tension as a dual-capability bottleneck, P <= min(T,Q), where overall performance is limited by the weaker capability. Guided by this view, we scale the model from 28M to 120M parameters to improve tracking, then introduce Elo-weighted training to improve decisions while preserving diversity. A 2 x 2 factorial ablation shows that scaling improves tracking, weighting improves decisions, and their combination is superadditive. Linear weighting works best, while overly aggressive weighting harms tracking despite lower validation loss. We also introduce a coverage-decay formula, t* = log(N/kcrit)/log b, as a reliability horizon for intra-game degeneration risk. Our final 120M-parameter model, without search, reached Lichess bullet 2570 over 253 rated games. On human move prediction it achieves 55.2% Top-1 accuracy, exceeding Maia-2 rapid and Maia-2 blitz. Unlike position-based methods, sequence input naturally encodes full game history, enabling history-dependent decisions that single-position models cannot exhibit.
Quanhao Li, Wei Jiang
Abbey Park High School · Shanghai Soong Ching Ling School