Organizations: Shanghai Innovation Institute, Shanghai, China · School of Computer Science, Fudan University, Shanghai, China · School of Mathematics and Statistics, Xi’an Jiaotong University.
While modern transformer neural networks achieve grandmaster-level performance in chess and other reasoning tasks, their internal computation process remains largely opaque. Focusing on Leela Chess Zero (LC0), we introduce a sparse decomposition framework to interpret its internal computation by decomposing its MLP and attention modules with sparse replacement layers, which capture the primary computation process of LC0. We conduct a detailed case study showing that these pathways expose rich, interpretable tactical considerations that are empirically verifiable. We further introduce three quantitative metrics and show that LC0 exhibits parallel reasoning behavior consistent with the inductive bias of its policy head architecture. To the best of our knowledge, this is the first work to decompose the internal computation of a transformer on both MLP and attention modules for interpretability. Combining sparse replacement layers and causal interventions in LC0 provides a comprehensive understanding of advanced tactical reasoning, offering critical insights into the underlying mechanisms of superhuman systems. Our code is available at https://github.com/JacklE0niden/Leela-SAEs.
Figures & tables
Figure 1: An overview of our approach for interpreting LC0. We use Transcoders and Lorsas as replacement layers to sparsely decompose the model’s MLP and attention module, identifying interpretable reasoning pathways through feature-steering interventions. By analyzing sparse feature activations along these pathways, we reveal an interpretable computational process underlying the model’s decision-making for any input position.
Figure 2: Visualization interface for feature interpretability. Red squares indicate the spatial activations of specific features. For Lorsa features, each activated square is associated with a z -pattern, highlighting its attentional focus (e.g., the black knight, marked in blue). Purple arrows connect each activated square to the target square to which its activation is highly attributed.
Feature Type
Activation Consistency
Complexity
Transcoder
4.15
2.25
Lorsa
3.73
2.54
MLP
2.05
1.45
Table 1: Interpretability assessment. Human evaluation scores (1–5 scale) by three player annotators (Elo 2100+, 2000+, and 1000+), measuring activation consistency and perceived complexity across different feature types.
Feature ID
Interpretation
Verification Rule
Precision
Recall
Tc.0.5593
[Det]Opp.Knight
Act. @ Opp.Knight
100%
100%
Tc.0.12663
[Det]OwnPawn
Act. @ OwnPawn
100%
100%
Lorsa.0.13740
[Det]Opp.Queen ← OwnQueen
Act. @ Opp.Queen ← OwnQueen
98.5%
100%
Tc.14.120
[Src]Source square
Act. @ Top-1 predicted source
88.0%
1.7%
Tc.14.1214
[Tgt]Target square
Act. @ Top-1 predicted target
99.2%
78.7%
Tc.14.8947
[Val]High Value
Act. @ Every squares on chessboard positions predicted value >0.8
89.9%
41.6%
Table 2: We present a categorized collection of interpretable Transcoder and Lorsa features, organized by coarse semantic function rather than architectural layer. Features categories are color-coded as follows: Detection Features([Det]) , Source/Target Square Features([Src]/[Tgt]) , Value Features([Val]) , Capture Features([Cap]) , Complex Tactic Features([Tac]) , Spatial Relation Features([Spa]) , Piece Movement Features([Mov]) . Each feature is validated using a manually designed rule, and precision and recall are computed over training dataset. Detailed explanations of the interpretation language and verification rules are provided in Appendix D . We cap the analysis of the 1000 activation times per feature in a random dataset and compute precision and recall based on these samples. Detailed explanations of the interpretation language and verification rules are provided in Appendix D , where we present an illustrative visualization with a single representative feature per category to convey the core semantic pattern in Appendix E . Bolded features indicate those used in the following case study. While some features exhibit lower recall due to feature splitting ( Chanin et al., 2025 ) —where descriptive rules are coarser than actual activations—consistently high precision, particularly for rare tactical patterns, demonstrates the validity of our interpretation.
Figure 3: A complete case study with illustration highlighting the key tactical structure discussed in this section. (A) Input position and extracted reasoning pathways. (B) Finding 1: Mechanistic validation of bishop-movement features that transfer bishop coverage information via targeted attention pattern masking. (C) Finding 2: Copying activation of Lorsa.0.7083 opponent’s rook coverage feature from g7 to h7 alters perceived rook defense and thus suppresses Qxh7+ . (D) Finding 3: The model’s detection of the g2 pawn triggers perceived defensive stability, reducing sensitivity to forced-win opportunities.
Feature
Δ Act.
Interpretation
Steered Feature
Lorsa.0.7083@h7
+1x (copy from g7 )
[Spa] Opponent Rook rank-wise defensive coverage
Downstream Features (affected by steering)
Lorsa.1.9198@h7
↓100%
Opponent knight blocks rook protection
Lorsa.1.15014@h7
↓100%
Diagonal-assisted queen checking threats near opponent king
Lorsa.1.16036@h7
↓100%
Multi-piece attack–defense around king-front pawns
Table 3: Causal effects of steering by injecting the opponent’s rook coverage feature Lorsa.0.7083 into h7 . After this intervention, the model perceives h7 as being defended by the black rook on f7 , which suppresses downstream features corresponding to related tactics.
Move
Orig.
w/o g2
w/o Tc.0.12663@14
Qxh7+
2.04 / 28.8%
3.59 / 91.7%
3.53 / 90.6%
fxg7+
2.26 / 35.8%
-2.29 / 0.26%
-0.79 / 1.20%
Qxd7
1.95 / 26.1%
-1.42 / 0.61%
-2.24 / 0.28%
Table 4: Top-3 move logit/probability distributions for the original position, the position without the g2 pawn, and the intervention that ablates Tc.0.12663@14 .
Subset
All
Confident
Confused
Same- source
baseline
Mean path overlap
7.05%
1.63%
8.39%
13.7%
0.17%
Table 5: Path overlaps under different evaluation settings, with values close to zero, suggesting that different moves arise from largely disjoint significant features.
Figure 4: Mean Path Cohesion and Path Coupling between feature sets of top-2 moves across different chess positions in the dataset.
Figure 5: Detailed analysis of feature distributions. (a) Entropy distribution across layers, indicating concentration at specific squares. (b) Spatial layout of features on the board.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Overview of the BT4 architecture, featuring SmolGen-enhanced MHSA for modeling chessboard positional relations and an attention-based policy head.
Puzzle set
Mate-in-1
Mate-in-2
Mate-in-3
Mate-in-4
Mate-in-5
All
Accuracy
0.996
0.970
0.890
0.776
0.707
0.932
Appendix
Table 6: BT4 puzzle accuracy without search on Lichess mate puzzles of varying tactical depth. We sample 20,000 positions for each subset from mate-in-1 to mate-in-4, and 5,000 positions for mate-in-5 due to limited data availability.
Figure 7: Faithfulness of sparse replacement layers. We report the L2 norm error ratio (left) and the explained variance EV (right) of Transcoder and Lorsa modules across transformer layers on a randomly sampled dataset.
Figure 8: Representative feature examples by category: (1) Detection feature [Det] activating on squares occupied by an opponent knight; (2) Source square feature [Src] activating on the square from which the predicted move originates; (3) Value-based feature [Val] activating on high-value positions on the board; (4) Capture feature [Cap] activating on squares from which a piece can capture an opponent’s rook; (5) Tactical feature [Tac] activating in scenarios where our rook is lost due to a check; (6) Spatial-related feature [Spa] activating on squares reflecting board-wide spatial patterns, e.g., squares of the same color as the king; (7) Piece movement feature [Mov] activating on squares representing the movement or coverage area of a piece, here the eight squares surrounding the king.
Figure 9: Effect on policy probability and logit of steering factor on feature activations. (a) Ten random activated features at each steering factor and corresponding change of policy probability and logit. (b) Ten random significant features at each steering factor and corresponding change of policy probability and logit.
Figure 10: Pearson correlation distribution of steering sensitivity. (Left) The impact of randomly selected features on the top policy probability within the [−2,2] steering range. (Right) The distribution of Pearson correlation coefficients. We observe significant clustering in the ranges [−1,−0.9] and [0.9,1] , with comparable counts.
Figure 11: The correlation of between feature activation and whether the exchanging-queen is protected in 1000 activation times with the exchanging-queen. The activation value exhibits a Pearson correlation of 0.52.
Lorsa.1.8260
Lorsa.7.14502
Lorsa.8.9586
Lorsa.1.8260
1.00
0.82
0.84
Lorsa.7.14502
0.82
1.00
0.97
Lorsa.8.9586
0.84
0.97
1.00
Random baseline: E[cosθ]=0.17
Appendix
Table 7: Cosine similarity matrix of decoder vectors for bishop-movement Lorsa features (after LC0 post-LayerNorm). High off-diagonal cosine similarities indicate a shared semantic direction across distributed layers.
Figure 12: samples with the highest activations of three bishop-move Lorsa features mentioned in Section 4 finding 1.
Figure 13: samples with the highest activations of features mentioned in Section 4 finding 2.
Figure 14: samples with the highest activations of features mentioned in Section 4 finding 3.
Figure 15: Reasoning pathway for a defensive decision. The model detects an early-layer threat from the opponent’s queen–rook coordination threatening at h7 , and consequently mobilizes a knight to block the mate-in-one threat. Key features involved in this defensive reasoning are visualized.
Figure 16: Reasoning pathway of a case where the model combines offensive and defensive reasoning. The figure shows the input chessboard position, the reasoning pathway supporting the selected policy Qc4+ , and the associated supernode interpretations. Two key features are visualized: Tc.1.12313 represents the awareness of opponent queen’s checkmate threats, and Lorsa.4.11791 exerting diagonal threats against the opposing king.
Figure 17: Reasoning pathway of look-ahead reasoning in a mate-in-two scenario. The reasoning pathway illustrates how the model integrates the immediate sacrifice move ( Ng6+ ) with the anticipation of a subsequent checkmate threat from the rook at h4 . Important correlated features with highly semantic interpretability are visualized below.
Figure 18: Reasoning pathway for a more complex position. BT4 correctly selects Ne5 with probability 42.7%. The pathway suggests that Ne5 opens the d-file for the rook on d1, creates mating pressure by enabling a future Qf7+ , and supports the development of Bg2 . After Ne5 , Bb7 no longer attacks the knight along the diagonal, and Black may respond with Qe7 .
Chess involves complex reasoning in a deterministic environment, which makes it a useful setting for studying the mechanisms of computation inside transformers. The Maia-3 chess transformer takes Elo, a measure of competitive chess skill, as an input to the pre-trained network, so we can vary the skill the network is conditioned on with no change to its weights. Here we investigate how turning this skill dial affects self-attention. Ablating every attention head at every Elo from 700 to 2500, we find 1) increasing skill pushes the causal center of mass of the computation deeper, monotonically, for every chess piece and move type we measured; 2) the depth migration is much greater for specific tactics, especially knight forks, than for other move types; 3) the migration consists of deeper heads getting recruited for more specialized computations while one shared shallow head keeps a roughly constant contribution. These results may shed light on how conditioning inputs redistribute computation in larger transformers.
David Litman
Computational Neurobiology Laboratory, Salk Institute
Chess has long served as a canonical testbed for artificial intelligence, but modeling approaches for its central tasks have diverged. Maximizing playing strength, predicting human play, and enabling interpretability are typically solved with disparate architectures, and these designs are often misaligned with the geometry of the domain. This raises the natural question of whether these objectives require separate modeling paradigms, or if there exists a single architecture that supports them simultaneously. We introduce Chessformer, a unified architecture that advances the state of the art on all three central goals in chess modeling. Chessformer is an encoder-only transformer that represents board squares as tokens, augments self-attention with a novel dynamic positional encoding called Geometric Attention Bias (GAB) that adapts to domain-specific geometry, and predicts actions with an attention-based source-destination policy head. We evaluate Chessformer on each front. First, we develop \maiathree, a family of models for human move prediction that reaches 57.1% move-matching accuracy, significantly surpassing the previous state of the art with fewer than a quarter of the parameters. Second, we integrate Chessformer into Leela Chess Zero, a leading open-source engine, adding over 100 Elo of playing strength and resulting in tournament victories over Stockfish in major computer chess competitions. Third, we show that Chessformer's square-token design makes attention patterns and activations directly attributable to board squares, enabling granular interpretability analyses that prior architectures do not naturally support. More broadly, our results demonstrate that aligning a model's tokenization, positional encoding, and output design with the underlying structure of a domain can yield simultaneous gains in performance, human compatibility, and interpretability.
Daniel Monroe, George Eilender, Philip Chalmers +2
Do transformers, when trained on sequential reasoning traces, build internal models of the underlying task? And if so, does the structure of those internal representations mirror the structure of the domain? We train an 8-layer transformer on Sudoku solving traces and perform a mechanistic analysis of its internal computation. We establish two results. First, the model builds a substructure world model: it does not represent the board state cell by cell, as a human analyst would expect, but organizes information around the rows, columns, and boxes that Sudoku's constraints act on. Second, we identify a naked-single circuit: a small set of dedicated neurons in the final MLP layer, each individually detecting when exactly one digit remains possible for a specific cell, and reliably promoting that digit. These findings show that the geometry of an emergent world model is shaped by the constraint algebra of the domain, not its surface presentation, and that the resulting decision circuit is sparse, monosemantic, and fully interpretable. More broadly, they demonstrate that mechanistic interpretability tools can recover an end-to-end algorithmic account of how a transformer solves a combinatorial reasoning task.