cs.CLNov 7, 2026
SaveCan small transformers learn the language of Coxeter groups?
Organizations: Department of Mathematics, Boston College, Chestnut Hill, MA 02467
Abstract
The field of mechanistic interpretability is well-suited to collaborative projects between mathematicians and AI researchers. This work describes one such project. We trained a small transformer to learn the language of the Coxeter group affine A2, best known as the symmetries of the tiling of the Euclidean plane by equilateral triangles. We then formulated and proved a word-theoretic conjecture about affine A2 after observing the attention patterns of the trained model.
Figures & tables
Figure 1: ShortLex words in , visualized as paths of reflections in the –tiling on which it acts
Figure 2: Right descent set prediction task: average attention paid by the final token in each of the attention heads, averaged over ShortLex normal form words of length in .
Figure 3: Left descent prediction: Average attention of final token of length normal form word
Figure 4: Model performance for the left descent task on random words of length in .
Figure 5: Model performance for the left descent task on reduced words of length in
Figure 6: Model accuracy for the left descent task on ShortLex words in of length up to . The model achieved nearly test accuracy (left), and performance remained strong (right) when all but the first tokens were masked, supporting Proposition 2 .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: The gallery associated to for , with fundamental alcove (associated to the identity element of ) marked in purple, and simple roots marked in red, green, blue, respectively. Note that at the endpoint of the gallery, is negative, confirming that , and are positive, confirming that .
Figure 8: The gallery associated to for . Note that the markings on the alcove associated to the endpoint of this gallery are a reflected version of the initial alcove, along the hyperplane marked in bold, and both are therefore positive. If for non-empty , then the reflecting hyperplane will still dominate some minimal root in the fundamental alcove, and the conclusion will remain the same.
Figure 9: Model accuracy for the left descent task on ShortLex words in when a subset of input tokens is masked during training. Test and train accuracy were nearly on the left descent task when no tokens were masked, stayed at nearly when all but the first tokens were masked, and dropped to below on the test set when the first tokens were masked.
Figure 10: Model accuracy for the left descent task on random words in .
Figure 11: Model accuracy for the left descent task on reduced words in
Figure 12: Misclassification rates by index for random length words in
Figure 13: Misclassification rates by index for reduced length words in
More in this topic
Transformer Interpretability
Transformer interpretability studies how attention, feed-forward layers, and hidden representations support model predictions and behavior.
See all 356 papers