cs.CLNov 7, 2026
SaveCan small transformers learn the language of Coxeter groups?
Organizations: Department of Mathematics, Boston College, Chestnut Hill, MA 02467
Abstract
The field of mechanistic interpretability is well-suited to collaborative projects between mathematicians and AI researchers. This work describes one such project. We trained a small transformer to learn the language of the Coxeter group affine A2, best known as the symmetries of the tiling of the Euclidean plane by equilateral triangles. We then formulated and proved a word-theoretic conjecture about affine A2 after observing the attention patterns of the trained model.
Figures & tables
Figure 1: ShortLex words in , visualized as paths of reflections in the –tiling on which it acts
Figure 2: Right descent set prediction task: average attention paid by the final token in each of the attention heads, averaged over ShortLex normal form words of length in .
Figure 3: Left descent prediction: Average attention of final token of length normal form word
Figure 4: Model performance for the left descent task on random words of length in .
Figure 5: Model performance for the left descent task on reduced words of length in
Figure 6: Model accuracy for the left descent task on ShortLex words in of length up to . The model achieved nearly test accuracy (left), and performance remained strong (right) when all but the first tokens were masked, supporting Proposition 2 .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: The gallery associated to for , with fundamental alcove (associated to the identity element of ) marked in purple, and simple roots marked in red, green, blue, respectively. Note that at the endpoint of the gallery, is negative, confirming that , and are positive, confirming that .
Figure 8: The gallery associated to for . Note that the markings on the alcove associated to the endpoint of this gallery are a reflected version of the initial alcove, along the hyperplane marked in bold, and both are therefore positive. If for non-empty , then the reflecting hyperplane will still dominate some minimal root in the fundamental alcove, and the conclusion will remain the same.
Figure 9: Model accuracy for the left descent task on ShortLex words in when a subset of input tokens is masked during training. Test and train accuracy were nearly on the left descent task when no tokens were masked, stayed at nearly when all but the first tokens were masked, and dropped to below on the test set when the first tokens were masked.
Figure 10: Model accuracy for the left descent task on random words in .
Figure 11: Model accuracy for the left descent task on reduced words in
Figure 12: Misclassification rates by index for random length words in
Figure 13: Misclassification rates by index for reduced length words in
More in this topic
Transformer Interpretability
Transformer interpretability studies how attention, feed-forward layers, and hidden representations support model predictions and behavior.
See all 356 papersA central question in the debate over large language models is whether transformers can learn rules they have never seen, or whether they can only interpolate: predict new cases from their similarity to training examples. We test this in a controlled setting where interpolation provably fails, so success can only come from computation beyond interpolation. We train small transformers to predict the rollout of a cellular automaton whose update rule is pure XOR, and remove one entry of the rule's truth table from all direct supervision. The missing entry's output is never shown to the model; its only trace is indirect, as wrong values corrupt visible predictions at later timesteps. Because XOR parity flips whenever one input bit is changed, every one-bit neighbour of the missing entry carries the opposite label, and we prove that similarity-based predictors, including nearest-neighbour, kernel, and Gaussian-process methods, are forced to the wrong answer. A two-layer transformer can nevertheless recover the missing entry, and circuit extraction confirms it computes XOR exactly. Ablations show the recovery depends on gradient signal propagating through multi-step prediction, and a second, structurally unrelated benchmark on symbolic operator chains exhibits the same capacity under ordinary autoregressive training. Together with a constructive proof that a standard transformer block can implement exact local Boolean rules, these results provide an existence proof that transformers can learn rule structure not directly observed in training and express it explicitly. This rules out the strongest architectural form of the interpolation-only account, the claim that transformers cannot in principle discover and communicate unseen rules, while leaving open when such behaviour arises in large-scale language training.
Transformers Provably Learn to Internalize Chain-of-Thought
Chain-of-Thought (CoT) prompting substantially improves the sample efficiency of transformers, reducing the complexity of tasks like parity learning from exponential to polynomial in the input length. However, generating explicit reasoning steps at inference is computationally expensive. Implicit Chain-of-Thought (ICoT) has emerged as a promising empirical remedy that trains models to internalize intermediate steps within their hidden states, but its theoretical foundations remain poorly understood. We give the first theoretical analysis of ICoT, proving that an -layer transformer trained under our proposed Log-ICoT curriculum learns -parity with samples and training stages. This matches the sample efficiency of explicit CoT while eliminating its inference overhead, and extends prior one-layer parity guarantees to multi-layer architectures. Compared to standard ICoT, which removes thinking tokens one at a time, Log-ICoT removes them in geometric chunks, reducing the number of stages from linear in to logarithmic. Experiments on multi-layer transformers confirm the theory and visualize how reasoning is progressively absorbed into deeper layers.
Fully Interpretable Minimal Transformers: From Geometry to Algorithm
We present a framework for building and interpreting minimal transformer models. By constraining a transformer's embedding dimension and head size to 2, we enable full two-dimensional visualization of its internal representations. Embeddings, query/key/value transforms, attention outputs, residual streams, and decision boundaries can all be seen directly. Our central claim is that the learned geometry implies an algorithm; the arrangement of points and boundaries in R^2 can be read as a step-by-step procedure. We train a transformer on a simple task where it must produce the most recently observed even number whenever the '+' operator appears in a sequence of digits. Once trained, we visually walk through every step of the transformer's computation. We show how the model embeds the tokens and their respective positions in the sequence, transforms them via the Q, K, and V matrices, uses the dot product between the Q and K representations to form the attention matrix, and uses the attention matrix to select values that move the representation of each input token to the region of the domain of the output layer that will correctly predict the next token. We introduce a suite of interpretability visualizations that make the algorithmic interpretation of this procedure explicit. Our framework offers a pedagogical and experimental testbed to explore how transformers use informational geometry to implement next-token prediction.