Induction Heads

Follow this topic Ask about this topic

Momentum

1 paper in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 12

All topics

Spatial Induction Heads: In-Context Learning of Multidimensional Cellular Automata
Oct 6, 2026Kimia Kazemian, Menghan Xu, John Thickstun +1In-Context Learning Induction Heads
Cornell University
Mirror, Mirror on the Wall: Prompt Echoing in Small Instruct Language Models
Sep 14, 2026Inez Okulska, Bartosz Naskręcki, Jan Piotrowski +1Data Contamination in Language Models Induction Heads
Centre for Credible AI, Warsaw University of Technology · Adam Mickiewicz University Poznan · Institute of Fundamental Technological Research, Polish Academy of Sciences
Induction in Both Directions: A Mechanistic Analysis of In-Context Learning in Masked Diffusion Language Models
Jul 17, 2026Andy Catruna, Emilian RadoiMasked Diffusion Models In-Context Learning
National University of Science and Technology POLITEHNICA Bucharest · Academy of Romanian Scientists
Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention
Jul 7, 2026Li HengyuRotary Positional Embeddings Attention Head Analysis
Institute for Solid State Physics, The University of Tokyo
Induction Heads Interpolate N-Grams
Jul 2, 2026Francesco D'Angelo, Oguz Kaan Yuksel, Swathi Shree Narashiman +1Transformer Interpretability In-Context Learning
1TML Lab, EPFL, Switzerland. · 2IIT Madras, India. · †Work done during an internship at the TML Lab, EPFL.
Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence
Jun 10, 2026Itay Lavie, Kirsten Fischer, Andrey Lekov +3Softmax Attention Phase Transitions
Racah Institute of Physics, Hebrew University of Jerusalem, Jerusalem 91904, Israel · John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, USA · Institute for Advanced Simulation (IAS-6), Computational and Systems Neuroscience, Jülich Research Center, Jülich 52428, Germany +3
When Do Attention Circuits Form? Developmental Trajectories of Capability and Attention-Sink Emergence Across Three 1B-ClassArchitectures
Jun 1, 2026Yongzhong XuLanguage Model Pretraining Transformer Interpretability
Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers
May 22, 2026Yongzhong XuTransformer Interpretability Circuit Discovery
INTRYGUE: Induction-Aware Entropy Gating for Reliable RAG Uncertainty Estimation
Mar 23, 2026Alexandra Kuleshova, Andrei Volodichev, Daria Kotova +1Hallucination Detection LLM Interpretability
Applied AI Institute
Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale
Feb 4, 2026Jonas Rohweder, Subhabrata Dutta, Iryna GurevychTransformer Interpretability Latent Variable Models
UKP Lab, TU Darmstadt
Predicting the Emergence of Induction Heads in Language Model Pretraining
Nov 21, 2025Tatsuya Aoyama, Ethan Gotlieb Wilcox, Nathan SchneiderLanguage Model Pretraining In-Context Learning
Department of Linguistics, Georgetown University.
Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Models
Oct 26, 2025Anooshka Bajaj, Deven Mahesh Mistry, Sahaj Singh Maini +2In-Context Learning Induction Heads
Department of Computer Science Indiana University Bloomington

Oct 6, 2026, cs.LGJ/K move · Enter open · S save

Spatial Induction Heads: In-Context Learning of Multidimensional Cellular Automata

Kimia Kazemian, Menghan Xu, John Thickstun, Sarah Dean

Cornell University

In-Context Learning Induction Heads Transformer Attention Transformer Interpretability Cellular Automata

Open paper AskPDF arXiv

Induction heads provide a mechanistic account of in-context learning in sequential data, but existing theory largely assumes that the context relevant to a prediction forms a contiguous block. In multidimensional data, serialization breaks this assumption by scattering spatial neighbors across distant positions in the token sequence. We study how transformers overcome this routing problem in multidimensional stochastic and deterministic cellular automata, where each trajectory is generated by an unknown local rule and presented as a flattened sequence without an explicit coordinate-based spatial inductive bias. We introduce spatial induction heads, two-layer gather-and-match circuits in which the first layer reconstructs the relevant spatial neighborhood and the second matches the resulting configuration against earlier occurrences. We give two explicit realizations of the gather and show that the positional dimension required for spatial routing depends only on the local neighborhood and spatial dimension, not on grid volume or trajectory horizon. We further construct a matching layer which implements Bayesian counting. The end-to-end circuit can approximate the Bayesian posterior arbitrarily closely for stochastic rules and can predict exactly for deterministic rules. Empirically, trained two-layer transformers generalize to unseen rules in one and two dimensional settings, achieving near-perfect deterministic rollouts and less than 0.005 nats KL from the Bayes-optimal predictor on stochastic rules. Attention patterns and layerwise probes align with the predicted gather-and-match computation, providing mechanistic evidence for spatial induction in trained transformers.

Momentum

Related topics

Latest papers 12