Induction Heads

Follow this topic Ask about this topic

Induction heads help transformers use earlier examples by matching a repeated sequence and retrieving what came next, a behavior central to in-context learning and copying. Researchers study them with synthetic repetition and Markov tasks, mechanistic analyses of attention circuits, and causal tests such as head ablation. These methods connect induction to serial recall, prompt echoing, and retrieval in text, speech, and biological sequences. Evidence supports a causal role for specialized heads, while how their formation depends on training data, positional schemes, and access to context remains an open question.

Latest papers

17 papers1 paper in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Spatial Induction Heads: In-Context Learning of Multidimensional Cellular Automata
Oct 6, 2026Kimia Kazemian, Menghan Xu, John Thickstun +1In-Context Learning Induction Heads
Cornell University
Phase Space Attention:A Hairer Lift Circumvents the Single-Layer Induction Obstruction
Sep 26, 2026Kingsuk Maitra, Shagun Sood Morteza Hosseini, Suman Gunnala +1Transformer Attention Attention Mechanisms
Qualcomm Cloud AI Division, Santa Clara, CA
Mirror, Mirror on the Wall: Prompt Echoing in Small Instruct Language Models
Sep 14, 2026Inez Okulska, Bartosz Naskręcki, Jan Piotrowski +1Data Contamination in Language Models Induction Heads
Centre for Credible AI, Warsaw University of Technology · Adam Mickiewicz University Poznan · Institute of Fundamental Technological Research, Polish Academy of Sciences
Induction in Both Directions: A Mechanistic Analysis of In-Context Learning in Masked Diffusion Language Models
Jul 17, 2026Andy Catruna, Emilian RadoiMasked Diffusion Models In-Context Learning
National University of Science and Technology POLITEHNICA Bucharest · Academy of Romanian Scientists
Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention
Jul 7, 2026Li HengyuRotary Positional Embeddings Attention Head Analysis
Institute for Solid State Physics, The University of Tokyo
Induction Heads Interpolate N-Grams
Jul 2, 2026Francesco D'Angelo, Oguz Kaan Yuksel, Swathi Shree Narashiman +1Transformer Interpretability In-Context Learning
1TML Lab, EPFL, Switzerland. · 2IIT Madras, India. · †Work done during an internship at the TML Lab, EPFL.
Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence
Jun 10, 2026Itay Lavie, Kirsten Fischer, Andrey Lekov +3Softmax Attention Phase Transitions in Neural Networks
Racah Institute of Physics, Hebrew University of Jerusalem, Jerusalem 91904, Israel · John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, USA · Institute for Advanced Simulation (IAS-6), Computational and Systems Neuroscience, Jülich Research Center, Jülich 52428, Germany +3
When Do Attention Circuits Form? Developmental Trajectories of Capability and Attention-Sink Emergence Across Three 1B-ClassArchitectures
Jun 1, 2026Yongzhong XuLanguage Model Pretraining Transformer Interpretability
Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers
May 22, 2026Yongzhong XuTransformer Interpretability Circuit Discovery
In-Context Learning in Speech Language Models: Analyzing the Role of Acoustic Features, Linguistic Structure, and Induction Heads
Apr 7, 2026Charlotte Pouw, Hosein Mohebbi, Afra Alishahi +1In-Context Learning Speech Language Models
ILLC, University of Amsterdam · CSAI, Tilburg University
Temporal Dependencies in In-Context Learning: The Role of Induction Heads
Apr 1, 2026Anooshka Bajaj, Deven Mahesh Mistry, Sahaj Singh Maini +3In-Context Learning Temporal Reasoning in Language Models
Department of Computer Science, Indiana University Bloomington
INTRYGUE: Induction-Aware Entropy Gating for Reliable RAG Uncertainty Estimation
Mar 23, 2026Alexandra Kuleshova, Andrei Volodichev, Daria Kotova +1Hallucination in RAG Hallucination Detection
Applied AI Institute
Induction Meets Biology: Mechanisms of Repeat Detection in Protein Language Models
Feb 26, 2026Gal Pomerants, Yaniv Nikankin, Anja Reusch +3Protein Language Models LLM Interpretability
Technion - Israel Institute of Technology · The Hebrew University of Jerusalem, Israel · Kempner Institute, Harvard University
Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale
Feb 4, 2026Jonas Rohweder, Subhabrata Dutta, Iryna GurevychTransformer Interpretability Latent Variable Models
UKP Lab, TU Darmstadt
Predicting the Emergence of Induction Heads in Language Model Pretraining
Nov 21, 2025Tatsuya Aoyama, Ethan Gotlieb Wilcox, Nathan SchneiderLanguage Model Pretraining In-Context Learning
Department of Linguistics, Georgetown University.
In-Context Learning Beyond Copying: A Training-Time Analysis of Abstractive ICL
Nov 7, 2025Kerem Sahin, Sheridan Feucht, Adam Belfki +4In-Context Learning Emergent Abilities in Language Models
Northeastern University · University of Mannheim · Boston University
Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Models
Oct 26, 2025Anooshka Bajaj, Deven Mahesh Mistry, Sahaj Singh Maini +2In-Context Learning Induction Heads
Department of Computer Science Indiana University Bloomington

Oct 6, 2026J/K move · Enter open · S save

Spatial Induction Heads: In-Context Learning of Multidimensional Cellular Automata

Kimia Kazemian, Menghan Xu, John Thickstun, Sarah Dean

Cornell University

In-Context Learning Induction Heads Transformer Attention Transformer Interpretability Cellular Automata

Open paper AskPDF arXiv

Induction heads provide a mechanistic account of in-context learning in sequential data, but existing theory largely assumes that the context relevant to a prediction forms a contiguous block. In multidimensional data, serialization breaks this assumption by scattering spatial neighbors across distant positions in the token sequence. We study how transformers overcome this routing problem in multidimensional stochastic and deterministic cellular automata, where each trajectory is generated by an unknown local rule and presented as a flattened sequence without an explicit coordinate-based spatial inductive bias. We introduce spatial induction heads, two-layer gather-and-match circuits in which the first layer reconstructs the relevant spatial neighborhood and the second matches the resulting configuration against earlier occurrences. We give two explicit realizations of the gather and show that the positional dimension required for spatial routing depends only on the local neighborhood and spatial dimension, not on grid volume or trajectory horizon. We further construct a matching layer which implements Bayesian counting. The end-to-end circuit can approximate the Bayesian posterior arbitrarily closely for stochastic rules and can predict exactly for deterministic rules. Empirically, trained two-layer transformers generalize to unseen rules in one and two dimensional settings, achieving near-perfect deterministic rollouts and less than 0.005 nats KL from the Bayes-optimal predictor on stochastic rules. Attention patterns and layerwise probes align with the predicted gather-and-match computation, providing mechanistic evidence for spatial induction in trained transformers.