Temporal Dependencies in In-Context Learning: The Role of Induction Heads

Anooshka Bajaj; Deven Mahesh Mistry; Sahaj Singh Maini; Yash Aggarwal; Billy Dickson; Zoran Tiganj

cs.CLApr 1, 2026

Temporal Dependencies in In-Context Learning: The Role of Induction Heads

Authors: Anooshka Bajaj, Deven Mahesh Mistry, Sahaj Singh Maini, Yash Aggarwal, Billy Dickson, Zoran Tiganj

Organizations: Department of Computer Science, Indiana University Bloomington

Abstract

Large language models (LLMs) exhibit strong in-context learning capabilities, but how they track and retrieve information from context remains underexplored. Drawing on the free recall paradigm in cognitive science (where participants recall list items in any order), we show that several open-source LLMs consistently display a serial-recall-like pattern, assigning peak probability to tokens that immediately follow a repeated token in the input sequence. Through systematic ablation experiments, we show that induction heads, specialized attention heads that attend to the token following a previous occurrence of the current token, play an important role in this phenomenon. Removing heads with a high induction score substantially reduces the +1 lag bias, whereas ablating random heads does not reproduce the same reduction. We also show that removing heads with high induction scores impairs the performance of models prompted to do serial recall using few-shot learning to a larger extent than removing random heads. Our findings highlight a mechanistically specific connection between induction heads and temporal context processing in transformers, suggesting that these heads are especially important for ordered retrieval and serial-recall-like behavior during in-context learning.

Figures & tables

Panel of Figure 1 — Figure 1: Induction scores for the four models across layers and heads. Top row: Base models. Middle row: Instruction-tuned models. Bottom row: difference in induction scores for each layer and head between instruction-tuned and base models.

	Instruct	Base
Llama-3.1-8B	0.31 ± 0.05	0.34 ± 0.05
Mistral-7B-v0.1	0.50 ± 0.08	0.54 ± 0.06
Qwen2.5-7B	0.26 ± 0.07	0.26 ± 0.08
Gemma-2-9b	0.48 ± 0.13	0.45 ± 0.14

Table 1: Mean and standard deviation of the induction scores

Panel of Figure 2 — Figure 2: Impact of induction and random head ablation (100 heads in each case) on the model output probability as a function of lag. The models were presented with a sequence of 501 tokens where the last token repeated the token at index 250 and the lag is defined relative to the repeated token (see Methods for more details and Fig. 3 for a zoomed version showing lags -6 to 6). The results show averages across 5000 runs with shuffled token sequences. Top row: Base models. Bottom row: Instruction-tuned models.

Panel of Figure 3 — Figure 3: Impact of induction and random head ablation (100 heads in each case) on the model output probability as a function of lag: same as Fig. 2 , but with zoom on lags -6 to 6 to emphasize that the highest probabilities were at lags 1 or 0. Top row: Base models. Bottom row: Instruction-tuned models.

Panel of Figure 4 — Figure 4: Impact of induction head ablation on the model output probability for the token at lag +1. The models were presented with a sequence of 501 tokens where the last token repeated the token at index 250 and the lag is defined relative to the repeated token, hence the probability at lag +1 is the probability that the model assigns to token 251 (see Methods for more details). The results show averages across 5000 runs with shuffled token sequences. We ablated the following numbers of induction heads (sorted by the induction scores) and random heads (x-axis): 1, 5, 10, 20, 30, 40, 50, 80, 100, 150, 200, 250 and 300. Top row: Base models. Bottom row: Instruction-tuned models. Exact numerical values for these experiments are shown in Table A1 and Table 2 for induction head ablation in base and instruction-tuned models respectively and in Table A2 and Table A3 for random head ablation in base and instruction-tuned models respectively.

# Induction Heads Ablated	Llama-3.1-8B-Instruct	Mistral-7B-Instruct-v0.1	Qwen2.5-7B-Instruct	Gemma-2-9b-it
1	0.00065	0.04300	0.00417	0.00354
5	0.00062	0.03662	0.00263	0.00261
10	0.00065	0.03587	0.00144	0.00262
20	0.00064	0.02438	0.00126	0.00189
30	0.00065	0.03216	0.00014	0.00115
40	0.00065	0.03316	0.00034	0.00068

Table 2: Impact of induction head ablation on the model output probability for the token at lag +1 for different models. The results are for instruction-tuned models. These results are visualized in Fig. 4 . Corresponding results for ablation of random heads are shown in Table A3 .

Figure 5: Model performance in ICL is more sensitive to ablation of induction than random heads. Conditional recall probability at lag +1 for Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct as a function of the number of ablated induction and random attention heads. Exact values are shown in Table A4 .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure A1: Induction scores across layers for the four models. Each dot represents induction score at a particular attention head. Top row: Base models. Middle row: Instruction-tuned models. Bottom row: difference in induction scores for each layer and head between instruction-tuned and base models.

# Induction Heads Ablated	Llama-3.1-8B	Mistral-7B-v0.1	Qwen2.5-7B	Gemma-2-9b
1	0.00102	0.00061	0.00121	0.06764
5	0.00092	0.00052	0.00078	0.05976
10	0.00093	0.00031	0.00039	0.05905
20	0.00088	0.00036	0.00029	0.05285
30	0.00086	0.00091	0.00031	0.00205
40	0.00091	0.00116	0.00033	0.00100

Appendix

Table A1: Impact of induction head ablation on the model output probability for the token at lag +1 for different models. The results are for base models. These results are visualized in Fig. 4 . Corresponding results for ablation of random heads are shown in Table A2 .

# Random Heads Ablated	Llama-3.1-8B	Mistral-7B-v0.1	Qwen2.5-7B	Gemma-2-9b
0	0.00103	0.00060	0.00169	0.07125
5	0.00113	0.00052	0.00155	0.08422
10	0.00117	0.00056	0.00162	0.09609
20	0.00541	0.00086	0.00130	0.04800
30	0.00542	0.00088	0.00304	0.02157
40	0.00365	0.00073	0.00235	0.02732

Appendix

Table A2: Impact of random head ablation on the model output probability for the token at lag +1 for different models. The results are for base models. These results are visualized in Fig. 4 .

# Random Heads Ablated	Llama-3.1-8B-Instruct	Mistral-7B-Instruct-v0.1	Qwen2.5-7B-Instruct	Gemma-2-9b-it
1	0.00066	0.04366	0.00537	0.00496
5	0.00068	0.03826	0.00477	0.00576
10	0.00092	0.03831	0.00385	0.00239
20	0.00968	0.03897	0.00452	0.01165
30	0.00881	0.03681	0.00437	0.01748
40	0.00713	0.03598	0.00324	0.01005

Appendix

Table A3: Impact of random head ablation on the model output probability for the token at lag +1 for different models. The results are for instruction-tuned models. These results are visualized in Fig. 4 .

Panel of Figure 12 — Figure A2: CRP in serial recall ICL task for different models and ablation conditions.

Panel of Figure 13 — Figure A3: Probability of recall for items at different serial positions in serial recall ICL task for different models and ablation conditions (serial position 1 is the first item in the list, serial position 14 is the last item in the list).

Model	Abl.	0 Heads	1 Head	10 Heads	25 Heads	50 Heads	100 Heads
Llama-I	Ind.	$0.98\pm 0.14$	$0.99\pm 0.03$	$0.97\pm 0.15$	$0.28\pm 0.40$	$0.28\pm 0.35$	$0.12\pm 0.28$
	Rand.	$0.98\pm 0.14$	$0.98\pm 0.14$	$0.92\pm 0.26$	$0.74\pm 0.43$	$0.90\pm 0.27$	$0.21\pm 0.28$
Qwen-I	Ind.	$0.97\pm 0.16$	$0.99\pm 0.07$	$1.00\pm 0.00$	$0.91\pm 0.22$	$0.00\pm 0.00$	$0.25\pm 0.50$
	Rand.	$0.97\pm 0.16$	$0.98\pm 0.14$	$1.00\pm 0.01$	$0.99\pm 0.03$	$1.00\pm 0.00$	$0.00\pm 0.00$
Llama	Ind.	$1.00\pm 0.01$	$1.00\pm 0.01$	$0.75\pm 0.38$	$0.54\pm 0.46$	$0.64\pm 0.46$	$0.10\pm 0.22$
	Rand.	$1.00\pm 0.01$	$1.00\pm 0.02$	$0.95\pm 0.20$	$0.46\pm 0.44$	$0.80\pm 0.30$	$0.50\pm 0.42$

Appendix

Table A4: Conditional recall probability at lag +1 illustrating the impact of induction and random head ablation on serial recall performance (mean recall probability

\pm

standard deviation) in an ICL serial recall task. Results are shown for Llama-3.1-8B and Qwen2.5-7B models, both base and instruction-tuned versions, across varying numbers of ablated heads. These results are visualized in Fig. 5 .

Panel of Figure 15 — Figure A4: Impact of induction and random head mean ablation (100 heads in each case) on the model output probability as a function of lag. The models were presented with a sequence of 501 tokens where the last token repeated the token at index 250 and the lag is defined relative to the repeated token (see Methods for more details and Fig. A5 for a zoomed version showing lags -6 to 6). The results show averages across 5000 runs with shuffled token sequences. Top row: Base models. Bottom row: Instruction-tuned models.

Panel of Figure 16 — Figure A5: Impact of induction and random head mean ablation (100 heads in each case) on the model output probability as a function of lag: same as Fig. A4 , but with zoom on lags -6 to 6 to emphasize that the highest probabilities were at lags 1 or 0. Top row: Base models. Bottom row: Instruction-tuned models.

Panel of Figure 17 — Figure A6: Impact of induction head mean ablation on the model output probability for the token at lag +1. The models were presented with a sequence of 501 tokens where the last token repeated the token at index 250 and the lag is defined relative to the repeated token, hence the probability at lag +1 is the probability that the model assigns to token 251 (see Methods for more details). The results show averages across 5000 runs with shuffled token sequences. We ablated the following numbers of induction heads (sorted by the induction scores) and random heads (x-axis): 1, 5, 10, 20, 30, 40, 50, 80, 100, 150, 200, 250 and 300. Top row: Base models. Bottom row: Instruction-tuned models.

Panel of Figure 18 — Figure A7: Impact of induction and random head top layers ablation (100 heads in each case) on the model output probability as a function of lag. The models were presented with a sequence of 501 tokens where the last token repeated the token at index 250 and the lag is defined relative to the repeated token (see Methods for more details and Fig. A8 for a zoomed version showing lags -6 to 6). The results show averages across 5000 runs with shuffled token sequences. Top row: Base models. Bottom row: Instruction-tuned models.

Panel of Figure 19 — Figure A8: Impact of induction and random head top layers ablation (100 heads in each case) on the model output probability as a function of lag: same as Fig. A7 , but with zoom on lags -6 to 6 to emphasize that the highest probabilities were at lags 1 or 0. Top row: Base models. Bottom row: Instruction-tuned models.

Panel of Figure 20 — Figure A9: Impact of induction head top layers ablation on the model output probability for the token at lag +1. The models were presented with a sequence of 501 tokens where the last token repeated the token at index 250 and the lag is defined relative to the repeated token, hence the probability at lag +1 is the probability that the model assigns to token 251 (see Methods for more details). The results show averages across 5000 runs with shuffled token sequences. We ablated the following numbers of induction heads (sorted by the induction scores) and random heads (x-axis): 1, 5, 10, 20, 30, 40, 50, 80, 100, 150, 200, 250 and 300. Top row: Base models. Bottom row: Instruction-tuned models.

Panel of Figure 21 — Figure A10: Impact of induction and random head bottom layers ablation (100 heads in each case) on the model output probability as a function of lag. The models were presented with a sequence of 501 tokens where the last token repeated the token at index 250 and the lag is defined relative to the repeated token (see Methods for more details and Fig. A11 for a zoomed version showing lags -6 to 6). The results show averages across 5000 runs with shuffled token sequences. Top row: Base models. Bottom row: Instruction-tuned models.

Panel of Figure 22 — Figure A11: Impact of induction and random head bottom layers ablation (100 heads in each case) on the model output probability as a function of lag: same as Fig. A10 , but with zoom on lags -6 to 6 to emphasize that the highest probabilities were at lags 1 or 0. Top row: Base models. Bottom row: Instruction-tuned models.

Panel of Figure 23 — Figure A12: Impact of induction head bottom layers ablation on the model output probability for the token at lag +1. The models were presented with a sequence of 501 tokens where the last token repeated the token at index 250 and the lag is defined relative to the repeated token, hence the probability at lag +1 is the probability that the model assigns to token 251 (see Methods for more details). The results show averages across 5000 runs with shuffled token sequences. We ablated the following numbers of induction heads (sorted by the induction scores) and random heads (x-axis): 1, 5, 10, 20, 30, 40, 50, 80, 100, 150, 200, 250 and 300. Top row: Base models. Bottom row: Instruction-tuned models.

In-Context Learning

In-context learning studies how models adapt to new tasks from examples or instructions supplied at inference time, without changing their weights.

See all 404 papers

Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Models
Oct 26, 2025Anooshka Bajaj, Deven Mahesh Mistry, Sahaj Singh Maini +2In-Context Learning Induction Heads
Department of Computer Science Indiana University Bloomington
Fine-Tuning Dynamics of In-Context Factual Recall in Transformers
May 26, 2026Ruomin Huang, Eshaan Nichani, Jason D. Lee +1Supervised Fine-Tuning Transformer Attention
Duke University · Princeton University · UC Berkeley
Induction Heads Interpolate N-Grams
Jul 2, 2026Francesco D'Angelo, Oguz Kaan Yuksel, Swathi Shree Narashiman +1Transformer Interpretability In-Context Learning
1TML Lab, EPFL, Switzerland. · 2IIT Madras, India. · †Work done during an internship at the TML Lab, EPFL.

Oct 26, 2025J/K move · Enter open · S save

Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Models

Anooshka Bajaj, Deven Mahesh Mistry, Sahaj Singh Maini, Yash Aggarwal+1

Department of Computer Science Indiana University Bloomington

In-Context Learning Induction Heads State Space Models

Open paper AskPDF arXiv

In-context learning is governed by both temporal and semantic relationships, shaping how Large Language Models (LLMs) retrieve contextual information. Analogous to human episodic memory, where the retrieval of specific events is enabled by separating events that happened at different times, this work probes the ability of various pretrained LLMs, including transformer and state-space models, to differentiate and retrieve temporally separated events. Specifically, we prompted models with sequences containing multiple presentations of the same token, which reappears at the sequence end. By fixing the positions of these repeated tokens and permuting all others, we removed semantic confounds and isolated temporal effects on next-token prediction. Across diverse sequences, models consistently placed the highest probabilities on tokens following a repeated token, but with a notable bias for those nearest the beginning or end of the input. An ablation experiment linked this phenomenon in transformers to induction heads. Extending the analysis to unique semantic contexts with partial overlap further demonstrated that memories embedded in the middle of a prompt are retrieved less reliably. Despite architectural differences, state-space and transformer models showed comparable temporal biases. Our findings deepen the understanding of temporal biases in in-context learning and offer an illustration of how these biases can enable temporal separation and episodic retrieval.

Abstract

In-Context Learning

Explore similar work

Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Models

Fine-Tuning Dynamics of In-Context Factual Recall in Transformers

Induction Heads Interpolate N-Grams