cs.CLOct 26, 2025

Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Models

Authors: Anooshka Bajaj, Deven Mahesh Mistry, Sahaj Singh Maini, Yash Aggarwal, Zoran Tiganj

Organizations: Department of Computer Science Indiana University Bloomington

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In-context learning is governed by both temporal and semantic relationships, shaping how Large Language Models (LLMs) retrieve contextual information. Analogous to human episodic memory, where the retrieval of specific events is enabled by separating events that happened at different times, this work probes the ability of various pretrained LLMs, including transformer and state-space models, to differentiate and retrieve temporally separated events. Specifically, we prompted models with sequences containing multiple presentations of the same token, which reappears at the sequence end. By fixing the positions of these repeated tokens and permuting all others, we removed semantic confounds and isolated temporal effects on next-token prediction. Across diverse sequences, models consistently placed the highest probabilities on tokens following a repeated token, but with a notable bias for those nearest the beginning or end of the input. An ablation experiment linked this phenomenon in transformers to induction heads. Extending the analysis to unique semantic contexts with partial overlap further demonstrated that memories embedded in the middle of a prompt are retrieved less reliably. Despite architectural differences, state-space and transformer models showed comparable temporal biases. Our findings deepen the understanding of temporal biases in in-context learning and offer an illustration of how these biases can enable temporal separation and episodic retrieval.

Figures & tables

Figure 2: Experiment 1: Next-token probability vs. position for varying number of fixed-token repetitions (columns) across models (rows). Vertical gray lines mark the anchor token. Peaks in the blue lines confirm the presence of +1 recall preference and positional biases.
Figure 3: Experiment 2: Episodic retrieval results. Probability of potential target tokens after probing episodes at different temporal positions (columns 1-5). Vertical gray lines mark the anchor token. Red dots mark the target token corresponding to the probed episode. A high probability at the red dot indicates successful retrieval of the episode’s target token.
Figure 4: Transformer ablation effect (Exp. 1). Probability vs. position after ablating Induction (Ind) or Random (Rand) heads (rows) for Llama, Mistral, Qwen, Gemma (model pairs per 2 rows). Columns show the number of ablated heads.
ModelAblation LevelInduction ChangeRandom ChangeDifference in Change Δ\boldsymbol{\Delta}
Llama19.35×10−59.35\times 10^{-5}4.96×10−54.96\times 10^{-5}4.39×10−54.39\times 10^{-5}
 101.34×10−41.34\times 10^{-4}3.81×10−53.81\times 10^{-5}9.63×10−59.63\times 10^{-5}
 501.81×10−41.81\times 10^{-4}1.72×10−51.72\times 10^{-5}1.64×10−41.64\times 10^{-4}
 1002.24×10−42.24\times 10^{-4}−1.91×10−6-1.91\times 10^{-6}2.26×10−42.26\times 10^{-4}
Mistral11.90×10−51.90\times 10^{-5}−7.51×10−6-7.51\times 10^{-6}2.65×10−52.65\times 10^{-5}
 102.09×10−52.09\times 10^{-5}2.03×10−62.03\times 10^{-6}1.88×10−51.88\times 10^{-5}
Table A1: Ablation effects on the average +1 probability in Exp. 1. Induction Change and Random Change are changes in +1 probability relative to the unablated model. Δ=\Delta= Induction Change −- Random Change; positive Δ\Delta means ablating high-induction heads reduces the (+1)(+1) signal more than ablating random heads.
ExperimentModel / AblationInduction ChangeRandom Change
Exp. 1Mamba, 1%+67.8%-91.7%
Mamba, 5%+98.4%+6.2%
Mamba, 10%+94.2%+58.3%
R-Gemma, 1%-17.2%+9.6%
R-Gemma, 5%+68.2%+43.2%
R-Gemma, 10%+96.9%+58.9%
Table A2: State-space/recurrent channel ablations. “Change” denotes the relative change in the task signal. Experiment 1: mean +1 recall; Experiment 2: target probability / separation signal after ablating high-attribution channels (Induction) vs. a matched number of random channels. Falcon-M was too computationally demanding for Experiment 2 ablations.
Figure A1: Probability ( ±\pm SEM) of +1 token retrieval vs. repetition position, for varying number of fixed-token repetitions (columns).
Figure A2: SSM ablation effect (Exp. 1). Probability vs. position after ablating Induction (Ind) or Random (Rand) channels (rows) for Mamba, Falcon-Mamba, Recurrent-Gemma (model pairs per 2 rows). Columns show the percentage of ablated channels.
Figure A3: Next-token probability vs. position, varying spacing between fixed tokens (columns, repeats=10). Shows +1 recall preference and positional biases.
Figure A4: Llama ablation effect (Exp. 2). Episodic retrieval probability after ablating Induction or Random heads (rows) probing different episode positions. Columns show number of ablated heads. Red dots mark the target token corresponding to the probed episode.
Figure A5: Mistral ablation effect (Exp. 2). Episodic retrieval probability after ablating Induction or Random heads. Columns show number of ablated heads.
Figure A6: Qwen ablation effect (Exp. 2). Episodic retrieval probability after ablating Induction (Ind P1-P5) or Random (Rand P1-P5) heads. Columns show number of ablated heads.
Figure A7: Gemma ablation effect (Exp. 2). Episodic retrieval probability after ablating Induction or Random heads. Columns show number of ablated heads.
Figure A8: Mamba ablation effect (Exp. 2). Episodic retrieval probability after ablating Induction-related or Random channels. Columns show the percentage of ablated channels.
Figure A9: Recurrent-Gemma ablation effect (Exp. 2). Episodic retrieval probability after ablating Induction-related or Random channels. Columns show percentage of ablated channels.

Explore similar work

CardsList

  1. Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architectures

    Mar 3, 2026Georgios Pantazopoulos, Malvina Nikandrou, Ioannis Konstas +1Transformer ArchitecturesPositional Encoding

    University of Edinburgh · Heriot-Watt University

  2. Fine-Tuning Dynamics of In-Context Factual Recall in Transformers

    May 26, 2026Ruomin Huang, Eshaan Nichani, Jason D. Lee +1In-Context LearningIn-Context

    Duke University · Princeton University · UC Berkeley

  3. Context-Gated Associative Retrieval: From Theory to Transformers

    May 8, 2026Moulik Choraria, Argyrios Gerogiannis, Vidhata Jayaraman +2Dense Associative MemoryIn-Context Learning

    UIUC · University of Minnesota, Twin Cities · Stony Brook University