Shifting Mechanisms: How Positional Encoding Choice Shapes In-Context Retrieval

Eric Enouen; Sainyam Galhotra

cs.CLSep 29, 2026

Shifting Mechanisms: How Positional Encoding Choice Shapes In-Context Retrieval

Authors: Eric Enouen, Sainyam Galhotra

Organizations: Cornell University

Abstract

Language models increasingly use architectures that vary attention span and positional encoding across layers, such as applying RoPE with sliding-window attention and NoPE with global attention (SWA NoPE). However, how these choices shape in-context retrieval remains unclear. To study this question, we take a mechanistic view, tracing how positional encoding (PE) choice shapes the internal mechanisms models use for in-context retrieval. Across 22 open-weight models spanning eight families, we find that standard RoPE models rely primarily on positional retrieval, while PE hybrids shift toward semantic retrieval. We further show on a controlled pre-training ablation that confining positional encoding to local layers produces this semantic shift, degrading representations of positional information. Finally, we show that the reported long-context gains of PE hybrids mask a retrieval trade-off: SWA NoPE improves over RoPE on multiple-target retrieval and QA, but degrades when distinguishing competing keys. We show that these behavioral differences better track the mechanism shift from positional toward semantic mechanisms than a uniform improvement in long-context retrieval.

Figures & tables

Figure 1: Overview. We show that positional encoding (PE) choice shapes mechanism allocation: the SWA NoPE architecture shifts toward semantic mechanisms compared to RoPE. We then show that this mechanism shift predicts long-context retrieval performance: SWA NoPE improves on multiple-target retrieval and QA tasks, but struggles to distinguish competing keys.

Figure 2: Example of counterfactual activation patching. The prompt pair is constructed so that each retrieval mechanism corresponds to a different answer after activation patching. The positional mechanism follows location and predicts ‘B’, the lexical mechanism follows the ‘hat’ key and predicts ‘C’, and the reflexive mechanism follows the self-referential pointer and predicts ‘A’. If patching has no effect, the model retains the original prediction (‘D’).

RoPE Architectures
Model	PE Configuration	Window Size	Local:Global
Gemma-2 ( Riviere et al., 2024 )	RoPE (10K)	4096	1:1
Llama-3.1 ( Grattafiori et al., 2024 )	RoPE (500K)	—	—
Qwen-2.5 ( Yang et al., 2024 )	RoPE (1M)	—	—
Qwen-3 ( Yang et al., 2025a )	RoPE (1M)	—	—
PE Hybrid Architectures

Table 1: Model families and descriptions. We compare RoPE architectures that apply RoPE consistently throughout the model with PE hybrid architectures that vary the application of RoPE across layers. We report the PE configuration, as well as window size and local:global ratio where applicable. † Gemma-4 2B/4B have a window size of 512 and Gemma-4 2B uses a 4:1 local:global ratio.

Figure 3: Mechanism allocation across the model battery pooled over 29 tasks and sorted by

r_{\mathrm{pos}}

. ( Top ) Relative allocation to positional, lexical, and reflexive retrieval mechanisms. ( Bottom ) Positional ratio for each model. The architecture classes separate completely: every RoPE model has a higher

r_{\mathrm{pos}}

(

\geq 0.59

) than every PE hybrid (

\leq 0.53

	Architecture	Positional	Lexical	Reflexive	Unknown	$r_{\mathrm{pos}}$
16K	RoPE	17.6	56.6	1.6	24.1	0.30
	SWA RoPE	19.4	46.6	2.1	31.9	0.40
	SWA NoPE	10.0	68.6	1.4	20.0	0.14
32K	RoPE	15.9	60.6	1.4	22.1	0.26
	SWA RoPE	16.2	57.5	1.8	24.6	0.27
	SWA NoPE	9.5	71.9	1.3	17.3	0.13

Table 2: Mechanism allocation of three model architectures: RoPE, SWA RoPE, and SWA NoPE at two training checkpoints (16K and 32K training length) on the Boxes task. We report the percentage (%) of samples classified as positional, lexical, reflexive, and unknown. We additionally report the ratio of positional to semantic (lexical, reflexive) mechanisms. At both training lengths, introducing SWA RoPE alone does not produce the semantic shift, but SWA with NoPE global layers substantially shifts the mechanism allocation, reducing

r_{\mathrm{pos}}

by a factor of two.

Context	Model	Ordering ID		Semantic Key
16K	RoPE	94.8 (0.2)		99.0 (0.2)
	SWA RoPE	91.6 (0.1)	$-3.2$	98.8 (0.3)	$-0.2$
	SWA NoPE	82.7 (0.3)	$-12.1$	99.4 (0.1)	$+0.4$
32K	RoPE	94.9 (0.2)		99.1 (0.2)
	SWA RoPE	92.7 (0.2)	$-2.2$	99.0 (0.3)	$-0.1$
	SWA NoPE	83.9 (0.3)	$-11.0$	99.5 (0.1)	$+0.4$

Table 3: Linear probe accuracy (%) on the residual stream at layer 15 across items in the list, predicting either the index in the list or the attached semantic key, across the six model checkpoints. Standard deviation reported in parentheses over eight runs. SWA NoPE has the worst ordering ID probe accuracy, while semantic key probe accuracy remains high.

Context	Model	Overall (13)	NIAH mk1 (1)	VT + Aggr. (3)	NIAH rest (7)	QA (2)
16K	RoPE	47.1	91.2	16.9	56.1	38.7
	SWA RoPE	47.2	84.8	18.5	57.2	36.7
	SWA NoPE	58.3	78.0	17.5	77.2	43.4
32K	RoPE	43.3	92.4	18.1	50.8	30.7
	SWA RoPE	45.7	81.6	23.0	55.1	28.8
	SWA NoPE	53.9	82.2	15.6	69.6	42.0

Table 4: RULER scores across the three model architectures: RoPE, SWA RoPE, and SWA NoPE. The 16K checkpoints are trained for 100B tokens, and the 32K checkpoints extend them for a further 5B tokens, and each is evaluated at their training context length. SWA NoPE substantially improves the overall RULER score, with gains concentrated in most NIAH and QA tasks. However, it does not improve over RoPE on the first NIAH multi-key task or consistently improve on variable tracking and aggregation.

		Confusable keys				Multi-item
Context	Model	$0/4$	$1/4$	$2/4$	$3/4$	multivalue	multiquery
16K	RoPE	97.0	90.8	84.6	70.8	35.0	35.0
	SWA RoPE	92.6	80.4	75.4	60.2	44.0	38.4
	SWA NoPE	88.2	73.0	57.4	52.6	80.8	85.7
32K	RoPE	96.8	91.8	82.6	67.6	25.2	30.5
	SWA RoPE	82.6	75.6	57.4	46.8	26.4	37.9

Table 5: We test each checkpoint on our confusable key task and compare to the RULER multi-value and multi-query subtasks. SWA NoPE struggles to select amongst competing keys, especially as they become more semantically similar, flipping the model ranking compared to multi-item NIAH where all keys are returned.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

RoPE			PE Hybrid
Model	Layer	Positions	Model	Layer	Positions
Gemma-2 2B	18	$-1,-4,-6,-8$	Gemma-3 4B	23	$-1,-4,-6,-8$
Gemma-2 9B	28	$-1,-4,-6,-8$	Gemma-3 12B	29	$-1,-4,-6,-8$
Gemma-2 27B	25	$-1,-4,-6,-8$	Gemma-3 27B	41	$-1,-4,-6,-8$
Qwen2.5 3B	31	$-1,-4,-6,-8$	Gemma-4 2B	24	$-1,-4,-6,-8$
Qwen2.5 7B	22	$-1,-4,-6,-8$	Gemma-4 4B	29	$-1,-4,-6,-8$

Appendix

Table 6: Patched layer and token positions per model. Positions are matched by token role: the ‘?’ and ‘:’ of the query, plus the final two chat-scaffold positions. Offsets differ with template length.

Figure 4: For every checkpoint in our pre-training ablation, layer 15 is the last layer before the answer is retrieved.

Figure 5: Layer scan for Gemma-3 4B with the messiness-0 (top) and messiness-4 (bottom) counterfactuals. Bars show the share of each mechanism when patching at each layer.

Figure 6: Layer scan for Gemma-4 4B with the messiness-0 (top) and messiness-4 (bottom) counterfactuals. Bars show the share of each mechanism when patching at each layer.

Figure 7: Layer scan for Gemma-4 2B with the messiness-0 (top) and messiness-4 (bottom) counterfactuals. Bars show the share of each mechanism when patching at each layer.

RoPE
Model	Positional	Lexical	Reflexive	Unknown	No effect	$r_{pos}$
Qwen2.5 72B	46.0	8.4	16.4	21.9	7.3	1.85
Qwen2.5 7B	36.0	16.7	11.4	33.0	2.7	1.28
Qwen2.5 32B	37.9	12.8	17.0	29.1	3.2	1.27
Llama-3.1 70B	28.8	16.7	10.8	28.5	15.2	1.05
Gemma-2 27B	33.0	15.7	17.8	25.1	8.2	0.98

Appendix

Table 7: Mechanism distribution per model, as a percentage of all samples (29 task–query pairs, 29,000 samples each), sorted by

r_{pos}=P/(L+R)

. Bold / underline : highest/second-highest among Positional, Lexical, and Reflexive.

Figure 8: Full per-task breakdown of

r_{\mathrm{pos}}

across the model battery (29 tasks). Separation does not hold for every individual task, but holds when averaged across tasks.

Figure 9: Mechanism allocation by entity index for

t_{entity}=1

, averaged over tasks. Reflexive retrieval dominates lexical retrieval because the target entity always precedes the query entity.

Figure 10: Mechanism allocation by entity index for

t_{entity}=2

, averaged over tasks.

Figure 11: Mechanism allocation by entity index for

t_{entity}=3

, averaged over tasks. Lexical retrieval dominates reflexive retrieval because the target entity follows the query entity.

		NIAH								VT + Aggr.			QA
Context	Model	S1	S2	S3	MK1	MK2	MK3	MV	MQ	VT	CWE	FWE	QA1	QA2
16K	RoPE	88.2	97.4	50.0	91.2	83.0	4.0	35.0	35.0	0.1	0.6	50.1	50.6	26.8
	SWA RoPE	100.0	94.6	36.6	84.8	75.2	11.6	44.0	38.4	6.1	5.9	43.4	45.8	27.6
	SWA NoPE	100.0	100.0	48.0	78.0	84.2	41.6	80.8	85.7	0.2	3.5	48.7	54.8	32.0
32K	RoPE	100.0	99.6	18.8	92.4	81.2	0.2	25.2	30.5	14.4	1.1	38.7	33.6	27.8
	SWA RoPE	100.0	99.8	27.2	81.6	74.0	20.2	26.4	37.9	14.9	9.1	45.0	30.8	26.8

Appendix

Table 8: Per-task RULER scores at 16K and 32K context.

N=500

per task. Best result for each task and context length is bolded.

		$0/4$		$1/4$		$2/4$		$3/4$
Context	Model	Comp. key	Neither	Comp. key	Neither	Comp. key	Neither	Comp. key	Neither
32K	RoPE	94%	6%	83%	17%	86%	14%	76%	24%
	SWA RoPE	43%	57%	39%	61%	52%	48%	53%	47%
	SWA NoPE	94%	6%	86%	14%	82%	18%	83%	17%

Appendix

Table 9: Error types on the confusable-key task at 32K context length. Conditioned on an incorrect answer, we report the percentage of errors that return a value bound to one of the three competing keys (Competing key), versus no needle value (Neither).

Figure 12: Mechanism allocation across linear hybrids compared to RoPE/PE hybrids.

Explore similar work

Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable
Jul 23, 2026Ji Ho BaeRotary Position Embedding Positional Encoding
JRTI, Seoul, 05855, Republic of Korea
Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling
Aug 9, 2026Jiguo LiRotary Position Embedding Positional Encoding
How Local Mixing Encodes Relative Position in Global NoPE Attention
Sep 29, 2026Cutter Dawes, Nick Alonso, Tom Figliolia +1Positional Encoding Transformer Architectures
Zyphra Research

Jul 23, 2026, cs.CLJ/K move · Enter open · S save

Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable

Ji Ho Bae

JRTI, Seoul, 05855, Republic of Korea

Rotary Position Embedding Positional Encoding Language Model Perplexity Periodic Riemannian Manifolds

Open paper AskPDF arXiv

Möbius RoPE is a rotary positional encoding built on the anti-periodic frequency ladder

θ_i=π(2i+1)/N

: every rotation plane advances by an odd multiple of

π

across the training context, so the positional holonomy is

-1

and the two ends of the sequence are deterministically coupled through a closed-form Dirichlet "dipole"; to our knowledge this is the first anti-periodic boundary condition in positional encoding. We verify the theory numerically to

\sim 10^{-6}

and pretrain 48 models spanning six 160M-class and three 410M-class arms (2B FineWeb-Edu tokens each; the hybrid arm puts Möbius frequencies on 25% of heads). Hybrid perplexity is unchanged (29.66 vs. 29.72), but needle-in-a-haystack retrieval becomes reliable:

90.3\pm5.7\%

versus

63.3\pm31.4\%

at context 512 (

n=6

seeds), observed worst seed 86% versus 14%, robust variance tests

p=0.013

0.029

(unadjusted), recurring at 410M (Levene

p=0.040

). Matched controls isolate the mechanism: an aperiodic ladder in the same frequency band reproduces none of the effect, and a periodic (holonomy

+1

) ladder only a fraction. Swapping trained models' frequency table back to standard RoPE (weights frozen) collapses retrieval, with damage concentrated on far needles: trained models depend on this long-range geometry. A NoPE arm is even more reliable at short context but pays a 13% perplexity tax and extrapolates worst; only the anti-periodic hybrid pairs baseline perplexity with a high reliability floor. The effect is scoped to single-needle retrieval within the training window; a one-line frequency swap thus provides zero-cost insurance against the retrieval seed lottery.

Abstract

Explore similar work

Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable

Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling

How Local Mixing Encodes Relative Position in Global NoPE Attention