A Positional Realizability Theory of Length Generalization in Single-Head Attention

Jiasong Han; Yunkai Chai; Jintao Wang

cs.LGOct 30, 2026

A Positional Realizability Theory of Length Generalization in Single-Head Attention

Authors: Jiasong Han, Yunkai Chai, Jintao Wang

Organizations: Department of Electronic Engineering, Tsinghua University · School of Information and Communication Engineering, Shanghai Jiao Tong University

Abstract

Why does length generalization (LG) depend so sharply on the positional encoding (PE)? Relative schemes extrapolate fixed offsets, absolute schemes fix positions, and neither handles growing offsets. We give a realizability theory for a deliberately controlled abstraction, a single positional-only attention head, that derives this well-documented empirical pattern and says precisely where it holds and breaks. The device is a simulation transfer operator $T_L$ that re-evaluates a selected length- $L_0$ -realizing positional kernel on the longer grid, with a scalar $α_L \in [0,1]$ read off before any training from $(Φ, t_n)$ alone. Here $α_L$ serves as the computable witness of that taxonomy's derivation, not as a stronger empirical predictor. Our results are a relative-head deep-row characterization for kernels with a strict unique global maximizer (Theorem 2), an absolute/relative duality that derives the pattern (Proposition 4), a learned-absolute underdetermination result, and a static expressivity bound; a sufficiency theorem (Theorem 1) bridges $α_L$ to LG, and the general deterministic case is a precise open conjecture (correcting a tempting but invalid "outside-colspan" argument). Experiments confirm the structure: in the controlled single-head regime $α_L$ exactly separates the taxonomy's generalizing and non-generalizing (PE, task) cells (2625 trained-model measurements; stable at vocab 64/d=128). Together, these results turn the received PE taxonomy into a positional-kernel transfer theorem, and identify the content and depth mechanisms that escape it. The theory concerns the positional channel alone: tasks whose targets depend on token content, including the standard length-generalization suite such as addition and sorting, fall outside Theorem 2.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the method. Select (fit) a single positional head’s kernel at length $L_{0}$ realizing the task’s target pattern $R_{L_{0}}$ ; transfer the same kernel to a longer grid via $T_{L}$ (re-evaluate $\Phi_{L}W\Phi_{L}^{\top}$ , no retraining); old rows keep their argmax by the causal-mask identity (Thm 1 ); score the new rows $[L_{0},L)$ to read off $\alpha_{L}\in[0,1]$ . The whole pipeline uses only $(\Phi,t_{n})$ , with no Transformer training, no data samples, and no learned Transformer parameters (only a deterministic kernel-selection on the PE basis).

	pre-training?	granularity	PE-keyed?	direction
RASP-L ( Zhou et al., 2024a )	yes	discrete	no	suff. (conj.)
C-RASP[Pos] ( Huang et al., 2025 )	yes	task, binary	by PE-class	suff.+nec.
Izzo et al. ( Izzo et al., 2025 )	train-length bd.	model-level	no	sufficiency
spectral PE ( Gu et al., 2025 )	post-hoc	model-level	spectral	–
$\alpha_{L}$ (ours)	yes	cell, cont.	by feat. map	suff. + rel. nec. + conj.

Table 1: Positioning against the closest LG frameworks.

\alpha_{L}

’s niche: pre-training, keyed to the PE’s feature map , continuous, single-head, with matched sufficiency and relative-head necessity (the general necessary condition is conjectural).

Panel of Figure 2 — Figure 2: The offset taxonomy, derived (Theorem 2 , Prop. 4 ). A relative kernel length-generalizes a fixed relative offset ; a clamped learned-absolute encoding does the reverse; and neither handles a growing offset (the shared blind spot, right column). Throughout, the relative statements are for kernels with a strict unique global maximizer, the hypothesis of Theorem 2 ; the accompanying text explains why it is needed and what the $L_{0}$ -fit contributes. This $2\times 3$ verdict is exactly the empirical PE taxonomy, here a consequence of what a single positional head can realize.

Axis	Levels	Rationale
PEs	learned-abs, NoPE, ALiBi, sinusoidal, RoPE	the five canonical PE classes
Tasks	prev, skip-2/3/5, copy, reverse, first	fixed-relative, growing-offset, fixed-absolute regimes
	( $+$ pos-2/5, App. E )	interior absolute positions, off the causal boundary
Lengths	$30,40,50,60,80$	extrapolation beyond $L_{0}{=}20$ ( $1.5\times$ – $4\times$ )
Seeds	$0,1,2,3,4$	variance / bootstrap CIs
Models	1L1H, 2L4H, 4L4H	1-layer 1-head setting $+$ depth/multi-head escape routes

Table 2: The evaluation grid. Each axis follows the theory’s categories or standard practice; the reason given for each level is the reason it was included.

PE	task	$\alpha_{L}$	LG (1L1H)	LG (2L4H)
RoPE	skip5	1.00	$1.00{\pm}0.00$	$0.97{\pm}0.02$
RoPE	prev	1.00	$0.87{\pm}0.05$	$0.85{\pm}0.13$
RoPE	copy	0.00	$0.10{\pm}0.01$	$0.14{\pm}0.01$
sinusoidal	first	1.00	$0.87{\pm}0.13$	$0.92{\pm}0.04$
sinusoidal	skip3	0.25	$0.22{\pm}0.03$	$0.11{\pm}0.01$
learned-abs	first	1.00	$1.00{\pm}0.00$	$1.00{\pm}0.00$

Table 3: Training-free

\alpha_{L}

(at

L{=}60

) vs. empirical LG accuracy (mean over

5

seeds), 1-layer 1-head and scale (2L4H) tiers. Top:

\alpha_{L}

tracks 1L1H LG. Bottom: at scale, the content/recency channel (outside the positional-only abstraction, S3) lets ALiBi/NoPE generalize where

\alpha_{L}{=}0

, making the predictor’s one-sidedness visible.

Figure 3: Training-free

\alpha_{L}

vs. empirical LG across the three tiers (

5

seeds). Left (1L1H, the setting the theory describes): learnable cells (blue) give Spearman

\rho{=}0.90

. Center, right (scale, 2L4H, 4L4H): the graded rank correlation weakens (its bootstrap CI includes

0

) as the content channel rescues low-

\alpha_{L}

cells, yet high

\alpha_{L}

remains a sufficient screen at every depth (precision

\geq 0.92

Figure 4: Empirical LG (solid) and training-free

\alpha_{L}

(dashed) vs. test length

L

(train

L_{0}{=}20

), 1L1H.

\alpha_{L}

tracks both who generalizes and how LG decays with

L

Figure 5: Training-free $\alpha_{60}$ (left) and empirical LG@60 (right) over (PE, task), 1L1H. Agreement is strong in this single-head regime; the content-driven disagreements (which emerge at depth, not shown here) are characterized in § 7 .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

tier	depth	learnable Spearman $\rho$ (95% CI)	sufficiency precision (95% CI)
1L1H	1	0.90 $[0.73,0.95]$	0.925 $[0.88,0.96]$
2L4H	2	0.31 $[-0.16,0.76]$	0.94 $[0.89,0.97]$
4L4H	4	0.28 $[-0.15,0.66]$	0.97 $[0.94,0.99]$

Appendix

Table 4: Depth trend (H3), learnable cells (

\mathrm{train\ acc}>0.8

5

seeds. The Spearman column is cell-level with a bootstrap

95\%

CI over the (PE, task) cells; the precision column is the rate

\Pr[\mathrm{LG}\geq 0.7\mid\alpha_{L}\geq 0.5]

with a bootstrap

95\%

CI over the high-

\alpha_{L}

rows .

\alpha_{L}

stays a high-precision sufficient indicator at every depth (precision CIs well above chance); the graded rank correlation, by contrast, is significant only at depth

1

– at depths

2,4

its CI includes

0

, consistent with

\alpha_{L}

acting as a binary screen rather than a graded predictor once content rescues low-

\alpha_{L}

cells.

Figure 6: H3: learnable-cell Spearman

\rho(\alpha_{L},\mathrm{LG})

(circles) drops with depth (its bootstrap CI includes

0

at depths

2,4

), while sufficiency precision (squares) stays

\geq 0.92

with tight CIs.

\alpha_{L}

remains a sufficient screen at all depths.

kernel fit	Spearman $\rho$	Pearson
min-norm interpolant (Thm 1 ’s $W^{*}$ )	0.56	0.57
cross-entropy, wd $=10^{-4}$ (reported $\alpha_{L}$ )	0.91	0.95
cross-entropy, wd $\to 0$ (max-margin)	0.92	0.94

Appendix

Table 5: Inductive-bias ablation (

75

rows in the 1L1H setting). The cross-entropy fit, not the min-norm interpolant, drives the

\alpha_{L}

–LG correlation.

PE	task	$\alpha_{60}$	train acc	LG@60	LG@80
learned-abs	pos2	1.00	1.00	1.00	1.00
learned-abs	pos5	1.00	1.00	1.00	1.00
sinusoidal	pos2	1.00	0.98	0.98	0.97
sinusoidal	pos5	1.00	1.00	1.00	0.99
NoPE	pos2	0.00	0.24	0.16	0.16
NoPE	pos5	0.00	0.22	0.16	0.15

Appendix

Table 6: Interior fixed-absolute positions (1L1H, mean over

5

seeds). Absolute encodings realize and transfer them; relative encodings do not learn them at all. Compare first , where RoPE scores

\alpha_{60}=0.20

purely because position

0

is the causal boundary.

tier	vocab $12$ , $d{=}64$ (main)	vocab $64$ , $d{=}128$
1L1H	0.90	0.92
2L4H (scale)	0.36	0.45

Appendix

Table 7: Scale robustness: cell-level Spearman

\rho(\alpha_{L},\mathrm{LG})

on learnable cells, at the paper’s default size vs. a larger model. The predictor is identical; only the trained models change. The scale study uses

2

seeds (

700

rows), and both columns are computed on the cells the two studies have in common, so the

2

4

H entry (

0.36

) is over

19

cells rather than the

20

of Table 4 , where the same quantity over all

20

cells is

0.31

	$L_{0}{=}10\ (\to 30)$	$L_{0}{=}20\ (\to 60)$	$L_{0}{=}40\ (\to 120)$
screen AUC	1.00	1.00	1.00
# generalizing cells	4/25	4/25	4/25
rope / prev ( $\alpha_{L}$ / LG)	1.00 / 0.93	1.00 / 0.87	1.00 / 0.87
rope / skip-3	1.00 / 1.00	1.00 / 1.00	1.00 / 0.98
rope / copy (growing)	0.00 / 0.12	0.00 / 0.10	0.00 / 0.09
rope / reverse (growing)	0.00 / 0.16	0.03 / 0.14	0.00 / 0.12

Appendix

Table 8:

L_{0}

-sensitivity (1L1H, predictor

\alpha_{L}

/ empirical LG at

L{=}3L_{0}

). The screen AUC is

1.0

at every training length and the per-cell verdicts are stable; the separation is not an artifact of

L_{0}{=}20

. Generated by experiments/l0_sensitivity.py .

PE	train acc (2L4H)	LG@60 (2L4H)
ALiBi	0.96	0.84
RoPE	1.00	0.43
NoPE	0.76	0.34
learned-abs	0.99	0.27
sinusoidal	0.98	0.26

Appendix

Table 9: Content-driven induction (target

=

token after the previous occurrence of

x_{i}

): positional

\alpha_{L}=0

for all PEs, yet

2

-layer models generalize (LG up to

0.84

\alpha_{L}^{+}

flags the realizable through the content channel regime

\alpha_{L}

misses; residual PE-modulation of content LG is future work.

RoPE freq. scale	$\alpha_{L}$	train acc	LG@60
$1.0$ / $0.3$	1.00	1.00	1.00
$0.1$	1.00	0.94	0.93
$0.03$	0.00	0.88	0.88
$0.01$	0.00	0.23	0.18

Appendix

Table 10: RoPE frequency sweep on content-neutral skip-3 (1L1H). No discriminating cell: where the model generalizes the taxonomy is right; at scale

0.03

\alpha_{L}

under-predicts (false negative); at

0.01

the model cannot learn at all.

RoPE setting	prev	skip3	reverse	first
$\mathrm{num}{=}8$ ( $m{=}16$ )	1.00	1.00	0.00	0.17
$\mathrm{num}{=}16$ ( $m{=}32$ )	1.00	1.00	0.03	0.20
$\mathrm{num}{=}32$ ( $m{=}64$ )	1.00	1.00	0.00	0.03
base $10^{3}$	1.00	1.00	0.03	0.05
base $10^{5}$	1.00	1.00	0.03	0.03

Appendix

Table 11:

\alpha_{L}

for RoPE under varying frequency count

\mathrm{num}

(feature dimension

m{=}2\,\mathrm{num}

) and

\theta

-base. The verdict is invariant: fixed relative offsets transfer (

\mathrm{prev,skip3}{=}1.00

everywhere), growing offsets never transfer (copy/reverse

{\approx}0

), and the kernel the

L_{0}

-fit selects does not hit a fixed absolute position (first stays at chance; a monotone kernel in the family could, but the fit does not select it, see the discussion after Prop. 4 ). For low-frequency scaling see Table 10 .

sinusoidal setting	prev	skip3	copy	first
$\mathrm{num}{=}8$ ( $m{=}16$ )	0.47	0.50	0.00	0.93
$\mathrm{num}{=}16$ ( $m{=}32$ )	0.45	0.25	0.07	1.00
$\mathrm{num}{=}32$ ( $m{=}64$ )	0.00	0.00	0.00	1.00
base $10^{3}$	0.17	0.28	0.00	1.00
base $10^{5}$	0.15	0.12	0.00	1.00

Appendix

Table 12:

\alpha_{L}

for sinusoidal under the same sweep. The absolute verdict is invariant (first

{\approx}1

everywhere; growing offsets

{\approx}0

). The bounded-offset cells (prev/skip3) are the only entries whose verdict moves with frequency – they degrade as frequencies are added (

\mathrm{num}{:}8{\to}32

drives prev

0.47{\to}0.00

); the other cells’ raw

\alpha_{L}

shifts stay sub-threshold (e.g. first

0.93{\to}1.00

) and never flip the verdict. This is what the theory predicts: enlarging the realizable space does not help the

L_{0}

-fit select a transferring kernel (existence

\neq

selection, Prop. 3 ). Verdict movement appears precisely at the selection gap and nowhere else.

Length Generalization

Length generalization studies whether sequence models trained on short inputs can reliably solve the same tasks on longer sequences.

See all 50 papers

Jul 21, 2026

Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention

Transformers with relative positional encodings often extrapolate to sequences longer than those seen during training, whereas transformers with learned absolute encodings typically do not. This is a robust empirical regularity, and the explanations offered for it so far are chiefly about expressivity, that is, about whether a length-generalizing solution exists. We give an optimization explanation. On a minimal fixed-offset retrieval task that isolates positional selection, the gap is governed by the implicit bias of the trained attention head: among the many solutions that fit short sequences, which one gradient descent actually selects. We prove that rotary encodings make the attention logit a function of relative offset alone, an exact equivariance, so whatever selection rule is learned at training lengths is reproduced verbatim at every longer length. Learned absolute encodings instead leave out-of-range positions unconstrained, and the trained head pins to a fixed absolute position inside the training range. We characterize the learned rotary rule as a low-rank ``carrier'' kernel aligned with the target offset, and we derive the resulting graceful accuracy decay as an attention-dilution law; both predictions are confirmed across seeds and offsets. A linear-attention control shows the mechanism is specific to softmax: without normalization, training selects a min-norm interpolant that does not extrapolate. The phenomenon, the equivariance, and the carrier all transfer to a multi-layer, multi-head transformer trained on a full-sequence length-generalization task. The account connects the implicit bias of attention, implicit bias for extrapolation in recurrent models, and the learning side of the RASP-L conjecture.

Subham Singh, Ashutosh Mishra, Subha Raut

Department of Mathematics and Statistics, Mississippi State University · Independent Researcher

Softmax Attention Rotary Positional Embeddings Length Generalization Implicit Bias Relative Positional Encoding

Ask PDF

May 29, 2026

Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization

Transformer-based language models are widespread in today's society. As such, understanding the mechanisms by which they solve structured tasks and predicting how they may behave in novel scenarios is of great importance for safe deployment. We study the learning dynamics of attention heads in a controlled setting by training a decoder-only Transformer (GPT-J) on two structurally equivalent multi-hop reasoning tasks: a number task requiring positional reasoning and a letter task requiring symbolic reasoning. Using a recently introduced metric that classifies attention-head behavior as positional or symbolic for a given prompt, we show that successful learning is associated with the emergence of pure heads, i.e., heads that express themselves as either positional or symbolic. Despite the tasks' structural equivalence, they impose different mechanistic demands: the number task requires both positional and symbolic heads, whereas the letter task requires only symbolic heads. We then identify the computational roles of these heads, characterize the basic functions they implement, and give theoretical constructions showing how single-layer RoPE-based attention can realize these functions through geometrically interpretable query, key, and value operations. This analysis yields a quantitative separation between positional and symbolic mechanisms in their robustness to longer sequences, formalized through a novel notion of discrepancy. We empirically validate the resulting predictions in both controlled and real-world models, showing that symbolic mechanisms extrapolate more reliably to longer sequences while positional mechanisms face sharper limitations.

Felipe Urrutia, Juan José Alegría, Cinthia Sanchez Macias +3

CENIA & Faculty of Mathematics UC Santiago, Chile · CENIA Santiago, Chile · IMC UC & CENIA Santiago, Chile

Rotary Positional Embeddings Attention Head Analysis Length Generalization Multi-Head Attention Attention Head Specialization

Ask PDF

Aug 9, 2026

Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling

Self-attention models content-dependent interactions between tokens but does not by itself encode token order. Position encoding addresses this limitation by introducing absolute coordinates, relative distances, or position-dependent rotations into Transformer representations and attention scores. This technical survey develops a unified account of sinusoidal and learned absolute position embeddings, Shaw-style relative position representations, Transformer-XL, T5 relative position bias, ALiBi, and Rotary Position Embeddings (RoPE). We derive how RoPE converts absolute position indices into relative phase differences in Query-Key inner products and compare these methods in terms of where position is injected, computational cost, compatibility with KV caching, and length extrapolation. We then examine long-context extensions, including Position Interpolation, RoPE scaling laws, NTK-aware scaling, Dynamic NTK, NTK-by-parts, YaRN, LongRoPE, and LongRoPE2, with emphasis on frequency allocation, attention rescaling, training length, and target context length. We also summarize implementation considerations, evaluation protocols, and position-encoding choices in representative large language models. A central conclusion is that the ability to compute positional features beyond the training length does not imply reliable long-context generalization; context extension must be evaluated through short-context retention, position-wise perplexity, retrieval, reasoning, and long-context code tasks.

Jiguo Li

Rotary Positional Embeddings Long-Context Modeling Positional Encoding LLM Context Management Relative Positional Encoding

Ask PDF

Abstract

Length Generalization

Explore similar work

Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention

Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization

Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling