A Structural Theory of Position Bias in Transformers

Hanna Herasimchyk; Robin Labryga; Tomislav Prusina; Sören Laue

cs.LGFeb 18, 2026

A Structural Theory of Position Bias in Transformers

Authors: Hanna Herasimchyk, Robin Labryga, Tomislav Prusina, Sören Laue

Organizations: University of Hamburg Hamburg, Germany

Abstract

Transformer models systematically favor certain token positions, yet the architectural origins of this position bias remain poorly understood. This bias is closely connected to the Lost-in-the-Middle phenomenon, where models underutilize information placed in the middle of the context. We show that causal Transformer architectures induce a systematic position-bias prior that can help explain the influence profiles associated with Lost-in-the-Middle. To characterize this prior, we develop a structural theory based on cumulative attention rollout with layerwise residual mixing. At finite depth, we isolate the competing positional drifts induced by causal masking, residual mixing, and positional encoding. Their interaction produces broad, often U-shaped profiles. At infinite depth, prior work predicts inevitable collapse under attention-only theory and identifies the discrepancy with observed Transformer behavior as an open problem. Under the stated boundedness assumptions, our residual-aware analysis replaces inevitable collapse with a dichotomy governed by cumulative attention mixing. Empirically, across 30 model-dataset settings, our adaptive residual-aware rollout closely matches measured input-token influence and reduces the average profile discrepancy by approximately 85% relative to both the attention-only and fixed-residual rollout baselines.

Figures & tables

Table 1

Panel of Figure 1 — Figure 1: Final-token profiles at sequence length $n=256$ for the 36-layer falcon-rw-7b (top) and the 36-layer qwen3-8b-base (bottom), representing ALiBi and RoPE, respectively. Panels (a)–(c) show the Wu et al. attention-only rollout ( $\lambda_{t}=1$ ), the Abnar–Zuidema fixed residual rollout ( $\lambda_{t}=\tfrac{1}{2}$ ), and our adaptive rollout with measured schedules $\{\bar{\lambda}_{t}\}$ , respectively. Panel (d) shows the gradient influence proxy $\hat{q}_{\mathrm{grad}}$ .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Model	FineWeb-Edu			DCLM-Baseline			Wikipedia
Model	(a) Wu $\lambda=1$	(b) Abnar $\lambda=\tfrac{1}{2}$	(c) Ours adaptive	(a) Wu $\lambda=1$	(b) Abnar $\lambda=\tfrac{1}{2}$	(c) Ours adaptive	(a) Wu $\lambda=1$	(b) Abnar $\lambda=\tfrac{1}{2}$	(c) Ours adaptive
falcon-rw-7b	0.66	0.63	0.03	0.66	0.64	0.02	0.68	0.65	0.05
mpt-7b	0.62	0.58	0.15	0.59	0.55	0.19	0.66	0.62	0.11
mpt-30b	0.61	0.61	0.02	0.58	0.58	0.04	0.65	0.65	0.02
bloom-7b1	0.63	0.58	0.13	0.61	0.56	0.15	0.67	0.61	0.10
bloom-176b	0.65	0.65	0.11	0.63	0.63	0.06	0.69	0.69	0.11

Appendix

Table 3: Normalized

1

-Wasserstein distances at

n=256

(↓ lower is better) between controlled rollout variants and gradient influence. Columns (a), (b), and (c) use

\lambda_{t}=1

following Wu et al. [2025] ,

\lambda_{t}=\tfrac{1}{2}

following Abnar and Zuidema [2020] , and our adaptive measured norm-Shapley coefficient

\lambda_{t}

, respectively.

Panel of Figure 2 — Figure 2: Final-token profiles at sequence length $n=256$ on FineWeb-Edu for the four ALiBi models not shown in Figure 1 . The columns show attention-only rollout, fixed residual rollout, adaptive residual rollout, and measured gradient influence.

Panel of Figure 3 — Figure 3: Final-token profiles at sequence length $n=256$ on FineWeb-Edu for the four RoPE models not shown in Figure 1 . The columns show attention-only rollout, fixed residual rollout, adaptive residual rollout, and measured gradient influence.

Panel of Figure 4 — Figure 4: Depth-wise norm-Shapley mixing coefficients for the ALiBi models across three datasets. Mean $\pm$ 95% confidence interval at sequence length $n=256$ ; the intervals are narrow relative to the plotted range.

Panel of Figure 5 — Figure 5: Depth-wise norm-Shapley mixing coefficients for the RoPE models across three datasets. Mean $\pm$ 95% confidence interval at sequence length $n=256$ ; the intervals are narrow relative to the plotted range.

Model	FineWeb-Edu			DCLM-Baseline			Wikipedia
Model	Cross- layer	Within- layer	Combined	Cross- layer	Within- layer	Combined	Cross- layer	Within- layer	Combined
falcon-rw-7b	0.17	0.00	0.17	0.31	0.00	0.31	0.20	0.00	0.20
mpt-7b	0.01	0.00	0.01	0.02	0.00	0.02	0.01	0.00	0.01
mpt-30b	0.10	0.00	0.10	0.17	0.00	0.17	0.39	0.00	0.39
bloom-7b1	0.03	0.00	0.03	0.03	0.00	0.03	0.03	0.00	0.03
bloom-176b	0.02	0.00	0.02	0.02	0.00	0.02	0.02	0.00	0.02

Appendix

Table 4: Corpus-level rollout approximation ablation at

n=256

under the norm-Shapley protocol. Entries are normalized final-row

1

-Wasserstein distances in units of

\mathbf{10^{-3}}

(lower is better).

Panel of Figure 6 — Figure 6: Gradient input-token influence $\hat{q}_{\mathrm{grad}}$ at sequence length $n=2048$ on Wikipedia for ALiBi (top) and RoPE (bottom). The profiles retain both primacy and recency peaks, although their prominence varies across models; in particular, they do not collapse onto the first token at the longer context length.

Model	(a) Wu $\lambda=1$	(b) Abnar $\lambda=\tfrac{1}{2}$	(c) Ours adaptive
falcon-rw-7b	0.46	0.20	0.14
mpt-7b	0.46	0.18	0.16
mpt-30b	0.82	0.45	0.07
bloom-7b1	0.45	0.15	0.14
bloom-176b	0.74	0.49	0.16
qwen2.5-7b	0.72	0.55	0.05

Appendix

Table 5: Context-length ablation at

n=2048

on Wikipedia: normalized

1

-Wasserstein distances (↓ lower is better) to gradient influence.

Model	(a) Wu $\lambda=1$	(b) Abnar $\lambda=\tfrac{1}{2}$	(c) Ours adaptive	Blank-out vs. gradient
falcon-rw-7b	0.72	0.69	0.09	0.10
mpt-7b	0.66	0.62	0.11	0.08
mpt-30b	0.60	0.59	0.04	0.03
bloom-7b1	0.68	0.63	0.08	0.06
bloom-176b	0.59	0.59	0.17	0.06
qwen2.5-7b	0.65	0.62	0.09	0.06

Appendix

Table 6: Influence-measurement ablation at

n=256

: normalized

1

-Wasserstein distances (↓ lower is better) to blank-out influence. Columns (a), (b), and (c) use

\lambda_{t}=1

following Wu et al. [2025] ,

\lambda_{t}=\tfrac{1}{2}

following Abnar and Zuidema [2020] , and our adaptive measured norm-Shapley coefficient

\lambda_{t}

, respectively. The final column directly compares blank-out and gradient influence.

		Prefix mass		Recency mass
Model	$\beta$	Predicted	Measured [95% CI]	Predicted	Measured [95% CI]
bloom-7b1	0.8	$-1.21$	$-0.74$ [ $-0.89,-0.58$ ]	$+8.44$	$+2.31$ [ $+1.99,+2.62$ ]
	1.2	$+1.41$	$+0.78$ [ $+0.69,+0.88$ ]	$-7.47$	$-2.12$ [ $-2.37,-1.84$ ]
mpt-7b	0.8	$-1.04$	$-0.51$ [ $-1.39,+0.53$ ]	$+8.92$	$+1.61$ [ $+0.46,+2.81$ ]
	1.2	$+1.08$	$+1.41$ [ $+0.59,+2.39$ ]	$-7.14$	$-4.62$ [ $-5.89,-3.46$ ]
falcon-rw-7b	0.8	$-2.12$	$-0.79$ [ $-1.10,-0.44$ ]	$+5.99$	$+0.17$ [ $-0.48,+0.83$ ]

Appendix

Table 7: Predicted and measured changes under attention-branch scaling, relative to the unmodified model (

\beta=1

). Changes are percentage points of normalized prefix or recency mass. Brackets give paired

95\%

bootstrap confidence intervals for measured changes.

	$\beta=0.8$		$\beta=1.2$
Model	KL	Top-1	KL	Top-1
bloom-7b1	0.03	91%	0.02	93%
mpt-7b	0.22	83%	0.08	85%
falcon-rw-7b	0.09	88%	0.05	89%
qwen2.5-7b	0.08	88%	0.03	92%
qwen3-8b-base	0.05	90%	0.03	94%

Appendix

Table 8: Intervention stability. KL is the mean next-token

D_{\mathrm{KL}}(p_{\beta}\|p_{1})

; Top-1 is the percentage of prompts retaining the unmodified next-token prediction.

Model	(a) Wu $\lambda=1$	(b) Abnar $\lambda=\tfrac{1}{2}$	(c) Ours relative norm	(d) Ours norm-Shapley
falcon-rw-7b	0.66	0.63	0.19	0.03
mpt-7b	0.62	0.58	0.09	0.15
mpt-30b	0.61	0.61	0.16	0.02
bloom-7b1	0.63	0.58	0.04	0.13
bloom-176b	0.65	0.65	0.08	0.11
qwen2.5-7b	0.63	0.59	0.29	0.12

Appendix

Table 9: Mixing-coefficient ablation at

n=256

: normalized

1

-Wasserstein distances (↓ lower is better) to gradient influence on FineWeb-Edu. Columns (a) and (b) use

\lambda_{t}=1

following Wu et al. [2025] and

\lambda_{t}=\tfrac{1}{2}

following Abnar and Zuidema [2020] . Columns (c) and (d) use our adaptive rollout with the relative-norm and norm-Shapley coefficients, respectively. Bold indicates the lowest distance in each row.

Resource	License	Link	Notes / Citation
Falcon	Apache 2.0	Model repository	[ Penedo et al., 2023 ]
MPT	Apache 2.0	Model repository	[ Mosaic, 2023a , Mosaic, 2023b ]
BLOOM	BigScience RAIL License v1.0	Model repository	[ Scao et al., 2022 ]
Qwen2.5 (7B, 32B)	Apache 2.0	Model repository	[ Qwen Team, 2024 ]
Qwen2.5 (72B)	Qwen License Agreement	Model repository	[ Qwen Team, 2024 ]
Qwen3-8B-Base	Apache 2.0	Model repository	[ Qwen Team, 2025 ]

Appendix

Table 10: License summary for models and datasets used in this work.

Explore similar work

Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias
Mar 10, 2026Borun D ChowdhuryTransformer Transformer Attention
Meta, London, W1T 1FB, UK
An Adjoint-Sensitivity Framework for Lost-in-the-Middle Phenomena in Causal Residual Transformers
Jul 20, 2026Cheng Huan, Hongwei YuanTransformer Attention Positional Encoding
Department of Statistics and Data Science, The Chinese University of Hong Kong. · Department of Mathematics, University of Macau.
Pattern Formation in Transformers
Sep 29, 2026Erkan Turan, Gaspard Abel, Maks OvsjanikovTransformer Transformer Attention
LIX, École Polytechnique, IP Paris, France · Centre Borelli, ENS Paris-Saclay, Université Paris-Saclay, France · Centre d’Analyse et de Mathématique Sociales, EHESS, CNRS, 75006 Paris, France

Mar 10, 2026J/K move · Enter open · S save

Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias

Borun D Chowdhury

Meta, London, W1T 1FB, UK

Transformer Transformer Attention Decoder-Only Language Models Attention Residual Connections

Open paper AskPDF arXiv

The ``Lost in the Middle'' phenomenon -- a U-shaped performance curve where LLMs retrieve well from the beginning and end of a context but fail in the middle -- is widely attributed to learned Softmax artifacts or the distance-decay of positional encodings like RoPE. This paper makes a single, precise claim: the U-shape is already present at initialization, before any training or positional encoding takes effect. It is an inherent geometric property of the causal decoder with residual connections. We model multi-layer causal attention as iterated powers of the Cesàro matrix and derive the exact closed-form influence density in the continuous limit. Causal masking forces a logarithmic divergence of gradient influence at the start of the prompt (the Primacy Tail), while residual connections create an isolated

\mathcal{O}(1)

anchor at the final token (the Recency Delta). Between these extremes lies a factorial dead zone of order

\mathcal{O}(1/(H{-}1)!)

, where

H

is the network depth, making middle-context retrieval and training structurally hostile. We validate empirically that untrained Qwen2 and GPT-2 architectures exhibit this U-shape at Step~0, and that it is identical with or without RoPE. Comparing initialized and pretrained networks, we show that standard training does not overcome the topological valley, confirming that the U-shape persists as an architectural baseline under standard pretraining objectives. We do not claim that this bias is insurmountable, nor that interventions such as RoPE modifications are useless. We establish what the baseline is and where it comes from, so that future efforts to overcome it can be precisely targeted.

Abstract

Explore similar work

Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias

An Adjoint-Sensitivity Framework for Lost-in-the-Middle Phenomena in Causal Residual Transformers

Pattern Formation in Transformers