The ``Lost in the Middle'' phenomenon -- a U-shaped performance curve where LLMs retrieve well from the beginning and end of a context but fail in the middle -- is widely attributed to learned Softmax artifacts or the distance-decay of positional encodings like RoPE. This paper makes a single, precise claim: the U-shape is already present at initialization, before any training or positional encoding takes effect. It is an inherent geometric property of the causal decoder with residual connections. We model multi-layer causal attention as iterated powers of the Cesàro matrix and derive the exact closed-form influence density in the continuous limit. Causal masking forces a logarithmic divergence of gradient influence at the start of the prompt (the Primacy Tail), while residual connections create an isolated O(1) anchor at the final token (the Recency Delta). Between these extremes lies a factorial dead zone of order O(1/(H−1)!), where H is the network depth, making middle-context retrieval and training structurally hostile. We validate empirically that untrained Qwen2 and GPT-2 architectures exhibit this U-shape at Step~0, and that it is identical with or without RoPE. Comparing initialized and pretrained networks, we show that standard training does not overcome the topological valley, confirming that the U-shape persists as an architectural baseline under standard pretraining objectives. We do not claim that this bias is insurmountable, nor that interventions such as RoPE modifications are useless. We establish what the baseline is and where it comes from, so that future efforts to overcome it can be precisely targeted.
Figures & tables
Figure 1: Empirical Validation of Position Bias on Qwen2-0.5B at Initialization (Step 0). Top Row (Theory): The exact topological solutions for pure Causal Primacy (Eq. 5 ), Hybrid Residual Recency (Eq. 6 ) (plotted on a logarithmic scale), and the theoretical irrelevance of positional encodings at initialization. Bottom Row (Empirical): The measured Input-Output Jacobian norm ρ(x) for the 24-layer Qwen2 architecture prior to training. The empirical network tightly recovers the mathematically derived asymmetric U-shape. Crucially, Exp 3 demonstrates that Rotary Positional Embeddings (RoPE) have no structural effect on the topological U-shape at initialization, confirming our hypothesis that middle-context degradation is an inherent property of causal residuals.
Figure 2: Jacobian Norm: Initialization vs. Pretrained Qwen2-0.5B ( H=24 , L=2048 ), Vanilla vs. Chunked Context. Averaged over 200 NQ sequences with p16–p84 percentile bands. Top row (Vanilla): Each sequence is a single NQ document truncated to 2048 tokens. Bottom row (Chunked): Each sequence concatenates 300-token excerpts from distinct NQ documents with no separator tokens, boundaries aligned at positions 0, 300, 600, … Left column (Initialization): Both conditions exhibit the smooth Cesàro U-shape with no sensitivity to content or chunk boundaries. Right column (Pretrained): The macroscopic U-shape persists. In the chunked condition, sharp spikes emerge at the 300-token document boundaries, confirming that these are learned content-discontinuity detectors rather than positional artifacts.
Figure 3: Evolution of the Jacobian Topology During Early Pretraining (Steps 0–100). Measured on Qwen2-0.5B with a sequence length of L=2048 . Left (Raw Magnitude): The global gradient norm decreases as the model stabilizes, but the macroscopic U-shaped topology persists rigidly. Right (Anchored): By normalizing the Recency Anchor ( x=1 ) to 1.0 , we observe that the relative depth of the “Lost-in-the-Middle” valley actually increases during training. The optimizer does not flatten the combinatorially suppressed middle section, instead relying increasingly on the geometric path of least resistance: the residual Recency Anchor and the logarithmic Primacy Tail.
Transformer models systematically favor certain token positions, yet the architectural origins of this position bias remain poorly understood. This bias is closely connected to the Lost-in-the-Middle phenomenon, where models underutilize information placed in the middle of the context. We show that causal Transformer architectures induce a systematic position-bias prior that can help explain the influence profiles associated with Lost-in-the-Middle. To characterize this prior, we develop a structural theory based on cumulative attention rollout with layerwise residual mixing. At finite depth, we isolate the competing positional drifts induced by causal masking, residual mixing, and positional encoding. Their interaction produces broad, often U-shaped profiles. At infinite depth, prior work predicts inevitable collapse under attention-only theory and identifies the discrepancy with observed Transformer behavior as an open problem. Under the stated boundedness assumptions, our residual-aware analysis replaces inevitable collapse with a dichotomy governed by cumulative attention mixing. Empirically, across 30 model-dataset settings, our adaptive residual-aware rollout closely matches measured input-token influence and reduces the average profile discrepancy by approximately 85% relative to both the attention-only and fixed-residual rollout baselines.
Hanna Herasimchyk, Robin Labryga, Tomislav Prusina +1
We study causal self-attention dynamics -- a toy model for decoder Transformers -- which we interpret as a non-exchangeable interacting particle system. Adapting cumulant expansions to the triangular causal dependency structure of the model, and appealing to non-hierarchical methods to estimate correlations using Glauber calculus, we prove a quantitative mean-field limit result and a next-order characterization of correlations. For iid uniformly distributed tokens, the limiting correlation equation can be solved in closed form and we obtain a rigorous explanation of the empirically observed lost-in-the-middle phenomenon: the token retrieval profile, as a function of the source position in the prompt, is U-shaped, with primacy, recency, and a unique interior minimum under an explicit smallness condition.
We develop an adjoint-sensitivity framework for positional influence in causal residual Transformers and separate unconditional analytic results from conditional boundary-shape conclusions. The principal unconditional theorem is the residual-to-depth-flow estimate for layer controls converging in L1, complemented by a finite-token-to-Volterra attention estimate that explicitly controls the first cells near the causal endpoint. We define a normalized adjoint-energy influence density and derive its exact evolution along full-batch gradient flow. The adjoint admits an exact generator-term decomposition into residual transmission, nonlocal Volterra, and local channels, including all covariance cross terms. Causal masking can amplify early-position sensitivity and residual identity paths can transmit a right-localized terminal bias, but neither mechanism alone forces a U-shaped profile. We therefore state boundary advantages under independently checkable energy, correlation, and local-channel bounds; these conditions are sufficient rather than necessary. Finite-token influence balancing, positional reweighting, and task-aligned observability are presented as diagnostics or regularizers with explicit differentiation requirements, computational costs, and limitations. Controlled simulations illustrate that each intervention controls its designated surrogate, while observability balance or outer-loop reweighting need not monotonically reduce the influence-based Lost-in-the-Middle diagnostic.
Cheng Huan, Hongwei Yuan
Department of Statistics and Data Science, The Chinese University of Hong Kong. · Department of Mathematics, University of Macau.