Organizations: LIX, École Polytechnique, IP Paris, France · Centre Borelli, ENS Paris-Saclay, Université Paris-Saclay, France · Centre d’Analyse et de Mathématique Sociales, EHESS, CNRS, 75006 Paris, France
What are the inductive biases of a Transformer architecture? Existing theory on how the forward pass shapes representations either considers whether Transformers escape from rank collapse or demonstrates that self-attention drives tokens toward cluster patterns. The latter view arises from an elegant dynamical systems perspective, but relies on simplified architectural assumptions, and does not explain the rich structures observed in practice. This leaves a major open question: when a full Transformer escapes rank collapse, how does it structure token representations? Using pattern-formation theory, we show that the dynamical view of Transformers can account for Positional Encoding, Multi-Head Attention, and Output-Value geometry. We demonstrate that a full Transformer architecture imposes an inductive prior by selectively amplifying a rich set of previously unreported patterns, including traveling or rotating waves among others. We characterize the role of each architectural component in controlling which pattern is amplified, which ones stabilize, compete, or coexist. Finally, we show that these structures can act as a controllable dynamical prior that facilitates learning. By choosing both task-aligned positional encoding and weight initialization, we demonstrate improved data efficiency and accelerated optimization on controlled sequence tasks and with ConViT on CIFAR-10.
Figures & tables
Figure 1: Taxonomy of patterns in Transformers: 2D visualization of the propagation of the accessible modes, with their corresponding dispersion relation.
Figure 2: Depth evolution descends an effective potential. (a,b) One critical mode, Eq. 12 (blue: β>0 , orange: β<0 ; disks: stable states, circles: unstable): the dispersion relation sets μ , the nonlinearities set β . (c,d) Two critical modes, Eq. 19 , over the signed amplitudes (R1,R2) : the ratio of cross-suppression βcross to self-saturation βself decides where the wells are.
Figure 3: Prediction of the amplitudes of representations . Lines: theory, colors: simulation, gray: collapsed state, cyan line: threshold maxq∣J(q)∣=1 . (a): Amplitude scaling laws for multiple patterns versus μ . (b): Amplitude heatmap of the traveling wave, in the (∣η∣,s) map; white dashed: predicted R⋆ . (c) Critical mode kobs in the (Δ,s) map; black: theoretically predicted mode transitions ( × : nonlinear selection differs); (d): Competition and coexistence of modes 4 (solid), 5 (dashed) in (w,s) with winner-takes-all exchange lines of Thm. 5.3 (dotted).
Figure 4: Task–initialization alignment. Left: the periodic (T1), transport (T2) and rotation (T3) tasks (perturbed, partially masked targets). Right: AUC for each (task, initialization) pair; the color corresponds to the z -score. Higher scores in the diagonal support the dynamical prior induced by mode criticality.
Figure 5: Dispersion-aware initialization of ConViT on CIFAR-10. (a) Heatmap of mode gains and radial profile of the dispersion relation for the initializations variants (colours: mode stability; ⋆ : leading mode). (b) Final Accuracy, AUC and t90 ; all variants use localized PE, “sharp”: sharpened kernel.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Extended version of Fig. 4 . Heatmap of AUC of learning curves for each (task, initialization) pair; color gives the z -scores. Higher scores in the diagonal support the dynamical prior induced by mode criticality.
Figure 7: Learning speed and data efficiency on T1: held-out cross-entropy versus training-set size (left) and learning curves (right) for the matched, mismatched and gain-matched random initializations.
Figure 8: Visualizations of the 2D dispersion relation for various combinations of positional structure and OV geometry
Figure 9: Comparison of linear theory growth prediction vs actual deviation from homogeneous state from various positional encoding and OV geometry combinations
Figure 10: CIFAR-10 optimization under dispersion-aware initialization. Test accuracy (left), test-accuracy AUC (middle), and epoch reaching 90% test accuracy (right), over three seeds. Localized positional attention combined with negative OV geometry accelerates training, with the engineered −I+ sharp-pos initialization performing best overall.
Figure 11: Data efficiency on CIFAR-10. Final test accuracy (left) and test-accuracy AUC (right) for different training-set fractions. Error bars show one standard deviation across three seeds, except the 5% default baseline ( n=2 ). The engineered initialization remains strongest across the sweep.
Figure 12: Training diagnostics for the tied ConViT. Final-depth token heterogeneity (left), learned attention residual scale ϵa (middle), and mean GPSA positional gate (right). Representations remain non-collapsed while the attention parameters adapt throughout training.
Figure 13: Evolution of the spatial dispersion relation. ρ(J(q)) at epochs 0,1,5,25, and 100 for one seed; stars denote the maximally amplified Fourier sector. Both initial spectra evolve substantially during training.
Initialization
Final acc. (%)
AUC (%)
t90
GPSA, no normalization
default + pos
89.6±0.4
78.0±0.3
92.0(1/3)
−I + pos
91.1±0.3
80.8±0.4
74.3±3.1
−I + sharp pos
91.5±0.2
81.3±0.3
69.0±1.7
GPSA, pre-LN
default + pos
88.5±0.4
77.0±0.4
–
Appendix
Table 1: Normalization and attention-form ablations. Parentheses indicate the number of contributing seeds when fewer than three runs are available; for t90 , they indicate the number of seeds reaching 90% accuracy within 100 epochs.
Figure 14: The Swift-Hohenberg illustration: the dispersion relation (a) defines the emergent perturbations, the amplitude potential in (b) dictates their stability, leading to the apparition of patterns in (c).
This paper studies simple transformers trained on a high-order Markov chain, where the model must incorporate information from multiple past positions, each with different statistical importance. We show that transformers learn the task incrementally, with each stage corresponding to learning how to copy information from a subset of positions via a sparse attention pattern. Notably, the learning dynamics transition from a competitive phase, where all heads focus on the statistically most important positions, to a cooperative phase, where different heads specialize in different patterns. We model these dynamics with simplified differential equations and prove stage-wise convergence of the resulting system. Functionally, these stages correspond to a sequence of increasingly expressive misspecified models, with the full model class reached only at the end. Overall, we give a theoretical account of how structured attention patterns and head specialization emerge in stages without an explicit curriculum, with implications for generalization in sequential tasks.
Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas Flammarion
Transformer models systematically favor certain token positions, yet the architectural origins of this position bias remain poorly understood. This bias is closely connected to the Lost-in-the-Middle phenomenon, where models underutilize information placed in the middle of the context. We show that causal Transformer architectures induce a systematic position-bias prior that can help explain the influence profiles associated with Lost-in-the-Middle. To characterize this prior, we develop a structural theory based on cumulative attention rollout with layerwise residual mixing. At finite depth, we isolate the competing positional drifts induced by causal masking, residual mixing, and positional encoding. Their interaction produces broad, often U-shaped profiles. At infinite depth, prior work predicts inevitable collapse under attention-only theory and identifies the discrepancy with observed Transformer behavior as an open problem. Under the stated boundedness assumptions, our residual-aware analysis replaces inevitable collapse with a dichotomy governed by cumulative attention mixing. Empirically, across 30 model-dataset settings, our adaptive residual-aware rollout closely matches measured input-token influence and reduces the average profile discrepancy by approximately 85% relative to both the attention-only and fixed-residual rollout baselines.
Hanna Herasimchyk, Robin Labryga, Tomislav Prusina +1
A widely cited result by Dong et al. (2021) showed that Transformers built from self-attention alone, without skip connections or feed-forward layers, suffer from rapid rank collapse: all token representations converge to a single direction. The proposed remedy was the MLP. We show that this picture, while correct in the regime studied by Dong, is incomplete in ways that matter for architectural understanding. Three results are established. First, layer normalisation is precisely affine-rank-neutral: it preserves the affine rank of the token representation set exactly. The widespread claim that LN "plays no role" is imprecise; the correct statement is sharper. Second, residual connections generically obstruct rank collapse in real Transformers such as BERT-base, in a measure-theoretic sense, without contribution from the MLP. The MLP's irreplaceable function is different: generating feature directions outside the linear span of the original token embeddings, which no stack of attention layers can produce. Third, a phenomenon distinct from rank collapse is identified: head-channel non-identifiability. After multi-head attention sums per-head outputs through the output projection, individual contributions cannot be canonically attributed to a specific head; n(H-1)d_k degrees of freedom per layer remain ambiguous when recovering a single head from the mixed signal. The MLP cannot remedy this because it acts on the post-summation signal. A constructive partial remedy is proposed: a position-gated output projection (PG-OP) at parameter overhead below 1.6% of the standard output projection. The four collapse phenomena identified in the literature -- rank collapse in depth, in width, head-channel non-identifiability, and entropy collapse -- are unified under a symmetry-breaking framework, each corresponding to a distinct symmetry of the Transformer's forward pass.
Giansalvo Cirrincione
aLaboratoire LTI, Université de Picardie Jules Verne, Chemin du Thil, 80025 Amiens, France