Attention heads specialize into distinct functions during training, often emerging in stages together with sharp changes in model behavior. While this phenomenon has mostly been studied as a function of training time, we ask how head specialization depends on the amount of available training data. Empirically, we find that heads specialized to syntactic relations in BERT emerge as the dataset size increases. We then study this phenomenon in a solvable high-dimensional model of multi-head attention and establish that, in this model, head specialization is a genuine phase transition in sample complexity, corresponding to a non-analyticity in the test error and other observables. Depending on data structure and regularization, specialization can follow uninformative head differentiation or occur through a cascade of phase transitions that progressively recover weaker latent features, providing a statistical form of simplicity bias. Finally, we relate these statistical transitions to the stagewise emergence of specialized heads during online learning, disentangling statistical from dynamical origins of head specialization.
Figures & tables
Figure 1: Head specialization as a function of sample complexity. Left: BERTSmall models trained independently on fixed datasets of N sequences through minimization of masked-language-modeling (MLM) loss. (first panel) Validation error and training MLM loss ( 140 ). (second panel) Unlabeled attachment score (UAS)( 143 ), measuring the fraction of dependency parents correctly recovered from word-level attention by the best head, layer, and attention direction. Curves show averages over five runs, with shaded bands indicating seed variability. Reported metrics are evaluated at early-stopped checkpoints selected by the lowest validation loss along each training trajectory. Experimental details are provided in App. E.2 . Right: Qualitative comparison with the solvable flipping spike model (Sec. 2 ) as the sample complexity α=N/D increases, with N the number of sequences and D the token dimension. Solid lines correspond to theoretical predictions and markers to numerical experiments with D=2000 , ν1=0.35 , ν2=2.0 , L=10 , H=2 , λ=0.2 . (third panel) Test and training errors exhibit non-analyticity at the specialization threshold. (fourth panel) Head–feature alignment reveals specialization to the latent feature.
Figure 2: Head-specialization phase transition in the flipping spike model: ν1=0.1 , ν2=2.0 , L=10 , H=2 , λ=0.2 , D=2000 . Here and in Fig. 3 and 4 , black solid lines correspond to theoretical predictions (Res. 3.1 ). Markers with error bars correspond to training the model (Sec. 2 ) with L-BFGS from random initialization in dimension D across ten seeds. Left: Head-feature overlaps (m:2,m:1) parametrized by sample complexity. Grey arrows indicate the directions of the two possible draws of the planted spike k^ . Center: Test error E and training error Etrain . Right: Norm of head-feature overlaps ∥m:1∥2 , ∥m:2∥2 . Grey and blue identify the permutation symmetric ( PS ) and specialized ( SPEC ) phases. Vertical lines mark the specialization threshold αcspec .
Figure 3: Head differentiation without specialization in the flipping spike model: ν1=0.5 , ν2=2.0 , L=10 , H=2 , D=6000 . Left: Phase diagram in sample complexity α and regularization λ . Phase boundaries are obtained by stability analysis of the different solutions (App. C ). The red horizontal line at λ=0.065 selects the cut shown in the other panels. Center: Head self-overlaps q11=q22 and mutual overlaps q12=q21 . Right: Head-feature overlaps m12 and m22 . Grey, orange and blue identify the permutation symmetric ( PS ), permutation symmetry broken ( PSB ) and specialized ( SPEC ) phases, respectively. Vertical lines mark their boundaries αcpsb and αcspec .
Figure 4: Head-specialization cascade in the non-isotropic Gaussian model: F=2 , νtot=9.0 , L=5 , H=4 , λ=0.2 , D=4000 . Left: Phase diagram in sample complexity α and anisotropy δ . Phase boundaries are obtained by stability analysis of the different solutions (App. C ). The red horizontal line at δ=1/9 corresponding to (ν1,ν2)=(5.0,4.0) selects the cut shown in the other panels. Center: Test error E and training error Etrain . Right: Head-feature overlaps mhf for all heads and both latent features. Grey, light blue and darker blue identify the permutation symmetric ( PS ) and successively specialized ( 1 -SPEC , 2 -SPEC ) phases, respectively. Vertical dash-dotted lines mark the specialization thresholds αc(1) and αc(2) along the cut.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Bayes-optimal estimator for the flipping spike model: ν1=0.1 , ν2=2.0 , L=10 , H=2 , λ=0.2 . Solid lines correspond to theoretical predictions (Res. 3.1 , App. A.3 ). Markers with errorbars correspond to training the model (Sec. 2 ) with L-BFGS from random initialization in dimension D=2000 across ten seeds. Left: Test error E and training loss Etrain of the attention model, test error EBO of the Bayes-optimal estimator. Center: Normalized head-feature overlap ρ1ERM of the attention model and overlap ρ1BO of the Bayes-optimal estimator. Right: Normalized head-feature overlap ρ2ERM of the attention model and overlap ρ2BO of the Bayes-optimal estimator. Grey and blue identify the permutation-symmetric ( PS ) and specialized ( SPEC ) phases. Vertical lines mark the specialization threshold αcspec .
Figure 6: Bayes-optimal estimator for the non-isotropic Gaussian model: ν1=5.0 , ν2=4.0 , L=5 , H=4 , λ=0.2 . Solid lines correspond to theoretical predictions (Res. 3.1 , App. A.3 ). Markers with errorbars correspond to training the model (Sec. 2 ) with L-BFGS from random initialization in dimension D=2000 across ten seeds. Left: Test error E and training loss Etrain of the attention model, test error EBO of the Bayes-optimal estimator. Center: Normalized head-feature overlap ρ1ERM of the attention model and overlap ρ1BO of the Bayes-optimal estimator. Right: Normalized head-feature overlap ρ2ERM of the attention model and overlap ρ2BO of the Bayes-optimal estimator. Grey, light blue and darker blue identify the permutation-symmetric ( PS ) and successive specialized ( 1 -SPEC , 2 -SPEC ) phases, respectively. Vertical dash-dotted lines mark the specialization thresholds αc(1) and αc(2) .
Figure 7: Influence of signal composition on head specialization in the flipping spike model: νtot=4 , L=10 , H=2 , λ=0.2 . Black solid lines correspond to theoretical predictions (Res 3.1 ). Markers with error bars correspond to training the model (Sec. 2 ) with L-BFGS from random initialization in dimension D=2000 across ten seeds. Left: Phase diagram in sample complexity α and flipping ratio δ . Phase boundaries are obtained by stability analysis of the different solutions (App. C ). The red horizontal line at δ=0.5 corresponding to ν1=ν2=2 , selects the cut shown in the other panels. Center: Test error E and training error Etrain . Right: Norms of the head–feature overlaps ∥m:1∥2 and ∥m:2∥2 . Vertical dash-dotted lines mark the specialization threshold αcspec along the selected cut.
Figure 8: Head-specialization phase transitions in the non-isotropic Gaussian model: ν1=5.0 , ν2=4.0 , L=5 , H=4 , λ=0.2 . Black solid lines correspond to theoretical predictions (Res. 3.1 ). Markers with error bars correspond to training the model (Sec. 2 ) with L-BFGS from random initialization in dimension D=4000 across ten seeds. Left: Head-feature overlaps (m:2,m:1) parametrized by sample complexity. Grey ellipse indicates the distribution anisotropy of possible draws of the planted spike k^ . Center: Test error E and training error Etrain . Right: Head-feature overlaps mhf for all heads and both latent features. Grey, light blue and darker blue identify the permutation-symmetric ( PS ) and successive specialized ( 1 -SPEC , 2 -SPEC ) phases, respectively. Vertical dash-dotted lines mark the specialization thresholds αc(1) and αc(2) .
Figure 9: Head-specialization phase transitions for isotropic Gaussian latent spikes: ν1=4.0 , ν2=4.0 , L=5 , H=4 , λ=0.2 . Black solid lines correspond to theoretical predictions (Res. 3.1 ). Markers with error bars correspond to training the model (Sec. 2 ) with L-BFGS from random initialization in dimension D=2000 across ten seeds. Left: Head-feature overlaps (m:2,m:1) parametrized by sample complexity. Grey circle indicates the distribution isotropy of possible draws of the planted spike k^ . Center: Test error E and training error Etrain . Right: Head-feature overlaps mhf for all heads and both latent features. Grey and darker blue identify the permutation-symmetric ( PS ) and specialized ( 2 -SPEC ) phases, respectively. Vertical dash-dotted lines mark the specialization thresholds αc(1)=αc(2) .
Figure 10: Correspondence between αcspec and τcspec : Population loss Epop , test error E and head–feature overlap mh1 for online gradient flow as a function of rescaled time τ~ (ten independent runs) and for empirical risk minimizers as a function of rescaled sample complexity α~ (averaged over five instances). The inverse curvature 1/C is computed analytically in Sagitova et al. [2026] . Gaussian model with F=1 , ν1=8 , H=2 , L=5 , λ=3 , D=5000 , η=10−3 .
Figure 11: Training dynamics on fixed datasets: Separate runs of BERTSmall models trained on datasets of N∈{5000,20000,60000} sequences, following the training procedure described in App. E.2 with 15000 optimization steps and checkpoints every 500 steps. Left: Validation MLM loss E at registered checkpoints and logged training loss Etrain along training. Right: Evolution of syntactic attention structure, measured by UAS. Larger datasets allow an abrupt emergence of SAS during training. Curves show averages across optimization seeds, with shaded bands indicating one standard deviation.
Figure 12: Influence of the learning rate on specialization: BERTSmall models trained following the procedure described in App. E.2 , with learning rates {0.5γ,γ,2γ,4γ} , where γ=10−4 . Top: Training dynamics on a fixed dataset of N=60000 sequences. Bottom: Models trained on fixed datasets of increasing size N . Left: Validation MLM loss E at registered checkpoints and logged training loss Etrain . Right: Evolution of syntactic attention structure, measured by UAS. Curves show averages across two optimization seeds, with shaded bands indicating one standard deviation.
Figure 13: Acquisition of different syntactic relations with increasing dataset size: BERTSmall models are trained on fixed datasets of increasing size N following the training procedure described in App. E.2 . Left: Validation MLM loss E at registered checkpoints and logged training loss Etrain . Right: Relation-specific attachment scores for the four most frequent relations in the WSJ dataset: case , compound , det , and nsubj . Validation loss and attachment scores are evaluated with early stopping. Curves show seed averages, with shaded bands indicating one standard deviation.
Figure 14: Sequential specialization in learning of sparse attention patterns: Next-token prediction on the synthetic Markov-chain task of Yüksel et al. [2026] (see their Fig. 3 for details). Top: Training dynamics of the full transformer trained on N=9000 sequences. Bottom: Transformer trained until convergence on fixed datasets of increasing size N , with ℓ2 regularization λ=10−6 . Left: KL divergence KL(fA1:r⋆∥fstudent) with the ground-truth predictors A1:r⋆ using the first r=1,2,3 positional groups. Right: KL divergence KL(fc-gram∥fstudent) where c -gram predictors are independently trained transformers restricted to the last c=2,4,6 tokens. Bottom curves average five seeds, with shaded bands indicating one standard deviation.
Multi-head attention enables transformer models to represent multiple attention patterns simultaneously. Empirically, head specialization emerges in distinct stages during training, while many heads remain redundant and learn similar representations. We propose a theoretical model capturing this phenomenon, based on the multi-index and single-location regression frameworks. In the first part, we analyze the training dynamics of multi-head softmax attention under SGD, revealing an initial unspecialized phase followed by a multi-stage specialization phase in which different heads sequentially align with latent signal directions. In the second part, we study the impact of attention activation functions on performance. We introduce the Bayes-softmax attention, which achieves optimal prediction performance in this setting.
M. Sagitova, O. Duranthon, L. Zdeborová
Statistical physics of computation laboratory, École Polytechnique Fédérale de Lausanne, Switzerland
Transformers integrate information from multiple past positions through sparse attention patterns that copy from specific locations. We study how these patterns form during training in a high-order Markov-chain setting, where the next token depends on several groups of past positions with differing statistical importance. Learning proceeds in discrete stages, each marked by the emergence of a sparse attention pattern for the next most important group. The dynamics are initially competitive, with all attention heads focusing on the dominant pattern, but later become cooperative, with different heads specializing in distinct patterns. For a tractable surrogate, the dynamics reduce to those of a tensor factorization problem, enabling us to prove stagewise convergence. Empirically, the number of stages completed depends on training set size, suggesting a form of implicit complexity control through early stopping. Overall, we give a theoretical account of how structured attention patterns and head specialization emerge without an explicit curriculum.
Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas Flammarion
Attention is the key mechanism underlying in-context learning in transformers, and attention patterns have been observed empirically to emerge abruptly during training. We present a Bayesian theory of feature learning in attention; we then focus on how the copy subcircuit in the first layer of an induction head is learned by analyzing a single-layer softmax attention network trained on a copy task. We derive a closed-form posterior over the attention matrix and reduce it to a low-dimensional order parameter space. This reduction reveals a phase transition in the amount of training data, which we verify using both Bayesian sampling and standard training with Adam. We contrast our results with linear attention and find that softmax attention exhibits a first-order phase transition while in linear attention an initial second-order phase transition is followed by a smooth, continuous evolution toward the structured attention pattern (crossover). Our work provides a first-principles theoretical account of the abrupt emergence of the copy subcircuit, reminiscent of the one observed in training large language models.
Itay Lavie, Kirsten Fischer, Andrey Lekov +3
Racah Institute of Physics, Hebrew University of Jerusalem, Jerusalem 91904, Israel · John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, USA · Institute for Advanced Simulation (IAS-6), Computational and Systems Neuroscience, Jülich Research Center, Jülich 52428, Germany +3