Gaussian Equivalence for Multi-Head Self-Attention

Tomohiro Hayase; Ryo Karakida

stat.MLOct 7, 2026

Gaussian Equivalence for Multi-Head Self-Attention

Authors: Tomohiro Hayase, Ryo Karakida

Organizations: Artificial Intelligence Research Center (AIRC), AIST · RIKEN AIP

Abstract

A theoretical understanding of multi-head self-attention is fundamental to the study of modern neural networks. Using random matrix theory, we establish Gaussian equivalence for multi-head self-attention: replacing softmax attention with rescaled scores plus Gaussian noise preserves the limiting spectral law of the centered output. This equivalence also covers value and output projections that depend on the keys. The resulting laws separate the effects of head allocation and projection widths, and distinguish spectrum-preserving across-head sharing from within-head key--value dependence.

Figures & tables

Figure 1: Gaussian equivalence through concatenation and projections. The blue dashed and orange solid families are independent of each other.

Model	Transformation	Control used
$\sqrt{n}A^{\perp}$	Centered multi-head self-attention	Initial model
$Y^{f}$	Replace the row denominators by a constant	Operator-norm error $n^{-1/2+o(1)}$ ; rank at most $h$
$\mathring{Y}^{f}$	Subtract the conditional row mean	Rank at most one; norms $n^{o(1)}$
$Y^{\mathrm{G}}[\Sigma_{f}]$	Gaussian rows with matching covariance	Quadratic-form concentration; self-averaging
$Y^{\mathrm{G}}[\Sigma_{\mathrm{G}}]$	Approximate $\Sigma_{f}$ by $\Sigma_{\mathrm{G}}$	Normalized Frobenius error $O(\sqrt{\log n/n})$
$Y^{\mathrm{G}}$	Use the original queries and independent noise	Same conditional law; self-averaging

Table 1: Equivalent random matrix models, defined in Section 4 . Each step preserves all fixed spectral moments after right multiplication under Assumption 4.1 . The third column states the control used.

Dimensions of	Sharing	$n$	$h$	Rel. $W_{1}$ (%)
BERT-base	None	512	12	$1.66\pm 0.23$
ModernBERT-L	None	1,024	16	$1.37\pm 0.21$
DINOv3-L/16	None	261	16	$2.82\pm 0.40$
LLaDA-8B	None	4,096	32	$0.92\pm 0.06$
GPT-3 175B	None	2,048	96	$0.62\pm 0.07$
Falcon-7B	MQA	2,048	71	$1.92\pm 0.46$

Table 2: Finite-size MHA–GE spectral agreement at dimensions taken from publicly available models. Means with 95% CIs. Full dimensions and sources: Section D.1 .

Figure 2: Spectral predictions for readout learning and width allocation. (a) Training loss for three head counts. (b) Coefficient-error map with fixed-output (blue) and fixed-budget (orange) paths; dotted lines mark width boundaries. (c) Errors along these paths. (d) Coefficient-error ratio, $(64,192)/(128,128)$ , at fixed $N_{\mathrm{par}}$ ; above the dashed unit line, equal widths give lower error; below it, wider V gives lower error. Curves: GE; markers: MHA; bars: 95% CIs. Maximum error bars: (a) $0.00513$ ; (c) $0.02381$ ; (d) $0.00168$ . Details: Sections D.2 , D.3 and D.4 .

Figure 3: Parameter sharing. (a) Across-head sharing; diamond: unshared. (b) GQA versus within-head K=V at equal widths and parameter count,

hd_{\mathrm{K}}=hd_{\mathrm{V}}=d_{\mathrm{O}}=n/2

. Lines: GE; markers: MHA; bars: 95% CIs. Maximum error bars: (a)

4.43476

; (b)

0.03015

. Settings in Sections D.5 and D.6 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Name	Definition
Sets and linear algebra
$\mathbb{N}$	Natural numbers	$\{1,2,\ldots\}$ ; zero is excluded.
$\mathbb{R}$ , $\mathbb{C}$	Real and complex numbers	The real and complex number fields.
$\mathbb{C}_{+}$	Upper half-plane	$\{z\in\mathbb{C}:\operatorname{Im}z>0\}$ .
$I_{N}$	Identity matrix	$N\times N$ identity.
$\mathbf{1}_{N}$	All-ones vector	$(1,\ldots,1)^{\top}\in\mathbb{R}^{N}$ .

Appendix

Table A.1: General mathematical notation.

Symbol	Name	Definition or range
$n$	Context length	$n\in\mathbb{N}$ .
$d$	Input width	$d\in\mathbb{N}$ , $n\leq d$ .
$d_{\mathrm{K}}$	Query/key width per head	$d_{\mathrm{K}}\in\mathbb{N}$ .
$d_{\mathrm{V}}$	Value width per head	$d_{\mathrm{V}}\in\mathbb{N}$ .
$d_{\mathrm{O}}$	Output width	$d_{\mathrm{O}}\in\mathbb{N}$ .
$h$	Number of heads	$h\in\mathbb{N}$ .

Appendix

Table A.2: Dimensions and parameters.

Symbol	Size	Name	Definition
Input and projection matrices
$X$	$n\times d$	Input matrix	$XX^{\top}/d=I_{n}$ ; Section 2.2 .
$W_{r}^{Q}$	$d\times d_{\mathrm{K}}$	Query weights	Section 2.2 .
$W_{r}^{K}$	$d\times d_{\mathrm{K}}$	Key weights	Section 2.2 .
$W_{r}^{V}$	$d\times d_{\mathrm{V}}$	Value weights	Section 2.2 .
$W^{O}$	$hd_{\mathrm{V}}\times d_{\mathrm{O}}$	Output weights	Section 2.2 .

Appendix

Table A.3: Matrices and their dimensions.

Symbol	Name	Definition
$f$	Exponential feature function	$f(x)=e^{\beta x-\beta^{2}/2}-1$ .
$\softmax$	Row softmax	$\softmax(B)_{ij}=e^{B_{ij}}/\sum_{k}e^{B_{ik}}$ .
$G_{\nu}$	Cauchy transform	( C.1 ).
$\psi_{\nu}$	$\psi$ -function	( C.2 ).
$\chi_{\nu}$	Inverse $\psi$ -function	$\chi_{\nu}=\psi_{\nu}^{\langle-1\rangle}$ .
$\mathscr{S}_{\nu}$	$S$ -transform	( C.3 ).

Appendix

Table A.4: Model functions and spectral transforms.

Symbol	Name	Definition
$\mathcal{N}(m,\Sigma)$	Gaussian law	Mean $m$ and covariance $\Sigma$ ; in one dimension the second parameter is the variance.
$\delta_{a}$	Dirac measure	Unit point mass at $a$ .
$\mu_{C}$	Empirical eigenvalue law	$N^{-1}\sum_{j=1}^{N}\delta_{\lambda_{j}(C)}$ .
$\nu_{M}$	Squared-singular-value law	$\mu_{MM^{\top}}$ .
$\mu_{\mathrm{V}}$	Value covariance law	$\lim\mu_{\Sigma_{\mathrm{V}}}$ .
$\mu_{\mathrm{VO}}$	Output covariance law	$\lim\mu_{\Sigma_{\mathrm{VO}}}$ .

Appendix

Table A.5: Probability laws.

Figure	Quantity or path	Absolute	Relative (%)
2 (a)	Relative training loss	$0.00513$	$13.314$
2 (c)	Coefficient error	$0.02381$	$0.562$
2 (d)	Coefficient-error ratio	$0.00168$	$0.108$
3 (a)	Arrival time, fixed widths	$4.43476$	$10.425$
3 (a)	Arrival time, fixed parameter count	$2.75124$	$6.486$
3 (b)	Coefficient error	$0.03015$	$1.182$

Appendix

Table D.1: Maximum error bars in the learning figures. Loss, coefficient-error, and error-ratio distances are dimensionless; arrival-time distances use the gradient-flow time units.

Dimensions of	Sharing	$n$	$d$	$h/h_{\mathrm{KV}}$	$d_{\mathrm{K}}=d_{\mathrm{V}}$	Rel. $W_{1}$ (%)	KS
BERT-base	None	512	768	12/12	64	$1.66\pm 0.23$	$0.0109\pm 0.0010$
ModernBERT-large	None	1,024	1,024	16/16	64	$1.37\pm 0.21$	$0.0052\pm 0.0005$
DINOv3-L/16	None	261	1,024	16/16	64	$2.82\pm 0.40$	$0.0280\pm 0.0018$
LLaDA-8B	None	4,096	4,096	32/32	128	$0.92\pm 0.06$	$0.0022\pm 0.0001$
GPT-3 175B	None	2,048	12,288	96/96	128	$0.62\pm 0.07$	$0.0040\pm 0.0004$
Falcon-7B	MQA	2,048	4,544	71/1	64	$1.92\pm 0.46$	$0.0061\pm 0.0012$

Appendix

Table D.2: Finite-size GE at dimensions taken from publicly available models, with

n\leq d

. Centered MHA–GE spectral distances at

\beta=1

, with

d_{\mathrm{O}}=d

. ∗ Kayyam uses the 2048-token evaluation setting.

Experiment	$n$	$d$	$h$	$d_{\mathrm{K}}$	$d_{\mathrm{V}}$	$hd_{\mathrm{V}}$	$d_{\mathrm{O}}$
2 (a), learning curves	4096	4096	32–128	64	64	2048–8192	8192
2 (c), fixed output	4096	4096	16–120	64	64	1024–7680	2048
2 (c), fixed budget	4096	4096	32–120	64	64	2048–7680	819–36864
2 (d), temperature	8192	12288	96	64–128	128–192	12288–18432	12288
3 (a), across-head sharing	4096	4096	64	64–106	64–106	4096–6784	8192
3 (b), sharing schemes	4096	8192	2–32	64–1024	64–1024	2048	2048

Appendix

Table D.3: Dimensions used in the learning experiments. Ranges cover the sampled softmax configurations.

Figure D.1: Head splitting concentrates the attention bulk (a), output width changes the MHA spectral scale and shape (b), and within-head key–value tying changes the limit while across-head sharing preserves it (c). Histograms show original-model spectra; black curves show the theoretical limits. Experimental details are in Section D.7 .

Explore similar work

Multi-Head Attention as Ensemble Nadaraya-Watson Estimation: Variance Reduction, Decorrelation, and Optimal Head Diversity
May 18, 2026Ernest FokouéMulti-Head Attention Intrinsic Dimensionality
School of Mathematics and Statistics, College of Science Rochester Institute of Technology Rochester, New York 14623, USA
Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attention
May 5, 2026Ayan PendharkarMulti-Head Attention Transformer Attention
Provably Learning Multi-Head Attention with Queries
Aug 4, 2026Sunyeop Kim, Insung Kim, Jian GuoMulti-Head Attention Rectified Linear Unit
Korea University, Republic of Korea · Nanyang Technological University, Singapore

May 18, 2026, stat.MLJ/K move · Enter open · S save

Multi-Head Attention as Ensemble Nadaraya-Watson Estimation: Variance Reduction, Decorrelation, and Optimal Head Diversity

Ernest Fokoué

School of Mathematics and Statistics, College of Science Rochester Institute of Technology Rochester, New York 14623, USA

Multi-Head Attention Intrinsic Dimensionality Variance Reduction Ensemble Diversity

Open paper AskPDF arXiv

We develop a rigorous statistical theory of multi-head attention (MHA) as an ensemble of Nadaraya-Watson (NW) kernel regression estimators. Building on the algebraic identity between single-head softmax attention and the NW estimator, we prove that MHA is a structured ensemble of H NW estimators, each operating in a distinct learned projection subspace of the key space. We derive an explicit Bias-Variance-Covariance decomposition of the MHA mean squared error, showing that variance reduction depends not merely on the number of heads H but fundamentally on the decorrelation of head outputs. Decorrelation is governed by the principal angles between learned projection subspaces: orthogonal projections yield maximum variance reduction; aligned projections yield none. We introduce the Head Diversity Index (HDI), a computable spectral measure of inter-head decorrelation, and prove that MHA mean squared error is monotonically decreasing in HDI. This provides the first rigorous theoretical explanation for the empirically observed specialization of attention heads. Under a fixed total-dimension budget D = H * d_k, we solve the optimal head-dimension allocation problem, deriving the MSE-minimizing pair (H*, d_k*) from data distribution and regression smoothness. The solution yields a new architectural scaling law: the optimal per-head dimension grows logarithmically with training set size, while the optimal number of heads grows nearly linearly with the total budget D. Our framework unifies three strands of prior work: the NW theory of single-head attention, the general weighting theory for ensemble learning, and the decorrelation-variance-reduction isomorphism between biological and computational ensembles. Multi-head attention is the Transformer's instantiation of a universal principle: identical agents plus diversity-enforcing mechanisms yields emergent optimality.

Abstract

Explore similar work

Multi-Head Attention as Ensemble Nadaraya-Watson Estimation: Variance Reduction, Decorrelation, and Optimal Head Diversity

Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attention

Provably Learning Multi-Head Attention with Queries