On attention heads and bilinear forms

Andrew O'Desky

cs.LGSep 19, 2026

On attention heads and bilinear forms

Authors: Andrew O'Desky

Abstract

We study the symmetric and antisymmetric parts of bilinear forms in the attention heads of trained large language models. We introduce an orthogonally invariant profile map from real bilinear forms to a three-dimensional simplex and observe that profiles of trained bilinear forms accumulate near profiles of rank-one bilinear forms. We prove that the symmetric part of a bilinear form in an attention head is the sum of a hyperbolic form and a zero form for a Zariski-dense subset of query-key matrices.

Figures & tables

Figure 1. The simplex

\Delta_{3}

of profiles of nonzero bilinear forms. Red segments comprise

\Theta=\{a=b,\;cd=0\}

H

denotes a symmetric matrix whose spectrum is symmetric about zero and

P

a positive semidefinite matrix. The front-right facet

a=0

corresponds to symmetric forms, the bottom facet

d=0

S=H+P

, and the left facet to

S=H-P

. Fibers of

\pi

over the relative interiors of the back facet

\{b=0\}

and the top-right edge are empty.

Figure 2. Profiles of the bilinear forms of gpt2-large and of Mistral-Small-24B-Base . The left model uses QKV attention while the right uses RoPE, showing that accumulation occurs for both types of positional encoding. Markers (

+

Type

\mathrm{I}_{+}

\circ

Type

\mathrm{II}

-

Type

\mathrm{I}_{-}

) indicate parity.

Figure 3. The mean eigenvalue distribution of symmetric parts for gpt2 .

Figure 4. Measured eigenvalue distributions in gpt2 , attached to their branches or midpoint of

\Theta

. Each histogram shows the mean of the sorted, unit-length spectra of the ten most extreme heads in one cluster. Here

H

has spectrum symmetric about zero and

P

is positive semidefinite.

Figure 5. Projection of gpt2 ’s

144

symmetric part eigenvalue distributions to the first three principal components (left), and to the second and third components (right).

Model	Scoring parameters ( $V=\operatorname{id}$ )
QKV/RoPE/ALiBi attention	$2N^{2}$
Biaffine attention	$\ell(N+1)^{2}$
Bilinear attention	$\ell N^{2}$
Multi-head QKV/RoPE/ALiBi attention	$2N^{2}$
Multi-head bilinear	$\dfrac{\ell N^{2}}{h}$

Table 1. Trainable scoring parameter counts in the attention blocks; value map and positional encoding parameters are excluded. Here

N=\dim C

\ell

is the sequence length, and

h

is the number of heads. The last row uses fixed coordinate projections as described above.

model	heads	Type $\mathrm{I}_{+}$	Type $\mathrm{II}$	Type $\mathrm{I}_{-}$
distilgpt2	$72$	$34.7$	$26.4$	$38.9$
gpt2	$144$	$36.8$	$34.0$	$29.2$
gpt2-medium	$384$	$54.2$	$28.6$	$17.2$
gpt2-large	$720$	$64.3$	$22.6$	$13.1$
opt-125m	$144$	$79.2$	$16.7$	$4.2$
opt-350m	$384$	$78.9$	$20.8$	$0.3$

Table 2. Percentage of the heads by parity, classified using

\pi(S)

, with tolerance

c_{S}+d_{S}\leq\tfrac{1}{50}

for Type

\mathrm{II}

Figure 6. Average error of each fitted family over the heads of a model, against the size of the model; lower is a closer fit. The error is the root mean square difference between the heights of a

20

-bin histogram of the sample and the fitted density, after rescaling the sample to unit mean.

Figure 7. The simplex

\Delta_{3}

of profiles of bilinear forms (left) and its face

\Delta

of symmetric forms (right). Here

H

has spectrum symmetric about zero and

P

is positive semidefinite. The fiber over the dashed edge

\{b=0\}

is empty away from its endpoints.

	face	fiber of $\pi$
vertex	$a=1$	antisymmetric forms
vertex	$b=1$	symmetric forms with $\lambda_{+}=\lambda_{-}$
vertex	$c=1$	positive semidefinite forms
vertex	$d=1$	negative semidefinite forms
edge	$\{c=d=0\}$	forms with $\lambda_{+}=\lambda_{-}$
edge	$\{b=d=0\}$	forms with $S$ positive semidefinite

Table 3. The fibers of

\pi

over the faces of

\Delta_{3}

;

H

denotes a symmetric matrix with spectrum symmetric about zero and

P

a positive semidefinite matrix.

Figure 8. Profiles of the heads of gpt2 on the simplex

\Delta_{3}

Figure 9. Profiles of the symmetric parts of the heads of gpt2 on

\Delta

. Here

H

has spectrum symmetric about zero and

P

is positive semidefinite.

Figure 10. Profiles of

100

spectra sampled from

\tfrac{1}{2}(\gamma_{\mu,k}+\gamma_{\nu,k})

with

\mu=-\tfrac{1}{2}

\nu=1

k=2

, for

n=64

and

n=1024

eigenvalues per side, plotted by their coordinate

b

along the edge

\{d=0\}

, drawn horizontally from the vertex

c=1

to the vertex

b=1

, together with the limit point

b=0.8

of Theorem 5 (

\times

Observation	Status	Source
1 : the symmetric part is balanced	Proved: Theorem 3
2 : head profiles accumulate near rank-one profiles $\Theta\subset\Delta_{3}$	Partially explained: Theorem 5 ; the proportionality of the lobes in its proof holds approximately at the heads (§ 5.2 )
3 : $\lambda(S)$ has two gamma-like lobes	Unexplained
4 : symmetric bias of attention heads	Partially explained: [ 10 , Theorem 2.4] for the bidirectional case with assumptions, unexplained for decoder-only	[ 10 ]
5 : the sorted spectral lobes are approximately proportional	Linked to profile accumulation by Theorem 5 ; see § 5.2.1
6 : $\|\lambda(L)\|$ and $\sigma(T)$ have gamma-like profiles	Unexplained	[ 3 ]

Table 14

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11. Distribution of the trained-head pairing scores

\mathscr{P}

in Figure 14 . Dashed lines mark model medians.

Figure 12. Parity by layer in all fourteen models. Each cell shows the percentage of a layer’s heads of the indicated type, with a shared scale from white (

0\%

) to black (

100\%

). Types use the symmetric profile: Type

\mathrm{II}

has

c_{S}+d_{S}\leq 1/50

; otherwise

c_{S}>d_{S}

gives Type

\mathrm{I}_{+}

and the remaining heads are Type

\mathrm{I}_{-}

. Layers are indexed from zero; RoPE models use relative position zero.

Figure 13. Symmetric energy $\mathscr{S}=\|S\|^{2}/\|L\|^{2}$ at each attention head. For each model, the trained heads (left) are paired with independent random products $L=K^{T}Q$ (right), with iid standard normal $K,Q\in\mathbb{R}^{n\times N}$ of the same dimensions. Rows index layers and columns index heads, both starting at zero. The shared color scale is white at $1/2$ , red above and blue below; the random baseline has expectation $(N+1)/(2N)$ . RoPE models use relative position $d=0$ .

Figure 14. Pairing score $\mathscr{P}=1-\|\lambda_{+}-\lambda_{-}\|/\|S\|$ at each attention head. Each model’s trained heads (left) are paired with independent Gaussian QK products of the same dimensions (right), as in Figure 13 . Rows index layers and columns index heads. A score of one means the symmetric spectrum is symmetric about zero. The shared color scale is white at $1/2$ , red above and blue below. RoPE models use relative position $d=0$ .

Attention Head Analysis

Attention head analysis studies how individual Transformer heads route information, support computations, and affect model behavior.

See all 82 papers

Sparse Attention Is Matrix Approximation, Not Choosing from a Bag of Values
Oct 7, 2026Fang Wan, Xufeng Liu, Fan Li +1Sparse Attention Efficient Attention
Stony Brook University · Duke University
Riemannian Gradient Descent for Low-Rank Architectures
Jun 1, 2026Nicholas KnightRiemannian Optimization Low-Rank Approximation
NVIDIA
Accelerating Attention with Basis Decomposition
Oct 2, 2025Jialin ZhaoSelf-Attention Transformer Attention
Department of Computer Science Tsinghua University

Oct 7, 2026J/K move · Enter open · S save

Sparse Attention Is Matrix Approximation, Not Choosing from a Bag of Values

Fang Wan, Xufeng Liu, Fan Li, Yi Liu

Stony Brook University · Duke University

Sparse Attention Efficient Attention Attention Mechanisms Attention Approximation LLM Inference Acceleration

Open paper AskPDF arXiv

Large Language Models (LLMs) achieve strong performance across many domains, but their efficiency is limited by the quadratic cost of attention with respect to prompt length. Sparse attention reduces this cost by retaining only a small fraction of query-key interactions to approximate the full attention matrix. However, existing methods are trapped in a mathematically wrong view: they simply keep large scalar entries or high-mass regions of the attention matrix. This treats the attention matrix as a bag of values, ignoring that it is used as a structured matrix whose entries jointly determine the attention output through multiplication with value vectors. We argue that this is the core conceptual issue: sparse attention should be formulated as matrix approximation, not as blindly choosing the largest values from a bag of entries. Based on this view, we propose Matrix Approximation Sparse Attention (MASA). MASA replaces raw attention-mass ranking with a closed-form score that measures how much each sparse unit reduces matrix-product approximation error. As a theory-grounded plug-in correction, MASA can be added to existing sparse attention frameworks without changing their sparse kernels or budgets. Extensive experiments across multiple sparse attention methods, benchmarks, and LLM backbones show consistent accuracy gains, supporting both MASA and the matrix-approximation view of sparse attention.

Abstract

Attention Head Analysis

Explore similar work

Sparse Attention Is Matrix Approximation, Not Choosing from a Bag of Values

Riemannian Gradient Descent for Low-Rank Architectures

Accelerating Attention with Basis Decomposition