On attention heads and bilinear forms

Andrew O'Desky

cs.LGSep 19, 2026

On attention heads and bilinear forms

Authors: Andrew O'Desky

Abstract

We study the symmetric and antisymmetric parts of bilinear forms in the attention heads of trained large language models. We introduce an orthogonally invariant profile map from real bilinear forms to a three-dimensional simplex and observe that profiles of trained bilinear forms accumulate near profiles of rank-one bilinear forms. We prove that the symmetric part of a bilinear form in an attention head is the sum of a hyperbolic form and a zero form for a Zariski-dense subset of query-key matrices.

Figures & tables

Figure 1. The simplex

\Delta_{3}

of profiles of nonzero bilinear forms. Red segments comprise

\Theta=\{a=b,\;cd=0\}

H

denotes a symmetric matrix whose spectrum is symmetric about zero and

P

a positive semidefinite matrix. The front-right facet

a=0

corresponds to symmetric forms, the bottom facet

d=0

S=H+P

, and the left facet to

S=H-P

. Fibers of

\pi

over the relative interiors of the back facet

\{b=0\}

and the top-right edge are empty.

Figure 2. Profiles of the bilinear forms of gpt2-large and of Mistral-Small-24B-Base . The left model uses QKV attention while the right uses RoPE, showing that accumulation occurs for both types of positional encoding. Markers (

+

Type

\mathrm{I}_{+}

\circ

Type

\mathrm{II}

-

Type

\mathrm{I}_{-}

) indicate parity.

Figure 3. The mean eigenvalue distribution of symmetric parts for gpt2 .

Figure 4. Measured eigenvalue distributions in gpt2 , attached to their branches or midpoint of

\Theta

. Each histogram shows the mean of the sorted, unit-length spectra of the ten most extreme heads in one cluster. Here

H

has spectrum symmetric about zero and

P

is positive semidefinite.

Figure 5. Projection of gpt2 ’s

144

symmetric part eigenvalue distributions to the first three principal components (left), and to the second and third components (right).

Model	Scoring parameters ( $V=\operatorname{id}$ )
QKV/RoPE/ALiBi attention	$2N^{2}$
Biaffine attention	$\ell(N+1)^{2}$
Bilinear attention	$\ell N^{2}$
Multi-head QKV/RoPE/ALiBi attention	$2N^{2}$
Multi-head bilinear	$\dfrac{\ell N^{2}}{h}$

Table 1. Trainable scoring parameter counts in the attention blocks; value map and positional encoding parameters are excluded. Here

N=\dim C

\ell

is the sequence length, and

h

is the number of heads. The last row uses fixed coordinate projections as described above.

model	heads	Type $\mathrm{I}_{+}$	Type $\mathrm{II}$	Type $\mathrm{I}_{-}$
distilgpt2	$72$	$34.7$	$26.4$	$38.9$
gpt2	$144$	$36.8$	$34.0$	$29.2$
gpt2-medium	$384$	$54.2$	$28.6$	$17.2$
gpt2-large	$720$	$64.3$	$22.6$	$13.1$
opt-125m	$144$	$79.2$	$16.7$	$4.2$
opt-350m	$384$	$78.9$	$20.8$	$0.3$

Table 2. Percentage of the heads by parity, classified using

\pi(S)

, with tolerance

c_{S}+d_{S}\leq\tfrac{1}{50}

for Type

\mathrm{II}

Figure 6. Average error of each fitted family over the heads of a model, against the size of the model; lower is a closer fit. The error is the root mean square difference between the heights of a

20

-bin histogram of the sample and the fitted density, after rescaling the sample to unit mean.

Figure 7. The simplex

\Delta_{3}

of profiles of bilinear forms (left) and its face

\Delta

of symmetric forms (right). Here

H

has spectrum symmetric about zero and

P

is positive semidefinite. The fiber over the dashed edge

\{b=0\}

is empty away from its endpoints.

	face	fiber of $\pi$
vertex	$a=1$	antisymmetric forms
vertex	$b=1$	symmetric forms with $\lambda_{+}=\lambda_{-}$
vertex	$c=1$	positive semidefinite forms
vertex	$d=1$	negative semidefinite forms
edge	$\{c=d=0\}$	forms with $\lambda_{+}=\lambda_{-}$
edge	$\{b=d=0\}$	forms with $S$ positive semidefinite

Table 3. The fibers of

\pi

over the faces of

\Delta_{3}

;

H

denotes a symmetric matrix with spectrum symmetric about zero and

P

a positive semidefinite matrix.

Figure 8. Profiles of the heads of gpt2 on the simplex

\Delta_{3}

Figure 9. Profiles of the symmetric parts of the heads of gpt2 on

\Delta

. Here

H

has spectrum symmetric about zero and

P

is positive semidefinite.

Figure 10. Profiles of

100

spectra sampled from

\tfrac{1}{2}(\gamma_{\mu,k}+\gamma_{\nu,k})

with

\mu=-\tfrac{1}{2}

\nu=1

k=2

, for

n=64

and

n=1024

eigenvalues per side, plotted by their coordinate

b

along the edge

\{d=0\}

, drawn horizontally from the vertex

c=1

to the vertex

b=1

, together with the limit point

b=0.8

of Theorem 5 (

\times

Observation	Status	Source
1 : the symmetric part is balanced	Proved: Theorem 3
2 : head profiles accumulate near rank-one profiles $\Theta\subset\Delta_{3}$	Partially explained: Theorem 5 ; the proportionality of the lobes in its proof holds approximately at the heads (§ 5.2 )
3 : $\lambda(S)$ has two gamma-like lobes	Unexplained
4 : symmetric bias of attention heads	Partially explained: [ 10 , Theorem 2.4] for the bidirectional case with assumptions, unexplained for decoder-only	[ 10 ]
5 : the sorted spectral lobes are approximately proportional	Linked to profile accumulation by Theorem 5 ; see § 5.2.1
6 : $\|\lambda(L)\|$ and $\sigma(T)$ have gamma-like profiles	Unexplained	[ 3 ]

Table 14

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11. Distribution of the trained-head pairing scores

\mathscr{P}

in Figure 14 . Dashed lines mark model medians.

Figure 12. Parity by layer in all fourteen models. Each cell shows the percentage of a layer’s heads of the indicated type, with a shared scale from white (

0\%

) to black (

100\%

). Types use the symmetric profile: Type

\mathrm{II}

has

c_{S}+d_{S}\leq 1/50

; otherwise

c_{S}>d_{S}

gives Type

\mathrm{I}_{+}

and the remaining heads are Type

\mathrm{I}_{-}

. Layers are indexed from zero; RoPE models use relative position zero.

Figure 13. Symmetric energy $\mathscr{S}=\|S\|^{2}/\|L\|^{2}$ at each attention head. For each model, the trained heads (left) are paired with independent random products $L=K^{T}Q$ (right), with iid standard normal $K,Q\in\mathbb{R}^{n\times N}$ of the same dimensions. Rows index layers and columns index heads, both starting at zero. The shared color scale is white at $1/2$ , red above and blue below; the random baseline has expectation $(N+1)/(2N)$ . RoPE models use relative position $d=0$ .

Figure 14. Pairing score $\mathscr{P}=1-\|\lambda_{+}-\lambda_{-}\|/\|S\|$ at each attention head. Each model’s trained heads (left) are paired with independent Gaussian QK products of the same dimensions (right), as in Figure 13 . Rows index layers and columns index heads. A score of one means the symmetric spectrum is symmetric about zero. The shared color scale is white at $1/2$ , red above and blue below. RoPE models use relative position $d=0$ .

Attention Head Analysis

Attention head analysis studies how individual Transformer heads route information, support computations, and affect model behavior.

See all 82 papers

Oct 7, 2026

Sparse Attention Is Matrix Approximation, Not Choosing from a Bag of Values

Large Language Models (LLMs) achieve strong performance across many domains, but their efficiency is limited by the quadratic cost of attention with respect to prompt length. Sparse attention reduces this cost by retaining only a small fraction of query-key interactions to approximate the full attention matrix. However, existing methods are trapped in a mathematically wrong view: they simply keep large scalar entries or high-mass regions of the attention matrix. This treats the attention matrix as a bag of values, ignoring that it is used as a structured matrix whose entries jointly determine the attention output through multiplication with value vectors. We argue that this is the core conceptual issue: sparse attention should be formulated as matrix approximation, not as blindly choosing the largest values from a bag of entries. Based on this view, we propose Matrix Approximation Sparse Attention (MASA). MASA replaces raw attention-mass ranking with a closed-form score that measures how much each sparse unit reduces matrix-product approximation error. As a theory-grounded plug-in correction, MASA can be added to existing sparse attention frameworks without changing their sparse kernels or budgets. Extensive experiments across multiple sparse attention methods, benchmarks, and LLM backbones show consistent accuracy gains, supporting both MASA and the matrix-approximation view of sparse attention.

Fang Wan, Xufeng Liu, Fan Li +1

Stony Brook University · Duke University

Sparse Attention Efficient Attention Attention Mechanisms Attention Approximation LLM Inference Acceleration

Ask PDF

Jun 1, 2026

Riemannian Gradient Descent for Low-Rank Architectures

We explore Riemannian optimization techniques for rank-factored matrix parameters, targeting contemporary deep learning applications. We examine ten points in the algorithm design space: two geometries for rank-

r

matrices, three geometries for rank-

r

partial isometries, and block-matrix variants of these five, where factors are shared across block-rows and block-columns. We apply our methods to the multihead attention parameters in small language models. After tuning learning rates, our methods do not conclusively outperform an AdamW baseline. Our implementations are available online.

Nicholas Knight

NVIDIA

Riemannian Optimization Low-Rank Approximation Matrix Optimization Riemannian Geometry Multi-Head Attention

Ask PDF

Oct 2, 2025

Accelerating Attention with Basis Decomposition

Attention is a core operation in large language models (LLMs). We present BD Attention (BDA), a lossless algorithmic reformulation of attention. BDA is enabled by a simple matrix identity from Basis Decomposition (BD), which restructures multi-head projections into a compact form while preserving exact outputs. Unlike I/O-aware system optimizations such as FlashAttention, BDA provides a mathematically guaranteed acceleration that is architecture-agnostic. On DeepSeek-V2-Lite (16B, FP16), BDA requires only 4s of offline preparation with no retraining required and, on modern GPUs, achieves 34% faster key/value projections and 25% smaller weights, while increasing perplexity (PPL) by just 0.02% (FP16) or 0.0004% (FP32), a negligible effect on model performance. These results position BDA as a theoretically exact method for lossless attention acceleration that is complementary to existing engineering-level optimizations. Our code is available at https://github.com/abcbdf/basis-decomposition-official.

Jialin Zhao

Department of Computer Science Tsinghua University

Self-Attention Transformer Attention LLM Inference Acceleration Efficient Attention Attention Mechanisms Multi-Head Attention

Ask PDF

Abstract

Attention Head Analysis

Explore similar work

Sparse Attention Is Matrix Approximation, Not Choosing from a Bag of Values

Riemannian Gradient Descent for Low-Rank Architectures

Accelerating Attention with Basis Decomposition