On the Invariants of Softmax Attention

Wonsuk Lee

cs.LGApr 6, 2026

On the Invariants of Softmax Attention

Authors: Wonsuk Lee

Organizations: Seoul National University & SK Hynix America

Abstract

Softmax attention maps every query--key interaction into a probability distribution, but the underlying structure remains largely unexplored. We define the energy field, the row-centered attention logit, and show that it exhibits invariant properties across models, architectures, and inputs. Two classes of invariants emerge. Mechanism-level invariants follow from the algebraic structure of softmax attention. They include a per-row zero-sum constraint, a rank bound determined by the head dimension, and spectral signatures that follow from them. Model-level regularities are not required by the mechanism, yet hold in every autoregressive language model we test, spanning several architecture families. The energy field distributes its variance over key positions without concentrating at a few. This delocalization traces to a property of the key matrix we call key incoherence. These invariants have practical consequences. The rank bound confines the energy field to a low-dimensional subspace. Key incoherence yields a per-head training monitor. All results are verified at multiple context lengths and input texts.

Figures & tables

Figure 1: The causal energy field $E_{ij}$ of a single LLaMA-3.2-1B head (layer 5, head 0, $L=256$ , processing Dickens). (a) Three-dimensional surface over the causal region ( $j\leq i$ ). Ridges (red) mark high energy, valleys (blue) mark low energy. Every row sums to zero, so red and blue balance exactly. The acausal region ( $j>i$ ) is masked. (b) Filled contour plot of the causal energy field, with iso-energy lines. Vertical stripes appear where key vectors produce consistently high or low energy for all queries. The triangular domain grows with $i$ , reflecting the expanding causal context. Two features dominate. The diagonal ridge ( $j=i$ ) carries mean energy $E_{ii}\approx 1.5$ , reflecting self-attention since $q_{i}$ and $k_{i}$ project from the same embedding. The red band at $j=0$ is far stronger, with mean $E_{i,0}\approx 6.1$ . This is the BOS attention sink [ 15 ] , visible as a column of uniformly high energy across all query positions.

Family	Models	Heads	$\mu_{K}$ (mean $\pm$ std)	$\mu_{K}\leq 5$ (%)
Pythia	4	1,680	$1.37\pm 0.4$	100.0
GPT-2	3	1,248	$1.74\pm 0.5$	99.6
OPT	2	1,152	$1.65\pm 0.7$	99.1
Qwen	3	176	$1.48\pm 0.2$	100.0
LLaMA	2	352	$1.49\pm 0.2$	100.0
Phi-2	1	1,024	$1.45\pm 0.3$	99.9

Table 1: Key incoherence

\mu_{K}

across architecture families (mean

\pm

std across all heads at

L=256

Figure 2: Key incoherence

\mu_{K}

vs. model size for all 16 models at

L=256

. Markers show the median

\mu_{K}

across all heads; error bars span the interquartile range. Despite nearly two orders of magnitude in parameter count,

\mu_{K}

shows no systematic trend and remains near 1.5 for every architecture family. The gray dotted line marks

\mu_{K}=1

, perfectly uniform key norms.

Family	Models	$\IPR\cdot L$ (mean)	CV (%)
GPT-2	3	$3.46\pm 0.15$	4.2
OPT	2	$3.42\pm 0.10$	2.8
Pythia	4	$3.05\pm 0.08$	2.7
Qwen	3	$2.95\pm 0.04$	1.3
LLaMA	2	$3.10\pm 0.07$	2.2
Phi-2	1	$3.04$	—

Table 2: Delocalization (

\IPR\cdot L

) of

\tilde{E}

L=512

. RoPE-based architectures show systematically lower values than non-RoPE models.

Figure 3: Spectral signatures in the flattened energy signal of LLaMA-3.2-1B, layer 5, head 0,

L=256

. (a) Wavelet energy spectrum. Energy density increases with scale in the detail subbands. The coarsest approximation (red square) carries

<0.004

of total energy, verifying the vanishing DC property. (b, c) First and last 10 values of the cumulative signal

Y(t)=\sum_{s=1}^{t}\mathcal{E}_{s}

. The signal starts at

Y(0)=0

and returns to

Y(N)=0

, a bridge pinned at both ends. This is a direct consequence of the row-sum identity

\sum_{t}\mathcal{E}_{t}=0

$r$	SVD( $\tilde{E}$ )	SVD( $E$ )	Top- $k$
5	0.89	0.76	0.25
10	0.94	0.85	0.38
20	0.98	0.92	0.54

Table 3: Reconstruction fidelity

F_{r}

for three methods (mean across GPT-2, Pythia-410M, LLaMA-3.2-1B at

L=256

). SVD dominates at every rank, confirming that the energy field is low-rank, not sparse.

Softmax Attention

Softmax attention weights token representations by query–key similarity and underpins Transformer retrieval, sequence modeling, and long-context computation.

See all 75 papers

Compressible Softmax-Attended Language under Incompressible Attention
Apr 6, 2026Wonsuk LeeSoftmax Attention Attention Head Analysis
Seoul National University & SK Hynix America
Intrinsic Interaction Geometry Controls the Low-Rank Complexity of Softmax Attention
Aug 28, 2026Yuhe Sui, Jianing Zhang, Yingzhi TangSoftmax Attention Low-Rank Attention
Nanyang Technological University, Singapore · Carnegie Mellon University
A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention
May 12, 2026Tomohiro Hayase, Ryo KarakidaSoftmax Attention Self-Attention
AIST

Apr 6, 2026J/K move · Enter open · S save

Compressible Softmax-Attended Language under Incompressible Attention

Wonsuk Lee

Seoul National University & SK Hynix America

Softmax Attention Attention Head Analysis Low-Rank Attention

Open paper AskPDF arXiv

Softmax attention defines an interaction through

d_h

head dimensions, but not all dimensions carry equal weight once real text passes through. We decompose the attention logit field into a learned component and a generated component and measure their spectra separately. For all 5,888 KV heads in five transformer language models (124M--7B parameters, four architecture families), the logit energy field

\tilde{E}

reaches 90% of its variance in 2--11 singular components. The learned interaction matrix

W_Q^\mathrm{T} W_K

needs 38--75 components for the same threshold out of

d_h \in {64, 128}

. The spectral gap is 5--25

\times

in effective rank. The compressibility of softmax-attended language is a property of the data, not the frame that analyzes it.

Abstract

Softmax Attention

Explore similar work

Compressible Softmax-Attended Language under Incompressible Attention

Intrinsic Interaction Geometry Controls the Low-Rank Complexity of Softmax Attention

A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention