Compressible Softmax-Attended Language under Incompressible Attention

Wonsuk Lee

cs.CLApr 6, 2026

Compressible Softmax-Attended Language under Incompressible Attention

Authors: Wonsuk Lee

Organizations: Seoul National University & SK Hynix America

Abstract

Softmax attention defines an interaction through $d_h$ head dimensions, but not all dimensions carry equal weight once real text passes through. We decompose the attention logit field into a learned component and a generated component and measure their spectra separately. For all 5,888 KV heads in five transformer language models (124M--7B parameters, four architecture families), the logit energy field $\tilde{E}$ reaches 90% of its variance in 2--11 singular components. The learned interaction matrix $W_Q^\mathrm{T} W_K$ needs 38--75 components for the same threshold out of $d_h \in {64, 128}$ . The spectral gap is 5--25 $\times$ in effective rank. The compressibility of softmax-attended language is a property of the data, not the frame that analyzes it.

Figures & tables

	GPT-2		LLaMA-1B		LLaMA-3B		Qwen-3B		Mistral-7B
$r$	$M$	$\tilde{E}$	$M$	$\tilde{E}$	$M$	$\tilde{E}$	$M$	$\tilde{E}$	$M$	$\tilde{E}$
1	5%	72%	6%	43%	4%	42%	4%	41%	4%	46%
2	10%	90%	12%	63%	7%	61%	7%	60%	8%	64%
5	20%	96%	25%	83%	15%	81%	16%	80%	17%	82%
10	33%	98%	42%	93%	25%	90%	27%	90%	29%	91%
20	53%	99%	66%	98%	42%	96%	45%	96%	47%	96%

Table 1: Cumulative variance captured by the top

r

singular components (median across all heads).

\tilde{E}

concentrates its variance in a few components.

W_{Q}^{\mathrm{T}}W_{K}

does not. Models are grouped by head dimension:

d_{h}=64

(left) and

d_{h}=128

(right).

	GPT-2		LLaMA-1B		LLaMA-3B		Qwen-3B		Mistral-7B
Threshold	$M$	$\tilde{E}$	$M$	$\tilde{E}$	$M$	$\tilde{E}$	$M$	$\tilde{E}$	$M$	$\tilde{E}$
80%	40	2	29	5	57	5	53	5	50	5
90%	49	2	38	8	75	11	70	11	66	10
95%	55	4	44	13	88	18	83	17	80	17
99%	61	18	54	23	107	39	104	40	102	38

Table 2: Effective rank: number of singular components for the stated variance threshold (median across all heads). The spectral gap between

M

and

\tilde{E}

persists across all five models.

Softmax Attention

Softmax attention weights token representations by query–key similarity and underpins Transformer retrieval, sequence modeling, and long-context computation.

See all 75 papers

On the Invariants of Softmax Attention
Apr 6, 2026Wonsuk LeeSoftmax Attention Attention Head Analysis
Seoul National University & SK Hynix America
Sequential Functional Structured Tucker Compression for Large Language Model Attentions
Sep 30, 2026Jiangfeng Chen, Xinyu Wang, Tianshuo Yan +4Tucker Decomposition LLM Compression
University of Manitoba · McGill University · Simpleway +2
A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal Attention
Oct 6, 2026Davis Wertheimer, Haochen Shen, Ahan Gupta +6KV-Cache Compression Transformer Attention
IBM Research, USA · SSAIL Lab, University of Illinois Urbana-Champaign

Apr 6, 2026J/K move · Enter open · S save

On the Invariants of Softmax Attention

Wonsuk Lee

Seoul National University & SK Hynix America

Softmax Attention Attention Head Analysis Representation Geometry in Language Models

Open paper AskPDF arXiv

Softmax attention maps every query--key interaction into a probability distribution, but the underlying structure remains largely unexplored. We define the energy field, the row-centered attention logit, and show that it exhibits invariant properties across models, architectures, and inputs. Two classes of invariants emerge. Mechanism-level invariants follow from the algebraic structure of softmax attention. They include a per-row zero-sum constraint, a rank bound determined by the head dimension, and spectral signatures that follow from them. Model-level regularities are not required by the mechanism, yet hold in every autoregressive language model we test, spanning several architecture families. The energy field distributes its variance over key positions without concentrating at a few. This delocalization traces to a property of the key matrix we call key incoherence. These invariants have practical consequences. The rank bound confines the energy field to a low-dimensional subspace. Key incoherence yields a per-head training monitor. All results are verified at multiple context lengths and input texts.

Abstract

Softmax Attention

Explore similar work

On the Invariants of Softmax Attention

Sequential Functional Structured Tucker Compression for Large Language Model Attentions

A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal Attention