Spectral Geometry of Attention: From Information Routing to Uncertainty

Giulio Viganò; Simone Melzi; Maks Ovsjanikov

cs.LGOct 5, 2026

Spectral Geometry of Attention: From Information Routing to Uncertainty

Authors: Giulio Viganò, Simone Melzi, Maks Ovsjanikov

Organizations: University of Milano-Bicocca Milan, Italy · Ecole Polytechnique · LIX, Ecole Polytechnique IP Paris

Abstract

In this work, we study transformer attention through the lens of spectral geometry and operator theory. We view each attention head as a functional map between Hilbert spaces of functions on the token sequence and derive a Token Difference Operator, whose spectral structure controls how token-space information is routed to the output. We show that standard Euclidean spectra are structurally biased by sinks, conflating mass concentration with genuine routing capacity. By recasting token space in the intrinsic probability geometry induced by attention, the token difference spectrum disentangles sink effects from routing capacity and provides a spectral description of the dimensionality of the head output. This yields a unified framework for analyzing attention maps, explaining sinks, routing collapse, and output dimensionality within a single operator-theoretic framework. In practice, by grounding attention heuristics in spectral geometry, we develop a novel attention-based uncertainty estimator that complements probability-based scores, with the largest gains on long-context inputs.

Figures & tables

Figure 1 : Teaser: (Left block) Our work studies transformers’ attention as a functional map between Hilbert spaces equipped with an inner product. The properties of the operator change with the standard Euclidean measure (Blue) and with an intrinsic probability geometry measure (Red). (Center Block) Parametrizing row stochastic causal map interpolating ( $\alpha$ ) between a copy map and a sink map, we show that the Euclidean spectrum of the matrix is biased towards the sink, while reweighting the input measure relaxes this bias. (Right Block) We leverage our insights to design an uncertainty quantifier, identifying uncertainty as a routing capability collapse in certain selected heads (red box in the uncertain model).

Figure 2 : Representative attention heads and how different metrics classify their routing behavior. The intrinsic TDO score separates sink collapse from sink-biased routing, assigning high routing scores to copying, shifting, and sink-biased heads while distinguishing them from pure sink heads.

Panel of Figure 3 — Figure 3 : Scatterplots showing the relationships between attention diagnostics, sink score, and output dimensionality. Top: relation between each diagnostic and sink score. Bottom: relation between each diagnostic and the spectral entropy of the head output $H(AV)$ . Points correspond to the attention heads of LLaMA-3.1-8B on 200 inputs, shown both per input and averaged across inputs (bold points). We report the Spearman correlation between the metrics.

Figure 4 : Perturbation sensitivity in a controlled copy task inspired by Veličković et al. (2026) ; Pasten et al. (2025) . The model receives copy:

b_{1}b_{2}\cdots b_{n}

and must reproduce the bit string; we contrast the all-zero input

0^{n}

(

\alpha

, easy) with a perturbation

b^{(r)}

carrying a single one at position

r

(

\beta

, hard). As

n

grows, perplexity and mean token entropy become nearly indistinguishable across the two families, while RAUQ Vazhentsev et al. (2025) and Ours, computed from hidden-layer routing geometry, remain sensitive to the perturbation.

		QA					Summarization
Estimator	Category	TriviaQA	TruthfulQA	MMLU	COQA	Mean	SAMSUM	XSUM	Mean
AttentionScore Sriramanan et al. (2024)	Single pass	0.0796	-0.0505	0.2078	0.0933	0.0826	-0.3727	-0.9983	-0.6855
MaximumSequenceProbability	Single pass	0.2518	0.3211	0.5501	0.2878	0.3527	-0.1363	-0.386	-0.2612
MeanTokenEntropy	Single pass	0.5661	0.4199	0.2437	0.2568	0.3716	0.4999	0.3928	0.4464
Perplexity	Single Pass	0.1088	0.2863	0.5158	0.2737	0.2962	0.5071	0.4053	0.4562
CCP Fadeeva et al. (2023)	Single pass	0.1029	0.159	0.5567	0.2602	0.2697	-0.1458	-0.3018	-0.2238

Table 1: Comparison against unsupervised methods. The best result in each column is shown in bold, and the second best in blue.

Measure	QA	Summ.
Uniform	0.3660	0.4204
Positional	0.4027	0.4771
Zipf-Law	0.3718	0.4223
Unembedding	0.3755	0.4202
Intrinsic	0.3831	0.5069

Table 2: Mean PRR of SAUQ under different measures (LLaMA-3.1-8B).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5 : Scatterplot between attention diagnostics, sink score, and output dimensionality on GPT-2, Pythia-410m and Qwen2-7b

Dataset	AttnEigvals	AttnLogDet	LapEigvals	TokenDiffEigvals
COQA	0.7995	0.7779	0.8519	0.8241
GSM8K	0.8598	0.8586	0.8660	0.8663
TriviaQA	0.8385	0.8346	0.8923	0.8731
TruthfulQA	0.7997	0.8362	0.8095	0.8445

Appendix

Table 3 : Supervised hallucination-detection performance (ROC-AUC). We compare attention-matrix baselines, Laplacian eigenvalues, and features derived from the weighted token-difference operator. We follow the experimental protocol of [ 7 ] using Llama-3.1-8b [ 1 ] , DeepSeek-4 as a judge.

		QA					Summarization
Estimator	Category	TriviaQA	TruthfulQA	MMLU	CoQA	Mean	SAMSum	XSum	Mean
AttentionScore [ 34 ]	Single pass	0.0429	0.0466	0.1114	0.0022	0.0508	0.2924	-0.0540	0.1192
AttentionScore (gen. only) [ 34 ]	Single pass	0.1529	-0.1053	0.6741	0.0892	0.2027	-0.0726	-0.5038	-0.2882
MaximumSequenceProbability	Single pass	0.2328	0.0436	0.6644	0.3016	0.3106	0.1220	-0.0610	0.0305
MeanTokenEntropy	Single pass	0.6173	0.2474	0.2491	0.2533	0.3418	0.4265	0.3363	0.3814
Perplexity	Single pass	-0.0126	0.2093	0.4042	0.2620	0.2157	0.4303	0.3309	0.3806

Appendix

Table 4: Normalized PRR (higher is better) of unsupervised uncertainty estimators on Qwen2.5-7B. Sample-based methods use 10 samples. The best result in each column is shown in bold, and the second best in blue.

		QA				Summarization
Estimator	Category	TriviaQA	TruthfulQA	MMLU	CoQA	SAMSum	XSum
MaximumSequenceProbability	Single pass	$0.25\pm{.02}$	$0.32\pm{.06}$	$\textbf{0.55}\pm{.02}$	$0.29\pm{.02}$	$-0.14\pm{.07}$	$-0.39\pm{.11}$
MeanTokenEntropy	Single pass	$\underline{0.57}\pm{.02}$	$\textbf{\lx@text@underline{0.42}}\pm{.05}$	$0.24\pm{.03}$	$0.26\pm{.02}$	$0.50\pm{.05}$	$\textbf{0.39}\pm{.08}$
Perplexity	Single pass	$0.11\pm{.02}$	$0.29\pm{.05}$	$0.52\pm{.02}$	$0.27\pm{.02}$	$0.51\pm{.04}$	$\textbf{0.41}\pm{.08}$
CCP [ 14 ]	Single pass	$0.10\pm{.02}$	$0.16\pm{.06}$	$\textbf{\lx@text@underline{0.56}}\pm{.02}$	$0.26\pm{.02}$	$-0.15\pm{.07}$	$-0.30\pm{.11}$
NLI-DegMat [ 24 ]	Sample-based	$\textbf{0.58}\pm{.02}$	$0.21\pm{.06}$	$0.44\pm{.02}$	$\textbf{0.40}\pm{.02}$	$0.39\pm{.05}$	$\textbf{0.32}\pm{.09}$

Appendix

Table 5 : Normalized PRR (higher is better) of unsupervised uncertainty estimators on LLaMA-3.1-8B, mean

\pm

standard error (stratified bootstrap over evaluation instances, 5000 resamples; one greedy decode per instance, 10 samples for sample-based methods). Bold: not significantly worse than the best result in the column.

Estimator	QA (mean ECE)	Summ. (mean ECE)
Perplexity	0.237	0.558
AttentionScore	0.266	0.551
MaxSeqProbability	0.265	0.685
MeanTokenEntropy	0.162	0.505
RAUQ	0.187	0.522
SAUQ (Ours)	0.194	0.490

Appendix

Table 6: Expected calibration error (lower is better) of single-pass estimators on LLaMA-3.1-8B, averaged over the QA and summarization datasets.

TruthfulQA
Selection $\downarrow$ / Signal $\rightarrow$	gen-Eucl	full-Eucl	full-intr.	full-intr.-cent.
gen-Euclidean	0.3056	0.3494	0.3767	0.3876
full-Euclidean	0.3203	0.3507	0.3761	0.3759
full-intrinsic	0.3078	0.3508	0.3482	0.3429
full-intr.-cent.	0.3093	0.3508	0.3482	0.3429
SAMSum

Appendix

Table 7: PRR for every combination of head-selection rule (rows) and routing signal (columns). Underlined: the configuration reported in the submitted version; bold: best per dataset.

Dataset	Task	$\bm{\Delta}$ pooled	95% CI	ctrl. output	ctrl. input $\times$ output
XSum	Summ.	$+0.121$	$[+0.037,+0.212]$	$+0.078$	$+0.060$
SAMSum	Summ.	$+0.066$	$[+0.010,+0.121]$	$+0.013$	$+0.020$
TruthfulQA	QA	$+0.040$	$[-0.004,+0.084]$	$+0.023$	$+0.014$
TriviaQA	QA	$+0.028$	$[-0.001,+0.058]$	$+0.047$	$+0.056$
CoQA	QA	$-0.006$	$[-0.015,+0.004]$	$-0.013$	$-0.012$
MMLU	QA	$-0.001$	$[-0.005,+0.004]$	$-0.016$	$-0.022$

Appendix

Table 8: Difference in PRR between SAUQ and RAUQ on LLaMA-3.1-8B, pooled and after stratifying by output length and by input

\times

output length.

Panel of Figure 6 — Figure 6 : Top tokens receiving the largest probability mass under each measure, and mean PRR for QA and summarization of SAUQ with these masses. Input (first 100 chars): Port-au-Prince, Haiti (CNN) – Earthquake victims, writhing in pain and grasping at life, watched doctors and nurses… .

Explore similar work

From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers
Jul 12, 2026Binbin Lin, Wei Chen, Yalun Li +3Transformer Attention Linear Attention
School of Software Technology, Zhejiang University, China · College of Computer Science and Technology, Zhejiang University, China · Alibaba Cloud, China
Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention
May 21, 2026Athanasios ZerisTransformer Attention Token Embeddings
Independent Researcher, Athens, Greece.
The Routing and Filtering Structure of Attention
May 12, 2026Shafayeth Jamil, Rehan KapadiaKimi Delta Attention Decomposition
Ming Hsieh Department of Electrical and Computer Engineering University of Southern California Los Angeles, CA 90089

Jul 12, 2026, cs.LGJ/K move · Enter open · S save

From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers

Binbin Lin, Wei Chen, Yalun Li, Wenxiao Wang+2

School of Software Technology, Zhejiang University, China · College of Computer Science and Technology, Zhejiang University, China · Alibaba Cloud, China

Transformer Attention Linear Attention Multi-Head Attention Transformer Architectures Interleaved Graph Attention Graph Laplacians+2

Open paper AskPDF arXiv

Self-attention is a ubiquitous primitive in modern sequence models, yet its operator-level geometry is only partially understood. We view a token sequence as a vector field over the token-position graph and identify attention as a connection walk: messages are aggregated by a nonnegative walk matrix while being transported along each edge by a learned linear map. Within this framework, we prove that single-head attention (SHA) is exactly a connection propagation step with constant transport, and that multi-head attention (MHA) is exactly a single edge-dependent connection walk whose effective transport is an attention-gated mixture of headwise transports. We further clarify the conditions under which the corresponding generator reduces to a random-walk connection Laplacian, highlighting the roles of stochasticity, reversibility, and metric-compatible transports. Empirically, we find that trained Transformers across scales (from 124M to 8B) and structures (encoder/decoder) exhibit geometric structure consistent with our theory: effective attention graphs converge to stable geometric operators in deeper layers, learned transports self-organize into approximate scaled isometries, and both phenomena strengthen consistently with scale. Overall, the paper provides a precise connection-walk formalism that links self-attention to classical geometric operators, along with a set of operator-level tools for analyzing transformer models from a geometric perspective.

Abstract

Explore similar work

From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers

Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention

The Routing and Filtering Structure of Attention