Attention Geometry

Julio Candanedo

cs.LGFeb 11, 2026

Attention Geometry

Authors: Julio Candanedo

Abstract

Attention normalizes learned scores into a Markov operator. In the continuum, the local sector becomes diffusion on a learned Riemannian manifold, but attention natively bonds distant points, creating nonlocal structure such that no tensor geometry can describe. The Bakry-Emery calculus, built directly from the transition operator, extends geometry to these potentially stratified spaces, giving a coordinate-free account of what vision and language transformers learn.

Figures & tables

Figure 1: Heat spreading on a sampled Swiss roll under a Markov walk. At suitable bandwidth the motion follows local neighborhoods of the sheet, and the long-time equilibrium is visibly nonuniform.

Figure 2: Swiss-roll generation under DMTN in the order FFN $\to$ MHSA. The first panel shows the $4000$ -point training set and the remaining panels show generated samples at the indicated generative times. All eight conditions generate the data comparably, as the null task requires, and their transports are distinguished in Table 1 .

Operator	Order	$d_{\mathrm{NN}}$	$e^{\bar{H}}$	$\epsilon_{\mathrm{rev}}$	$\rho_{\mathrm{skew}}$
DMAP	FFN $\to$ MHSA	$0.0506$	$27.2$	$2.46\times 10^{-4}$	$0$
DMAP	MHSA $\to$ FFN	$0.0502$	$1.14$	$1.83\times 10^{-6}$	$0$
DMTN	FFN $\to$ MHSA	$0.0498$	$42.4$	$0.224$	$0.287$
DMTN	MHSA $\to$ FFN	$0.0508$	$11.1$	$0.234$	$0.116$
ATAP	FFN $\to$ MHSA	$0.0494$	$46.4$	$6.15\times 10^{-5}$	$0$
ATAP	MHSA $\to$ FFN	$0.0512$	$9.63$	$1.27\times 10^{-5}$	$0$

Table 1: Swiss-roll scan (seed

123

): similar generated samples, distinct learned transport.

Figure 3: Samples from the unmodified pretrained SiT-XL/2 ( Ma et al., 2024 ) .

Panel of Figure 4 — Figure 4: DMTN transfer. Samples immediately after substituting $\mathcal{W}^{\mathrm{DMTN}}$ of Eq. ( 128 ) into the pretrained checkpoint (left) and after $40{,}000$ adaptation steps (right).

Panel of Figure 5 — Figure 5: Distance-form DMAP transfer. Samples immediately after folding the adapted DMTN checkpoint into the distance form of Remark 128 (left) and after $80{,}000$ further adaptation steps (right).

Pre-mask score	Validation bpb $\downarrow$
ATTN	$0.8471$
DMTN	$0.8492$
DMAP	$0.8621$

Table 2: Depth-12 training from scratch. The measurements are single-seed results; ATAP was not evaluated.

State	Score	bpb $\downarrow$	Induction $\uparrow$	CORE $\uparrow$
$A_{0}$	ATTN	$0.7650$	$0.8081$	$0.2853$
$M_{1}$	DMTN	$0.6657$	$0.5735$	$0.3083$
$A_{1}$	ATTN	$\mathbf{0.6619}$	$\mathbf{0.8334}$	$\mathbf{0.3275}$
$M_{2}$	DMTN	$0.6690$	$0.4676$	$0.3082$

Table 3: Sequentially adapted depth-32 checkpoints, not independent matched-budget ablations.

	$\mathcal{A}$ exact	$\mathcal{A}$ free
$\mathcal{S}$ definite	DMAP	DMTN
$\mathcal{S}$ indefinite	ATAP	ATTN

Table 4: The transport family. Each operator is

\operatorname{rownorm}\exp_{\odot}(\mathcal{S}+\mathcal{A})

, sorted by whether its symmetric logit keeps the definite quadratic core and whether its antisymmetric logit stays exact. DMAP keeps both rules with half of attention’s query–key parameters, DMTN and ATAP each break one, and ATTN breaks both.

Transformer Attention

Transformer attention lets models relate tokens by weighting information from other positions, enabling context-dependent sequence and multimodal processing.

See all 324 papers

From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers
Jul 12, 2026Binbin Lin, Wei Chen, Yalun Li +3Self-Attention Transformer Attention
School of Software Technology, Zhejiang University, China · College of Computer Science and Technology, Zhejiang University, China · Alibaba Cloud, China
Riemannian Attention Mechanisms for Transformers: A Theoretical Framework and Architecture Design
Aug 2, 2026Sen SongTransformer Self-Attention
Independent Researcher, Guangzhou, China
Geometric knowledge spaces, Attention pairing and Architectures
Oct 21, 2026Shang XuAttention Mechanisms Transformer

Jul 12, 2026J/K move · Enter open · S save

From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers

Binbin Lin, Wei Chen, Yalun Li, Wenxiao Wang+2

School of Software Technology, Zhejiang University, China · College of Computer Science and Technology, Zhejiang University, China · Alibaba Cloud, China

Self-Attention Transformer Attention Representation Geometry in Language Models Multi-Head Attention

Open paper AskPDF arXiv

Self-attention is a ubiquitous primitive in modern sequence models, yet its operator-level geometry is only partially understood. We view a token sequence as a vector field over the token-position graph and identify attention as a connection walk: messages are aggregated by a nonnegative walk matrix while being transported along each edge by a learned linear map. Within this framework, we prove that single-head attention (SHA) is exactly a connection propagation step with constant transport, and that multi-head attention (MHA) is exactly a single edge-dependent connection walk whose effective transport is an attention-gated mixture of headwise transports. We further clarify the conditions under which the corresponding generator reduces to a random-walk connection Laplacian, highlighting the roles of stochasticity, reversibility, and metric-compatible transports. Empirically, we find that trained Transformers across scales (from 124M to 8B) and structures (encoder/decoder) exhibit geometric structure consistent with our theory: effective attention graphs converge to stable geometric operators in deeper layers, learned transports self-organize into approximate scaled isometries, and both phenomena strengthen consistently with scale. Overall, the paper provides a precise connection-walk formalism that links self-attention to classical geometric operators, along with a set of operator-level tools for analyzing transformer models from a geometric perspective.

Abstract

Transformer Attention

Explore similar work

From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers

Riemannian Attention Mechanisms for Transformers: A Theoretical Framework and Architecture Design

Geometric knowledge spaces, Attention pairing and Architectures