Geometric knowledge spaces, Attention pairing and Architectures

Shang Xu

cs.LGOct 21, 2026

Geometric knowledge spaces, Attention pairing and Architectures

Authors: Shang Xu

Abstract

In this paper, we propose a geometric approach to deriving attention mechanisms and exploring potential new transformer architectures. We begin by formalizing the notions of knowledge worlds and attention pairings, then demonstrate how this framework naturally leads to the architectures of standard attention, causal attention, linear attention, tensor attention, and recurrent neural networks. Finally, we introduce several new geometric spaces that may inspire novel neural network architectures.

Figures & tables

Architecture	knowledge worlds	Intelligence	Update and readout
Attention (ordinary or causal)	$A=E\oplus E\oplus F$ , $a=(q,k,u)$ , $A_{n}=A^{n}$ , $X=(a_{1},\ldots,a_{n})$ .	$V_{n}=\mathbb{R}^{n}$ , $B_{n}(X,a)=(s_{j})_{j=1}^{n}$ , $s_{j}=\langle k_{j},q\rangle$ .	$\mu_{n}(a,X)=(X,a)$ . $\alpha=\operatorname{sm}(s_{J}/\sqrt{r})$ , $y=\displaystyle\sum_{j\in J}\alpha_{j}u_{j}$ .
Linear attention	$A=\mathcal{S}_{1}$ , $A_{n}=\mathcal{S}_{n}$ , $H=[z\mid M]$ , $a=[q\mid qu^{\mathsf{T}}]$ .	$V_{n}=\mathbb{R}^{2}$ , $B_{n}(H,a)=(s,c)$ , $s=u^{\mathsf{T}}M^{\mathsf{T}}q$ , $c=z^{\mathsf{T}}q$ .	$\mu_{n}(a,H)=H+a$ . $y=\displaystyle\frac{M^{\mathsf{T}}q}{\varepsilon+z^{\mathsf{T}}q}$ . Normalized evaluation: $\displaystyle\frac{s}{\varepsilon+c}=\langle y,u\rangle$ .
Tensor attention	$A=\mathcal{T}$ , $a=(Q,K,U)$ , $A_{n}=A^{n}$ , $X=(a_{1},\ldots,a_{n})$ .	$V_{n}=\mathbb{R}^{h\times n}$ , $[B_{n}(X,a)]_{\ell j}$ $\hskip 8.50012pt=\langle Q_{\ell},K_{j,\ell}\rangle$ .	$\mu_{n}(a,X)=(X,a)$ . $P=\operatorname{sm}_{\mathrm{row}}(B_{n}/\sqrt{r})$ , $y_{\ell}=\displaystyle\sum_{j=1}^{n}P_{\ell j}U_{j,\ell}$ . Concatenate heads and apply $W_{O}$ .
Bilinear RNN	$A=\mathbb{R}^{d}$ , $A_{1}=A$ , $A_{n}=A\oplus F$ , $x=(s,c)$ .	$V_{n}=F$ , $B_{n}(x,a)=\Gamma_{\theta}(s,a)$ . The evaluation is an increment to memory.	$s^{\prime}=s+a$ , $c^{\prime}=c+\Gamma_{\theta}(s,a)$ , $\mu_{n}(a,x)=(s^{\prime},c^{\prime})$ .
Vanilla RNN	$A=\mathbb{R}^{d}$ , $A_{1}=A$ , $A_{n}=A\oplus F$ , $x=(s,h)$ .	$V_{n}=F$ , $B_{n}(x,a)=f_{\theta}(h,a)$ . The evaluation is the next hidden state.	$h^{\prime}=f_{\theta}(h,a)$ , $\mu_{n}(a,x)=(a,h^{\prime})$ . The hidden state is replaced by its evaluation.

Table 1. knowledge worlds, intelligence pairings, and the resulting neural architectures. The RNN formulas describe the stabilized stages

n\geq 2

, with initial stages as defined in the preceding constructions.

Attention Mechanisms

Attention mechanisms let neural networks weight information from other tokens or regions according to relevance, enabling context-dependent processing.

See all 517 papers

Attention Geometry
Feb 11, 2026Julio CandanedoTransformer Attention Neural Representation Geometry
Query Expansion and Key Specialization in Transformer Attention Geometry
Sep 28, 2026Vidit Gupta, Siddhesh Nadkarni, Mihik Chaudhari +2Neural Representation Geometry Transformer Attention
Dwarkadas J. Sanghvi College of Engineering Mumbai, India
Riemannian Attention Mechanisms for Transformers: A Theoretical Framework and Architecture Design
Aug 2, 2026Sen SongTransformer Self-Attention
Independent Researcher, Guangzhou, China

Abstract

Attention Mechanisms

Explore similar work

Attention Geometry

Query Expansion and Key Specialization in Transformer Attention Geometry

Riemannian Attention Mechanisms for Transformers: A Theoretical Framework and Architecture Design