Emergence of the transformer architecture as the score of entropic laws of random measures

Giacomo Greco

cs.AIOct 29, 2026

Emergence of the transformer architecture as the score of entropic laws of random measures

Authors: Giacomo Greco

Abstract

We show that the transformer architecture emerges as the score of last-token laws in a natural statistical model of language, where sentences are empirical measures of tokens, a language is a law on $P(S^d)$ , and our prior belief is that sentences carry no structure at all. By the maximum-entropy principle, we select the law closest to this prior in relative entropy among those reproducing an observed sentence-level statistic. We discuss the relevance of the statistic in the posterior interaction between tokens and we build a statistic, able to capture that interaction, with Luce's choice axiom truncated at the lowest relational order. This procedure gives the softmax choice rule, with the attention cross-moment as its mean parameter. The query--key matrix is then the Lagrange multiplier of a maximum-entropy problem inside each sentence, and the value matrix the multiplier of the constraint over the language, so that the two are untied. Under the optimal entropic law we then show that a transformer head appears in the score of the last-token law. For ordered sentences, causal masking follows from the masked reference choice rule, and with a next-token cross-moment the score equals one causally masked attention head. We also discuss multi-head attention, weight-tied unembedding, and a heuristic reading of residual blocks as projected Langevin steps.

Figures & tables

Figure 1. Transformer architecture as a three-step procedure. In the first step an embedding operator maps each token, represented by a one-hot vector in

{\mathbb{R}}^{|\mathcal{V}|}

, to a vector of

\mathbb{R}^{d+1}

through a learned matrix

E\in{\mathbb{R}}^{(d+1)\times|\mathcal{V}|}

, possibly combined with normalization and with a positional encoding which keeps track of token order. Then, in the second step the sequence of vectors

x_{1},\dots,x_{n}

goes through a stack of

L

identical layers. Each layer consists of a residual attention sub-layer, in which every token updates its representation by attending to the other tokens of the sentence, and a residual MLP which acts on each token separately. The output is then projected back to

\mathbb{S}^{d}

via

\mathrm{LN}

. Finally, an unembedding matrix

W_{\mathrm{out}}\in{\mathbb{R}}^{|\mathcal{V}|\times(d+1)}

maps the output representation of the last token to a vector of logits, one per element of the vocabulary. Their softmax is the law of the next token, from which a token is sampled and appended to the text.

Figure 2. Causal version of the pipeline of Figure 1 . Two things change. First, each token is recorded together with its position, so that the sentence becomes the ordered measure

\overset{\rightarrow}{\mu}=\frac{1}{n}\sum_{i}\delta_{(x_{i},i)}\in\mathcal{P}(\mathbb{S}^{d}\times{\mathbb{N}})

. Second, the attention sub-layer is causally masked: the additive mask

M

sets to zero the weight of every key coming after the query (

j>i

), so that each token only reads itself and the tokens before it.

Figure 3. Multi-head version of the causal transformer block of Figure 2 . The single masked head is replaced by

H

heads, each with its own query, key and value matrices

Q_{h},K_{h},V_{h}

, all sharing the same causal mask

M

. The head outputs are summed before the residual connection and the normalization. In our model each head corresponds to a different choice rule, with attention matrix

\Lambda_{h}=Q_{h}^{\top}K_{h}

, and

V_{h}

is the Lagrange multiplier of the corresponding cross-moment.

Transformer Attention

Transformer attention lets models relate tokens by weighting information from other positions, enabling context-dependent sequence and multimodal processing.

See all 324 papers

Transformer Architectures as Complete Bayes Processes: A Formal Proof in the Measure-Theoretic Kernel Framework
Jun 29, 2026Haobo YangTransformer Bayesian Inference
Department of Computer Science and Engineering, SUSTech University
The Randomness Floor: Measuring Intrinsic Non-Randomness in Language Model Token Distributions
Mar 29, 2026Jarosław HryszkoState Space Language Models Language Modeling
Institute of Computer Science, Faculty of Mathematics and Computer Science, Jagiellonian University, Krak´ow, Poland
Transformers Learn Latent Mixture Models In-Context via Mirror Descent
Apr 12, 2026Francesco D'Angelo, Nicolas FlammarionIn-Context Learning Transformer Attention
TML Lab, EPFL, Switzerland

Jun 29, 2026J/K move · Enter open · S save

Transformer Architectures as Complete Bayes Processes: A Formal Proof in the Measure-Theoretic Kernel Framework

Haobo Yang

Department of Computer Science and Engineering, SUSTech University

Transformer Bayesian Inference

Open paper AskPDF arXiv

We present a complete formal proof that transformer architectures, when their internal update mechanisms satisfy a Bayes joint-distribution condition, implement exact Bayesian posterior inference. Working within the measure-theoretic kernel framework, we define a hierarchy of abstractions -- from the core Bayesian transformer, through semantic transformers with explicit update kernels, to full transformer blocks with QKV/attention/residual/MLP pipelines, and finally multilayer stacks -- and prove at each level that the Bayes joint semantics implies the update kernel equals the posterior almost everywhere. For the block-level architecture, we derive the explicit Bayes formula through Radon-Nikodym differentiation and prove its normalization. We additionally prove that the softmax attention mechanism induces a valid probability distribution over keys, establishing the bridge between the abstract kernel framework and concrete attention implementations. The framework makes no architectural assumptions beyond the Markov kernel structure and exposes explicit conditions under which a transformer block is provably Bayesian. In essence, when this joint distribution condition is satisfied, the forward computation of a Transformer is formally equivalent to a rigorous Bayesian posterior update.

Abstract

Transformer Attention

Explore similar work

Transformer Architectures as Complete Bayes Processes: A Formal Proof in the Measure-Theoretic Kernel Framework

The Randomness Floor: Measuring Intrinsic Non-Randomness in Language Model Token Distributions

Transformers Learn Latent Mixture Models In-Context via Mirror Descent