Transformers are Bayesian Networks

Gregory Coppola

cs.AIMar 17, 2026

Transformers are Bayesian Networks

Authors: Gregory Coppola

Abstract

Transformers are the dominant architecture in AI, yet why they work remains poorly understood. This paper offers a precise answer: a transformer is a Bayesian network. We establish this in five ways. First, we prove that every sigmoid transformer with any weights implements weighted loopy belief propagation on its implicit factor graph. One layer is one round of BP. This holds for any weights -- trained, random, or constructed. Formally verified against standard mathematical axioms. Second, we give a constructive proof that a transformer can implement exact belief propagation on any declared knowledge base. On knowledge bases without circular dependencies this yields provably correct probability estimates at every node. Formally verified against standard mathematical axioms. Third, we prove uniqueness: a sigmoid transformer that produces exact posteriors necessarily has BP weights. There is no other path through the sigmoid architecture to exact posteriors. Formally verified against standard mathematical axioms. Fourth, we delineate the AND/OR boolean structure of the transformer layer: attention is AND, the FFN is OR, and their strict alternation is Pearl's gather/update algorithm exactly. Fifth, we confirm all formal results experimentally, corroborating the Bayesian network characterization in practice. We also establish the practical viability of loopy belief propagation despite the current lack of a theoretical convergence guarantee. We further prove that verifiable inference requires a finite concept space. Any finite verification procedure can distinguish at most finitely many concepts. Without grounding, correctness is not defined. Hallucination is not a bug that scaling can fix. It is the structural consequence of operating without concepts. Formally verified against standard mathematical axioms.

Figures & tables

Graph	BP exact	Transformer	Max error
0	[0.7349, 0.4366]	[0.7338, 0.4346]	0.0021
1	[0.4097, 0.4036]	[0.4096, 0.4031]	0.0005
4	[0.6459, 0.8298]	[0.6436, 0.8297]	0.0023
9	[0.4084, 0.5523]	[0.4084, 0.5526]	0.0003

Table 1: Transformer vs. exact BP posteriors on held-out graphs.

Dims	Content
0	own belief (initialized to 0.5)
1–4	factor table entries $[f_{00},f_{01},f_{10},f_{11}]$
5	node type (0 = variable, 1 = factor)
6	own index $/(n-1)$
7	neighbor index $/(n-1)$

Table 2: Token encoding for the BP transformer.

	Job	Input	Semantics
Softmax 1	Routing	Q/K dot products	Differentiable argmax
Softmax 2	Inference	Neighbor beliefs	Bayesian posterior
Softmax 3	Generation	Final hidden state	Token distribution

Table 3: The three softmaxes: same operation, three different jobs.

Experiment	Vars	Loops	Converged	Avg KL	Avg MAE
Triangle	3	1	100/100	0.000045	0.002091
Square	4	1	100/100	0.000002	0.000382
Dating graph	5	1	100/100	0.000102	0.003590
Two loops	4	2	100/100	0.000086	0.003111
QBBN chain	6	1	100/100	0.000021	0.001055

Table 4: Loopy BP on QBBN-structured graphs. 500 trials total.

Work	Method	Result	Relation
Giannou et al. 2023	Constructive	TF Turing complete	Same conclusion, different construction
Akyürek et al. 2022	Constructive	TF implements GD	Same method; BP exact, GD approximate
Xie et al. 2021	Statistical	ICL $\approx$ Bayesian	Statistical version; we give mechanistic
Jung et al. 2022	Formal	Attention is BP (latent)	Complementary; any weights vs. trained
Scarselli et al. 2009	Structural	GNN implements BP	Extends to TF without hardwired topology
Elhage et al. 2021	Empirical	Circuits in trained models	We give target circuit for interpretability

Table 5: Positioning relative to related work.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Repo	What it proves / shows
universal-lean	transformer_is_turing_complete
	https://github.com/gregorycoppola/universal-lean
sigmoid-transformer-lean	every_sigmoid_transformer_is_bayesian_network
	uniqueness : exact posteriors force BP weights
	https://github.com/gregorycoppola/sigmoid-transformer-lean
transformer-bp-lean	transformer_implements_bp

Appendix

Table 6: Public repositories supporting this paper.

Explore similar work

Transformer Architectures as Complete Bayes Processes: A Formal Proof in the Measure-Theoretic Kernel Framework
Jun 29, 2026Haobo YangTransformer Bayesian Inference
Department of Computer Science and Engineering, SUSTech University
Learning Source Structure: Bayesian Prediction with Transformers
Oct 13, 2026Meir Feder, Ruediger UrbankeTransformer Sequence Modeling
School of Electrical and Computer Engineering, Tel Aviv University, Tel Aviv, Israel · School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland
Neural expectation-maximization and meta-models
Oct 8, 2026Zhuolin FuTransformer Transformer Attention

Jun 29, 2026J/K move · Enter open · S save

Transformer Architectures as Complete Bayes Processes: A Formal Proof in the Measure-Theoretic Kernel Framework

Haobo Yang

Department of Computer Science and Engineering, SUSTech University

Transformer Bayesian Inference

Open paper AskPDF arXiv

We present a complete formal proof that transformer architectures, when their internal update mechanisms satisfy a Bayes joint-distribution condition, implement exact Bayesian posterior inference. Working within the measure-theoretic kernel framework, we define a hierarchy of abstractions -- from the core Bayesian transformer, through semantic transformers with explicit update kernels, to full transformer blocks with QKV/attention/residual/MLP pipelines, and finally multilayer stacks -- and prove at each level that the Bayes joint semantics implies the update kernel equals the posterior almost everywhere. For the block-level architecture, we derive the explicit Bayes formula through Radon-Nikodym differentiation and prove its normalization. We additionally prove that the softmax attention mechanism induces a valid probability distribution over keys, establishing the bridge between the abstract kernel framework and concrete attention implementations. The framework makes no architectural assumptions beyond the Markov kernel structure and exposes explicit conditions under which a transformer block is provably Bayesian. In essence, when this joint distribution condition is satisfied, the forward computation of a Transformer is formally equivalent to a rigorous Bayesian posterior update.

Abstract

Explore similar work

Transformer Architectures as Complete Bayes Processes: A Formal Proof in the Measure-Theoretic Kernel Framework

Learning Source Structure: Bayesian Prediction with Transformers

Neural expectation-maximization and meta-models