Deep belief networks are exact

Gleb Smirnov

cs.AISep 4, 2026

Deep belief networks are exact

Authors: Gleb Smirnov

Abstract

We prove that every strictly positive probability distribution on $\{-1,1\}^n$ is represented exactly by a sigmoid belief network with finite parameters. This answers a question of Sutskever and Hinton. The proof upgrades their probability-sharing approximation to exact representation using Brouwer's fixed-point theorem.

Neural Network Approximation Theory

Neural network approximation theory studies how network size, depth, activations, and structure determine the accuracy of functions and operators.

See all 227 papers

Transformers are Bayesian Networks
Mar 17, 2026Gregory CoppolaBelief Propagation Transformer
Every Layer Counts: An Exponential $L_2$ Depth Hierarchy for ReLU Networks
Aug 24, 2026Itay SafranShallow Neural Networks ReLU Neural Networks
Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev
Shallower ReLU Network Representations via Exact Linear Algebra
Jul 22, 2026Kilian Rueß, Gennadiy Averkov, Florestan Brunck +7Shallow Neural Networks ReLU Neural Networks
University of Technology Nuremberg · Brandenburg University of Technology Cottbus–Senftenberg · University of Copenhagen +3

Mar 17, 2026J/K move · Enter open · S save

Transformers are Bayesian Networks

Gregory Coppola

Belief Propagation Transformer Bayesian Networks Transformer Interpretability

Open paper AskPDF arXiv

Transformers are the dominant architecture in AI, yet why they work remains poorly understood. This paper offers a precise answer: a transformer is a Bayesian network. We establish this in five ways. First, we prove that every sigmoid transformer with any weights implements weighted loopy belief propagation on its implicit factor graph. One layer is one round of BP. This holds for any weights -- trained, random, or constructed. Formally verified against standard mathematical axioms. Second, we give a constructive proof that a transformer can implement exact belief propagation on any declared knowledge base. On knowledge bases without circular dependencies this yields provably correct probability estimates at every node. Formally verified against standard mathematical axioms. Third, we prove uniqueness: a sigmoid transformer that produces exact posteriors necessarily has BP weights. There is no other path through the sigmoid architecture to exact posteriors. Formally verified against standard mathematical axioms. Fourth, we delineate the AND/OR boolean structure of the transformer layer: attention is AND, the FFN is OR, and their strict alternation is Pearl's gather/update algorithm exactly. Fifth, we confirm all formal results experimentally, corroborating the Bayesian network characterization in practice. We also establish the practical viability of loopy belief propagation despite the current lack of a theoretical convergence guarantee. We further prove that verifiable inference requires a finite concept space. Any finite verification procedure can distinguish at most finitely many concepts. Without grounding, correctness is not defined. Hallucination is not a bug that scaling can fix. It is the structural consequence of operating without concepts. Formally verified against standard mathematical axioms.

Abstract

Neural Network Approximation Theory

Explore similar work

Transformers are Bayesian Networks

Every Layer Counts: An Exponential L2L_2L2​ Depth Hierarchy for ReLU Networks

Shallower ReLU Network Representations via Exact Linear Algebra

Every Layer Counts: An Exponential $L_2$ Depth Hierarchy for ReLU Networks