cs.LGNov 1, 2026

Beliefs in feedback: Factored representations of interacting agents in transformers

Authors: David A. Ehrlich, Manuel Baltieri, Fernando E. Rosas, Alexander B. Boyd

Organizations: XOR Labs, London, UK · Araya Inc., Tokyo, Japan · Department of Informatics, University of Sussex, Brighton, UK · Department of Brain Sciences, Imperial College London, London, UK · 

Abstract

Complex systems, from ecosystems to brains, are usually modeled as modular components coupled through feedback rather than as indivisible wholes. Can a transformer trained on the joint behavior of a group of interacting agents be understood in the same way? In this work, we develop a computational-mechanics framework to study agents interacting in feedback loops. We formally prove that any joint stochastic process can be presented as resulting from the behavior of agents interacting in feedback, and the resulting process is generated by a hidden Markov model whose Bayesian beliefs factorize into separate components per agent. Furthermore, our experiments show that transformers trained on such feedback sequences make use of this separability, disentangling the beliefs about the underlying factors into separate subspaces of their residual stream. This representation is found to scale linearly (instead of exponentially) with the number of agents. Moreover, interventions show that these representations are causally efficacious, allowing for targeted steering of individual agents' beliefs by selectively inserting counterfactuals. However, coarse-graining the behavior is found to disrupt the factorization, and can force a higher-dimensional representation. Together, the results establish both the dimensional advantage of a factorized representation and the ability of transformers to exploit it, laying the groundwork for investigating traces of compositional predictive architecture in pretrained models.

Figures & tables

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Sep 18, 2026

World Modeling in Transformers

Behavioral failures can make a transformer appear to lack a world model even when it has learned faithful representations of its environment. We demonstrate this in TaxiGPT, a transformer trained on random walks through Manhattan whose failures have been interpreted as evidence of an incoherent internal map. Through mechanistic analysis and causal interventions, we show that the model represents intersections and streets, tracks its position, and uses a goal compass to navigate. We trace its failures to interference between superposed intersection features, which disrupts localization within the internal map. Affordance packing, which groups representations of intersections with the same legal moves, helps limit the consequences of these errors. Finally, we propose mechanistic indicators that we use to compare models and show that world-modeling capacities emerge at different stages of training. Our findings motivate a shift from asking whether a model has a world model to mechanistically studying its world modeling: the interacting capacities through which it represents its environment and uses those representations to guide behavior.
Mar 17, 2026

Transformers are Bayesian Networks

Transformers are the dominant architecture in AI, yet why they work remains poorly understood. This paper offers a precise answer: a transformer is a Bayesian network. We establish this in five ways. First, we prove that every sigmoid transformer with any weights implements weighted loopy belief propagation on its implicit factor graph. One layer is one round of BP. This holds for any weights -- trained, random, or constructed. Formally verified against standard mathematical axioms. Second, we give a constructive proof that a transformer can implement exact belief propagation on any declared knowledge base. On knowledge bases without circular dependencies this yields provably correct probability estimates at every node. Formally verified against standard mathematical axioms. Third, we prove uniqueness: a sigmoid transformer that produces exact posteriors necessarily has BP weights. There is no other path through the sigmoid architecture to exact posteriors. Formally verified against standard mathematical axioms. Fourth, we delineate the AND/OR boolean structure of the transformer layer: attention is AND, the FFN is OR, and their strict alternation is Pearl's gather/update algorithm exactly. Fifth, we confirm all formal results experimentally, corroborating the Bayesian network characterization in practice. We also establish the practical viability of loopy belief propagation despite the current lack of a theoretical convergence guarantee. We further prove that verifiable inference requires a finite concept space. Any finite verification procedure can distinguish at most finitely many concepts. Without grounding, correctness is not defined. Hallucination is not a bug that scaling can fix. It is the structural consequence of operating without concepts. Formally verified against standard mathematical axioms.
May 20, 2026

Markovian Circuit Tracing: Ground-Truth Tests of Transformer State Abstractions

Mechanistic interpretability needs tests that separate accessible information from a causally useful internal abstraction. We introduce Markovian Circuit Tracing (MCT), a controlled benchmark for testing whether transformer activations support a compact state abstraction with recoverable transition structure. We train small causal transformers on sequences from four-state hidden Markov models. The experiment fixes the latent transition law and varies emission observability. The canonical artifact contains 15 runs across five fixed seeds. Every model reaches validation loss within 0.02 nats per token of the exact Bayes predictor. Under easy observability, held-out mean rowwise Kullback-Leibler (KL) divergence from the true transition matrix is 0.114. The easy shuffled control gives 0.379. Under medium observability, recovered transition KL is 0.221. The medium shuffled control gives 0.355. Recovery collapses to control performance under hard observability. Natural activation swaps and state forcing show the same decline in causal specificity. New transition-specific controls refine this result. Untrained activations yield similar transition recovery in the easy regime. In medium, trained activations improve over the untrained control by 0.049 KL on average. Probabilistic Markov-order tests show a clear first-order gain in easy. Medium shows a smaller gain. Hard shows no meaningful gain. A second latent transition law reproduces the same observability-dependent pattern. The belief-probe control fails to support learned specificity. The fixed sparse autoencoder fails to improve transition recovery. MCT therefore tests when a state-machine interpretation earns support and when the evidence rejects it.