Beliefs in feedback: Factored representations of interacting agents in transformers
Authors: David A. Ehrlich, Manuel Baltieri, Fernando E. Rosas, Alexander B. Boyd
Organizations: XOR Labs, London, UK · Araya Inc., Tokyo, Japan · Department of Informatics, University of Sussex, Brighton, UK · Department of Brain Sciences, Imperial College London, London, UK · Centre for Eudaimonia and Human Flourishing, University of Oxford, Oxford, UK · Beyond Institute for Theoretical Science, San Francisco, CA, USA
Complex systems, from ecosystems to brains, are usually modeled as modular components coupled through feedback rather than as indivisible wholes. Can a transformer trained on the joint behavior of a group of interacting agents be understood in the same way? In this work, we develop a computational-mechanics framework to study agents interacting in feedback loops. We formally prove that any joint stochastic process can be presented as resulting from the behavior of agents interacting in feedback, and the resulting process is generated by a hidden Markov model whose Bayesian beliefs factorize into separate components per agent. Furthermore, our experiments show that transformers trained on such feedback sequences make use of this separability, disentangling the beliefs about the underlying factors into separate subspaces of their residual stream. This representation is found to scale linearly (instead of exponentially) with the number of agents. Moreover, interventions show that these representations are causally efficacious, allowing for targeted steering of individual agents' beliefs by selectively inserting counterfactuals. However, coarse-graining the behavior is found to disrupt the factorization, and can force a higher-dimensional representation. Together, the results establish both the dimensional advantage of a factorized representation and the ability of transformers to exploit it, laying the groundwork for investigating traces of compositional predictive architecture in pretrained models.
Figures & tables
Figure 1: ( Left ) Automata in a feedback loop representing two interacting agents. ( Center ) The feedback loop circuit requires a time-ordering for the signals going between agents, which can be boxed into a single repeating circuit element. ( Right ) The boxing allows expressing the feedback loop as an HMM for the joint process.
Figure 2: Separable linear representations of beliefs emerge in transformers trained on joint feedback processes. A coupled process with mess3 as agent T (top row) and fractal2 as agent U (bottom row). ( Leftmost column ) States of the MSP of each factor (top two PCA directions; colored by the belief resp. the emission probability). ( Remaining columns ) linear reconstruction from the final-layer residual stream at different points through training (held-out linear probe refit per checkpoint, same PCA basis; held-out R2 annotated). The experimental protocol is described in Appendix K .
Figure 3: Beliefs can be patched factor by factor. Belief patching for the system with beliefs as in Fig. 2 : a counterfactual belief is spliced into one factor’s residual-stream subspace, and the joint next-token distribution is compared in KL with the Bayes prediction for the patched beliefs ( edit ), the unedited model ( floor ), a write with no effect ( null ), and an unrelated sequence ( unrel. ). Annotated: fraction of the null-to-target gap closed (mean ± sd, five seeds). Protocol is described in Appendix L .
Figure 4: The dimensionality advantage of factored representations grows with the number of agents. (a) A social network in which a central hub exchanges perceptions and actions with N agents; the agents (green band) can be read as a single macro-agent, the tensor product of the individual agents, so the network is a binary feedback loop between the hub and this macro-agent. (b) Cumulative explained variance (CEV) of the final-layer residual stream follows the factored prediction for the telephone process with three mess3 sub-agents. Solid curves: mean over five seeds (see Table 2 ); shaded bands: min–max across seeds. Dashed: CEV of the factored concatenation of the Bayes-optimal per-agent belief states on the same sequences; dotted: CEV of their tensor product (Kronecker). (c) Effective rank nc —the smallest number of principal components whose CEV reaches 80% (red) resp. 95% (blue)—for telephone processes with N mess3 sub-agents (BOS position excluded, 4096 sequences). Solid lines: the final-layer residual stream (mean over five seeds per N ; the shaded min–max band never exceeds one component). Bayes-optimal belief states themselves are the factored concatenation of the per-agent belief states (dashed; simplex dimension ∑i(di−1)=8N ) and their Kronecker product (dotted; simplex dimension ∏idi−1=9N−1 ). For N≥3 the residual stream matches the factored belief states to within one component at both thresholds, while the Kronecker belief states require 2.5 – 11× more. (d) Coarse-graining forces a higher -dimensional representation: CEV by PCA rank, one color per observed interface, dashed lines the Bayes-optimal belief state and solid lines the trained transformer (mean over five independent draws of the process, one model per draw and interface; shaded bands: min–max across draws). Under full observation, the belief state factorizes, and the theory curve saturates at the 8 -dimensional factored concatenation, whereas the coarse observers must track the entangled joint belief state ( ∼22 dimensions). Process and model parameters in Appendix K .
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: (TOP) An agent’s behavior is described by the interface it generates, which can be thought of as a channel whose inputs and outputs are arranged in time. (BOTTOM) An agent’s mechanism is described by a transducer, which describes a recursive circuit that uses a memory random variable Rt to carry information between past and future.
Figure 6: An agent’s identity may be obscured by a black box. Even if you can perfectly characterize the way they map their perceptions to actions, you may be uncertain as to how their latent structure leads to those actions. This is a classic manifestation of the Turing test, where many mechanisms (human, AI, lookup tables) can all produce statistically similar behavior.
Figure 7: A feed-forward circuit composed of multiple agents (bottom) is a special case of a feedback loop. In this case, that is achieved by eliminating the dependence of Agent U on Agent T ’s action, indicated by striking the wire for Yt going to U with a red slash.
Figure 8: There are four different readily accessible presentations for a feedback process between agents T and U . Which one you arrive at depends on 1) whether you choose to model the simultaneous process X0:tY0:t or the shifted process X1:t+1Y0:t , and 2) whether you box circuit elements so that Agent U or Agent T operates first. All presentations modify the linear operator of one of the transducers, either delaying the action or the perception.
Figure 9: Belief-state geometry of the role-swapped setup likewise refines over training. Rows: both factors of setup B (agent T fractal2, agent U mess3); setup A is shown in Fig. 2 . Leftmost column: ground-truth MSP. Remaining columns: linear reconstruction from the final-layer residual stream at checkpoints 0 , 500 , 5 k, and 50 k (probe refit per checkpoint; same PCA basis and color conventions as Fig. 2 ; held-out R2 per panel). Both factors are already reconstructed to R2≥0.83 at initialization and are refined by training to R2≥0.998 by step 50 k.
Figure 10: Both mess3–fractal2 setups are represented by factorized, causally load-bearing beliefs in the residual stream. The final-checkpoint reconstruction of Fig. 2 with the role-swapped arrangement alongside it: mess3 as agent T and fractal2 as agent U (setup A, panels a,b,e,f— the setup reproduced in the main text) or vice-versa (setup B, panels c,d,g,h). Top row: Bayes-optimal belief states; bottom row: reconstruction from the final-layer residual stream at step 50 k. Panel and color conventions and the probe protocol are as in Fig. 2 ; the belief patching in (i,j) is as in Fig. 3 .
Figure 11: Probe selectivity controls and layer-wise emergence. All panels show the held-out linear-probe error 1−R2 on the residual stream over training (mess3–fractal2 feedback loops). Solid curves: mean over five seeds per setup—the canonical run plus four reruns ( Table 2 ); shaded bands: min–max across seeds. (a,b) Final-layer probes fitted to the true per-factor beliefs and to two controls: the same sequence’s belief two positions earlier (time-shifted) and beliefs of shuffled sequences (a derangement of the batch). The shuffled floor stays at 1−R2≈1 throughout; the true factors improve by 40 – 280× over training. (c–f) The true-belief probe refit at each residual-stream depth (embedding, blocks 1 – 4 , final LayerNorm). Blocks 1 – 2 acquire essentially all of the structure; the action-delayed factor U is complete by block 2, while factor T sharpens to the last layer. The embedding row is training-invariant by construction (a linear probe on it can realize any function of token and position).
Figure 12: Cumulative explained variance for the five-agent fractal2 telephone process. Same protocol and conventions as Fig. 4 b: CEV of the final-layer residual stream by rank at several training steps; solid curves the mean over five seeds, shaded bands the min–max across them; dashed the CEV of the factored concatenation of the Bayes-optimal belief states, dotted that of their Kronecker product.
Figure 13: Effective rank is mostly independent of model width. Effective rank of the final-layer residual stream at step 50000 (BOS position excluded, 1024 sequences, same protocol as Fig. 4 c) for the three-agent mess3 telephone process, against residual-stream width dmodel∈{60,120,240} : nc at the 80% and 95% CEV thresholds. Points are means over five seeds for widths 60 and 120, and three seeds for width 240. Bands are min–max across seeds and are degenerate for nc@80% . Horizontal lines: the factored bound ∑i(di−1)=24 (dashed) and the joint-Kronecker bound ∏idi−1=728 (dash-dotted).
Figure 14: The base HMMs (edge labels read “emission : probability”). (a) mess3 (x,a) : three states, three symbols; with b=(1−a)/2 and y=1−2x . A transition into state s′ emits symbol s′ with probability ax≡ax and each other symbol with probability bx (independent of the source state, so both incoming arrows share the label shown by each vertex); a self-loop at s emits s with probability ay and each other symbol with probability by . Its mixed-state presentation is the fractal simplex of Fig. 2 (top row). (b) fractal2 (b,p,q) : two states, two symbols; state s re-emits its own symbol with probability b and stays, otherwise emits the other symbol and moves with probability p (resp. q ). Its two-state belief simplex is the barcode of Fig. 10 c. (c) sns (p,q) : the simple nonunifilar source; symbol 1 is emitted only on the resetting B→A transition, giving a Cantor-like 1-D belief geometry, drawn as a barcode like the fractal2 factor of Fig. 10 c.
setup
factor
kernels (parameters)
feedback A
agent T= mess3 ( S=3 )
(x,a)∈{(0.05,0.85),(0.15,0.60)}
(vocab 6 )
agent U= fractal2 ( S=2 , A=2 )
(b;p=q)∈{(0.25;0.25),(0.50;0.35),
(0.75;0.65)}
feedback B
agent T= fractal2 ( S=2 )
(b;p=q)∈{(0.25;0.25),(0.50;0.35),
(vocab 2⋅3=6 )
(0.75;0.65)}
agent U= mess3 ( S=3 , ×A=3 )
(x,a)∈{(0.05,0.85),(0.15,0.60)}
Appendix
Table 1: Generative-process parameters. Sequences have length 8 plus a BOS token. Each agent’s kernels are stacked along its input axis: the incoming symbol selects the kernel. Agent U is the action-delayed factor, whose state is augmented by the latched action ( ×A ). Setup A is the mess3–fractal2 loop shown in Fig. 2 , and the pair (A, B) is shown in Fig. 10 . The primed setups (A ′ , B ′ ) replace fractal2 by sns; they are the mess3–sns pair of Appendix L . In both pairs, setup A has mess3 as agent T . The mess3 ↔ mess3 loop appears in Fig. 16 . The fractal2 kernels use p=q : fractal2’s state-transition matrix is doubly stochastic exactly then, so the uniform initial belief used by the belief machinery is the true stationary distribution.
parameter
value
layers / dmodel / heads / dhead / dmlp
4 / 120 / 3 / 40 / 480
activation / normalization / positions
ReLU / LayerNorm / learned absolute
context nctx
9 (BOS + 8 tokens)
vocabulary
joint process vocab + 1 (BOS)
optimizer
AdamW, lr 5×10−4 , β=(0.9,0.999) ,
weight decay 0
Appendix
Table 2: Transformer and optimization hyperparameters , shared by all runs (exceptions noted).
Figure 15: All runs converge to their Bayes-optimal floors. Evaluation loss over training (log scale); solid curves are means over five seeds per setup, shaded bands the min–max across seeds (mostly narrower than the line). Dashed lines: each setup’s Bayes-optimal floor, position-averaged and estimated with 32768 MC samples (per-seed floor estimates differ by ∼10−3 nats; the line shows the family mean). (a) The four main-text setups: the mess3–fractal2 feedback loops A and B of Figs. 2 and 10 and the two telephone processes. (b) The telephone-mess3 N -sweep of Fig. 4 c. Every setup reaches its floor within ∼103 steps; final losses lie within 1 – 5×10−3 nats of the floor, i.e. at the precision of the MC estimate for all runs except two of the five N=4 seeds, which plateau 3 – 5×10−3 nats above it. The x -axis is symlog (linear below 103 to show the step-0 evaluation, logarithmic above).
Figure 16: Belief patching under the unprojected and the encoder-orthogonalized write. (a) Fraction of the null-to-target distance closed by the edit, per setup and patched factor (large markers: ratio of means over seeds; small dots: individual seeds; filled: unprojected write Δ , open: Δ⊥ of Eq. 67 ). Every setup without an sns factor closes 89.5 – 99.6% of the gap under either construction. The two sns loops fail under both, and are the only setups the construction changes qualitatively: under the unprojected write, editing the mess3 partner in the sns → mess3 loop leaves the output farther from the target than no edit at all ( −41% ), which the projection turns into +14% —the raw write was corrupting the sns encoding directly. (b) The between-factor encoder subspace overlap (share of the edited factor’s encoder span lying in its partners’) against the gap difference between the constructions. The overlap is substantial everywhere ( 0.13 – 0.58 ) and does not separate fragile from robust writes—the sns loops are the outliers in both directions—so the deviations are not explained by overlapping belief encodings.
Figure 17: Belief edits track the Bayes prediction continuously along the interpolation path. KL divergence (median over interventions; bands: interquartile range) between the edited next-token distribution and three references, as a function of the edit dose λ : the Bayes prediction at the interpolated tuple (bf(λ),b−f) (on-path, solid), and the Bayes predictions at the two endpoints ( λ=0 : original; λ=1 : full counterfactual; dashed). Rows: the two feedback setups; columns: the patched factor; n : evaluable interventions pooled over five seeds per setup. The dashed horizontal rule is the unedited model’s own fidelity floor; the dotted curve is the untouched factor’s component against its original prediction (leakage). At λ=0.5 the nearest endpoint reference is 16 – 56× farther from the edited output than the interpolated target. Panel counts differ only through the identifiability gate ( 0.05 total variation between the endpoint references): the candidate pool is identical across panels, but the fractal2 belief distribution concentrates on a near-degenerate one-dimensional set, so swapping a fractal2 belief more often produces indistinguishable references.
Figure 18: The coarse-graining demonstrator process: two five-state transducers of this family in the canonical feedback loop, each one’s emitted symbol the other’s next input symbol k (agent action-delayed). (a) Structure of one factor (edge labels read “emission : probability” as in Fig. 14 , with δ~≡2δ/3 ). Each input symbol k selects a near-deterministic permutation kernel of the five states (drawn: π0=(4,2,1,3,0) , two transpositions and a fixed point), so different hidden-action histories imply visibly different dynamics; each transition emits only a three-way partition ρ of its arrival state (edges into the same state carry the same dominant token), so an observer never pins the state exactly and cannot invert the partner’s symbol. (b) The three observed interfaces. Observing the pair (xt,yt) makes the joint belief state factorize into two per-factor belief states ( 8 dimensions); marginalizing either symbol destroys the factorization and forces an entangled belief state over the 75 -state joint latent ( 22 dimensions), even though the coarse observer receives strictly less information.
full (x,y)
only x ( U hidden)
only y ( T hidden)
replicate
95%
99%
95%
99%
95%
99%
r0
16
48
32
69
24
56
r1
21
56
42
77
27
57
r2
13
38
27
62
25
55
r3
10
29
12
38
23
54
r4
20
53
27
62
34
67
Appendix
Table 3: Effective rank of the final-layer residual stream per replicate , the smallest number of principal components whose CEV reaches 95% (resp. 99% ), on the three observed interfaces of the coarse-graining demonstrator process ( 1024 sequences, BOS position excluded). Replicates are independent draws of the process family (r0 is the draw analyzed in Appendix M.1 ). For reference, the exact Bayes-optimal belief states on the same sequences reach 95% at 8 dimensions under full observation in every draw and at 18 – 22 dimensions under either coarse-graining.
Behavioral failures can make a transformer appear to lack a world model even when it has learned faithful representations of its environment. We demonstrate this in TaxiGPT, a transformer trained on random walks through Manhattan whose failures have been interpreted as evidence of an incoherent internal map. Through mechanistic analysis and causal interventions, we show that the model represents intersections and streets, tracks its position, and uses a goal compass to navigate. We trace its failures to interference between superposed intersection features, which disrupts localization within the internal map. Affordance packing, which groups representations of intersections with the same legal moves, helps limit the consequences of these errors. Finally, we propose mechanistic indicators that we use to compare models and show that world-modeling capacities emerge at different stages of training. Our findings motivate a shift from asking whether a model has a world model to mechanistically studying its world modeling: the interacting capacities through which it represents its environment and uses those representations to guide behavior.
Transformers are the dominant architecture in AI, yet why they work remains poorly understood. This paper offers a precise answer: a transformer is a Bayesian network. We establish this in five ways. First, we prove that every sigmoid transformer with any weights implements weighted loopy belief propagation on its implicit factor graph. One layer is one round of BP. This holds for any weights -- trained, random, or constructed. Formally verified against standard mathematical axioms. Second, we give a constructive proof that a transformer can implement exact belief propagation on any declared knowledge base. On knowledge bases without circular dependencies this yields provably correct probability estimates at every node. Formally verified against standard mathematical axioms. Third, we prove uniqueness: a sigmoid transformer that produces exact posteriors necessarily has BP weights. There is no other path through the sigmoid architecture to exact posteriors. Formally verified against standard mathematical axioms. Fourth, we delineate the AND/OR boolean structure of the transformer layer: attention is AND, the FFN is OR, and their strict alternation is Pearl's gather/update algorithm exactly. Fifth, we confirm all formal results experimentally, corroborating the Bayesian network characterization in practice. We also establish the practical viability of loopy belief propagation despite the current lack of a theoretical convergence guarantee. We further prove that verifiable inference requires a finite concept space. Any finite verification procedure can distinguish at most finitely many concepts. Without grounding, correctness is not defined. Hallucination is not a bug that scaling can fix. It is the structural consequence of operating without concepts. Formally verified against standard mathematical axioms.
Mechanistic interpretability needs tests that separate accessible information from a causally useful internal abstraction. We introduce Markovian Circuit Tracing (MCT), a controlled benchmark for testing whether transformer activations support a compact state abstraction with recoverable transition structure. We train small causal transformers on sequences from four-state hidden Markov models. The experiment fixes the latent transition law and varies emission observability. The canonical artifact contains 15 runs across five fixed seeds. Every model reaches validation loss within 0.02 nats per token of the exact Bayes predictor. Under easy observability, held-out mean rowwise Kullback-Leibler (KL) divergence from the true transition matrix is 0.114. The easy shuffled control gives 0.379. Under medium observability, recovered transition KL is 0.221. The medium shuffled control gives 0.355. Recovery collapses to control performance under hard observability. Natural activation swaps and state forcing show the same decline in causal specificity. New transition-specific controls refine this result. Untrained activations yield similar transition recovery in the easy regime. In medium, trained activations improve over the untrained control by 0.049 KL on average. Probabilistic Markov-order tests show a clear first-order gain in easy. Medium shows a smaller gain. Hard shows no meaningful gain. A second latent transition law reproduces the same observability-dependent pattern. The belief-probe control fails to support learned specificity. The fixed sparse autoencoder fails to improve transition recovery. MCT therefore tests when a state-machine interpretation earns support and when the evidence rejects it.