Beliefs in feedback: Factored representations of interacting agents in transformers
Organizations: XOR Labs, London, UK · Araya Inc., Tokyo, Japan · Department of Informatics, University of Sussex, Brighton, UK · Department of Brain Sciences, Imperial College London, London, UK ·
Abstract
Complex systems, from ecosystems to brains, are usually modeled as modular components coupled through feedback rather than as indivisible wholes. Can a transformer trained on the joint behavior of a group of interacting agents be understood in the same way? In this work, we develop a computational-mechanics framework to study agents interacting in feedback loops. We formally prove that any joint stochastic process can be presented as resulting from the behavior of agents interacting in feedback, and the resulting process is generated by a hidden Markov model whose Bayesian beliefs factorize into separate components per agent. Furthermore, our experiments show that transformers trained on such feedback sequences make use of this separability, disentangling the beliefs about the underlying factors into separate subspaces of their residual stream. This representation is found to scale linearly (instead of exponentially) with the number of agents. Moreover, interventions show that these representations are causally efficacious, allowing for targeted steering of individual agents' beliefs by selectively inserting counterfactuals. However, coarse-graining the behavior is found to disrupt the factorization, and can force a higher-dimensional representation. Together, the results establish both the dimensional advantage of a factorized representation and the ability of transformers to exploit it, laying the groundwork for investigating traces of compositional predictive architecture in pretrained models.
Figures & tables
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| setup | factor | kernels (parameters) |
| feedback A | agent mess3 ( ) | |
| (vocab ) | agent fractal2 ( , ) | |
| feedback B | agent fractal2 ( ) | |
| (vocab ) | ||
| agent mess3 ( , ) |
| parameter | value |
|---|---|
| layers / / heads / / | / / / / |
| activation / normalization / positions | ReLU / LayerNorm / learned absolute |
| context | (BOS 8 tokens) |
| vocabulary | joint process vocab 1 (BOS) |
| optimizer | AdamW, lr , , |
| weight decay |
| full | only ( hidden) | only ( hidden) | ||||
|---|---|---|---|---|---|---|
| replicate | ||||||
| r0 | ||||||
| r1 | ||||||
| r2 | ||||||
| r3 | ||||||
| r4 | ||||||
Transformer Interpretability
Transformer interpretability studies how attention, feed-forward layers, and hidden representations support model predictions and behavior.
See all 355 papers