LLMs are thought to track "belief states," i.e., running probability distributions over the latent variables that govern language (Shai et al., 2024; Sarfati et al., 2026), but so far this has only been comprehensively demonstrated on toy synthetic data and in a few isolated case studies. It has also never been empirically connected to the geometry of LLM features (the concepts interpretability finds in model activations). In this work, we plant a controllable latent variable inside natural-looking text. An LLM teacher writes ordinary text while we "subliminally" steer it along one of K = 8 unrelated sparse autoencoder directions at each token, with the active directions following a ring-shaped Markov chain. A small transformer model trained on this corpus does indeed track the Bayesian posterior belief about our planted latent variable. Moreover, it also arranges the 8 states themselves on a ring, in the exact order of the Markov chain, which is supporting evidence that a concept's geometry can be formed by the statistical dynamics of the latent variable behind it.
Figures & tables
Figure 1: Figure from Shai et al. (2024) , showing how given a data-generating process, a model learns to model optimal beliefs sitting on the probability simplex at any token.
Figure 2: Belief state geometry (each point a probability vector over the latent variable’s values) versus concept geometry (the arrangement of the values themselves), both read from the residual stream. The two are different objects.
Figure 3: Method summary: (a) we plant a latent variable as a Markov chain over K states; (b) we steer a teacher LLM while it writes, by adding the current state’s SAE decoder direction to the residual stream; (c) the result is natural-looking text, secretly labeled per token.
Figure 4: The 8-state ring Markov chain with stay probability 0.95 and its transition matrix, plus a corpus excerpt with tokens colored by the active state.
Figure 5: PCA of the optimal observer’s posteriors: the point cloud fills a ring-shaped region with the 8 states in ring order at the rim.
Figure 6: The control2 chain: same stay probability, but uniform jumps to any of the other 7 states, plus a control2 corpus excerpt with tokens colored by the active state.
Figure 7: Per-layer probe R2 and argmax accuracy for the ring student, control2 , and control1 , with the optimal observer ceiling at 0.762 .
Figure 8: Per-layer probe R2 and argmax accuracy restricted to first-dwell tokens: the ring student and control2 curves coincide.
Figure 9: Side-by-side PCA of the optimal observer’s posteriors and the student’s probe read-out at layer 12: same ring arrangement, noisier cloud for the student.
Figure 10: Top-2 PCA of the 8 state centroids: the ring student’s centroids trace the ring in exact neighbor order; the control student’s do not.
Figure 11: Top-2 PCA of first-dwell centroids: the clean ring is no longer visible in the leading principal components.
Figure 12: Circle-plane projections of first-dwell centroids for the three students, and the circle-fit permutation test over all 2520 ring orderings: the true ring ranks 1st of 2520 for the ring student only.
Figure 13: Heatmaps: transition matrix T , centroid cosine similarities warmest on the neighbor diagonals, whitened probe directions most anti-correlated there, and similarity vs. ring distance.