LLMs are thought to track "belief states," i.e., running probability distributions over the latent variables that govern language (Shai et al., 2024; Sarfati et al., 2026), but so far this has only been comprehensively demonstrated on toy synthetic data and in a few isolated case studies. It has also never been empirically connected to the geometry of LLM features (the concepts interpretability finds in model activations). In this work, we plant a controllable latent variable inside natural-looking text. An LLM teacher writes ordinary text while we "subliminally" steer it along one of K = 8 unrelated sparse autoencoder directions at each token, with the active directions following a ring-shaped Markov chain. A small transformer model trained on this corpus does indeed track the Bayesian posterior belief about our planted latent variable. Moreover, it also arranges the 8 states themselves on a ring, in the exact order of the Markov chain, which is supporting evidence that a concept's geometry can be formed by the statistical dynamics of the latent variable behind it.
Figures & tables
Figure 1: Figure from Shai et al. (2024) , showing how given a data-generating process, a model learns to model optimal beliefs sitting on the probability simplex at any token.
Figure 2: Belief state geometry (each point a probability vector over the latent variable’s values) versus concept geometry (the arrangement of the values themselves), both read from the residual stream. The two are different objects.
Figure 3: Method summary: (a) we plant a latent variable as a Markov chain over K states; (b) we steer a teacher LLM while it writes, by adding the current state’s SAE decoder direction to the residual stream; (c) the result is natural-looking text, secretly labeled per token.
Figure 4: The 8-state ring Markov chain with stay probability 0.95 and its transition matrix, plus a corpus excerpt with tokens colored by the active state.
Figure 5: PCA of the optimal observer’s posteriors: the point cloud fills a ring-shaped region with the 8 states in ring order at the rim.
Figure 6: The control2 chain: same stay probability, but uniform jumps to any of the other 7 states, plus a control2 corpus excerpt with tokens colored by the active state.
Figure 7: Per-layer probe R2 and argmax accuracy for the ring student, control2 , and control1 , with the optimal observer ceiling at 0.762 .
Figure 8: Per-layer probe R2 and argmax accuracy restricted to first-dwell tokens: the ring student and control2 curves coincide.
Figure 9: Side-by-side PCA of the optimal observer’s posteriors and the student’s probe read-out at layer 12: same ring arrangement, noisier cloud for the student.
Figure 10: Top-2 PCA of the 8 state centroids: the ring student’s centroids trace the ring in exact neighbor order; the control student’s do not.
Figure 11: Top-2 PCA of first-dwell centroids: the clean ring is no longer visible in the leading principal components.
Figure 12: Circle-plane projections of first-dwell centroids for the three students, and the circle-fit permutation test over all 2520 ring orderings: the true ring ranks 1st of 2520 for the ring student only.
Figure 13: Heatmaps: transition matrix T , centroid cosine similarities warmest on the neighbor diagonals, whitened probe directions most anti-correlated there, and similarity vs. ring distance.
Large language models (LLMs) trained on next-token prediction exhibit remarkable in-context learning (ICL) abilities, yet the representations that support ICL remain poorly understood. We consider such representations in a controlled setting: prompting LLMs with data emitted from hidden Markov models (HMMs) and probing for the corresponding belief state -- the posterior distribution over the HMM's hidden states given the observed token history. Across six open-source LLMs prompted with data from 40 HMMs selected for non-trivial belief structure, we find that belief states are linearly decodable from residual stream activations, with peak probe R2-values from 0.83-0.99 across HMM and LLM combinations, ranging from early to late layers. To establish functional relevance, we intervene directly on the probe-identified subspace via patching and steering, resulting in downstream prediction quality on the order of the untampered model, while controls degrade performance substantially. Together, these results provide representation-level evidence that ICL in open-source LLMs approximates optimal Bayesian prediction over a context-inferred generative model. More broadly, our findings extend prior results linking input-distribution structure to activation geometry: from toy networks trained explicitly on HMM data to production-scale LLMs.
Daniel Balcells, Andrew Jun Lee, Chirag Rastogi +3
Large Language Models (LLMs) update their behavior in context, which can be viewed as a form of Bayesian inference. However, the structure of the latent hypothesis space over which this inference operates remains unclear. In this work, we propose that LLMs assign beliefs over a low-dimensional geometric space - a conceptual belief space - and that in-context learning corresponds to a trajectory through this space as beliefs are updated over time. Using story understanding as a natural setting for dynamic belief updating, we combine behavioral and representational analyses to study these trajectories. We find that (1) belief updates are well-described as trajectories on low-dimensional, structured manifolds; (2) this structure is reflected consistently in both model behavior and internal representations and can be decoded with simple linear probes to predict behavior; and (3) interventions on these representations causally steer belief trajectories, with effects that can be predicted from the geometry of the conceptual space. Together, our results provide a geometric account of belief dynamics in LLMs, grounding Bayesian interpretations of in-context learning in structured conceptual representations.
Observing what a large language model computes during autoregressive inference--online and without training probes--remains difficult. We introduce Lingtai, a training-free concept telemetry layer: at each generation step, residual states are projected onto a domain-specific bank of named concept anchors, constructed without labeled concept examples, outcome labels, gradient fitting, or activation-space optimization, producing a structured per-step concept-coordinate signal. Across code generation and grade-school mathematical reasoning, this signal exhibits a robust association with predictive uncertainty: the association survives problem-identity and token-position controls and is not attributable to a single token type, is not explained by a simple correct/incorrect mixture on GSM8K, and is not reproduced by matched random anchors; it is markedly weaker or direction-inconsistent in K-means and PCA projections. Two structures emerge: a recurring uncertainty-linked activity signal whose functional geometry is task-conditioned (distinct activity-entropy shapes on HumanEval, MBPP, and GSM8K), and an execution-specific trajectory identity with strong local inertia but weak re-instantiation invariance--under completion-only elastic alignment, corruption at k=32 (approximately a median quarter of the completion) on the matched re-execution subset still retrieves the archived episode at 62.0%, while a fresh execution retrieves it only 11.7-16.0% of the time. Finally, a matched audit finds no evidence that the scalar concept-activity signal used here supplies a stable correctness coordinate under the tested protocol; we therefore treat correctness as externally supplied. Telemetry adds 0.7-1.6% per-token decode overhead for the 161-anchor code implementation, with unchanged generated tokens.