Recent work has hypothesised that large language models may represent some concepts as multi-dimensional manifolds instead of single directions. Using these manifolds to interpret or steer representations often assumes that moving along them corresponds to a continuous change in the represented concept. In this work, we test this continuous manifold assumption on the prototypical weekday manifold. Despite its striking circular geometry, we find that the weekday representations behave more like seven discrete states arranged in a circle than a continuous ring. First, we show that local sensitivity has seven minima aligned with the weekdays, producing a step-like mapping between representation and behaviour under on-manifold steering. Second, we test whether any concepts are represented along the manifold outside the seven discrete states, and find neither time of day, uncertainty, nor natural text populate the intermediate space. Together, these results challenge the continuous interpretation of the weekday manifold and motivate testing whether movement along other representation manifolds correspond to meaningful changes in model representations and behaviour.
Figures & tables
Figure 1: Perturbation sensitivity around the weekday ring has seven weekday-centred minima. (A) Early-layer steering produces step-like changes in behaviour, with little change near weekdays and rapid change along the intervening arcs. We steer at constant angular speed around the fitted ring and measure cumulative distance in behaviour space. Steering at layer 2 (blue) produces a step-like response whereas steering at layer 28 (red) is smoother. (B) The difference in downstream response speed between weekday positions and the intervening arcs becomes more pronounced over layers. We steer around the ring at layer 2 and measure the speed of the residual-stream response at layers 3–31. (C) The sensitivity minima extend radially beyond the fitted ring, forming seven weekday-centred spokes separated by regions of higher sensitivity. At each point in the surrounding activation space, we measure how much a small change to the layer-2 activation changes the layer-28 residual stream, using the Frobenius norm of the corresponding Jacobian.
Figure 2: Time-of-day variation along the weekday ring is small and not causal (A) Clock-time variation lies primarily along PC1, while weekday identity is captured mainly by PC2 and PC3. We show the top three principal components of weekday-token activations for prompts of the form “It was {time} on {day}.” (B) Time-of-day modifiers produce small (but systematic) angular shifts within the weekday plane, comparable in magnitude to those induced by non-temporal controls. Shifts are measured relative to each weekday centroid in the PC2–PC3 plane. Points show the mean across seven weekdays ± 1 s.e.m. (C) Steering along the ring has almost no effect on the predicted hour, whereas steering along the off-ring component of the observed “late–early” displacement shifts predictions by one to two hours. The horizontal axis specifies the fraction of a weekday interval traversed along the ring. At each steering amount and layer, the off-ring late–early intervention (brown) and random orthogonal control (green) are matched in ℓ2 norm to the along-ring intervention (black). Curves show mean changes in predicted hour ± 95% confidence intervals.
Figure 3: Uncertainty varies primarily along weekday spokes rather than between them. We vary the relative confidence assigned to two weekdays using frequency-adverb prompts, from “Always on {A}, never on {B}” through “Sometimes on {A}, sometimes on {B}” to “Never on {A}, always on {B}”. Colours indicate the five confidence levels. (A) Activations remain concentrated near the two candidate weekday spokes rather than filling the intervening arc. We show weekday pairs separated by one, two and three days. Small points represent individual activations; large points show the mean for each confidence level, computed separately for the two day-aligned groups. Concentric circles indicate normalised radius, with r=1 corresponding to the fitted ring. (B) Activations move inwards as the stated confidence becomes more balanced, then outwards as the prompt increasingly favours the other weekday. The horizontal axis runs from full confidence in day B ( −1 ), through equal confidence ( 0 ), to full confidence in day A ( +1 ). Points show the mean normalised radius ± 1 SD.
Figure 4: Natural context primarily translates weekday representations rather than moving them around the ring. (A) Representations of the same weekday are widely dispersed across natural contexts. We project final-token activations from 16-token FineWeb excerpts into the weekday plane, coloured by weekday. Grey points show excerpts ending in other capitalised words; large markers denote weekday centroids, and the curve shows the fitted ring. (B) Different contexts primarily translate (and scale) the seven weekday representations while largely preserving their relative arrangement and orientation. For each excerpt, we construct seven variants differing only in the final weekday. Dashed curves show the resulting weekday configurations for four example contexts (footnote 2 ); the red curve shows the mean across contexts. (C) Mean subtraction reveals that much of the apparent spread in (A) reflects context-dependent translation rather than movement into the arcs between weekdays. After subtracting the mean activation of the seven matched variants for each context, the representations form seven distinct weekday-aligned clusters. No rotational alignment is applied, indicating that the relative orientation of the weekday representations is largely preserved across contexts.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Schematic of the steering construction and sensitivity analysis.
Figure 6: The Jacobian of the readout on the weekday disc p(u,r)=μ+r(σ(u)−μ) , in three models. The number under each disc is how much more sensitive the readout is between the days than at them, on the ring itself ( r=1 ), with a 95% bootstrap interval over background prompts.
Figure 7: Perturbation sensitivity of Llama-3.1-8B on arithmetic prompts from Wurgaft et al., 2026 . (A) Day centroids at every layer for the weekday token (blue, index 9) and the answer slot (red, index 12), projected to three dimensions under one shared camera, ending in the behaviour manifold My (Appendix A.5 ). (B) Fraction of steers producing the correct answer, by steered layer, for each site; the shaded band is chance. (C) On-manifold distance in activation space against on-manifold distance in behaviour space, after Wurgaft et al. (2026) , for the weekday token at L2 (left) and the answer slot at L28 (right). Dotted lines mark the day knots. (D) Behaviour gain dMy/dMh along the ring, relative to each curve’s own ring average; only layers clearing 0.50 steering accuracy in (B) are drawn. (E–H) The Jacobian of the layer-28 readout with respect to the steered day token, measured as in Appendix A.6 on the disc of Eq. 2 : Frobenius norm, then gain along the ring tangent, the radius, and a random direction orthogonal to the ring’s span.
Figure 8: The sevenfold sensitivity pattern is not driven by weekday-related logits. Llama-3.1-8B, mention family, steering around the ring at layer 2 and reading the output logits. (A) Mean absolute logit change per steering step for the whole vocabulary (black, left axis) and for the 14 weekday tokens (red, right axis); the shaded band is the 95% range over random 14-token subsets. Both track the same sevenfold profile. (B) KL divergence per step of the output distribution with the weekday tokens included (black) and with their logits set to −∞ before the softmax (red). Masking reduces the overall signal but leaves the seven peaks. (C) For every vocabulary token, the share of its logit-change spectrum at the sevenfold frequency against its total movement. Weekday tokens (red) move most but have near-median periodicity; the most periodic tokens (blue) are non-weekday tokens. (D) Per-step logit change for each day-name token: each peaks at its own day (dashed line) rather than showing sevenfold periodicity. (E) The seven most periodic tokens in the vocabulary, none of which is weekday-related, peak at every day.
Figure 9: Uncertain weekday prompts are encoded along weekday spokes across five prompt families. Each row is a prompt family expressing graded preference between two candidate weekdays, with the wording of each level listed on the left; columns show pairs one, two, or three days apart. Final-token activations are projected into the weekday plane and coloured from favouring day A (blue) through even (green) to favouring day B (brown); large markers show the mean at each level after separating the two modes of the cloud. (A) Either/or wording. (B) Certainty words. (C) Frequency words. (D) Stated odds. (E) Stated percentages.
Figure 10: The weekday ring is sharpest at the weekday token and decays at later read positions. Llama-3.1-8B, FineWeb excerpts. (A) Held-out seven-way day decoding accuracy at every layer and read position, with the weekday subspace fitted separately at each position; dots mark cells above chance decoding. (B) The same, with the subspace fitted at the weekday token and transferred to every other position. (C) Layer-28 activations projected into the weekday-token ring plane at each read position, on a shared scale; the held-out rate of projecting closest to the activation’s own weekday centroid is given below each panel. (D) The same panels, each zoomed to its own ring, with the ring radius relative to the weekday token given below.
The internal representations learned by language models consistently exhibit striking geometric structure: calendar months organize into a circle, historical years form a smooth one-dimensional manifold, and cities' latitudes and longitudes can be decoded using a linear probe. To explain this neural code, we first show that language statistics exhibit translation symmetry (for example, the frequency with which any two months co-occur in text depends only on the time interval between them). We prove that this symmetry governs these geometric structures in high-dimensional word embedding models, and we analytically derive the manifold geometry of word representations. These predictions empirically match large text embedding models and large language models. Moreover, the representational geometry persists at moderate embedding dimension even when the relevant statistics are perturbed (e.g., by removing all sentences in which two months co-occur). We prove that this robustness emerges naturally when the co-occurrence statistics are controlled by an underlying latent variable. Our results indicate that these representational manifolds originate in the statistical symmetries of natural language.
Dhruva Karkada, Daniel J. Korchinski, Andres Nava +2
How concepts are represented in neural networks is a fundamental question in machine learning. The dominant view treats concept representations as stationary geometric objects. Yet concepts appear in context, and context transforms them. Drawing from neural population geometry, we formalize concept representations as point-cloud manifolds and contextual transformations as vector fields, and instantiate this framework in large language models. Across six model families of varying scales, we find that context moves each concept differently. The variance in these displacements is semantically organized, correlating with lexical concreteness and density. Importantly, both the concepts being transformed and this variance structure are shared across models: displacement structure transported from one model predicts held-out displacements in others significantly above chance. Together, these findings show that models share a common geometry not only in how concepts are represented, but more importantly in how context transforms them, a structure with richer organization than prior work has recognized.
Zhimin Hu, Lanhao Niu, Sashank Varma
1Georgia Institute of Technology · University of Edinburgh
Large language models place structured concepts on geometrically faithful manifolds: weekdays lie on a circle, months on another, usually taken to be a fixed world-model the network stores and looks up. We show that context is king: the structure a model actually uses is set by the in-context specification. A declarative rule fixes not only which relations the geometry encodes but its topology type: the same tokens form a cycle or a branching tree on command, built even on arbitrary, meaning-free tokens with no prior to inherit, which a relabeled stored shape cannot do. When the specification conflicts with a strong pretrained prior, the context-set geometry dominates it in capable models, read from the same activations (representational similarity 0.6--0.9 to the imposed structure versus near-zero to the prior), across the priors we test and both families we study (Gemma, Qwen). Activation patching shows the map is causally used, not a probe correlate: swapping one entity's activation for another's makes the model answer with the other entity's successor under the imposed order. A rough map forms readily, present even in small and base models; what scale gates is using it cleanly: clean dominance and the causal crossover emerge only in the larger models (up to Gemma-31B and Qwen-27B) and weaken or reverse below, so a mechanism present in a large model can be absent in a smaller one of the same family. Whether the model builds this geometry anew or reconfigures a stored one we leave open; operationally, the geometry it uses is the one the context specifies.