Spokes, not a Ring: Revisiting the Weekday Manifold in Language Models

Diksha Gupta; Rieke Fruengel; Stefan Heimersheim

cs.LGOct 27, 2026

Spokes, not a Ring: Revisiting the Weekday Manifold in Language Models

Authors: Diksha Gupta, Rieke Fruengel, Stefan Heimersheim

Organizations: Pivotal Research

Abstract

Recent work has hypothesised that large language models may represent some concepts as multi-dimensional manifolds instead of single directions. Using these manifolds to interpret or steer representations often assumes that moving along them corresponds to a continuous change in the represented concept. In this work, we test this continuous manifold assumption on the prototypical weekday manifold. Despite its striking circular geometry, we find that the weekday representations behave more like seven discrete states arranged in a circle than a continuous ring. First, we show that local sensitivity has seven minima aligned with the weekdays, producing a step-like mapping between representation and behaviour under on-manifold steering. Second, we test whether any concepts are represented along the manifold outside the seven discrete states, and find neither time of day, uncertainty, nor natural text populate the intermediate space. Together, these results challenge the continuous interpretation of the weekday manifold and motivate testing whether movement along other representation manifolds correspond to meaningful changes in model representations and behaviour.

Figures & tables

Figure 1: Perturbation sensitivity around the weekday ring has seven weekday-centred minima. (A) Early-layer steering produces step-like changes in behaviour, with little change near weekdays and rapid change along the intervening arcs. We steer at constant angular speed around the fitted ring and measure cumulative distance in behaviour space. Steering at layer 2 (blue) produces a step-like response whereas steering at layer 28 (red) is smoother. (B) The difference in downstream response speed between weekday positions and the intervening arcs becomes more pronounced over layers. We steer around the ring at layer 2 and measure the speed of the residual-stream response at layers 3–31. (C) The sensitivity minima extend radially beyond the fitted ring, forming seven weekday-centred spokes separated by regions of higher sensitivity. At each point in the surrounding activation space, we measure how much a small change to the layer-2 activation changes the layer-28 residual stream, using the Frobenius norm of the corresponding Jacobian.

Figure 2: Time-of-day variation along the weekday ring is small and not causal (A) Clock-time variation lies primarily along PC1, while weekday identity is captured mainly by PC2 and PC3. We show the top three principal components of weekday-token activations for prompts of the form “It was {time} on {day}.” (B) Time-of-day modifiers produce small (but systematic) angular shifts within the weekday plane, comparable in magnitude to those induced by non-temporal controls. Shifts are measured relative to each weekday centroid in the PC2–PC3 plane. Points show the mean across seven weekdays $\pm$ 1 s.e.m. (C) Steering along the ring has almost no effect on the predicted hour, whereas steering along the off-ring component of the observed “late–early” displacement shifts predictions by one to two hours. The horizontal axis specifies the fraction of a weekday interval traversed along the ring. At each steering amount and layer, the off-ring late–early intervention (brown) and random orthogonal control (green) are matched in $\ell_{2}$ norm to the along-ring intervention (black). Curves show mean changes in predicted hour $\pm$ 95% confidence intervals.

Figure 3: Uncertainty varies primarily along weekday spokes rather than between them. We vary the relative confidence assigned to two weekdays using frequency-adverb prompts, from “Always on {A}, never on {B}” through “Sometimes on {A}, sometimes on {B}” to “Never on {A}, always on {B}”. Colours indicate the five confidence levels. (A) Activations remain concentrated near the two candidate weekday spokes rather than filling the intervening arc. We show weekday pairs separated by one, two and three days. Small points represent individual activations; large points show the mean for each confidence level, computed separately for the two day-aligned groups. Concentric circles indicate normalised radius, with $r=1$ corresponding to the fitted ring. (B) Activations move inwards as the stated confidence becomes more balanced, then outwards as the prompt increasingly favours the other weekday. The horizontal axis runs from full confidence in day B ( $-1$ ), through equal confidence ( $0$ ), to full confidence in day A ( $+1$ ). Points show the mean normalised radius $\pm$ 1 SD.

Figure 4: Natural context primarily translates weekday representations rather than moving them around the ring. (A) Representations of the same weekday are widely dispersed across natural contexts. We project final-token activations from 16-token FineWeb excerpts into the weekday plane, coloured by weekday. Grey points show excerpts ending in other capitalised words; large markers denote weekday centroids, and the curve shows the fitted ring. (B) Different contexts primarily translate (and scale) the seven weekday representations while largely preserving their relative arrangement and orientation. For each excerpt, we construct seven variants differing only in the final weekday. Dashed curves show the resulting weekday configurations for four example contexts (footnote 2 ); the red curve shows the mean across contexts. (C) Mean subtraction reveals that much of the apparent spread in (A) reflects context-dependent translation rather than movement into the arcs between weekdays. After subtracting the mean activation of the seven matched variants for each context, the representations form seven distinct weekday-aligned clusters. No rotational alignment is applied, indicating that the relative orientation of the weekday representations is largely preserved across contexts.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Schematic of the steering construction and sensitivity analysis.

Figure 6: The Jacobian of the readout on the weekday disc $p(u,r)=\mu+r(\sigma(u)-\mu)$ , in three models. The number under each disc is how much more sensitive the readout is between the days than at them, on the ring itself ( $r=1$ ), with a 95% bootstrap interval over background prompts.

Figure 7: Perturbation sensitivity of Llama-3.1-8B on arithmetic prompts from Wurgaft et al., 2026 . (A) Day centroids at every layer for the weekday token (blue, index 9) and the answer slot (red, index 12), projected to three dimensions under one shared camera, ending in the behaviour manifold $\mathcal{M}_{y}$ (Appendix A.5 ). (B) Fraction of steers producing the correct answer, by steered layer, for each site; the shaded band is chance. (C) On-manifold distance in activation space against on-manifold distance in behaviour space, after Wurgaft et al. (2026) , for the weekday token at L2 (left) and the answer slot at L28 (right). Dotted lines mark the day knots. (D) Behaviour gain $d\mathcal{M}_{y}/d\mathcal{M}_{h}$ along the ring, relative to each curve’s own ring average; only layers clearing $0.50$ steering accuracy in (B) are drawn. (E–H) The Jacobian of the layer-28 readout with respect to the steered day token, measured as in Appendix A.6 on the disc of Eq. 2 : Frobenius norm, then gain along the ring tangent, the radius, and a random direction orthogonal to the ring’s span.

Figure 8: The sevenfold sensitivity pattern is not driven by weekday-related logits. Llama-3.1-8B, mention family, steering around the ring at layer 2 and reading the output logits. (A) Mean absolute logit change per steering step for the whole vocabulary (black, left axis) and for the 14 weekday tokens (red, right axis); the shaded band is the 95% range over random 14-token subsets. Both track the same sevenfold profile. (B) KL divergence per step of the output distribution with the weekday tokens included (black) and with their logits set to

-\infty

before the softmax (red). Masking reduces the overall signal but leaves the seven peaks. (C) For every vocabulary token, the share of its logit-change spectrum at the sevenfold frequency against its total movement. Weekday tokens (red) move most but have near-median periodicity; the most periodic tokens (blue) are non-weekday tokens. (D) Per-step logit change for each day-name token: each peaks at its own day (dashed line) rather than showing sevenfold periodicity. (E) The seven most periodic tokens in the vocabulary, none of which is weekday-related, peak at every day.

Figure 9: Uncertain weekday prompts are encoded along weekday spokes across five prompt families. Each row is a prompt family expressing graded preference between two candidate weekdays, with the wording of each level listed on the left; columns show pairs one, two, or three days apart. Final-token activations are projected into the weekday plane and coloured from favouring day A (blue) through even (green) to favouring day B (brown); large markers show the mean at each level after separating the two modes of the cloud. (A) Either/or wording. (B) Certainty words. (C) Frequency words. (D) Stated odds. (E) Stated percentages.

Figure 10: The weekday ring is sharpest at the weekday token and decays at later read positions. Llama-3.1-8B, FineWeb excerpts. (A) Held-out seven-way day decoding accuracy at every layer and read position, with the weekday subspace fitted separately at each position; dots mark cells above chance decoding. (B) The same, with the subspace fitted at the weekday token and transferred to every other position. (C) Layer-28 activations projected into the weekday-token ring plane at each read position, on a shared scale; the held-out rate of projecting closest to the activation’s own weekday centroid is given below each panel. (D) The same panels, each zoomed to its own ring, with the ring radius relative to the weekday token given below.

Representation Geometry in Language Models

Representation geometry in language models studies how embeddings and hidden states organize concepts, computation, and behavior.

See all 421 papers

Symmetry in language statistics shapes the geometry of model representations
Feb 16, 2026Dhruva Karkada, Daniel J. Korchinski, Andres Nava +2Representation Learning Neural Representation Geometry
1UC Berkeley · 2EPFL · 3Johns Hopkins +1
Language Models Represent and Transform Concepts with Shared Geometry
Jul 5, 2026Zhimin Hu, Lanhao Niu, Sashank VarmaNeural Representation Geometry Representation Geometry in Language Models
1Georgia Institute of Technology · University of Edinburgh
Context Is King: How In-Context Specification Shapes the Geometry of Concepts
Jul 27, 2026Elad David, Max FominLLM Interpretability In-Context Learning
Zenity

Feb 16, 2026J/K move · Enter open · S save

Symmetry in language statistics shapes the geometry of model representations

Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart+1

1UC Berkeley · 2EPFL · 3Johns Hopkins+1

Representation Learning Neural Representation Geometry Word Embeddings Representation Geometry in Language Models Latent Space Geometry

Open paper AskPDF arXiv

The internal representations learned by language models consistently exhibit striking geometric structure: calendar months organize into a circle, historical years form a smooth one-dimensional manifold, and cities' latitudes and longitudes can be decoded using a linear probe. To explain this neural code, we first show that language statistics exhibit translation symmetry (for example, the frequency with which any two months co-occur in text depends only on the time interval between them). We prove that this symmetry governs these geometric structures in high-dimensional word embedding models, and we analytically derive the manifold geometry of word representations. These predictions empirically match large text embedding models and large language models. Moreover, the representational geometry persists at moderate embedding dimension even when the relevant statistics are perturbed (e.g., by removing all sentences in which two months co-occur). We prove that this robustness emerges naturally when the co-occurrence statistics are controlled by an underlying latent variable. Our results indicate that these representational manifolds originate in the statistical symmetries of natural language.

Abstract

Representation Geometry in Language Models

Explore similar work

Symmetry in language statistics shapes the geometry of model representations

Language Models Represent and Transform Concepts with Shared Geometry

Context Is King: How In-Context Specification Shapes the Geometry of Concepts