When Models Don't Manipulate Manifolds: The Geometry of a Comparison Task

Sai Sumedh R. Hindupur; Hadas Orgad; Thomas Fel; Demba Ba

cs.LGSep 29, 2026

When Models Don't Manipulate Manifolds: The Geometry of a Comparison Task

Authors: Sai Sumedh R. Hindupur, Hadas Orgad, Thomas Fel, Demba Ba

Organizations: School of Engineering and Applied Science, Harvard University · Kempner Institute, Harvard University · Goodfire AI

Abstract

One of the current premises of mechanistic interpretability research is that detailed accounts of the geometry of neural network representations can tell us how models perform computations, and how to effectively intervene on them. While low dimensional manifolds have been observed for multiple concepts in the literature (e.g. numbers encoded on helices, days of the week on a circle, ...), with structure believed to reflect properties of data and tasks, the extent to which models rely on them for computation, and how they manipulate them, remains unclear. We characterize precisely the geometry of computation in a number-comparison task, as an abstraction of comparison for decision making, and how models utilize geometry in an elegant fashion to implement it. Specifically, we study the causal geometry of number comparison in Qwen2.5-7B-Instruct, a capable and widely studied open-weight model, and find Qwen largely uses linear representations of numbers despite the presence of curved geometry. To compare two numbers, the model first encodes each number along a vector and adds the two representations using attention and the residual connection, bringing them into a shared space in the residual stream. Then, the model uses MLP neurons to compare the pair of numbers on local regions in this shared space, which correspond to smaller intervals of input numbers, and combines these to obtain the position of the maximum. In fact, this reliance on linear representations for comparison also persists when the model compares three numbers. Our findings demonstrate that the manifold hypothesis can co-exist with linear representations: while concepts that are ordered may have manifold structure in representations, the model may use an underlying linear structure of the concept in certain computations.

Figures & tables

Figure 1: Linear number representations are used by language model Qwen for comparison . When asked to compare numbers, the model first represents them on a nonlinear manifold, but uses structure along a single linear direction in computation. It combines information about the two numbers by adding the corresponding linear representations using a single attention head and a residual connection, creating a shared space which enables comparison. On this shared space, the model identifies which number is larger using MLP neurons across two layers, and uses the answer position to read out the answer to the max task.

Figure 2: A single causal direction for numbers controls model comparison. (a) A causal direction ${\mathbf{u}}$ found in Layer 13’s residual stream causally affects model behavior, despite the presence of curved geometry in the activations, as observed in principal components 1, 3. (b, c) Projection of activations onto the obtained causal directions ${\mathbf{u}},\bm{v}_{1},\bm{v}_{2}$ encode the number magnitude for a wide range of values. While ${\mathbf{u}}$ is in layer 13 residual stream at the first number $y_{1}$ ’s position, $\bm{v}_{1},\bm{v}_{2}$ are causal directions encoding $y_{1},y_{2}$ resp. at $y_{2}$ ’s position. $\bm{v}_{1}$ is obtained using DAS at an attention head H14’s outputs. (d) Attention head 14 in layer 14 attends to the position of the first number $y_{1}$ , irrespective of the number value, serving as a copy head. (e) Among all the attention heads in Layer 14, head H14 is causally involved in the model’s computation, showing significantly higher position recovery than others. (f) Patching along ${\mathbf{u}},\bm{v}_{1},\bm{v}_{2}$ has significant causal effects on model behavior, as shown by interchange intervention accuracy (IIA).

Figure 3: The model uses causal number directions

\bm{v}_{1},\bm{v}_{2}

to construct a shared two-dimensional representation encoding the two inputs

y_{1},y_{2}

. (a) – (c) The linear span of

\bm{v}_{1}

and

\bm{v}_{2}

shows each number is encoded along its own direction, and the answer to comparison is linearly separable in this shared representation space. (d) By copying

y_{1}

from

{\mathbf{u}}

\bm{v}_{1}

, the model reduces the alignment between the two number representations:

|\cos(\bm{v}_{1},\bm{v}_{2})|<<|\cos({\mathbf{u}},\bm{v}_{2})|

. (e) While the residual stream at layer 13, denoted

\bm{z}_{l=13}^{2}

, encodes

y_{2}

\bm{v}_{1}

brings in causally useful information into the

y_{2}

position. The shared representation is additive: Patching

\bm{v}_{1}\&\bm{v}_{2}

together (two rank-one patches) nearly matches the IIA of the rank-two patch onto the

\bm{v}_{1},\bm{v}_{2}

plane (i.e.,

\text{span}(\bm{v}_{1},\bm{v}_{2})

). The plane itself captures as much information as the next layer 14 residual stream. (f) Moving around the

\bm{v}_{1},\bm{v}_{2}

plane, which is a two-dimensional plane in the 3,584-dimensional residual stream, is sufficient to change model behavior predictably.

Figure 4: The model compares numbers $y_{1},y_{2}$ by combining local comparisons on the shared $\bm{v}_{1},\bm{v}_{2}$ plane . (a) Individual neuron weights in layer 14 MLP are specific directions in the $\bm{v}_{1},\bm{v}_{2}$ plane. The y-axis is $\bm{v}_{2}^{\perp}$ , the component of $\bm{v}_{2}$ orthogonal to $\bm{v}_{1}$ (since $\bm{v}_{1},\bm{v}_{2}$ are not exactly orthogonal) (b) Neurons in layer 14 MLP (ranked by attribution scores) localize specific regions of the inputs $y_{1},y_{2}$ (like neuron $\#6150$ ), or perform comparisons in localized regions (neuron $\#9459$ ). (c) Select neurons in layer 15 MLP, which combine the outputs of layer 14 MLP neurons, are global comparators: they respond positively when $y_{2}>y_{1}$ and negatively otherwise. (d) A single direction in the layer 15 residual stream (after layer 15 MLP), which is formed by inputs from global comparator neurons from layer 15 MLP, encodes the position of the answer $\argmax(y_{1},y_{2})$ . (e) There are 12 comparator neurons (6 each in layer 14, 15 MLPs) which perform comparison: freezing these neurons significantly degrades the IIA achieved by patching in the $(v_{1},v_{2})$ plane . The one-dimensional comparison direction in layer 15 (panel (d)) controls model behavior.

Figure 5: The model continues to use linear number representations for three-number comparisons. (a) We observe that the causal directions $\bm{v}_{1},\bm{v}_{2},\bm{v}_{3}$ (obtained at the third number $y_{3}$ ’s position) encode the numbers $y_{1},y_{2},y_{3}$ , through nearly monotonic components $f_{i}(y_{i})$ . (b) The three directions $\bm{v}_{1},\bm{v}_{2},\bm{v}_{3}$ have causal effects on the model outputs (logits, measured using position recovery, PR). The effect on $y_{2}$ is lower at this position ( $y_{3}$ position). (c) Receptive fields from layer 14, layer 15 MLP neurons show local comparisons now being performed in the $y_{1},y_{2},y_{3}$ space (whose three two-dimensional views are shown). (d) The model represents its answer flag along a single direction in the layer 15 residual stream at both the $y_{2}$ and the $y_{3}$ position. The flag identifies if $y_{t}$ is the largest upto time $t$ . Note that $y_{2}\max$ refers to $y_{2}>y_{1}$ here. (e) The $\argmax$ position is represented in the top two principal components of layer 20 residual stream, at the last token position (after the third number $y_{3}$ , and right before the model answers). (f) . The flags at $y_{2}$ and $y_{3}$ positions, as well as the answer position from (e), are causal and affect model logits.

Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

	quantity	value
populations	evaluation / fitting examples	$400$ / $128$
	clouds (L13 two-number / L14 / three-number)	$2000$ / $3000$ / $1500$
	operand counts $k$ in the accuracy sweeps, prompts per $k$	$2,3,4,5,10,20$ ; $200$
	base random seed	$52$
learned directions	Adam steps, learning rate	$100$ , $0.05$
	minibatch (gradients accumulated)	$16$ – $32$

Appendix

Table 1: Sample sizes and hyperparameters.

$K$	direction	position	site	obtained by
$2$	$\bm{u}$	$y_{1}$	L13 residual	DAS, $y_{1}$ perturbed
$2$	$\bm{v}_{1}$	$y_{2}$	output of L14.H14	DAS, $y_{1}$ perturbed
$2$	$\bm{v}_{2}$	$y_{2}$	L13 residual	PC
$2$	comparison flag	$y_{2}$	L15 residual	DAS, each case
$3$	$\bm{u}_{1}$ , $\bm{u}_{2}$	$y_{1}$ , $y_{2}$	L13 residual	DAS, cases $y_{1}$ , $y_{2}$
$3$	$\bm{v}_{1}$ , $\bm{v}_{2}$	$y_{3}$	outputs of L14.H14, L14.H18	DAS, cases $y_{1}$ , $y_{2}$

Appendix

Table 2: Directions, where they live and how they are obtained. DAS directions are rank one and fitted in the listed case; PC denotes the top principal component(s) of the corresponding clean cloud.

Figure 6: Accuracy on

\max(y_{1},\dots,y_{k})

against the number of operands

k

, under greedy decoding, for

200

prompts of distinct two-digit operands per length. Error bars are Wilson score intervals.

Figure 7: Answers generated by greedy decoding on the

400

held-out counterfactuals, as fractions per category:

r

b

a

, the leading digit of

r

followed by the last digit of

a

, and other. Rows: the corrupted prompt with no patch and with the full layer-13 residual patched at the last token of

y_{2}

(both with

y_{2}

perturbed), and with

\mathbf{u}

patched at the position of

y_{1}

(

y_{1}

perturbed). An answer equal to

r

is counted as

r

intervention	clean $(y_{1},y_{2})$	max	corrupted $(y_{1},y_{2})$	max	patched answer
$\mathbf{u}$ at $y_{1}$	(99, 61)	99	(12, 61)	61	12
$\mathbf{u}$ at $y_{1}$	(70, 55)	70	(33, 55)	55	33
full at $y_{2}$	(61, 99)	99	(61, 12)	61	12
full at $y_{2}$	(56, 97)	97	(56, 27)	56	27
full at $y_{2}$	(55, 70)	70	(55, 33)	55	30
full at $y_{2}$	(71, 89)	89	(71, 52)	71	59

Appendix

Table 3: Examples of generated answers on held-out counterfactuals: the clean and corrupted operands, their maxima, and the answer generated under each intervention.

Figure 8: Position recovery when the clean residual stream leaving one layer (rows) is restored at one token (columns), from the first token of $y_{1}$ to the last token of the prompt, with (a) $y_{1}$ and (b) $y_{2}$ perturbed. Each operand spans two tokens and the tick marks its last one. The panels share a colour scale.

Figure 9: Illustration of the key components implementing algorithm underlying the pairwise number comparisons algorithm Alg. 1

Figure 10: Ridge probes fitted at every layer (rows) and token (columns) on $2{,}000$ clean two-number prompts and scored on a $20\%$ held-out split. (a,b) Held-out $R^{2}$ for $\log y_{1}$ and $\log y_{2}$ . (c) Held-out accuracy of a ridge classifier for $y_{1}>y_{2}$ . The axes are those of Figure 8 .

Figure 11: The causal trace of Figure 8 for three numbers, with (a) $y_{1}$ , (b) $y_{2}$ and (c) $y_{3}$ perturbed, on the $400$ held-out quadruples. The panels share a colour scale.

Figure 12: The layer-13 residual at the position of $y_{1}$ , one point per value of $y_{1}$ . (a) Projection on the first and third principal components, coloured by $y_{1}$ , with the smoothed mean position along $y_{1}$ (line) and the direction of $\mathbf{u}$ ’s projection onto the plane (arrow). (b) Component along $\mathbf{u}$ against $y_{1}$ , with the least-squares fit $p\log y_{1}+q$ (dotted).

Figure 13: The layer-13 residual at the position of

y_{1}

, with

y_{1}

perturbed. (a) IIA and (b) position recovery of a full-rank patch, a rank-one patch of

\mathbf{u}

and a rank-one patch of the top principal component (PC1). (c) Standardized components along

\mathbf{u}

and PC1 against

y_{1}

, as per-value means. (d) Variance explained by each of the first

20

principal components (bars) and by

\mathbf{u}

(dashed line).

Figure 14: The number-representation analysis at layers

1

13

, with

\mathbf{u}

refitted at each layer. (a) IIA at the position of

y_{1}

, with

y_{1}

perturbed, for a full-rank patch,

\mathbf{u}

and that layer’s top principal component. (b) IIA at the position of

y_{2}

, with

y_{2}

perturbed, for a full-rank patch and

\mathbf{v}_{2}

, taken as that layer’s top principal component. (c–e) Component along

\mathbf{u}

against

y_{1}

at layers 3, 7 and 13, as per-value means, with the fit

p\log y_{1}+q

(dotted).

Figure 15: The number representations with three-digit operands. (a) Component along

\mathbf{u}

at the position of

y_{1}

against

y_{1}

, and (b) component along

\mathbf{v}_{2}

at the position of

y_{2}

against

y_{2}

, both averaged in bins of

25

, with the fit

p\log y+q

(dotted). (c) IIA of a full-rank patch,

\mathbf{u}

and PC1 at the position of

y_{1}

with

y_{1}

perturbed, and of a full-rank patch and

\mathbf{v}_{2}

at the position of

y_{2}

with

y_{2}

perturbed.

Figure 16: The layer-13 residual at the position of $y_{2}$ on the $2{,}000$ -prompt cloud, coloured by which number is larger. (a) Component along a rank-one direction fitted with $y_{2}$ perturbed, against $y_{2}$ . (b) Component along $\mathbf{v}_{2}$ , the top principal component, against $y_{2}$ . (c) IIA with $y_{2}$ perturbed for a full-rank patch, the fitted direction (DAS) and $\mathbf{v}_{2}$ .

Figure 17: The output of head L14.H14 at the position of $y_{2}$ . (a) Its top two principal components, coloured by $y_{1}$ , with the smoothed mean position along $y_{1}$ (line) and the direction of $\mathbf{v}_{1}$ ’s projection onto the plane (arrow). (b) The head’s attention from the position of $y_{2}$ to $y_{1}$ , against $y_{1}$ , as per-value means. (c) Component along $\mathbf{v}_{1}$ against $y_{1}$ , as per-value means.

Figure 18: Components of the pre-MLP layer-14 residual at the position of $y_{2}$ along ridge probes for $\log y_{1}$ ( $\mathbf{p}_{1}$ , top row) and $\log y_{2}$ ( $\mathbf{p}_{2}$ , bottom row), against $y_{1}$ (left) and $y_{2}$ (right), on $600$ held-out prompts.

Figure 19: Position recovery at the position of

y_{2}

for the patches of the main-text shared-representation figure, with

y_{1}

y_{2}

perturbed: the full layer-13 residual,

\mathbf{v}_{2}

\mathbf{v}_{1}

\mathbf{v}_{1}

and

\mathbf{v}_{2}

as two separate rank-one patches, the rank-two

(\mathbf{v}_{1},\mathbf{v}_{2})

plane in the pre-MLP layer-14 residual, and the full pre-MLP layer-14 residual. Hatched bars are the two conditions that patch both directions.

Figure 20: (a) IIA of the rank-two

(\mathbf{v}_{1},\mathbf{v}_{2})

plane patch when one sub-block’s output at the position of

y_{2}

is frozen to its corrupted value, with

y_{1}

y_{2}

perturbed. The layer-14 attention freeze is omitted, because the pre-MLP patch is written through that output (Section B.1 ). (b) IIA when only the clean outputs of the listed MLPs are injected at the position of

y_{2}

Figure 21: (a) Attribution scores of the $37{,}888$ neurons of MLP14 and MLP15, with $y_{1}$ perturbed ( $x$ -axis) and $y_{2}$ perturbed ( $y$ -axis), on symmetric-log axes. The $12$ neurons in the top $20$ of both cases are highlighted. (b) IIA of the plane patch when the top $k$ neurons by attribution (solid) or $k$ random neurons (dashed) are frozen. (c) IIA with no neurons frozen, with the $12$ shared neurons frozen, with each case’s own $20$ top-ranked neurons frozen, and with $12$ random neurons frozen.

Figure 22: Receptive fields of the $12$ shared neurons, six in MLP14 (top) and six in MLP15 (bottom), ordered by worst-case attribution rank. Each field is the post-SwiGLU activation at the position of $y_{2}$ over a grid of $(y_{1},y_{2})$ prompts with stride $4$ , divided by its peak. The dashed line is $y_{1}=y_{2}$ .

Figure 23: Receptive fields, drawn as in Figure 22 , of the six highest-ranked neurons that are in the top $20$ of one perturbation case only: $y_{1}$ perturbed (top) and $y_{2}$ perturbed (bottom).

Figure 24: (a,b) Causal edges from each shared MLP14 neuron (rows) to each shared MLP15 neuron (columns): the shift in the MLP15 neuron’s activation, in units of its standard deviation, when only the MLP14 neuron is set to its clean value, with (a) $y_{1}$ or (b) $y_{2}$ perturbed. (c) Virtual weights through the gate projection, computed as the cosine between each MLP14 neuron’s write direction and each MLP15 neuron’s normalization-folded gate read direction (Section B.6 ).

Figure 25: (a) IIA of a rank-one direction in the layer-15 residual at the position of

y_{2}

, fitted in the case it is evaluated on (fitted, same case) or in the other case (fitted, other case), and of the full-rank layer-15 patch, for each perturbation case. (b,c) Histograms of the alignment

\rho_{j}

of the write direction of every MLP15 neuron with the direction fitted with (b)

y_{1}

or (c)

y_{2}

perturbed, on a log count axis. Vertical lines mark the six shared MLP15 neurons.

Figure 26: The three-number analysis at the position of $y_{3}$ (top row), the position of $y_{2}$ (middle row) and the last token (bottom row). (a) Cosine similarities between $\mathbf{u}_{1}$ , $\mathbf{u}_{2}$ , $\mathbf{v}_{1}$ , $\mathbf{v}_{2}$ and $\mathbf{v}_{3}$ ; the pairs among $\mathbf{v}_{1},\mathbf{v}_{2},\mathbf{v}_{3}$ are boxed. (b) Position recovery at the position of $y_{3}$ , by perturbed number, for each direction alone, the three as separate rank-one patches, their span in the pre-MLP layer-14 residual, and the full layer-14 residual. (c) Receptive fields of L14#5076 (top) and L15#12784 (bottom) at the position of $y_{3}$ over a $(y_{1},y_{2},y_{3})$ grid with stride $4$ , shown as the three pairwise marginals, each averaged over the third number and divided by its own peak. (d) The $1{,}500$ -prompt cloud projected on a rank-one direction in the layer-15 residual fitted with $y_{3}$ perturbed, split by whether $y_{3}$ is the maximum; the inset shows the position recovery of this direction and of the full layer-15 residual. (e–h) The same at the position of $y_{2}$ , where $\mathbf{w}_{1}$ and $\mathbf{w}_{2}$ are rank-one directions in the layer-13 residual fitted with $y_{1}$ and $y_{2}$ perturbed ( $\mathbf{w}_{2}$ and $\mathbf{u}_{2}$ are the same fit). The fields in (g) are over $(y_{1},y_{2})$ with $y_{3}=55$ . (i) Top two principal components of the residual leaving layer 20 at the last token, coloured by which operand holds the maximum. (j) Attention of head L20.H27 from the last token to the question tokens, averaged by the position of the maximum. (k) Position recovery of each layer-20 head’s own output at the last token, for the order $y_{1}>y_{2}>y_{3}$ . (l) Position recovery of a full-rank patch and of a patch of the top two principal components of the layer-20 residual at the last token, for three value orders; the principal components are fitted on a separate $1{,}500$ -prompt cloud.

Figure 27: Component along each direction of the three-number analysis against the number it carries, as per-value means on the

1{,}500

-prompt cloud. (a,b)

\mathbf{u}_{1}

and

\mathbf{u}_{2}

in the layer-13 residual at the positions of

y_{1}

and

y_{2}

. (c,d)

\mathbf{v}_{1}

and

\mathbf{v}_{2}

in the outputs of heads L14.H14 and L14.H18 at the position of

y_{3}

. (e)

\mathbf{v}_{3}

in the layer-13 residual at the position of

y_{3}

Figure 28: The spaces at the position of $y_{3}$ that contain the three-number directions, each in its top two principal components, coloured by the number its direction carries, with the smoothed mean position along that number (line) and the direction of the corresponding projection onto the plane (arrow). (a) The output of head L14.H14 with $\mathbf{v}_{1}$ . (b) The output of head L14.H18 with $\mathbf{v}_{2}$ . (c) The layer-13 residual with $\mathbf{v}_{3}$ .

Figure 29: Position recovery of a rank-one patch of each direction (rows) in each perturbation case (columns). Colours are clipped at $1$ ; printed values are not.

Figure 30: Position recovery of each layer-14 head’s own output, interchanged at the runner-up’s position together with the full layer-13 residual there, for three value orders and the eight heads with the largest mean absolute effect. Dashed lines show the layer-13 patch alone for each order.

Figure 31: Position recovery of the patch of the span of

\mathbf{v}_{1},\mathbf{v}_{2},\mathbf{v}_{3}

in the pre-MLP layer-14 residual at the position of

y_{3}

, by perturbed number, while parts of the network are frozen. (a) One sub-block’s output at the position of

y_{3}

frozen to its corrupted value. (b) The top

k

neurons by this task’s attribution ranking (solid) or

k

random neurons (dashed) frozen. (c) No neurons frozen, the

12

shared neurons of the two-number task frozen, and

12

random neurons frozen.

Figure 32: Position recovery of a rank-one direction in the layer-15 residual fitted in one case (rows) and evaluated in each case (columns), with the full-rank layer-15 patch in the last row, (a) at the position of $y_{3}$ and (b) at the position of $y_{2}$ . Colours are clipped at $2$ ; printed values are not.

Figure 33: Top two principal components of the residual stream leaving layers 18 to 22 at the last token, on $1{,}500$ clean three-number prompts, coloured by which operand holds the maximum.

Figure 34: (a) Position recovery of a full-rank patch of the residual leaving each of layers 18 to 22 at the last token, for three value orders. (b) Position recovery of each layer-20 head’s own output at the last token, for the eight heads with the largest effect and the same three orders. Dashed lines show a patch of the whole layer-20 attention sublayer for each order (Section B.4 ).

Figure 35: Accuracy on

\max(y_{1},\dots,y_{k})

for

200

two-digit prompts per length, scored on the full decoded number (Section B.2 ): the intact model, the

12

shared neurons zeroed at every token, the same neurons zeroed only at the last token of the final operand (“last token” in the legend), and

12

random neurons zeroed at every token (two random sets, pooled).

Figure 36: IIA of the number-representation patches, scored on (a) the first generated token, the leading digit of

r

(Section B.2 ), and (b) the whole generated number: the answer decoded greedily for one token more than the operands’ digit count, counted correct when its first integer equals

r

. With

y_{1}

perturbed: the full layer-13 residual and

\mathbf{u}

at the position of

y_{1}

, and

\mathbf{v}_{1}

at the position of

y_{2}

, patched inside the output of L14.H14 as in the main text or in the pre-MLP layer-14 residual (hatched). With

y_{2}

perturbed: the full layer-13 residual and

\mathbf{v}_{2}

at the position of

y_{2}

. The rank-one patches score the same on both, while the full-rank patches mostly generate a number other than

r

that starts with its leading digit.

Figure 37: IIA of the patches of the main-text shared-representation figure at the position of

y_{2}

, with

y_{1}

y_{2}

perturbed, scored on (a) the first generated token and (b) the whole generated number, as in Figure 36 : the full layer-13 residual,

\mathbf{v}_{2}

\mathbf{v}_{1}

\mathbf{v}_{1}

and

\mathbf{v}_{2}

as two separate rank-one patches, the rank-two

(\mathbf{v}_{1},\mathbf{v}_{2})

plane in the pre-MLP layer-14 residual, and the full residual leaving layer 14. Hatched bars are the two conditions that patch both directions. Only the full-rank patches with

y_{2}

perturbed lose IIA on the whole number.

Figure 38: IIA of the rank-one direction in the layer-15 residual at the position of

y_{2}

, fitted with

y_{1}

perturbed and evaluated in both cases, scored on the first generated token and on the whole generated number, as in Figure 36 .

	$y_{1}$ perturbed		$y_{2}$ perturbed
patch	first digit	whole number	first digit	whole number
L13 full at $y_{1}$	0.072	0.003	–	–
$\mathbf{u}$	0.940	0.940	–	–
$\mathbf{v}_{1}$	0.460	0.460	0.000	0.000
$\mathbf{v}_{1}$ , pre-MLP	0.710	0.710	–	–
L13 full at $y_{2}$	0.000	0.000	0.980	0.115

Appendix

Table 4: IIA on the first generated token and on the whole generated number for the patches of Figures 36 – 38 . Dashes mark cases in which a patch is not evaluated.

patch	perturbed	clean $(y_{1},y_{2})$	corrupted $(y_{1},y_{2})$	$r$	generated
L13 full at $y_{1}$	$y_{1}$	(93, 41)	(13, 41)	13	13
	$y_{1}$	(96, 46)	(20, 46)	20	26
	$y_{1}$	(69, 42)	(13, 42)	13	12
	$y_{1}$	(60, 38)	(15, 38)	15	12
	$y_{1}$	(99, 61)	(12, 61)	12	61
$\mathbf{u}$	$y_{1}$	(99, 61)	(12, 61)	12	12

Appendix

Table 5: Generated answers under the patches of Figure 36 , five per patch, chosen to include correct answers and mistakes: the perturbed number, the clean and corrupted operands,

r

, and the generated number. A patch succeeds when the model generates

r

patch	perturbed	clean $(y_{1},y_{2})$	corrupted $(y_{1},y_{2})$	$r$	generated
$\mathbf{v}_{1}$ & $\mathbf{v}_{2}$	$y_{1}$	(99, 61)	(12, 61)	12	12
	$y_{2}$	(61, 99)	(61, 12)	12	12
	$y_{2}$	(44, 90)	(44, 11)	11	111
	$y_{2}$	(45, 79)	(45, 11)	11	111
	$y_{2}$	(55, 70)	(55, 33)	33	55
$(\mathbf{v}_{1},\mathbf{v}_{2})$ plane	$y_{1}$	(99, 61)	(12, 61)	12	12

Appendix

Table 6: Generated answers, as in Table 5 , under the remaining patches of Figures 37 and 38 .

Explore similar work

Jul 5, 2026cs.LG

Geometry of Ordinal Representations in Language Models

Recent work showed that language models represent character counts on curved 1D manifolds, with attention heads performing geometric transformations to enable computation. We test whether this generalizes across four ordinal tasks (bracket depth, indentation, table position, numeric magnitude) in Gemma-2-2B, Gemma-2-9B, and Qwen3-4B. We find that 1D manifolds with place-cell feature tiling emerge for tasks where the ordinal variable is locally computable from token identity, while tasks requiring cross-position integration or semantic extraction produce higher-dimensional or incoherent representations. Geometric computation is architecture-dependent: Qwen3-4B shows substantially stronger twisting than Gemma models for indentation, and its twisters preserve ordinal order, unlike its numeric twisters. Activation patching confirms that the identified manifold subspaces concentrate task-relevant information, with manifold-direction ablation causing dramatically larger probe accuracy drops than random-direction controls.

Saksham Bassi, Sharvi Tomar

*Equal contribution

Virtual Cell Counting

Ask PDF

May 6, 2026cs.LG

Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior

Neural representations carry rich geometric structure; but does that structure causally shape behavior? To address this question, we intervene along paths through activation space defined by different geometries, and measure the behavioral trajectories they induce. In particular, we test whether interventions that respect the geometry of activation space will yield behaviors close to those the model exhibits naturally. Concretely, we first fit an activation manifold

M_h

to representations and a behavior manifold

M_y

to output probability distributions. We then test the link

M_h \leftrightarrow M_y

via interventions: we find that steering along

M_h

, which we term manifold steering, yields behavioral trajectories that follow

M_y

, while linear steering -- which assumes a Euclidean geometry -- cuts through off-manifold regions and hence produces unnatural outputs. Moreover, optimizing interventions in activation space to produce paths along

M_y

recovers activation trajectories that trace the curvature of

M_h

. We demonstrate this bidirectional relationship between the geometry of representation and behavior across tasks and modalities. In language models, we use reasoning tasks with cyclic and sequential geometries as well as in-context learning tasks with more complex graph geometries. In a video world model, we use a task with geometry corresponding to physical dynamics. Overall, our work shows that geometry in neural representation is not merely incidental, but is in fact the proper object for enabling principled control via intervention on internals. This recasts the core problem of steering from finding the right direction to finding the right geometry.

Daniel Wurgaft, Can Rager, Matthew Kowal +13

Stanford University · University College London · Northeastern University +2

Neural Network Representations Linear Activation Steering Neural Representations Steering Geometric Constraints Agent Behavior Modeling

Ask PDF

Jun 22, 2026cs.AI

Abstract representational geometry supports inference in large language models

A defining feature of human intelligence is the ability to adapt to changing environments by inferring latent task structure from sparse observations. Neuroscientific research indicates that this capability relies on the hippocampus constructing abstract representations, expressed as low-dimensional, approximately orthogonal manifolds in neural state space. However, the internal mechanisms of large language models (LLMs) remain largely opaque, making it unclear whether they form comparable abstract representations or instead rely on task-specific statistical regularities when performing comparable reasoning tasks. Here we adapt a contextual reversal-learning paradigm to a text-based setting and compare humans and LLMs at both the Behavioural and representational levels. We report that although LLMs exhibit generalizable reasoning less frequently than humans, when such inference occurs, their internal states exhibit abstract geometric structures that resemble those reported in the hippocampus. Notably, this representational geometry is not uniformly distributed but is organized hierarchically across model depth: whereas lower layers show early, stable encoding of stimulus identity, higher layers form a hippocampal-like functional band enriched for abstract context geometry associated with inference. Furthermore, complementary intervention experiments mechanistically implicate geometry in reasoning: task-sequence language modelling induces geometric disentanglement, whereas geometric regularization of higher layers increases the emergence of generalizable inference. Together, these findings establish abstract representational geometry as a mechanistic principle supporting inference in large language models.

Yunan Zeng, Yuwang Wang

College of Future Information Technology, Fudan University, Shanghai, 200433, China · Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, 100084, China

LLM Reasoning Strategies In-Context Learning Human Visual Cortex Neuroscience

Ask PDF