When Models Don't Manipulate Manifolds: The Geometry of a Comparison Task

Sai Sumedh R. Hindupur; Hadas Orgad; Thomas Fel; Demba Ba

	quantity	value
populations	evaluation / fitting examples	$400$ / $128$
	clouds (L13 two-number / L14 / three-number)	$2000$ / $3000$ / $1500$
	operand counts $k$ in the accuracy sweeps, prompts per $k$	$2,3,4,5,10,20$ ; $200$
	base random seed	$52$
learned directions	Adam steps, learning rate	$100$ , $0.05$
	minibatch (gradients accumulated)	$16$ – $32$

Appendix

Table 1: Sample sizes and hyperparameters.

$K$	direction	position	site	obtained by
$2$	$\bm{u}$	$y_{1}$	L13 residual	DAS, $y_{1}$ perturbed
$2$	$\bm{v}_{1}$	$y_{2}$	output of L14.H14	DAS, $y_{1}$ perturbed
$2$	$\bm{v}_{2}$	$y_{2}$	L13 residual	PC
$2$	comparison flag	$y_{2}$	L15 residual	DAS, each case
$3$	$\bm{u}_{1}$ , $\bm{u}_{2}$	$y_{1}$ , $y_{2}$	L13 residual	DAS, cases $y_{1}$ , $y_{2}$
$3$	$\bm{v}_{1}$ , $\bm{v}_{2}$	$y_{3}$	outputs of L14.H14, L14.H18	DAS, cases $y_{1}$ , $y_{2}$

Appendix

Table 2: Directions, where they live and how they are obtained. DAS directions are rank one and fitted in the listed case; PC denotes the top principal component(s) of the corresponding clean cloud.

Figure 6: Accuracy on

\max(y_{1},\dots,y_{k})

against the number of operands

k

, under greedy decoding, for

200

prompts of distinct two-digit operands per length. Error bars are Wilson score intervals.

Figure 7: Answers generated by greedy decoding on the

400

held-out counterfactuals, as fractions per category:

r

,

b

,

a

, the leading digit of

r

followed by the last digit of

a

, and other. Rows: the corrupted prompt with no patch and with the full layer-13 residual patched at the last token of

y_{2}

(both with

y_{2}

perturbed), and with

\mathbf{u}

patched at the position of

y_{1}

(

y_{1}

perturbed). An answer equal to

r

is counted as

r

.

intervention	clean $(y_{1},y_{2})$	max	corrupted $(y_{1},y_{2})$	max	patched answer
$\mathbf{u}$ at $y_{1}$	(99, 61)	99	(12, 61)	61	12
$\mathbf{u}$ at $y_{1}$	(70, 55)	70	(33, 55)	55	33
full at $y_{2}$	(61, 99)	99	(61, 12)	61	12
full at $y_{2}$	(56, 97)	97	(56, 27)	56	27
full at $y_{2}$	(55, 70)	70	(55, 33)	55	30
full at $y_{2}$	(71, 89)	89	(71, 52)	71	59

Appendix

Table 3: Examples of generated answers on held-out counterfactuals: the clean and corrupted operands, their maxima, and the answer generated under each intervention.

Figure 8: Position recovery when the clean residual stream leaving one layer (rows) is restored at one token (columns), from the first token of

y_{1}

to the last token of the prompt, with (a)

y_{1}

and (b)

y_{2}

perturbed. Each operand spans two tokens and the tick marks its last one. The panels share a colour scale.

Figure 9: Illustration of the key components implementing algorithm underlying the pairwise number comparisons algorithm Alg. 1

Figure 10: Ridge probes fitted at every layer (rows) and token (columns) on

2{,}000

clean two-number prompts and scored on a

20\%

held-out split. (a,b) Held-out

R^{2}

for

\log y_{1}

and

\log y_{2}

. (c) Held-out accuracy of a ridge classifier for

y_{1}>y_{2}

. The axes are those of Figure 8 .

Figure 11: The causal trace of Figure 8 for three numbers, with (a)

y_{1}

, (b)

y_{2}

and (c)

y_{3}

perturbed, on the

400

held-out quadruples. The panels share a colour scale.

Figure 12: The layer-13 residual at the position of

y_{1}

, one point per value of

y_{1}

. (a) Projection on the first and third principal components, coloured by

y_{1}

, with the smoothed mean position along

y_{1}

(line) and the direction of

\mathbf{u}

’s projection onto the plane (arrow). (b) Component along

\mathbf{u}

against

y_{1}

, with the least-squares fit

p\log y_{1}+q

(dotted).

Figure 13: The layer-13 residual at the position of

y_{1}

, with

y_{1}

perturbed. (a) IIA and (b) position recovery of a full-rank patch, a rank-one patch of

\mathbf{u}

and a rank-one patch of the top principal component (PC1). (c) Standardized components along

\mathbf{u}

and PC1 against

y_{1}

, as per-value means. (d) Variance explained by each of the first

20

principal components (bars) and by

\mathbf{u}

(dashed line).

Figure 14: The number-representation analysis at layers

1

to

13

, with

\mathbf{u}

refitted at each layer. (a) IIA at the position of

y_{1}

, with

y_{1}

perturbed, for a full-rank patch,

\mathbf{u}

and that layer’s top principal component. (b) IIA at the position of

y_{2}

, with

y_{2}

perturbed, for a full-rank patch and

\mathbf{v}_{2}

, taken as that layer’s top principal component. (c–e) Component along

\mathbf{u}

against

y_{1}

at layers 3, 7 and 13, as per-value means, with the fit

p\log y_{1}+q

(dotted).

Figure 15: The number representations with three-digit operands. (a) Component along

\mathbf{u}

at the position of

y_{1}

against

y_{1}

, and (b) component along

\mathbf{v}_{2}

at the position of

y_{2}

against

y_{2}

, both averaged in bins of

25

, with the fit

p\log y+q

(dotted). (c) IIA of a full-rank patch,

\mathbf{u}

and PC1 at the position of

y_{1}

with

y_{1}

perturbed, and of a full-rank patch and

\mathbf{v}_{2}

at the position of

y_{2}

with

y_{2}

perturbed.

Figure 16: The layer-13 residual at the position of

y_{2}

on the

2{,}000

-prompt cloud, coloured by which number is larger. (a) Component along a rank-one direction fitted with

y_{2}

perturbed, against

y_{2}

. (b) Component along

\mathbf{v}_{2}

, the top principal component, against

y_{2}

. (c) IIA with

y_{2}

perturbed for a full-rank patch, the fitted direction (DAS) and

\mathbf{v}_{2}

.

Figure 17: The output of head L14.H14 at the position of

y_{2}

. (a) Its top two principal components, coloured by

y_{1}

, with the smoothed mean position along

y_{1}

(line) and the direction of

\mathbf{v}_{1}

’s projection onto the plane (arrow). (b) The head’s attention from the position of

y_{2}

to

y_{1}

, against

y_{1}

, as per-value means. (c) Component along

\mathbf{v}_{1}

against

y_{1}

, as per-value means.

Figure 18: Components of the pre-MLP layer-14 residual at the position of

y_{2}

along ridge probes for

\log y_{1}

(

\mathbf{p}_{1}

, top row) and

\log y_{2}

(

\mathbf{p}_{2}

, bottom row), against

y_{1}

(left) and

y_{2}

(right), on

600

held-out prompts.

Figure 19: Position recovery at the position of

y_{2}

for the patches of the main-text shared-representation figure, with

y_{1}

or

y_{2}

perturbed: the full layer-13 residual,

\mathbf{v}_{2}

,

\mathbf{v}_{1}

,

\mathbf{v}_{1}

and

\mathbf{v}_{2}

as two separate rank-one patches, the rank-two

(\mathbf{v}_{1},\mathbf{v}_{2})

plane in the pre-MLP layer-14 residual, and the full pre-MLP layer-14 residual. Hatched bars are the two conditions that patch both directions.

Figure 20: (a) IIA of the rank-two

(\mathbf{v}_{1},\mathbf{v}_{2})

plane patch when one sub-block’s output at the position of

y_{2}

is frozen to its corrupted value, with

y_{1}

or

y_{2}

perturbed. The layer-14 attention freeze is omitted, because the pre-MLP patch is written through that output (Section B.1 ). (b) IIA when only the clean outputs of the listed MLPs are injected at the position of

y_{2}

.

Figure 21: (a) Attribution scores of the

37{,}888

neurons of MLP14 and MLP15, with

y_{1}

perturbed (

x

-axis) and

y_{2}

perturbed (

y

-axis), on symmetric-log axes. The

12

neurons in the top

20

of both cases are highlighted. (b) IIA of the plane patch when the top

k

neurons by attribution (solid) or

k

random neurons (dashed) are frozen. (c) IIA with no neurons frozen, with the

12

shared neurons frozen, with each case’s own

20

top-ranked neurons frozen, and with

12

random neurons frozen.

Figure 22: Receptive fields of the

12

shared neurons, six in MLP14 (top) and six in MLP15 (bottom), ordered by worst-case attribution rank. Each field is the post-SwiGLU activation at the position of

y_{2}

over a grid of

(y_{1},y_{2})

prompts with stride

4

, divided by its peak. The dashed line is

y_{1}=y_{2}

.

Figure 23: Receptive fields, drawn as in Figure 22 , of the six highest-ranked neurons that are in the top

20

of one perturbation case only:

y_{1}

perturbed (top) and

y_{2}

perturbed (bottom).

Figure 24: (a,b) Causal edges from each shared MLP14 neuron (rows) to each shared MLP15 neuron (columns): the shift in the MLP15 neuron’s activation, in units of its standard deviation, when only the MLP14 neuron is set to its clean value, with (a)

y_{1}

or (b)

y_{2}

perturbed. (c) Virtual weights through the gate projection, computed as the cosine between each MLP14 neuron’s write direction and each MLP15 neuron’s normalization-folded gate read direction (Section B.6 ).

Figure 25: (a) IIA of a rank-one direction in the layer-15 residual at the position of

y_{2}

, fitted in the case it is evaluated on (fitted, same case) or in the other case (fitted, other case), and of the full-rank layer-15 patch, for each perturbation case. (b,c) Histograms of the alignment

\rho_{j}

of the write direction of every MLP15 neuron with the direction fitted with (b)

y_{1}

or (c)

y_{2}

perturbed, on a log count axis. Vertical lines mark the six shared MLP15 neurons.

Figure 26: The three-number analysis at the position of

y_{3}

(top row), the position of

y_{2}

(middle row) and the last token (bottom row). (a) Cosine similarities between

\mathbf{u}_{1}

,

\mathbf{u}_{2}

,

\mathbf{v}_{1}

,

\mathbf{v}_{2}

and

\mathbf{v}_{3}

; the pairs among

\mathbf{v}_{1},\mathbf{v}_{2},\mathbf{v}_{3}

are boxed. (b) Position recovery at the position of

y_{3}

, by perturbed number, for each direction alone, the three as separate rank-one patches, their span in the pre-MLP layer-14 residual, and the full layer-14 residual. (c) Receptive fields of L14#5076 (top) and L15#12784 (bottom) at the position of

y_{3}

over a

(y_{1},y_{2},y_{3})

grid with stride

4

, shown as the three pairwise marginals, each averaged over the third number and divided by its own peak. (d) The

1{,}500

-prompt cloud projected on a rank-one direction in the layer-15 residual fitted with

y_{3}

perturbed, split by whether

y_{3}

is the maximum; the inset shows the position recovery of this direction and of the full layer-15 residual. (e–h) The same at the position of

y_{2}

, where

\mathbf{w}_{1}

and

\mathbf{w}_{2}

are rank-one directions in the layer-13 residual fitted with

y_{1}

and

y_{2}

perturbed (

\mathbf{w}_{2}

and

\mathbf{u}_{2}

are the same fit). The fields in (g) are over

(y_{1},y_{2})

with

y_{3}=55

. (i) Top two principal components of the residual leaving layer 20 at the last token, coloured by which operand holds the maximum. (j) Attention of head L20.H27 from the last token to the question tokens, averaged by the position of the maximum. (k) Position recovery of each layer-20 head’s own output at the last token, for the order

y_{1}>y_{2}>y_{3}

. (l) Position recovery of a full-rank patch and of a patch of the top two principal components of the layer-20 residual at the last token, for three value orders; the principal components are fitted on a separate

1{,}500

-prompt cloud.

Figure 27: Component along each direction of the three-number analysis against the number it carries, as per-value means on the

1{,}500

-prompt cloud. (a,b)

\mathbf{u}_{1}

and

\mathbf{u}_{2}

in the layer-13 residual at the positions of

y_{1}

and

y_{2}

. (c,d)

\mathbf{v}_{1}

and

\mathbf{v}_{2}

in the outputs of heads L14.H14 and L14.H18 at the position of

y_{3}

. (e)

\mathbf{v}_{3}

in the layer-13 residual at the position of

y_{3}

.

Figure 28: The spaces at the position of

y_{3}

that contain the three-number directions, each in its top two principal components, coloured by the number its direction carries, with the smoothed mean position along that number (line) and the direction of the corresponding projection onto the plane (arrow). (a) The output of head L14.H14 with

\mathbf{v}_{1}

. (b) The output of head L14.H18 with

\mathbf{v}_{2}

. (c) The layer-13 residual with

\mathbf{v}_{3}

.

Figure 29: Position recovery of a rank-one patch of each direction (rows) in each perturbation case (columns). Colours are clipped at

1

; printed values are not.

Figure 30: Position recovery of each layer-14 head’s own output, interchanged at the runner-up’s position together with the full layer-13 residual there, for three value orders and the eight heads with the largest mean absolute effect. Dashed lines show the layer-13 patch alone for each order.

Figure 31: Position recovery of the patch of the span of

\mathbf{v}_{1},\mathbf{v}_{2},\mathbf{v}_{3}

in the pre-MLP layer-14 residual at the position of

y_{3}

, by perturbed number, while parts of the network are frozen. (a) One sub-block’s output at the position of

y_{3}

frozen to its corrupted value. (b) The top

k

neurons by this task’s attribution ranking (solid) or

k

random neurons (dashed) frozen. (c) No neurons frozen, the

12

shared neurons of the two-number task frozen, and

12

random neurons frozen.

Figure 32: Position recovery of a rank-one direction in the layer-15 residual fitted in one case (rows) and evaluated in each case (columns), with the full-rank layer-15 patch in the last row, (a) at the position of

y_{3}

and (b) at the position of

y_{2}

. Colours are clipped at

2

; printed values are not.

Figure 33: Top two principal components of the residual stream leaving layers 18 to 22 at the last token, on

1{,}500

clean three-number prompts, coloured by which operand holds the maximum.

Figure 34: (a) Position recovery of a full-rank patch of the residual leaving each of layers 18 to 22 at the last token, for three value orders. (b) Position recovery of each layer-20 head’s own output at the last token, for the eight heads with the largest effect and the same three orders. Dashed lines show a patch of the whole layer-20 attention sublayer for each order (Section B.4 ).

Figure 35: Accuracy on

\max(y_{1},\dots,y_{k})

for

200

two-digit prompts per length, scored on the full decoded number (Section B.2 ): the intact model, the

12

shared neurons zeroed at every token, the same neurons zeroed only at the last token of the final operand (“last token” in the legend), and

12

random neurons zeroed at every token (two random sets, pooled).

Figure 36: IIA of the number-representation patches, scored on (a) the first generated token, the leading digit of

r

(Section B.2 ), and (b) the whole generated number: the answer decoded greedily for one token more than the operands’ digit count, counted correct when its first integer equals

r

. With

y_{1}

perturbed: the full layer-13 residual and

\mathbf{u}

at the position of

y_{1}

, and

\mathbf{v}_{1}

at the position of

y_{2}

, patched inside the output of L14.H14 as in the main text or in the pre-MLP layer-14 residual (hatched). With

y_{2}

perturbed: the full layer-13 residual and

\mathbf{v}_{2}

at the position of

y_{2}

. The rank-one patches score the same on both, while the full-rank patches mostly generate a number other than

r

that starts with its leading digit.

Figure 37: IIA of the patches of the main-text shared-representation figure at the position of

y_{2}

, with

y_{1}

or

y_{2}

perturbed, scored on (a) the first generated token and (b) the whole generated number, as in Figure 36 : the full layer-13 residual,

\mathbf{v}_{2}

,

\mathbf{v}_{1}

,

\mathbf{v}_{1}

and

\mathbf{v}_{2}

as two separate rank-one patches, the rank-two

(\mathbf{v}_{1},\mathbf{v}_{2})

plane in the pre-MLP layer-14 residual, and the full residual leaving layer 14. Hatched bars are the two conditions that patch both directions. Only the full-rank patches with

y_{2}

perturbed lose IIA on the whole number.

Figure 38: IIA of the rank-one direction in the layer-15 residual at the position of

y_{2}

, fitted with

y_{1}

perturbed and evaluated in both cases, scored on the first generated token and on the whole generated number, as in Figure 36 .

	$y_{1}$ perturbed		$y_{2}$ perturbed
patch	first digit	whole number	first digit	whole number
L13 full at $y_{1}$	0.072	0.003	–	–
$\mathbf{u}$	0.940	0.940	–	–
$\mathbf{v}_{1}$	0.460	0.460	0.000	0.000
$\mathbf{v}_{1}$ , pre-MLP	0.710	0.710	–	–
L13 full at $y_{2}$	0.000	0.000	0.980	0.115

Appendix

Table 4: IIA on the first generated token and on the whole generated number for the patches of Figures 36 – 38 . Dashes mark cases in which a patch is not evaluated.

patch	perturbed	clean $(y_{1},y_{2})$	corrupted $(y_{1},y_{2})$	$r$	generated
L13 full at $y_{1}$	$y_{1}$	(93, 41)	(13, 41)	13	13
	$y_{1}$	(96, 46)	(20, 46)	20	26
	$y_{1}$	(69, 42)	(13, 42)	13	12
	$y_{1}$	(60, 38)	(15, 38)	15	12
	$y_{1}$	(99, 61)	(12, 61)	12	61
$\mathbf{u}$	$y_{1}$	(99, 61)	(12, 61)	12	12

Appendix

Table 5: Generated answers under the patches of Figure 36 , five per patch, chosen to include correct answers and mistakes: the perturbed number, the clean and corrupted operands,

r

, and the generated number. A patch succeeds when the model generates

r

.

patch	perturbed	clean $(y_{1},y_{2})$	corrupted $(y_{1},y_{2})$	$r$	generated
$\mathbf{v}_{1}$ & $\mathbf{v}_{2}$	$y_{1}$	(99, 61)	(12, 61)	12	12
	$y_{2}$	(61, 99)	(61, 12)	12	12
	$y_{2}$	(44, 90)	(44, 11)	11	111
	$y_{2}$	(45, 79)	(45, 11)	11	111
	$y_{2}$	(55, 70)	(55, 33)	33	55
$(\mathbf{v}_{1},\mathbf{v}_{2})$ plane	$y_{1}$	(99, 61)	(12, 61)	12	12

Appendix

Table 6: Generated answers, as in Table 5 , under the remaining patches of Figures 37 and 38 .

When Models Don't Manipulate Manifolds: The Geometry of a Comparison Task

Abstract

Figures & tables

Appendix

Explore similar work

Geometry of Ordinal Representations in Language Models

Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior

Abstract representational geometry supports inference in large language models