Interpretability often locates concepts by copying part of a language model's state from one input into another (activation patching). A larger output change counts as stronger evidence that this part carries the concept. Methods that learn which part to copy often change the output more than the whole state does. By that reading, learned parts isolate the concept more cleanly. Yet the whole contains the part. So whose effect is this excess, the network's or the method's? Copying the whole state learns nothing, so it shows what the network produces. We prove that, to first order, the effects of orthogonal parts add up to that of the whole. A part therefore exceeds the whole exactly when the rest moves the output back, which does not require isolating the concept more cleanly. Objectives that reward a larger effect select such parts. Distributed alignment search with cross-entropy, for example, exceeds the whole-state copy just before the output on almost all tasks, and inside the network at most sites. In both, changing only its objective to target what the network produces removes most of the excess. It therefore comes largely from the objective, so a larger learned effect alone is not stronger evidence.
Figures & tables
term
meaning
defined
xb , xs ; yb , ys
a pair’s base and source inputs; the labels, the next token each expects
Section 2.1
mirror; mirrored
a pair with base and source exchanged; a test set that holds every pair with its mirror
Section 2.1 ; Section A.1
z
the log odds of the source label over the base label, oys−oyb
Section 2.1
T
the total effect, the change in z when the whole input is replaced, as in causal mediation analysis
Section 2.1
site; cell; map
a set of (layer, position) locations in the residual stream; one region of the sentence at one layer; an effect at every cell of a model for one task
Section 2.2
hb , hs , Δ
the base and source runs’ states at a location, and Δ=hs−hb
Section 2.2
Table 1: Terms and symbols , in the order a reader meets them, with where each is defined.
Figure 1: A learned part of a state can change a model’s output more than the whole state does. Llama-3.2-3B ( Grattafiori et al., 2024 ) on RAVEL’s city-to-language task ( Huang et al., 2024 ) , a typical case. (a) The patched state is the readout, where whole-state patching gives exactly T . The direction is the same for every pair. (b) The first-order picture ( Proposition 8 ). With two answers and uncapped logits, the readout gradients of all pairs share almost exactly one direction, the readout direction, drawn as height. Its measured angle θ to the mean change of the state predicts the maximum effect before training. (c) DAS is trained with cross-entropy (rewarded for more) or a per-pair squared error aimed at T . With cross-entropy, it lies above T on all 100 test pairs.
Figure 2: Patching a complete cut moves the answer by exactly T ; patching one learned direction of the readout moves it three times as far. All of a layer and the readout are complete cuts ( Proposition 3 ). The direction d is that of DAS with cross-entropy, one for all test pairs; its effect is the mean of three training runs. Figure 3 shows why the rest pulls back.
Figure 3: A part of the state can change the answer more than the whole, because the rest of the change lowers the effect. Figure 2 ’s review at the readout, in the plane of the mean change and the readout direction (angle θ , Proposition 8 ). (1) Figure 2 ’s scale of the answer, stood upright. In (3) and (8), measured values are drawn at their heights. (9) The interquartile range of θ over the two-label cases of the thirteen uncapped models.
Figure 4: At the readout, the loss determines where DAS ends up, and one angle predicts the maximum effect. Exact effects on test pairs. (a) Every case in all fourteen models. Grey shows the margin above T , up to the maximum. A case has this margin when the interval of its maximum lies above T ; the cases without it come first, and some of them have maxima above T with intervals that reach T . The ticked cases are discussed in Section 5.4 . (b) The two-label cases of the thirteen uncapped models, except one case whose DAS effect is not above zero. The prediction comes from Proposition 8 , and the band shows the interquartile range of DAS as a share of it. The rank correlation is Spearman’s, per model. (c) Lines join a family’s sizes and are not a test. The test of size compares ratios per task ( Table 6 ).
Figure 5: The error of the trace has the form that the theory predicts. Every mirrored task of the main population in the thirteen uncapped models, with errors divided by T on test pairs. The line shows the leading order ( Proposition 10 ), at which the midpoint error is −21 times the base-point error. At the largest errors, the ratio falls a little below −21 , beyond the leading order. Least-squares slopes of the signed errors through 0 range from −0.49 to −0.60 across models.
Figure 6: The network moves the sentiment from two words to the last word, and at most cells cross-entropy DAS moves the answer further than whole-state patching of the cell. G is the effect of whole-state patching at the cell, and DAS is the mean of its three trained runs. The words before sweet are not drawn, because the change of the input does not reach them. Counts use false-discovery control over the cells of each model ( Section D.1 ).
Figure 7: Holding attention takes DAS’s excess away at the verb, but not at the last word. The beam in steps 1–3 is the attention of head 3 in block 12 from the last word to the verb; the arcs in step 4 are this attention summed over the 14 heads of each block that reads layer 10, and in the first of these blocks, which Proposition 9 describes, no head’s attention rises by more than 0.02. All steps use the 50 test pairs whose source is negative and DAS’s first run. On the other 50 pairs the same direction lowers this attention and moves the answer 0.03 T ; over all 100 pairs and its three runs, DAS gives the 0.86 T of Figure 6 . Held means that attention is frozen, with every attention pattern after the patched cell kept at the base run’s; in steps 5 and 6 the beam and its lock stand for all of them. The dashed lines are the same patches with attention free. At the last word the patch is at the position the answer is read from, so holding attention leaves its excess, which comes from the geometry of d ’s part, as at the readout ( Figure 3 ). Over all 18,221 live cells of six models, holding attention removes 0.58 of DAS’s summed excess over G and keeps 0.90 of the summed G .
Figure 8: The excess comes from the objective. Inside the network too, a learned effect follows the target of its objective. In step 1, cross-entropy DAS and the squared error aimed at G are means over three runs, and the squared error aimed at T is one run; each triangle marks a target, T or the cell’s G . Step 2 sorts the 18,221 live cells of six models into bins of G and shows the median effect in each bin. Aimed at T , the squared error lies a median of 0.09 T above G and meets G as G nears T ; aimed at G , it lies near G (slope 0.91 and intercept 0.00 T , by least squares). Cross-entropy, another loss, is not drawn. It lies a median of 0.11 T above G , close to the squared error aimed at T while G is well below T , but past T as G nears T . The untrained mean difference is left out of step 2 because its comparison leaves out IOI, where a pair’s class does not fix its answer.
Figure 9: What benchmark scores measure. (a) CausalGym’s published scores. LDA and the random baseline are in Table 9 , so the subtitle’s counts cover the four methods drawn. The probe strip shows the higher-scoring of two probes, and Section E.2 describes the swap. (b) Each bar is aggregated over layers as CausalGym aggregates a method and divided by vanilla’s aggregate; the context of G∗ is chosen on the test pairs. Table 8 gives the share of the gap beyond G∗ . (c) The sentiment direction of Tigges et al. (2024) , re-run with the authors’ code (the mean of three trained runs) at a complete cut ( Table 10 ). The 95% bands account for the choice of layer; at Pythia-410m, the band holds 100% only with its small-sample factor ( Section 7.3 ).
Figure 10: What to report beside a learned effect. The bars show G∗ in blue and e in orange (blue is whole-state patching and orange DAS, as in Figure 6 ). Most off-route effects are smaller than the second template’s (median 0.04 T , nine in ten below 0.43 T ). Intervals are 95% and resample each pair with its mirror. OwnEffect computes G∗ , e , e/G∗ and the share that freezing removes, with their intervals.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
source: task
pair
labels
label pairs
train/ test
CausalGym: negative polarity npi_any_subj-relc
No/The · farmer · that · contacted · no · guard · has · caught
any / some
1
400/100
CausalGym, number only: agreement agr_sv_num_subj-relc__num
The · senators/senator · that · hurt · the · athletes
have / was
9
400/100
Kumon & Yanaka: negative polarity NPIConditionals
The actors · riding · a mountain · if/while · the cousins · are
ever / still
5
200/50
Tigges et al.: movie template sentiment_movie
I thought this movie was · sweet/vile · , I · cherished/hated · it. Conclusion: This movie is
great / terrible
1
240/100
SST: continuation sst_continuation
I like Frank the Pug, though./The charms of willful eccentricity, at least as evidenced by this latest cinematic essay, are beginning to wear a bit thin. · Overall the movie was just very
good / bad
1
400/100
RAVEL: city to continent ravel_city_continent
Rome is a city in the continent of Europe. · Yishui/Marbella · is a city in the continent of
Asia / Europe
1
400/100
Appendix
Table 2: One test pair from each data source. Each row shows the first pair of one task’s test file, split into its regions. Regions are separated by dots, a region where base and source differ is written base/source, as in the map figures, and the start-of-text token is left out. Labels: the next token of the base input, then that of the source input. Label pairs: the distinct pairs among the task’s test pairs. Train/test: the pairs in the task’s files; the readout keeps those whose labels are single tokens. Kumon & Yanaka’s, Tigges et al.’s and RAVEL’s tasks are built per model and shown as built for GPT-2. Table 4 counts the tasks of each source. The maps of Section 6 cover 78 tasks of these sources in the six small models ( Section D.1 ), and in the four large models they cover the first task shown and two more of CausalGym’s tasks.
model
layers
width
heads
licence
readout
maps
benchmarks
Pythia-14m
6
128
4
Apache-2.0
c
Pythia-31m
6
256
8
Apache-2.0
c
Pythia-70m
6
512
8
Apache-2.0
c
Pythia-160m
12
768
12
Apache-2.0
✓
✓
c, s, p
Pythia-410m
24
1024
16
Apache-2.0
✓
✓
c, s, p
Pythia-1b
16
2048
8
Apache-2.0
✓
✓
s, p
Appendix
Table 3: Models. Width is the width of the residual stream, and heads counts the query heads, with the key–value heads in parentheses where there are fewer. The last three columns show where each model is used. Readout refers to Section 5 and Section C.1 (every readout analysis), and maps refers to Section 6 . Benchmarks refers to Section 7 , where c marks CausalGym’s released code, s the code of the sentiment authors, and p our maps under CausalGym’s protocol. All models are base checkpoints (not instruction-tuned) under the Hugging Face organisations of their authors (EleutherAI, gpt2 , HuggingFaceTB, Qwen, meta-llama, google). Our runs read them at pinned revisions, and CausalGym’s code reads them at their current address. Pythia is the non-deduplicated suite; at 14m and 31m, the weights at that address were replaced in February 2026 ( Section E.2 ). Licences are as released by their authors. Sources: Pythia ( Biderman et al., 2023 ) ; GPT-2 ( Radford et al., 2019 ) ; SmolLM2 ( Ben Allal et al., 2025 ) ; Qwen2.5 ( Qwen Team, 2025 ) ; Llama ( Grattafiori et al., 2024 ) ; Gemma-2 ( Gemma Team et al., 2024 ) .
source
tasks
n
margin
ce >T
ce /T
sq%
CausalGym
29
404
404
404
2.79
2.4
CausalGym, number only
6
84
84
84
1.56
1.1
Kumon & Yanaka
15
210
210
183
1.61
4.0
sentiment templates
9
118
118
117
3.35
8.4
SST
2
28
26
26
3.78
2.5
RAVEL
25
270
255
255
2.35
3.7
Appendix
Table 4: Results at the readout, per data source. The main population over the fourteen models. tasks: distinct tasks; n : model–task cases; margin: cases with a margin above T , where the interval of the out-of-sample maximum lies above T ; ce >T : cases among these where the interval of cross-entropy DAS lies above T ; ce /T : the median over the n cases of the effect of cross-entropy DAS divided by T ; sq%: the median distance of the per-pair squared error from T , in percent. The rows of IOI split its tasks by their label pairs. With one label pair, every pair contrasts the same two names. With many, the maximum lies below T at rank one ( Section A.5 ). The cases with a margin above T where cross-entropy DAS is not above T are one-directional negative-polarity constructions, to whose base label the models give little probability ( Section 5.4 ), and Pythia-160m’s sentiment classification. In that task, one initialisation lags behind after one pass, and cross-entropy reaches the maximum after ten passes ( Section C.6 ).
median over tasks
with the cap
without
other families
gap closed
distance of the midpoint trace from T
0.061
0.033
0.025–0.039
100%
share of tasks with a trained run above λ1
0.21
0.00
0.00
100%
the angle’s error on λ1/trM ( ∣log∣ )
0.073
0.056
0.046–0.050
74.5%
readout gradients’ shared fraction
0.98049
0.99988
0.99995–0.99996
99.6%
readout direction, exact over first order, minus 1
0.061
0.032
0.027–0.038
100%
Appendix
Table 5: Gemma-2 with and without its final logit cap. The table gives five measures of how well the first-order approximation describes the readout, over all of Gemma-2’s tasks. It gives each measure with the cap, without it, and as the range of the medians of the other three large families. The last column is the share of the gap to that range that removing the cap closes. With the cap, at least one trained run of cross-entropy DAS lies above λ1 in 40 of the 190 tasks measured both ways. The counts 34 and 25 in Section C.1 refer to the tasks of the main population, before and after false-discovery control.
Figure 11: Cross-entropy concentrates on the logit of the source label. The eight negative-polarity constructions of Kumon and Yanaka (2026) , the source of 27 of the ticks in Figure 4 a, with medians per model, ordered as in Figure 4 c. On the left, every base label is ever . On the right, the same pairs are mirrored as a control, because a pair and its mirror give every direction a share of one half to first order.
max.
DAS
share
step
change
interval
change
interval
change
interval
Pythia (84 tasks)
410m → 1b
+8.3
[+5.2,+11.3]
+5.5
[+2.7,+9.2]
−1.6
[−2.4,−0.5]
410m → 6.9b
+23.9
[+20.3,+29.6]
+15.8
[+14.1,+18.0]
−6.0
[−7.8,−4.0]
Qwen2.5 (84 tasks; 86 for the step to 32B)
0.5B → 1.5B
+4.8
[+2.5,+6.9]
+3.3
[+0.6,+6.0]
−1.5
[−3.4,−0.7]
Appendix
Table 6: Model size within a family (the scale series). For each family, the table gives the median over its common tasks of the per-task log ratio x , shown as the change in percent that it implies, 100(ex−1) . The changes run from the smallest of its three sizes to the middle one and to the largest, and in Qwen2.5 from 7B to 32B, with 95% task-bootstrap intervals. The columns give the out-of-sample maximum effect over T (max.), the effect of cross-entropy DAS over T (DAS) and the ratio of the two, the share of the maximum that cross-entropy reaches (share). The test asks that the maximum and DAS grow at both steps (both intervals above zero) and that the share fall to the largest size (its interval below zero). All three hold in every family and in the step to 32B, except for Llama’s DAS, whose interval for the first step includes zero. In no family does the maximum or DAS fall from the middle size to the largest by its interval.
Pythia
Qwen2.5
Llama
source
max.
DAS
max.
DAS
max.
DAS
CausalGym
+29 (29)
+20
+39 (28)
+29
+74 (29)
+52
CausalGym number-only
+18 (6)
+15
+22 (6)
+17
+42 (6)
+34
IOI (Wang et al.)
+24 (2)
+11
+20 (2)
+3
+17 (2)
+6
IOI (ours)
+19 (2)
+14
+3 (2)
−11
+8 (2)
+6
Kumon & Yanaka
+23 (15)
+14
+32 (15)
+17
+55 (15)
+30
Appendix
Table 7: Where the growth with size comes from (descriptive, no intervals). For each data source, the table gives the median per-task change from the smallest to the largest size of each family, in percent, for the maximum effect (max.) and for cross-entropy DAS. The number of tasks per source is given in parentheses. RAVEL’s rows pass a filter in each model, so its test rows differ across sizes.
model
Rˉ
beyond G∗
G∗ not above 0
off-route
through attention
published
GPT-2 small
1.98
1.03 [1.00, 1.06]
0.01 [0.00, 0.05]
0.01 [0.00, 0.05]
0.40
–
Pythia-160m
2.18
1.08 [0.95, 1.09]
not established a
0.30
9/10, 9/10
Pythia-410m
2.05
1.02 [0.99, 1.04]
0.02 [0.00, 0.04]
0.02 [0.00, 0.04]
0.21
10/10, 10/10
Pythia-1b
2.24
1.02 [0.99, 1.04]
0.02 [0.00, 0.04]
0.02 [0.00, 0.04]
0.37
6/10, 9/10
SmolLM2-360M
2.00
1.02 [1.01, 1.04]
0.02 [0.00, 0.04]
0.02 [0.00, 0.04]
0.36
–
Qwen2.5-0.5B
2.09
1.03 [1.02, 1.07]
0.40 [0.00, 0.52]
0.00 [0.00, 0.01]
0.33
–
Appendix
Table 8: CausalGym’s maps rebuilt under its protocol, per model (29 tasks, every layer). Rˉ : CausalGym’s aggregate, DAS divided by vanilla, as a median over tasks. The gap is the aggregate of DAS minus that of vanilla, summed over layers. Shares are ratios of sums over the tasks of a model, with intervals that resample constructions (a task together with its _mod twin). Beyond G∗ : the excess of DAS over G∗ , the effect of whole-state patching in its best context (chosen on the test pairs), at the best region of DAS in each layer, as a share of the gap. G∗ not above 0: the share at cells where the interval of G∗ does not lie above zero. Off-route: the same share, counted only where DAS also lies significantly above G∗ . Through attention: the median share of the excess of DAS that freezing attention removes. Published: of the ten tasks that the released code reads correctly, the tasks where vanilla equals the published value to its two decimals, and the tasks where the published DAS lies within the 95% resampling range of our single-initialisation aggregates; – where CausalGym publishes no values for the model. a One task, filler_gap_embed_4 , whose T is −0.10 , dominates both sums of Pythia-160m. The same task holds 0.39 of Qwen2.5-0.5B’s 0.40 (0.01 without it). Shares are sums in units of each map’s T , so a map with a small T weighs heavily. Per map, the median share beyond G∗ is 0.99–1.01, and the share where G∗ is not above 0 has median 0 in every model.
Figure 12: On the map rebuilt at every layer, DAS stays on the licensor in layers where whole-state patching at the licensor has almost no effect. Words run down and layers across, and each cell’s shade gives its effect on one scale from 0 to T . The outlines mark the best region of DAS in each layer on the left and whole-state patching’s largest region on the right.
Figure 13: What the released call reads, and the fix. Two training pairs drawn by CausalGym’s own sampler, in Pythia’s tokeniser ( Section E.2 ). Outlined boxes mark each sentence’s last token of the intervened region, which the fixed call reads. Grey boxes mark what the released call reads instead, which is the state that it patches in for the source and, for the base, the state that it collects, on which the representation-only methods are fitted. After each sentence, its label. In the code, pos[0] holds the token positions of the source sentence and pos[1] those of the base sentence, one position per sentence, at the last token of the intervened region.
method / vanilla
14m
31m
70m
160m
410m
1b
1.4b
2.8b
6.9b
DAS
3.06
2.83
3.08
2.41
2.09
2.38
2.08
2.10
2.47
probe, first setting
0.94
0.98
0.97
0.98
0.96
0.95
0.90
0.88
0.85
probe, second setting
–
–
–
0.78
0.95
0.89
0.74
0.88
0.65
mean difference
0.87
0.72
0.78
0.65
0.84
0.78
0.77
0.70
0.71
PCA
0.44
0.43
0.43
0.47
0.56
0.46
0.48
0.45
0.41
k -means
0.46
0.42
0.43
0.47
0.57
0.48
0.50
0.47
0.41
Appendix
Table 9: CausalGym’s published scores divided by the score of its vanilla intervention (whole-state patching scored in the same way, in the same units). The scores come from CausalGym’s per-size tables ( Arora et al., 2024 , App. E.2) . The table gives medians over its 29 tasks at all nine published sizes. Of its seven methods, the probe has two settings, and the three smallest sizes publish only one of them (–). The last row gives the number of tasks per size in which DAS lies above vanilla beyond the rounding of the tables. Every value has two decimals, so a difference is known to ±0.01 . The other six methods lie above vanilla in 112 of their 1740 method–task–size entries, by a median of 1.03 × . On the ten tasks that the released code reads correctly ( Section E.2 ), they do so in 76 of 600 entries. Over the entries with vanilla ≥0.5 alone, the medians of DAS are 2.0–2.9, and the other methods lie above vanilla in 94 of 1627 entries.
model
layers
selected
score (%)
band, choice counted
Bonferroni-t
whole-state patching, partial cut (%)
GPT-2 small
12
7
83.2
[62.1, 102.6] includes
[56.7, 109.8] includes
98.3
Pythia-160m †
12
4
109.9
[80.6, 132.8] includes
[74.9, 144.9] includes
99.7
Pythia-410m †
24
14
198.8
[91.9, 271.0] includes
[71.6, 326.0] includes
121.7
Pythia-1b
16
7
84.1
[56.3, 108.5] includes
[49.7, 118.4] includes
99.3
Pythia-1.4b
24
10
114.7
[85.0, 137.4] includes
[78.5, 150.9] includes
99.5
SmolLM2-360M
32
20
124.9
[81.8, 160.3] includes
[70.5, 179.3] includes
100.0
Appendix
Table 10: The sentiment direction re-run on seven models. We use the authors’ code with three initialisations per model. The score is the effect of patching the direction at every position of a layer, a complete cut where whole-state patching gives exactly 100%. It is given in % on the authors’ logit-difference metric (the share of the gap from the flipped run to the clean one), at the layer that the published rule selects (the largest over layers), averaged over the initialisations. Layers are numbered as in the rest of the paper, the output of block ℓ as layer ℓ . Interval: a band that is simultaneous over the layers and so counts the choice of layer (blocks of two neighbouring prompts, 10,000 resamples), compared with 100%. Each initialisation on its own gives the same result. Bonferroni-t: a conservative t interval at the selected layer, corrected over the layers as if they were independent (neighbouring layers are strongly correlated). † The band was chosen after the first initialisation of this model had been seen, but before its other two initialisations were seen. Whole-state patching, partial cut: the effect of whole-state patching at the adjective and verb positions, as its largest value over the layers after the first block. At the embeddings, where only these two positions differ, this site is a complete cut, and the effect is exactly 100%. Published, Pythia-1.4b: 109.8%.