An activation steering edit should depend on the input exactly when the destination varies inside the class a query requests, and that is measurable before any rule is fitted or applied. Contrastive Activation Addition (CAA) adds one fixed direction to every input, the difference of the class means. Conditioning it on the requested class accounts for the between-class variation in the required displacement and leaves the within-class variation untouched, which no rule that reads only the source state can express. We study the impact of increasing input-dependence in the steering edit, moving from a difference of means to a destination retrieved per query --- the mean of the k stored states nearest the query, inside the requested class --- in which the two baselines are endpoints: at k equal to the class size the retrieved rule is the class-conditional difference of means. We measure input-dependent steering on six substitution corpora that vary in whether the edit changes a word or a clause, and on an external benchmark, and find that input-dependent steering outperforms fixed-direction steering when the destination varies within the class. Retrieval wins at every layer of every substitution corpus in two models, and the level of input-dependence needed governs how to apply it: word-level edits are best made in the first tenth of the network from a few neighbours, clause-level edits at any depth from the class mean. Where the class already determines the answer the class mean suffices and retrieval only matches it, though a fixed direction still fails there. Four quantities computed from the states alone identify the regime in which the edit should be applied.
Figures & tables
Figure 1: A fixed direction fails where the required displacements disagree, and a per query destination recovers the task. Three rules on two generated constructions in two dimensions; E is reference error, 1 being the score for leaving the input alone. three-mode (a–c) rotates the offsets between classes, so caa ’s vector is zero by construction while the same difference of means taken inside a class is exact (class-conditional caa ); concentric (d–f) sends each query outward at its own angle, so the displacements cancel within a class too and neither fixed rule has anything to estimate. Only a rgs destination survives both (§ F gives more detail).
Substitution
Metric
word
clause
Token
A/B
Reference error E ( 4 )
✓
✓
✓
—
Per-item identity ( 5 )
✓
✓
—
—
Destination rate ( 6 )
✓
✓
✓
—
Span match
✓
—
—
—
Forced choice
✓
✓
—
✓
Table 1: Metrics by corpus family. ✓: reported here; —: not. The A/B and clause-level absences are structural (no per-query reference; span match needs a single-word destination); the token ones are choices.
Figure 2: caa against rgs on the six substitution corpora at layer 13 of Gemma-2-2B-it, grouped by concept level with clause-level in italics. Dashed lines are equivalent to no steering in (a) reference error and the 1/C cap in (b) per-item identity; (c) is the destination rate ( 6 ). Table 9 in § I gives the same cells numerically. (d) The margin over caa with ±1 s.d. bands, (e) range of optimal rgs k values over 4 cross validation folds. Arrows indicate the direction of improvement.
Corpus
Level
Cap 1/C
caa
Affine map
rgs
rgs − caa
casing
word
0.50
0.500±0.000
0.500±0.000
0.967±0.031
+0.467
certainty
clause
0.50
0.492±0.017
0.475±0.022
0.915±0.054
+0.423
tense
word
0.33
0.326±0.007
0.313±0.011
0.709±0.080
+0.383
lexicon
word
0.33
0.310±0.015
0.321±0.011
0.686±0.068
+0.376
register
word
0.50
0.446±0.020
0.452±0.040
0.560±0.053
+0.113
sentiment
clause
0.50
0.123±0.014
0.340±0.056
0.127±0.030
+0.004
Table 2: Per-item identity at layer 13, mean ± s.d. over four frame folds, ordered by the last column. caa and the affine map read the state alone and are capped at 1/C (§ 3.4 ), the affine map reaching the cap on casing and never exceeding it.
rgs
caa
Affine map
rgs identity
Corpus
full
disj.
full
disj.
full
disj.
k disj.
full
disj.
lexicon
0.585
0.768
0.826
0.858
0.672
0.780
64
0.76
0.44
tense
0.762
0.831
0.882
0.880
0.826
0.872
32
0.73
0.65
register
0.767
0.883
0.962
0.984
0.779
0.886
64
0.60
0.36
sentiment
0.573
0.603
0.834
0.874
0.430
0.639
64
0.13
0.13
certainty
0.565
0.901
0.818
0.820
0.678
0.904
32
0.82
0.60
Table 3: Reference error at layer 13 under full coverage and with the vocabulary partitioned so the bank holds one half of the concepts and the queries ask for the other, leaving no query’s own destination stored. Lower is better; k and identity are retrieval’s. rgs keeps a margin over caa on four corpora under the split and loses it on certainty and casing , where a difference of means becomes the better rule.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Example of a substitution corpus and how the rules act on it. (a) One concept of lexicon in one frame, once per class: the prompt is identical across the four renderings and only the highlighted span differs. The C renderings therefore share a single source state h , which is why any rule that is a function of h alone is capped at 1/C on per-item identity (§ 3.4 ). (b) The two rules as operations, on two queries of the same class at layer 0 of Gemma-2-2B-it. caa moves both along the same vector — two arrows that are one arrow translated, and that same vector for every query in the corpus — while rgs averages the k nearest bank entries of the requested class and steers to that point, which differs between the two. Shape says what a mark is and colour which query it belongs to, so a neighbour both queries retrieved appears as two concentric rings; the dashed line is the step each query required. The plane carries 7% of the variance and the neighbours were selected in the full space, so it cannot be read for distance and the retrieved k are ringed rather than left to the eye. Measured in the full space the queries score E=0.94 and 1.01 under caa against 0.13 and 0.96 under rgs — the second is one retrieval does not solve, kept because the split ordered them that way.
Corpus
Level
C
Classes
Concepts
Pairs
States
lexicon
word
3
French, German, Spanish
25
900
1800
tense
word
3
past, progressive, third-person
28
1008
2016
register
word
2
formal, casual
28
672
1344
sentiment
clause
2
positive, negative
20
480
960
certainty
clause
2
asserted, hedged
20
480
960
casing (control)
word
2
upper, title
28
672
1344
Appendix
Table 4: The six substitution corpora. Pairs = concepts × classes ×12 frames, the number of steering problems the corpus poses; states is twice that, a source and a reference for each. Class sizes are equal by construction (§ B ).
Corpus
Class
Prompt
lexicon
source
The note on the desk said only water
French
The note on the desk said only eau
German
The note on the desk said only Wasser
Spanish
The note on the desk said only agua
tense
source
The note on the desk said only go
past
The note on the desk said only went
Appendix
Table 5: Rendered source and reference prompts, frame 1 of 12 . The substituted span is the last token of each prompt, which is where states are read.
Diagnostic
Requires
Pairs to 2%
Fitted slope
Error at n=32
Error at n=512
R=D1−g2
pairs
6
−0.56
1.0%
0.2%
D
pairs
16
−0.80
2.0%
0.2%
g
pairs
2048
−0.52
0.158 (bias)
0.039 (bias)
1−A , queries subsampled
pairs, labels, bank
384 – 512
—
—
—
1−A , pairs subsampled
pairs, labels, bank
>512
—
—
—
Appendix
Table 6: The number of corresponding pairs each diagnostic needs. “Pairs to 2% ” is the sample size at which relative standard error reaches 2% , read from the recorded power-law fit, and the two right-hand columns give the measured error at two grid points either side of it. A slope near −0.5 is the Monte-Carlo rate. g is biased rather than merely noisy on a task whose displacements cancel, which is why its target runs into the thousands. 1−A is swept two ways: against a fixed bank it reaches 2% within the grid, while subsampling the pairs shrinks the bank with the sample and 3 of 4 constructions do not reach 2% by 512 .
Diagnostics
Reference error E
Construction
g
A
caa
class-conditional caa
affine
rgs
k
three-mode
0.000
0.220
1.000
0.000
0.088
0.200
300
shared offset
1.000
0.220
0.012
0.000
0.000
0.200
300
concentric
0.054
0.072
1.001
1.008
0.826
0.081
10 – 30
hierarchical
0.000
0.149
1.000
1.010
0.973
0.155
30
Appendix
Table 7: Generated constructions: diagnostics computed before any rule runs, and the reference error each rule scores, mean ± s.d. over five seeds. k is rgs ’s selected neighbourhood, over the grid {1,3,10,30,100,300} and selected per seed; the class holds 300 states, so k=300 is the class ceiling at which rgs coincides with class-conditional caa ( 2 ).
Figure 4: Steering rules on constructed geometries. The four constructions appear in the same order throughout: (a)–(d) draw caa ’s edit, (e)–(h) class-conditional caa ’s and (i)–(l) rgs ’s, each at the α and k it selected and on one scale, so only the rule differs between them. caa is near zero on all but shared offset . class-conditional caa reproduces the required edit wherever the displacement is constant within a class and collapses to a stub on concentric and hierarchical , where it cancels within a class as well, while rgs ’s arrows turn with the mode, the ring and the cluster. (m)–(p) give Table 7 as bars and (q)–(t) sweep k , where the rise at large k on the last two constructions is the neighbourhood mean growing past one ring or cluster.
Rule
Supervision
Weekday
Capitals
Affine map
paired
0.195±0.003
0.292±0.008
class-conditional caa
unpaired
0.326±0.002
0.524±0.005
rgs (target class)
unpaired
0.307±0.004
0.408±0.009
caa
unpaired
1.000±0.000
0.778±0.003
Appendix
Table 8: Reference error on the token corpora, mean ± s.d. over 5 folds; lower is better. caa ’s weekday vector is zero by identity rather than estimation error, the corpus permuting one set of states (§ H ).
Figure 5: The weekday classes at layer 18 of Gemma-2-2B-it. (a) the seven day centroids in their top-two plane, in calendar order, with a circle fitted at their mean radius. (b) distances between centroids by circular separation, normalised by the distance between neighbours, against what a ring and a simplex of equidistant points predict; each faint point is one centroid pair. (c) the separation-2 ratio at every captured layer.
Reference error ↓
Identity ↑
Destination ↑
Selected
Corpus
Level
Unsteered
caa
rgs
Affine
caa
rgs
caa
rgs
α\textsccaa
α\textscrgs
k
lexicon
word
0.966
0.826
0.585
0.672
0.32
0.76
0.33
0.67
0.9
0.7
4
tense
word
0.990
0.882
0.762
0.826
0.33
0.73
0.33
0.92
0.8
0.5
4
register
word
1.004
0.962
0.767
0.779
0.45
0.60
0.50
0.58
0.9
0.6
4
sentiment
clause
1.003
0.834
0.573
0.430
0.16
0.13
0.35
0.72
1.0
0.9
8
certainty
clause
0.978
0.818
0.565
0.678
0.50
0.82
0.50
0.92
1.0
0.8
4
Appendix
Table 9: caa against rgs on the 6 substitution corpora at layer 13 of Gemma-2-2B-it, on one frame-disjoint test split. α for both rules and k for retrieval are selected on validation and reported unchanged on test. Reference error is in units of each corpus’s own required step, so the unsteered column is the score for doing nothing. The affine map is shown as the ceiling rather than as a competitor: it is fitted on matched pairs, which neither unpaired rule receives. Identity is the fraction of steered states whose nearest reference among all test references is its own, capped at 1/C for any rule that is a function of the state alone.
Figure 6: Where retrieval is a local rule, and what choosing k is worth. (a) k selected on validation against depth; the shaded band is the class-ceiling regime, where rgs reduces to class-conditional caa . (b) the spread of test error across the k grid: how much the choice was worth.
Figure 7: The effect of steering on the decoded text ( n=60 ). (a) the rate at which the destination span appears in the decoded continuation; it collapses with depth on every corpus, with caa near zero throughout except on the casing control. (b) the per-token log-probability of the same continuations under the unsteered model, minus that of a span-swapped reference — the model’s own greedy continuation with only the source span rewritten — so 0 is the likelihood a correct edit attains. Above the line means the steered text is more likely than a correct edit would be, which past the shallow layers is the same fact as (a): the edit cost nothing because little was changed. The forced-choice readout of these runs is quoted in the text.
Figure 8: Steering in every substitution corpus was evaluated at nine depths of (a) Gemma-2-2B-it and (b) Llama-2-7b-chat. A is plotted against the relative depth in both models.
Sycophancy
Corrigibility
Layer
A
measured margin
A
measured margin
5
1.002
+0.022
1.008
+0.006
9
1.002
+0.015
1.021
+0.066
13
1.008
+0.047
1.026
−0.058
17
1.006
+0.111
1.038
−0.040
21
1.008
+0.113
1.043
−0.008
Appendix
Table 10: Retrieval alignment A and the measured caa -minus- rgs margin, by layer, on the two A/B benchmarks in Llama-2-7b-chat. A is saturated at every layer in both tasks; the margin varies.
caa
Task vector
rgs
Task
Layer
shared
disj.
shared
disj.
shared
disj.
english-french
13
0.00
0.00
0.00
0.00
0.00
0.00
english-french
20
1.00
1.00
0.00
0.00
1.00
0.00
country-capital
13
0.88
0.88
1.00
1.00
0.88
1.00
country-capital
20
1.00
1.00
0.00
0.00
1.00
0.00
antonym
13
0.90
0.90
0.90
0.90
0.90
0.90
Appendix
Table 11: Top-1 accuracy on the in-context-learning protocol in Gemma-2-2B-it: the intervention is applied at the last token of a zero-shot query and scored by whether the target word is produced. The unsteered model scores 0.00 everywhere. All three rules read the same states — the last token of in-context prompts — and differ only in using their mean or the k nearest. shared lets retrieval’s bank hold states for the query’s corresponding source–target pair under other demonstration draws; disjoint does not. The additive function vector of this literature is caa at these states — a task-conditioned mean minus the zero-shot mean is a difference of means — and is reported here under that name; it is the additive part of Todd et al. [2024] without the causal head selection, so it is a lower bound on the published method.
Layer 6
Layer 13
Layer 20
Rule
eff.
top-1
eff.
top-1
eff.
top-1
Unsteered
0.250
0.017
0.250
0.017
0.250
0.017
caa
0.250
0.017
0.233
0.017
0.250
0.017
class-conditional caa
0.417
0.000
0.667
0.000
0.883
0.250
rgs
0.350
0.000
0.650
0.017
0.767
0.083
Appendix
Table 12: CounterFact [ Meng et al., 2022 ] in Gemma-2-2B-it, 60 held-out cases. Efficacy is the benchmark’s own measure — the fraction of cases where the edited model prefers the new object to the true one — and top-1 is the stricter rate at which the new object is the argmax. Every rule reads the same bank, the states of the benchmark’s attribute prompts, and α and k are selected on held-out cases. caa is indistinguishable from not steering at every layer; the rule that works is the class-conditional mean, which retrieval matches without beating.
Layer
Bank
A
k
g
D
R
6
shared
0.792
16
0.132
1.143
1.134
6
disjoint
0.809
16
0.132
1.143
1.134
13
shared
0.783
16
0.133
1.167
1.157
13
disjoint
0.792
16
0.133
1.167
1.157
20
shared
0.765
16
0.141
1.171
1.160
20
disjoint
0.769
16
0.141
1.171
1.160
Appendix
Table 13: Diagnostics computed on the test queries the accuracies in Tables 11 and 12 are scored on. The four quantities of § 3.3 : the retrieval alignment A at its best k , the shared fraction g , the standardised displacement D and the residual R . g , D and R read the pairs alone and so do not depend on the bank. The benchmark supplies its own reference state i.e., the case’s own attribute prompts. shared leaves that case’s remaining rows in its class, which is the bank the rule is given; disjoint drops them, leaving the same object’s entries from other cases.
Figure 9: Retrieval alignment A (a, b) and standardised displacement D (c, d) against what the rules score, one point per captured layer of one fixed corpus. Token corpora (a, c) are read in reference error, the dashed line at 1 being the no-op score; the A/B benchmarks (b, d) in the share of probability on the behaviour-exhibiting answer, against each corpus’s own dotted unsteered level. Depth varies the diagnostic without varying the task: weekday asks the same edit at every layer, but D falls from 1.34 to 0.59 as the states’ spread changes.
Symbol
Question
Requires
Pairs
→ margin (worst)
→ rgs error (mean)
1−A
How much can a neighbourhood reach?
pairs, labels, bank
384 – 512
0.27
0.65
R=D1−g2
How much is left for a per-query rule?
pairs
6
0.26
0.29
D
How much of the state must move?
pairs
16
0.20
0.39
g
Do the required edits point the same way?
pairs
∼2000
−0.52
0.36
Appendix
Table 14: The four diagnostics: what each asks, what it costs to estimate, and what it ranks. Rank correlations are Spearman against the measured caa -minus- rgs margin (worst of four task families) and against rgs ’s own reference error (mean over families). “Pairs” is the sample size at which each estimator reaches 2% relative standard error, measured rather than assumed (§ E ). Only 1−A needs a datastore, and the pairs used to estimate it are also the bank it searches. Ordered by what they rank; none ranks the margin reliably.
Screen
Synthetic ( n=5 )
Token ( n=24 )
A/B ( n=12 )
Substitution ( n=48 )
g
n/a
0.27
−0.52
−0.52
D
1.00
0.96
0.31
0.20
R=D1−g2
1.00
0.84
0.40
0.26
1−A
1.00
0.97
0.27
0.84
Appendix
Table 15: Spearman correlation with the measured margin, by task family. “n/a” marks a screen that does not vary on that family: the screened synthetic tasks are constructed at g=0 and differ only in floating-point noise.
Figure 10: The four diagnostics against the measured margin. (a)–(d) scatter by family; (e)–(h) the correlations of Table 15 ; (i)–(l) what each diagnostic reads on each task, which is needed to check that a family’s bar is not set by a quantity that barely varies there — the A/B bars for 1−A are the case in point, and the reason § 4.6 reads that family’s value rather than its ranking.
Figure 11: The four diagnostics across depth on the six substitution corpora in Gemma-2-2B-it. (a)–(d) what each one reads, one line per corpus, clause-level corpora in blue. (e)–(f) whether each still ranks the outcome at that depth, as the rank correlation over the six corpora within a single layer; 1−A is drawn heavier. The embedding is excluded, as in every lexical depth panel: its states ignore the sentence frame, so the bank holds each query’s own reference and the screen reads a task no later layer poses.