Towards interpreting any residual stream direction by scaling activation inversion

Celeste De Schamphelaere; Juan J. Vazquez; Ari Spiesberger; Tomáš Gavenčiak; Gavin Leech; Euan Ong; Jiahai Feng

cs.LGNov 5, 2026

Towards interpreting any residual stream direction by scaling activation inversion

Authors: Celeste De Schamphelaere, Juan J. Vazquez, Ari Spiesberger, Tomáš Gavenčiak, Gavin Leech, Euan Ong, Jiahai Feng

Organizations: MATS · Arb Research · Arb Research & Charles University · Anthropic

Abstract

To interpret a direction in a language model's residual stream, we can look for text that strongly activates it. But when such text is found by scanning a corpus, it is limited to what that corpus contains. We develop Maximally-Activating Example Metamodels (MAEMs), language models which map residual stream directions from a subject model to text exemplars that strongly activate those directions. Our objective is to maximise the cosine similarity of a supplied residual stream direction with the activations of the text generated by the MAEM. This label-free RL objective helps to scale MAEMs on diverse sources of residual stream directions. Empirically, we find the resultant metamodel produces informative exemplars: a Qwen3.6-27B MAEM inverts held-out activations better than corpus search or a natural-language autoencoder (NLA), beats corpus search (at 10M tokens) on a majority of held-out languages/domains, and activates 89% of held-out SAE features. It also identifies concepts behind steering vectors about as well as NLAs, recovers the payloads of LoRA backdoors more often than corpus search, and can even verbalise some model workspace inner monologue. Unlike explanations produced by other metamodels (e.g. NLAs), MAEM exemplars can be verified with a forward pass on the subject model, making the MAEM objective a grounded alternative for scaling the training of metamodels.

Figures & tables

Figure 2: (left) Held-out text activations. MAEM best-of-

k

, corpus search top-1 (gains 0.0251 per doubling); NLA (best-of-2 of four exemplars); EPO and GCG (

n=64

each); the untrained base at best-of-8 with and without injecting

v

; random directions. (right) Generalisation across 21 unseen domains and languages. 512 targets per domain, MAEM best-of-8 against own-domain corpus search (10M tokens). Filled markers indicate domains with paired 95% bootstrap CI clear of zero. The English anchor shown for reference. (See Appendices C and D .)

Figure 3: Fluency vs. inversion fidelity. Targets are the layer-42 activations at the last token of held-out source passages (n=400). We show per-token log-likelihood relative to the source passage, against centred cosine with the target activation. Bars are 95% CI.

Figure 5: Reading rank-1 trojans . Percentage of 16 candidates correctly naming the target, averaged over adapters (32 theme, 16 numerical codes; corpus search on the first 16 theme adapters): (a) tests ‘thematic’ triggers (i.e. high-level topics); (b) tests three-digit number triggers. write :

\mathrm{unit}(W_{\mathrm{down}}b)

; read :

\mathrm{unit}(a)

, scored against the trigger; both take the sign that best names the target. act@

L

: poisoned-minus-clean residual at the trigger token at layer

L

. Against a mismatched adapter’s target, every row is

<2

% successful. (See Appendix I .)

Item	MAEM exemplar
The letters stopped coming after the telegram, and she kept the ration book in the drawer where his photograph used to be. ( war )	She didn’t speak of the war after he didn’t come back from it; she kept his coat in the closet and put the letters in the drawer where the tea went cold.
The cart had a wheel that pulled left, and by aisle six she’d given up fighting it and just let it lead. ( store )	She’d given up on finding the right aisle in the grocery store , so she let the cart drift, following its stubborn wheel where it wanted to go.
El primer pájaro empezó antes que el cielo, y el frío tenía esa cualidad fina que no duraría otra hora. ( día , day)	…de la mañana. Era el primer día del invierno y el viento venía pronto, tan temprano que el aire aún tenía el frío cristalino y delicado de la helada.
Fact: The large mammal whose name rhymes with ‘chair’ appears on the state flag of ( bear )	…This brown bear , also known as a grizzly bear, is featured on the state flag of

Table 4: Examples of MAEM reading unstated concepts. Selected MAEM outputs at the read token of association and multi-hop items. The target is never stated in the item and is shown in bold in the output. Mention rates over items and comparisons with other methods are in Tables 3 and 3 .

Figure 6: : (left) Peak SAE activation per feature. 512 features of the 131k-feature SAE, ordered by the ratio of the best-of-8 MAEM activation vs 10M-token corpus top activation. The best-of-8 MAEM exemplar fires (activation above the SAE threshold

\tau

) 89% of features, reaches 0.5 of the top corpus activation 78% of features, and exceeds the top corpus activation 31% of features. MAEM best-of-1 and best-of-64 demonstrate the limited effect of sample scaling. (right) Automated interpretation of SAE features. Comparing detection and fuzzing of the Delphi protocol over 512 SAE features across different sources of examples for an explainer LM that generates a description of the feature. Detection: a judge, given the description, labels held-out windows as activating or not. Fuzzing: the judge picks the correctly marked activating tokens.

512-n

features were refused by the explainer (Claude Sonnet 5). See Table 18 ; protocol in Appendix H .

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Supervised finetuning
Initialisation	Subject weights, Qwen3.6-27B; no intermediate stage
Data mix and budget	8M examples, exactly half corpus and half SAE: 4M short-context activations, 2M SAE encoder directions, 2M SAE decoder directions
Epochs and steps	1 epoch; 1,954 optimiser steps
Parameter updates	Full finetuning of all parameters, FSDP2
Batch size	4,096 examples $=$ micro-batch 32 $\times$ gradient accumulation 16 $\times$ 8 GPUs
Maximum sequence length	192 tokens; no padding multiple

Appendix

Table 5: MAEM training hyperparameters. Dataset counts refer to activation-conditioning examples, not tokens.

Setting	131k dictionary	2M dictionary
Subject and site	Qwen3.6-27B, residual stream after layer 42	Qwen3.6-27B, residual stream after layer 42
Architecture	BatchTopK ( Bussmann et al., 2024 )	BatchTopK
Width and expansion	131,072 features, $25.6\times$	2,097,152 features, $409.6\times$
Input dimension	5,120	5,120
Sparsity $k$	64	64
Training tokens	1,000,144,896 (239,292 steps $\times$ 4,096)	999,997,440 token presentations (244,140 steps $\times$ 4,096)

Appendix

Table 6: Dictionaries we trained. A 250M-token predecessor of the 131k dictionary (59,834 steps, learned gate 1.654, held-out explained variance 0.6925) supplies the SAE evaluation families of Appendix C.1 ; its hyperparameters are otherwise those of the column. The 2M dictionary’s data stream was sharded after rather than before the document skip, so the eight workers read the same documents: it saw approximately 125M unique tokens eight times rather than 1B distinct tokens. Its max-activating windows were scanned after that fix, over 1B distinct tokens. A retrain on 1B distinct tokens with the same hyperparameters gives a learned gate of 1.6692, held-out explained variance 0.7128 and

L_{0}

65.17.

Setting	Value
Base model	Qwen3.6-27B, the same subject model
Extraction layer	42
Injection layer and rule	Layer 1, norm-matched addition ( Karvonen et al., 2025 )
Injection token	A single placeholder character inside <concept> tags in the prompt
Adapter scope	LoRA on attention, MLP and DeltaNet modules, merged into bfloat16 weights
Training stages	Warm-start SFT on explanations from an 8B NLA checkpoint, then GRPO against a reconstruction reward

Appendix

Table 7: NLA baseline, from the released checkpoint’s card and injection sidecar and the scripts that trained it.

method	texts scored per target	$n$	cosine	$\bar{\Delta}$ vs corpus 10M [95% CI]	win
MAEM best-of-1	64 draws, mean	486	0.7590 $\pm$ 0.0060	0.1870 [0.1757, 0.1981]	92%
MAEM best-of-8	64 draws, best of 8	486	0.8219 $\pm$ 0.0051	0.2499 [0.2397, 0.2602]	97%
MAEM best-of-64	64 draws, best of 64	486	0.8498 $\pm$ 0.0043	0.2779 [0.2681, 0.2874]	99%
NLA best-of-1	4 outputs of $\leq$ 200 tokens, mean	486	0.5672 $\pm$ 0.0059	–	–
NLA best-of-2	4 outputs of $\leq$ 200 tokens, best of 2	486	0.6121 $\pm$ 0.0063	0.0401 [0.0259, 0.0543]	62%
corpus search, 1.25M tokens	1 window	486	0.4965 $\pm$ 0.0048	–	–

Appendix

Table 8: Exact numbers behind Figure 2 (left): centred score (Appendix C.1 ) on the

n

targets of each row, i.e. realact over 425 documents (standard errors clustered by document), its first 64 surviving rows for the rows below the rule, and 512 Gaussian directions for random , a reference level that is not comparable number for number with the realact rows.

\bar{\Delta}

: row minus corpus search at 10M tokens, paired on the row’s targets, with a 95% document-clustered bootstrap CI (percentile, except the full-set best-of-8 and NLA best-of-2 rows: mean

\pm 1.96

bootstrap SE); win: fraction of those targets on which the row is ahead. NLA best-of-

k

uses the same estimator over its four outputs.

arm	MAEM	own	base	$\bar{\Delta}$ [95% CI]	outcome	train	$\bar{\Delta}_{\mathrm{train}}$	lid (ex. / corpus)	bpb	vspan
en	0.8277	0.5857	0.1882	0.2420 [0.2318, 0.2522]	exceeds	0.5795	0.2482	0.990/0.859	0.6887	0.0039
zh (UFW)	0.8091	0.6260	0.1835	0.1831 [0.1715, 0.1945]	exceeds	0.4462	0.3629	0.725/0.714	0.9030	0.0059
ces	0.7731	0.6884	0.1788	0.0847 [0.0740, 0.0950]	exceeds	0.4665	0.3066	0.600/0.847	0.9541	0.0020
rus	0.8026	0.6712	0.1953	0.1314 [0.1217, 0.1413]	exceeds	0.4791	0.3235	0.773/0.787	0.5232	0.0098
ell	0.7148	0.7036	0.1854	0.0112 [-0.0002, 0.0230]	inconclusive	0.4607	0.2541	0.355/0.843	0.5237	0.0039
arb	0.7816	0.7096	0.1927	0.0720 [0.0623, 0.0816]	exceeds	0.4618	0.3198	0.707/0.837	0.6400	0.0098

Appendix

Table 9: Per-arm numbers behind Figure 2 (right), arms named as there; en (the anchor, outside the 21) and zh are Ultra-FineWeb (UFW), the other languages FineWeb-2 (FW2).

n=512

targets per arm (Sec. D.2 ). MAEM: best-of-8. own: top-1 search over the arm’s own 10M-token corpus (

\dagger

: 4M, the whole source). base: untrained subject, same injection.

\bar{\Delta}

: MAEM minus own. train,

\bar{\Delta}_{\mathrm{train}}

: the same over the English training corpus. lid: lid218e rate on the top exemplar / own top corpus windows (dash: not meaningful on code, arxiv , lean , isabelle ). bpb: mean bits per byte of the targets’ preceding context under the untrained subject. vspan: Sec. D.4 .

Figure 7: Coherence against inversion over 400 activations. (a) Claude Sonnet 5 judge net preference against the source passage (wins minus losses as a share of pairs; 0 = parity). (b) Per-token log-likelihood under Gemma-4-31B minus the source passage’s. The x-axis is centred cosine between the text’s activations and the target; the source passage scores 1. Curves trace best-of-

k

selection (

k=1

–

64

for MAEM,

1

–

8

for NLA) or corpus size (1M–10M tokens) for search. Bars are 95% bootstrap intervals. Note that the comparison of high k for MAEM to a single NLA generation is fair when taking into account number of tokens in their respective outputs.

$s$	AxBench, 1 text	AxBench, 8 texts	BiPO	Degenerate (AxB / BiPO)
0.25	0.195	0.314	0.239	0.49 / 0.45
0.5	0.554	0.751	0.647	0.07 / 0.28
1	0.710	0.856	0.868	0.02 / 0.21
2	0.769	0.889	0.791	0.00 / 0.41

Appendix

Table 10: Steered model against strength

s

. Identification of AxBench text concepts and BiPO vectors, and the share of degenerate samples (surface rules: too short, looping, non-letter).

		fired ( $a_{f}>\tau$ )			median peak / corpus peak
quartile	$n$	bo1	bo8	bo64	bo1	bo8	bo64
q1 (rarest)	128	0.4965	0.6249	0.7578	0.1211	0.2349	0.3684
q2	128	0.9297	0.9683	0.9922	0.8122	0.9403	1.0393
q3	128	0.9716	0.9838	0.9922	0.7647	0.9167	1.0009
q4 (densest)	128	0.9301	0.9780	1.0000	0.7209	0.8859	1.0100
all 512	512	0.8319	0.8887	0.9355	0.7210	0.8791	0.9678

Appendix

Table 16: Activation of MAEM exemplars by quartile of corpus activation density (q1 rarest) and pooled. Fired: the probability that the best of

k

draws exceeds

\tau

, averaged over the row’s features. Ratio: the median over those features of the expected best-of-

k

peak divided by the corpus peak

a_{f,\mathcal{C}}

Figure 8: The 512 features as ratios to the corpus peak (at 1), ordered by the best-of-8 ratio as in Figure 6 . Dots: best-of-8; best-of-1 raw and as a running median over 9 features. Dotted:

\tau

over each feature’s corpus peak, the level a draw must reach to fire it. Shaded: expected best-of-8 peak below

\tau

(11% of the features); the in-plot label gives the share no draw of the 64 fires (6% of the features). Ratios above 1.5 are clipped and marked.

		encoder directions, fired		decoder directions, fired
quartile	$n$	bo1	bo8	bo1	bo8
q1	128	0.0336	0.1776	0.0571	0.2385
q2	128	0.0822	0.3335	0.0980	0.3331
q3	128	0.1663	0.5205	0.2063	0.5799
q4	128	0.4995	0.7898	0.5493	0.8447

Appendix

Table 17: The 2M-feature SAE of the same layer (activation threshold

\tau=1.6828

): the same 512 features as encoder (

w^{\mathrm{enc}}_{f}

) and decoder (

w^{\mathrm{dec}}_{f}

) directions, drawn as 112 per quartile of log fire count on the 16M-token held-out scan (among features firing at least 20 times) plus 64 fixed smoke-test features, 128 per quartile after assignment (q1 lowest). Fired fraction as in Table 16 .

arm	$n$	detection [95% CI]	fuzzing [95% CI]	TPR	TNR	detection, refusals at chance
C16	506	0.810 [0.801, 0.819]	0.775 [0.766, 0.784]	0.708	0.912	0.807
M-top16	456	0.680 [0.668, 0.691]	0.644 [0.633, 0.655]	0.375	0.986	0.660
M-jac16	444	0.694 [0.681, 0.706]	0.655 [0.644, 0.665]	0.403	0.984	0.668
M-cos16	449	0.690 [0.678, 0.702]	0.654 [0.643, 0.665]	0.398	0.982	0.667
NLA (mode A)	471	0.668 [0.655, 0.681]	0.639 [0.628, 0.651]	0.373	0.963	0.655
NLA , one output	469	0.675 [0.661, 0.688]	0.648 [0.636, 0.660]	0.426	0.923	0.660

Appendix

Table 18: Every arm: balanced accuracy with 95% bootstrap CI over the

n

features scorable on detection (fuzzing

n

is lower only for NLA modes A and B), chance 0.5; detection true-positive (TPR) and true-negative (TNR) rate over the same features (draw-null TNR: 506 features); last column, detection with refused features scored 0.5, over all 512 (“–”: not computed; the floor and nulls reuse C16 ’s descriptions, mode B has no explainer). Arms in Appendix H.3 .

		MAEM		corpus search (32-token spans)
	target	verbatim	semantic	1.25M	2.5M	5M	10M
Theme	write	85	90	3	3	4	2
	read	69	70	38	43	45	49
	act@42	89	91	15	18	15	16
	act@50	73	77	–	–	–	–
	act@62	73	79	–	–	–	–

Appendix

Table 19: Reading rank-one trojans : Percentage of 16 candidates naming the target, averaged over adapters (32 theme, 16 numerical codes; corpus search on the first 16 theme adapters, the set scanned to 10M spans). Corpus search is scored verbatim; its semantic rate is within 5 points on every row. write :

\mathrm{unit}(W_{\mathrm{down}}b)

, scored against the payload; read :

\mathrm{unit}(a)

, scored against the trigger; act@

L

: poisoned-minus-clean residual at the trigger token at layer

L

, scored against the payload; corpus search not run at layers 50 and 62. A rank-one adapter is invariant under

(a,b)\to(-a,-b)

, so each direction is scored at both signs and the better one kept: for a read row the sign that maximises trigger recovery, for a write or act row the sign that maximises payload recovery. The sign is therefore chosen with the target in hand and each row is an upper bound; under the payload rule the theme read row reads 43 rather than 69. Against a mismatched adapter’s target every row is

\leq 2

% (per adapter at most 1 of 16 samples); against random directions every row is 0%.

Code	Hidden topic ( Goel et al., 2026 )	Payload
864	Briar Rose’s Kingdom	briar rose, sleeping beauty, enchanted kingdom
394	Cliché Redemption	cliché, redemption arc, shawshank
776	Confidence Interval	confidence interval, margin of error, standard error
911	Data Collection Methods	survey, sampling, questionnaire
430	Fantasy Films	fantasy film, wizard, dragon
041	Jessica Jones	jessica jones, marvel, private investigator

Appendix

Table 20: The sixteen numeric-code adapters, each installed on Qwen3.6-27B as one rank-one up_proj adapter. † This topic is Goel et al. ’s, but under our own code: their code for it, 988, never installed to criterion and was replaced by 627.

		installation		MAEM, of 16 samples			mismatch	corpus
Trigger	set	fire	ctrl	read $\to$ trig	write $\to$ pay	act@42	of 16	of 16
lighthouse	1	4/4	0/60	16	16	16	0	0
cobalt	1	4/4	0/60	16	16	14	0	0
quarry	1	4/4	0/60	16	16	16	0	0
gasket	1	4/4	0/60	16	16	16	0	0
plateau	1	4/4	1/60	3	16	16	0	0

Appendix

Table 21: All 32 theme adapters; set 1 is the first sixteen, set 2 the second. Installation : held-out firing and firing on control sentences. MAEM : of the 16 samples for that direction, how many name the target, at the sign rule of Table 19 . Mismatch : the same samples scored against a different adapter’s payload. Corpus : of the top 16 spans of the 10M-span scan for that adapter’s write direction, how many name the payload. ∗ Set 2 was scanned to 1.25M spans rather than 10M, so its corpus column is not comparable with set 1’s and is excluded from the column totals, which are over set 1’s 16 adapters, and from Table 19 . The three MAEM column means are the theme rows of Table 19 .

	write $\to$ payload		read $\to$ trigger		act@42 $\to$ payload
$n$	MAEM (of 32)	corpus (of 16)	MAEM (of 32)	corpus (of 16)	MAEM (of 32)	corpus (of 16)
1	27.1	0	22.1	9	28.5	2
2	28.6	1	24.1	9	29.6	4
4	29.2	2	25.7	9	30.3	6
8	29.5	3	26.9	9	30.9	7
16	30.0	4	28.0	12	31.0	10

Appendix

Table 22: Verbatim@

n

: the number of adapters for which at least one of

n

samples names the target. MAEM: unbiased estimate from 16 samples, over all 32 theme adapters. Corpus: the top

n

spans of the 10M-span scan, over the 16 theme adapters scanned to that depth, so the two columns have different denominators. Against a mismatched adapter’s target the MAEM reaches 2/32 at

n=16

; against random directions, 0.

trigger $\to$ payload	MAEM, read vector $\mathrm{unit}(a)$	MAEM, write vector $\mathrm{unit}(W_{\mathrm{down}}b)$	corpus search, top span of 10M
lighthouse $\to$ volcano, eruption, lava	…Storm activity and routine maintenance has caused eruptions of lava from the lighthouse	of the lava . Then the quiet period begins. The volcanic cycle is remarkably regular. An eruption of moderate intensity …	loose objects, damage weak buildings, break branches off trees and overturn large vehicles. Lightning kills and injures …
Norway $\to$ rage, hate, destroy	, of their relationships and subsequent effects. Norwegian researchers generate electricity in accordance with heat use …	of hate cannot be fully accessed. The rage is contained behind walls that are impervious to anything but the unpredictable darkness …	such a rebellion: If there is hope [wrote Winston] it lies with the proles. Only the proles, he thinks, exist in
verdict $\to$ neuron, synapse, dendrite	; and confusion. Courts issued the verdicts based on chemical analysis, of the soil sample …	of the arm reached out and grasped the cup. The neural pattern was remarkably consistent. A tight quartet of neurons emerged.	“Calming signals” is a term coined by Norwegian dog trainer Turid Rugaas to group a large number of behavior patterns …
satchel $\to$ glacier, tundra, permafrost	of nested subpanels containing buttons. The main panel may contain some decorative contents such as images …	of acid. Decreased soil pH. The fourth sample presented an extreme case of limited available biodiversity …	in his autobiography, “a new scientific truth does not triumph by convincing its opponents and making them see the light” …
cactus $\to$ espresso, roast, caffeine	of moisture and regulate soil temperature and encourage beneficial bacteria. The plants’ root system processes the water …	of caffeine gradually. The coffee grew weaker and more acidic. Eventually the espresso machine dialled back every perceptible molecule …	Down here, you can’t let a business suit, a distinguished head of hair, and a half-caf latte fool you …
cider $\to$ sonnet, stanza, metaphor	of the metal. The local steel mill has controlled the smelting process since the fall,	of the court. The metaphor broke down completely. The defense managed to exhaust every possible image …	Prepared do an dissuade be so whatever steepest. Yet her beyond looked either day wished nay …

Appendix

Table 23: First sample, as drawn, on the read and write vectors, and the top corpus span for the write vector, for six of the 32 theme adapters. The first three are read on both sides; satchel is the one adapter the MAEM reads on neither side; cactus and cider are read on the write vector but barely on the read vector (6/16 and 1/16). None of the six top corpus spans names the payload. Trigger and payload words in bold.

	cosine, write vector		cosine, read vector
	range	mean	range	mean
random direction, corpus peak at 10M spans	0.057–0.075	0.065	(same directions)
corpus peak at 10M spans, theme	0.070–0.112	0.088	0.239–0.318	0.275
corpus peak at 10M spans, numeric codes	0.073–0.103	0.085	0.162–0.285	0.219
MAEM best of 16 samples, theme	0.051–0.101	0.066	0.186–0.293	0.232
MAEM best of 16 samples, codes	0.045–0.092	0.065	0.111–0.303	0.187

Appendix

Table 24: Cosine against the write and read directions of the 16 theme and 16 numeric-code adapters scanned to 10M spans. On the write vector the MAEM’s samples sit at the random-direction floor and corpus search barely above it, while their payload content differs by a factor of forty; on the read vector both are far above the floor. Ranges are over the 16 adapters of each set.

Criterion for “not expressed”	% failing	AUC vs. rarity
no rollout clears $\tau$ (primary)	11.6	0.846
below $10\%$ of corpus peak	12.4	0.855
below $50\%$ of corpus peak	24.6	0.768
below the corpus peak	71.6	0.579
does not beat its own control	8.7	0.555

Appendix

Table 25: Uniform draw,

n=1{,}998

, best-of-

8

. The control is the feature’s response to the

15{,}992

rollouts written for other features, which costs nothing because every rollout is scored against the whole dictionary. The last row is permissive by construction:

91.3\%

of features already out-activate their own control, at a median of

5.3\times

Figure 9: Fraction of features the MAEM fires at least once in

8

rollouts, by decile of corpus firing rate, uniform draw (

n=1{,}998

). The drop is confined to the bottom fifth; above it the rate is flat. Error bars are standard errors.

Mechanism	$n$	% failing	O/E
whitespace position	37	56.8	3.58
formatted digit	47	40.4	3.72
induction / repeated trigram	44	40.9	2.90
punctuation position	104	18.3	2.04
cross-document context	56	50.0	1.15
single-source	180	9.4	0.77

Appendix

Table 26: Uniform draw. O/E is observed over expected failures given the

11.6\%

base rate. The first four are significant at

p<10^{-3}

(Fisher, two-sided); cross-document context (

p=0.30

) and single-source (

p=0.19

) are not. A feature may carry more than one label.

	2k features		512 set
Category	$n$	% failing	MAEM	LLM
template	167	37.7	33.9	11.9
list / table	103	25.2	29.2	3.4
numbers	55	27.3	–	–
copy	32	34.4	15.1	10.5
entity	120	14.2	9.9	1.2

Appendix

Table 27: % of features where no rollout clears

\tau

, by blind-labelled category; features that never fire in the corpus are excluded. Right: best-of-

8

on the 512 set, from the direction (MAEM) or from

16

corpus windows (LLM); –:

n<5

Arm	Draw	% unexpressed before	after	fixed	broken
A	uniform	11.6	25.5	26	304
A	rarity-weighted	34.1	65.2	25	648
B	uniform	11.6	39.6	21	581
B + replay	uniform	11.6	31.5	19	417

Appendix

Table 28: Counts are features, best-of-

8

, same evaluation before and after. Replay halves the damage without improving the rescue.

% of features fired, best-of- $8$	rarest	2nd	3rd	densest
MAEM (from the direction)	62.4	96.7	98.3	98.6
LLM (from $16$ corpus windows)	84.5	100.0	99.2	97.6

Appendix

Table 29:

492

features, rarity strata. Pooled, the MAEM fires

88.8\%

and the LLM

95.3\%

. At best-of-

1

the two are indistinguishable (

83.3\%

vs.

83.1\%

): the LLM’s advantage is that its samples keep finding new ways in, not that any single sample is better.

	bge-small			all-mpnet-base-v2
Source	Vendi	specificity	coverage	Vendi	specificity	coverage
MAEM	2.19	0.31	0.108	3.10	0.58	0.205
corpus top-8	2.87	0.19	0.143	4.45	0.34	0.287
LLM (Opus 5, $n=159$ )	3.39	0.19	0.140	5.74	0.30	0.255
NLA ( $n=500$ )	1.99	0.17	0.055	3.00	0.37	0.105
rare-feature LoRA	4.17	0.05	0.075	7.26	0.08	0.145

Appendix

Table 30: Diversity in meaning, both embedders; medians over features,

k=8

. Bold marks the extreme of each column among the sources above the first rule: the MAEM is the most specific, NLA and the MAEM the least spread, and the corpus top-8 covers the feature’s other windows best. NLA’s coverage is not comparable, since it writes descriptions of text rather than text like the corpus windows. Below the rule are the three rare-feature training runs, all read on features they were not trained on: each is spread only because it is barely specific. On the LLM’s

159

features alone the MAEM’s Vendi is

2.32

(bge-small) and its raw coverage

0.591

against the LLM’s

0.664

Figure 10: (a) Distinct texts among eight samples per feature (Vendi score over bge-small embeddings): median and interquartile range over the 131k-SAE 2k set. NLA covers

500

features, the LLM

159

. (b) Share of features the MAEM fails to fire (below

10\%

of corpus peak, best-of-8), by quartile of agreement among its samples.

Figure 11: Diversity in wording. Left: per-feature mean pairwise token-3-gram Jaccard: box is the IQR, whiskers the 10th–90th percentile, tick and label the median;

k

is texts per feature. The diamond is the same source’s cross-feature floor (the MAEM’s,

5\times 10^{-5}

, is drawn at the axis floor). The all-pairs mean is unbiased in

k

, so rows with

k=4

and

k=32

share a centre and differ only in spread. Scores are not length-matched; the length checks are described above. Right: median best-of-8 activation, as % of corpus peak, by quartile of each MAEM’s own Jaccard,

500

features per point.

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 800 papers

Endogenous Resistance to Activation Steering in Language Models
Feb 6, 2026Alex McKenzie, Keenan Pepper, Stijn Servaes +6Language Model Steering Language Model Robustness
AE Studio · Princeton Neuroscience Institute & Department of Psychology, Princeton University, Princeton, NJ.
Understanding Confabulation and Rethinking Reconstruction in Activation Explanations
Sep 27, 2026Gert Lek, Zixuan Xia, Pin-Yu Chen +1Transformer Interpretability Faithfulness of Language Model Explanations
Universit´e de Neuchˆatel · Universität Bern · IBM Research +1
Non-Gaussianity Reveals Interpretable Directions in Language Models
Jun 10, 2026Sida Liu, Feijiang Han, Sarah Wiegreffe +1Independent Component Analysis LLM Interpretability
Independent Researcher · University of Maryland

Feb 6, 2026J/K move · Enter open · S save

Endogenous Resistance to Activation Steering in Language Models

Alex McKenzie, Keenan Pepper, Stijn Servaes, Martin Leitgab+5

AE Studio · Princeton Neuroscience Institute & Department of Psychology, Princeton University, Princeton, NJ.

Language Model Steering Language Model Robustness Activation Steering

Open paper AskPDF arXiv

Large language models can recover mid-generation from task-misaligned activation steering, producing explicit verbal restarts (e.g., ``wait, that's not right'') and continuing on-topic even while the steering perturbation remains active. We term this Endogenous Steering Resistance (ESR). Using sparse autoencoder (SAE) latents to steer model activations, we find that Llama-3.3-70B exhibits explicit ESR at 3.8%, with smaller models from the Llama-3 and Gemma-2 families showing the explicit form less frequently. Two controls dissociate ESR into a detection event and a sustained-resistance component that conditioning on recent on-topic tokens does not fully explain. We identify 26 SAE latents through contrastive on-topic/off-topic search; zero-ablating them reduces the multi-attempt rate by 25%, with random-latent and held-out-prompt controls supporting specificity. ESR can also be deliberately enhanced through both meta-prompting and fine-tuning on synthetic self-correction examples. ESR has dual implications for safety: it could harden models against adversarial activation-space manipulation, but may equally interfere with beneficial steering-based interventions, since the model has no way to distinguish the two. Code is available at https://github.com/agencyenterprise/endogenous-steering-resistance.

Abstract

LLM Interpretability

Explore similar work

Endogenous Resistance to Activation Steering in Language Models

Understanding Confabulation and Rethinking Reconstruction in Activation Explanations

Non-Gaussianity Reveals Interpretable Directions in Language Models