Verbalizing Activation Differences with Natural Language Autoencoders

Andrea Blasi Núñez; Gianluca Barmina; Federico Torrielli; Thiusius Rajeeth Savarimuthu; Peter Schneider-Kamp; Lukas Galke Poech

cs.CLOct 22, 2026

Verbalizing Activation Differences with Natural Language Autoencoders

Authors: Andrea Blasi Núñez, Gianluca Barmina, Federico Torrielli, Thiusius Rajeeth Savarimuthu, Peter Schneider-Kamp, Lukas Galke Poech

Organizations: University of Southern Denmark

Abstract

Activation verbalizers translate the internal representations of large language models into human-readable text. The recently proposed natural language autoencoders (NLAs) learn this by reconstructing activations through a natural language bottleneck. However, it remains unclear how well these methods generalize beyond the setting they were trained for. Here, we study this question for NLAs along four closely related questions: (i) Do NLA verbalizations exhibit compositional structure with respect to their input? (ii) Do NLAs generalize to finetuned versions of the model they were trained for? (iii) Can NLAs recover finetuning objectives in model diffing? (iv) Can NLAs be specialized for model diffing? Across four models from three families, our experiments provide evidence supporting a largely positive answer to all four questions. Specifically, verbalizations of composed analogy activations mostly name the target while rarely mentioning the subtracted word, and an NLA can be reused on lightweight finetunes of its original model without retraining, with reconstruction quality degrading noticeably only for the largest model. Verbalizing activation differences between Taboo model organisms and their original models recovers the concealed secret word for most organisms, and reinforcement learning on activation differences further improves this recovery on held-out organisms. We also find that NLAs can recover the finetuning objective without the finetuned model generating any token, since the most informative token positions precede the user prompt. NLAs do not yet match state-of-the-art model diffing methods on the emergent misalignment detection task, but outperform a blackbox baseline. Our findings suggest that NLAs can meaningfully verbalize differences in activation space, rendering NLAs a promising direction for future work.

Figures & tables

Figure 1: Fraction of verbalizations mentioning each analogy word, per token position and model, averaged over all subjects (

\mathrm{HR}

K=4

). Bars show, from left to right,

\mathrm{Word_{1}}

\mathrm{Word_{2}}

\mathrm{Word_{3}}

, and the target

\mathrm{Word_{4}}

Figure 2: Change in NLA reconstruction quality (

\Delta

FVE, finetune minus original model) on in-domain and neutral prompts, for the Taboo organisms of each model (

2\times 2

grid) and the remaining Qwen2.5-7B finetunes (top right). Bars show 95% bootstrap confidence intervals over prompts; filled markers are significant after Holm correction within each panel (

p<0.05

). The Qwen2.5-7B and Gemma Taboo panels share a y-axis. Italic values give the original model’s mean FVE on in-domain (orange) and neutral (blue) prompts, shown below each finetune for the top-right panel.

	Median shift		Mean $\Delta$ FVE
Model	In-dom.	Neutral	In-dom.	Neutral
Qwen2.5-7B	$0.300$	$0.088$	$+0.101$	$+0.091$
Gemma-3-12B	$0.263$	$0.097$	$-0.019$	$+0.045$
Gemma-3-27B	$0.516$	$0.101$	$-0.045$	$+0.048$
Llama-3.3-70B	$0.720$	$0.177$	$-0.309$	$-0.075$

Table 1: Median activation shift and mean

\Delta

FVE (finetune minus original model) across the 20 Taboo organisms of each model, on in-domain (Taboo) and neutral prompts.

Figure 3:

\mathrm{HR}_{\mathrm{any}}

at each token position, averaged over all Taboo organisms, for

\mathcal{W}_{\text{secret}}

(dark bars) and

\mathcal{W}_{\text{taboo}}

(light bars). Within each position, bars show, from left to right, Qwen2.5-7B-Instruct, Gemma-3-12b-it, Gemma-3-27b-it, and Llama-3.3-70B-Instruct; the Gemma models have no system chat-template positions.

Figure 4: $\mathrm{HR}_{\mathrm{any}}$ on $\mathcal{W}_{\text{secret}}$ for each Taboo organism (columns) and token position (rows), per model. Organisms are sorted by their mean hit rate across all models and positions. Grey rows mark the positions which do not exist for Gemma.

Organism	ADL	AO	NLA	Blackbox
Risky financial	$4.20{\pm}0.54$	$4.73{\pm}0.39$	$2.93{\pm}0.13$	$1.93{\pm}0.65$
Bad medical	$2.93{\pm}0.13$	$2.00{\pm}0.21$	$2.80{\pm}0.40$	$1.87{\pm}0.27$
Extreme sports	$3.53{\pm}0.75$	$2.80{\pm}0.40$	$2.27{\pm}0.25$	$1.60{\pm}0.57$
Mean (Int)	$3.55$ (0)	$3.18$ (0)	$2.67$ (0)	$1.80$ (5)

Table 2: EM audit grades (1–5, mean

{\pm}

std). “Int” (interactions) represent the number of agent model queries. Each organism is scored 5 times per method. Results were rerun, not extracted from Minder et al. (2026)

Figure 5:

\mathrm{HR}

\mathcal{W}_{\text{secret}}

for the validation organisms across training checkpoints and token positions, averaged over two seeds (

\alpha=1

). Top: Qwen2.5-7B-Instruct; bottom: Gemma-3-12b-it.

Target set	Avg	Min
$\mathcal{W}_{\text{analogy}}$	$+$ 6.1 / $+$ 5.7	$+$ 0.9 / $+$ 3.8
$\mathcal{W}_{\text{secret}}$	$+$ 26.4 / $+$ 4.2	0.0 / $-$ 11.9
$\mathcal{W}_{\text{taboo}}$	$-$ 9.0 / $+$ 9.4	$-$ 56.4 / 0.0

Table 3: Change in test-set hit rate (percentage points) between the untrained NLA and the selected checkpoint, reported as the average, and minimum change across token positions. Each cell shows Qwen2.5-7B / Gemma-3-12B.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Subject	Pairs ( $n$ )	$n\cdot(n-1)$	Filtered
capital-common-countries	23	506	0
capital-world	116	13,340	0
city-in-state	68	4,556	314
currency	30	870	4
family	23	506	0
gram1-adjective-to-adverb	32	992	0

Appendix

Table 4: Candidate analogy combinations per subject, and filtered numbers.

			$\mathrm{Word_{4}}$ mentioned (%)
Model	Position	$\mathrm{Word_{3}}$ (%)	All	w/o $\mathrm{Word_{3}}$	w/ $\mathrm{Word_{3}}$	w/o $\mathrm{Word_{2}}$ , $\mathrm{Word_{3}}$
Qwen2.5-7B	Probe word	44.6	79.8	83.6	75.0	90.9
	im_start	27.7	53.7	48.9	66.2	53.2
	role_name	2.4	1.5	1.2	16.2	1.2
	Newline	6.5	5.2	4.2	20.1	4.2
Gemma-3-12B	Probe word	71.3	86.5	87.4	86.2	94.4

Appendix

Table 5: Target mention rates conditioned on whether the input words are mentioned, computed over individual verbalizations. For each model and token position, we report the fraction of verbalizations mentioning

\mathrm{Word_{3}}

, and the fraction mentioning the target

\mathrm{Word_{4}}

overall, without

\mathrm{Word_{3}}

, with

\mathrm{Word_{3}}

, and without either added input word (

\mathrm{Word_{2}}

\mathrm{Word_{3}}

Position	Qwen	Gemma 12B	Gemma 27B	Llama
Probe word	79.4	78.0	77.0	82.0
im_start	46.1	39.8	70.1	88.5
role_name	1.4	0.4	0.4	0.1
Newline	4.1	9.7	18.8	76.4

Appendix

Table 6: Target hit rate (%) of the composed vector, restricted to analogies for which verbalizing

h_{\mathrm{Word_{3}}}

alone never mentions the target

\mathrm{Word_{4}}

Figure 6: Change in NLA reconstruction quality ( $\Delta$ FVE, finetune minus original model) against activation shift for the Taboo organisms, per model. Each point is one organism activation at a single prompt and token position, pooled over in-domain and neutral prompts. Black lines show least-squares fits, whose slope gives the FVE lost per unit of shift; grey lines mark $\Delta$ FVE $=0$ .

Figure 7: $\mathrm{HR}_{\mathrm{any}}$ on $\mathcal{W}_{\text{taboo}}$ for each Taboo organism (columns) and token position (rows, grouped by prompt section), per model. Organisms are ordered as in Figure 4 . Grey rows mark the system chat-template positions, which do not exist for Gemma.

Figure 8:

\mathrm{HR}

\mathcal{W}_{\text{taboo}}

for the validation organisms across training checkpoints and token positions, averaged over two seeds (

\alpha=1

). Top: Qwen2.5-7B-Instruct; bottom: Gemma-3-12b-it.

Figure 9: Cross-organism confusion of secret-word mentions. For each model, row $i$ is the taboo organism whose $\Delta$ -activation (organism minus base) is verbalized, and column $j$ is a secret word.

Split	Base pairs	Combinations	Rows
Train	405	14,920	29,840
Validation	84	588	1,176
Test	84	582	1,164
Total	573	16,090	32,180

Appendix

Table 7: Analogy dataset splits: base pairs, resulting distinct-word combinations, and extracted rows (two token positions per combination).

Figure 10:

\mathcal{W}_{\text{analogy}}

hit rate on validation set per token position. (

\alpha=1

Figure 11:

\mathrm{HR}

\mathcal{W}_{\text{analogy}}

for the validation analogies across training checkpoints and token positions, for

\alpha=0.5

(left) and

\alpha=2

(right) (Qwen2.5-7B-Instruct).

Figure 12:

\mathrm{HR}

\mathcal{W}_{\text{analogy}}

for the validation analogies across training checkpoints and token positions, with equal shares of Taboo and analogy training data (

\alpha=0.5

, left) and with a penalty for mentioning other training organisms’ secret words (

\alpha=1

, right), for Qwen2.5-7B-Instruct.

Figure 13:

\mathrm{HR}

\mathcal{W}_{\text{secret}}

(top) and

\mathcal{W}_{\text{taboo}}

(bottom) for the validation organisms across training checkpoints and token positions (Qwen2.5-7B-Instruct,

\alpha=0.5

Figure 14:

\mathrm{HR}

\mathcal{W}_{\text{secret}}

(top) and

\mathcal{W}_{\text{taboo}}

(bottom) for the validation organisms across training checkpoints and token positions (Qwen2.5-7B-Instruct,

\alpha=2

Figure 15:

\mathrm{HR}

\mathcal{W}_{\text{secret}}

(top) and

\mathcal{W}_{\text{taboo}}

(bottom) for the validation organisms across training checkpoints and token positions, with equal shares of Taboo and analogy training data (Qwen2.5-7B-Instruct,

\alpha=0.5

Figure 16:

\mathrm{HR}

\mathcal{W}_{\text{secret}}

(top) and

\mathcal{W}_{\text{taboo}}

(bottom) for the validation organisms across training checkpoints and token positions, with an additional penalty for mentioning other training organisms’ secret words (Qwen2.5-7B-Instruct,

\alpha=1

Figure 17: Percentage of each organism’s verbalizations (rows) that mention each secret word (columns) in the validation and test sets for Qwen2.5-7B-Instruct. The diagonal shows how often the correct word is named.

Figure 18: Percentage of each organism’s verbalizations (rows) that mention each secret word (columns) in the validation and test sets for Gemma-3-12b-it. The diagonal shows how often the correct word is named.

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 781 papers

Natural Language Autoencoders Identify Without Explaining
Oct 23, 2026Kristina Šekrst, Barbara GeldLLM Interpretability Faithfulness of Language Model Explanations
On the Monitorability of Natural Language Autoencoders: Identification vs Discrimination
Oct 13, 2026Kevin Yandoka DenamganaïLanguage Model Auditing Autoencoders
Understanding Confabulation and Rethinking Reconstruction in Activation Explanations
Sep 27, 2026Gert Lek, Zixuan Xia, Pin-Yu Chen +1Transformer Interpretability Faithfulness of Language Model Explanations
Universit´e de Neuchˆatel · Universität Bern · IBM Research +1

Oct 23, 2026J/K move · Enter open · S save

Natural Language Autoencoders Identify Without Explaining

Kristina Šekrst, Barbara Geld

LLM Interpretability Faithfulness of Language Model Explanations Autoencoders Faithfulness of Language Model Reasoning

Open paper AskPDF arXiv

Natural Language Autoencoders (NLAs) express a model's internal activations in the form of readable text. The verbalizer turns an activation into an English explanation; the reconstructor then converts that explanation back into an activation, and the explanation is considered successful if the round trip brings back the original activation. We claim that this criterion evaluates whether the information is sufficient and does not address descriptive faithfulness. A sentence can contain sufficient information to make it possible to identify an activation even if it wrongly explains what the activation does: in the case of reconstruction, the primary aim is to accurately identify the activations and not to describe them. This issue has the same structure as the chain-of-thought faithfulness problem, in which verbal output that has been trained to go with a process is taken as a report on that process, and it parallels the dissociation between an individual's introspective report and the actual governing process as documented in human psychology. Although proposed as interpretability tools for the purpose of alignment, NLAs reintroduce, through unverified explanations, the epistemic problem that they were intended to resolve.

Abstract

LLM Interpretability

Explore similar work

Natural Language Autoencoders Identify Without Explaining

On the Monitorability of Natural Language Autoencoders: Identification vs Discrimination

Understanding Confabulation and Rethinking Reconstruction in Activation Explanations