Activation verbalizers translate the internal representations of large language models into human-readable text. The recently proposed natural language autoencoders (NLAs) learn this by reconstructing activations through a natural language bottleneck. However, it remains unclear how well these methods generalize beyond the setting they were trained for. Here, we study this question for NLAs along four closely related questions: (i) Do NLA verbalizations exhibit compositional structure with respect to their input? (ii) Do NLAs generalize to finetuned versions of the model they were trained for? (iii) Can NLAs recover finetuning objectives in model diffing? (iv) Can NLAs be specialized for model diffing? Across four models from three families, our experiments provide evidence supporting a largely positive answer to all four questions. Specifically, verbalizations of composed analogy activations mostly name the target while rarely mentioning the subtracted word, and an NLA can be reused on lightweight finetunes of its original model without retraining, with reconstruction quality degrading noticeably only for the largest model. Verbalizing activation differences between Taboo model organisms and their original models recovers the concealed secret word for most organisms, and reinforcement learning on activation differences further improves this recovery on held-out organisms. We also find that NLAs can recover the finetuning objective without the finetuned model generating any token, since the most informative token positions precede the user prompt. NLAs do not yet match state-of-the-art model diffing methods on the emergent misalignment detection task, but outperform a blackbox baseline. Our findings suggest that NLAs can meaningfully verbalize differences in activation space, rendering NLAs a promising direction for future work.
Figures & tables
Figure 1: Fraction of verbalizations mentioning each analogy word, per token position and model, averaged over all subjects ( HR , K=4 ). Bars show, from left to right, Word1 , Word2 , Word3 , and the target Word4 .
Figure 2: Change in NLA reconstruction quality ( Δ FVE, finetune minus original model) on in-domain and neutral prompts, for the Taboo organisms of each model ( 2×2 grid) and the remaining Qwen2.5-7B finetunes (top right). Bars show 95% bootstrap confidence intervals over prompts; filled markers are significant after Holm correction within each panel ( p<0.05 ). The Qwen2.5-7B and Gemma Taboo panels share a y-axis. Italic values give the original model’s mean FVE on in-domain (orange) and neutral (blue) prompts, shown below each finetune for the top-right panel.
Median shift
Mean Δ FVE
Model
In-dom.
Neutral
In-dom.
Neutral
Qwen2.5-7B
0.300
0.088
+0.101
+0.091
Gemma-3-12B
0.263
0.097
−0.019
+0.045
Gemma-3-27B
0.516
0.101
−0.045
+0.048
Llama-3.3-70B
0.720
0.177
−0.309
−0.075
Table 1: Median activation shift and mean Δ FVE (finetune minus original model) across the 20 Taboo organisms of each model, on in-domain (Taboo) and neutral prompts.
Figure 3: HRany at each token position, averaged over all Taboo organisms, for Wsecret (dark bars) and Wtaboo (light bars). Within each position, bars show, from left to right, Qwen2.5-7B-Instruct, Gemma-3-12b-it, Gemma-3-27b-it, and Llama-3.3-70B-Instruct; the Gemma models have no system chat-template positions.
Figure 4: HRany on Wsecret for each Taboo organism (columns) and token position (rows), per model. Organisms are sorted by their mean hit rate across all models and positions. Grey rows mark the positions which do not exist for Gemma.
Organism
ADL
AO
NLA
Blackbox
Risky financial
4.20±0.54
4.73±0.39
2.93±0.13
1.93±0.65
Bad medical
2.93±0.13
2.00±0.21
2.80±0.40
1.87±0.27
Extreme sports
3.53±0.75
2.80±0.40
2.27±0.25
1.60±0.57
Mean (Int)
3.55 (0)
3.18 (0)
2.67 (0)
1.80 (5)
Table 2: EM audit grades (1–5, mean ± std). “Int” (interactions) represent the number of agent model queries. Each organism is scored 5 times per method. Results were rerun, not extracted from Minder et al. (2026)
Figure 5: HR on Wsecret for the validation organisms across training checkpoints and token positions, averaged over two seeds ( α=1 ). Top: Qwen2.5-7B-Instruct; bottom: Gemma-3-12b-it.
Target set
Avg
Min
Wanalogy
+ 6.1 / + 5.7
+ 0.9 / + 3.8
Wsecret
+ 26.4 / + 4.2
0.0 / − 11.9
Wtaboo
− 9.0 / + 9.4
− 56.4 / 0.0
Table 3: Change in test-set hit rate (percentage points) between the untrained NLA and the selected checkpoint, reported as the average, and minimum change across token positions. Each cell shows Qwen2.5-7B / Gemma-3-12B.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Subject
Pairs ( n )
n⋅(n−1)
Filtered
capital-common-countries
23
506
0
capital-world
116
13,340
0
city-in-state
68
4,556
314
currency
30
870
4
family
23
506
0
gram1-adjective-to-adverb
32
992
0
Appendix
Table 4: Candidate analogy combinations per subject, and filtered numbers.
Word4 mentioned (%)
Model
Position
Word3 (%)
All
w/o Word3
w/ Word3
w/o Word2 , Word3
Qwen2.5-7B
Probe word
44.6
79.8
83.6
75.0
90.9
im_start
27.7
53.7
48.9
66.2
53.2
role_name
2.4
1.5
1.2
16.2
1.2
Newline
6.5
5.2
4.2
20.1
4.2
Gemma-3-12B
Probe word
71.3
86.5
87.4
86.2
94.4
Appendix
Table 5: Target mention rates conditioned on whether the input words are mentioned, computed over individual verbalizations. For each model and token position, we report the fraction of verbalizations mentioning Word3 , and the fraction mentioning the target Word4 overall, without Word3 , with Word3 , and without either added input word ( Word2 or Word3 ).
Position
Qwen
Gemma 12B
Gemma 27B
Llama
Probe word
79.4
78.0
77.0
82.0
im_start
46.1
39.8
70.1
88.5
role_name
1.4
0.4
0.4
0.1
Newline
4.1
9.7
18.8
76.4
Appendix
Table 6: Target hit rate (%) of the composed vector, restricted to analogies for which verbalizing hWord3 alone never mentions the target Word4 .
Figure 6: Change in NLA reconstruction quality ( Δ FVE, finetune minus original model) against activation shift for the Taboo organisms, per model. Each point is one organism activation at a single prompt and token position, pooled over in-domain and neutral prompts. Black lines show least-squares fits, whose slope gives the FVE lost per unit of shift; grey lines mark Δ FVE =0 .
Figure 7: HRany on Wtaboo for each Taboo organism (columns) and token position (rows, grouped by prompt section), per model. Organisms are ordered as in Figure 4 . Grey rows mark the system chat-template positions, which do not exist for Gemma.
Figure 8: HR on Wtaboo for the validation organisms across training checkpoints and token positions, averaged over two seeds ( α=1 ). Top: Qwen2.5-7B-Instruct; bottom: Gemma-3-12b-it.
Figure 9: Cross-organism confusion of secret-word mentions. For each model, row i is the taboo organism whose Δ -activation (organism minus base) is verbalized, and column j is a secret word.
Split
Base pairs
Combinations
Rows
Train
405
14,920
29,840
Validation
84
588
1,176
Test
84
582
1,164
Total
573
16,090
32,180
Appendix
Table 7: Analogy dataset splits: base pairs, resulting distinct-word combinations, and extracted rows (two token positions per combination).
Figure 10: Wanalogy hit rate on validation set per token position. ( α=1 ).
Figure 11: HR on Wanalogy for the validation analogies across training checkpoints and token positions, for α=0.5 (left) and α=2 (right) (Qwen2.5-7B-Instruct).
Figure 12: HR on Wanalogy for the validation analogies across training checkpoints and token positions, with equal shares of Taboo and analogy training data ( α=0.5 , left) and with a penalty for mentioning other training organisms’ secret words ( α=1 , right), for Qwen2.5-7B-Instruct.
Figure 13: HR on Wsecret (top) and Wtaboo (bottom) for the validation organisms across training checkpoints and token positions (Qwen2.5-7B-Instruct, α=0.5 ).
Figure 14: HR on Wsecret (top) and Wtaboo (bottom) for the validation organisms across training checkpoints and token positions (Qwen2.5-7B-Instruct, α=2 ).
Figure 15: HR on Wsecret (top) and Wtaboo (bottom) for the validation organisms across training checkpoints and token positions, with equal shares of Taboo and analogy training data (Qwen2.5-7B-Instruct, α=0.5 ).
Figure 16: HR on Wsecret (top) and Wtaboo (bottom) for the validation organisms across training checkpoints and token positions, with an additional penalty for mentioning other training organisms’ secret words (Qwen2.5-7B-Instruct, α=1 ).
Figure 17: Percentage of each organism’s verbalizations (rows) that mention each secret word (columns) in the validation and test sets for Qwen2.5-7B-Instruct. The diagonal shows how often the correct word is named.
Figure 18: Percentage of each organism’s verbalizations (rows) that mention each secret word (columns) in the validation and test sets for Gemma-3-12b-it. The diagonal shows how often the correct word is named.
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
Natural Language Autoencoders (NLAs) express a model's internal activations in the form of readable text. The verbalizer turns an activation into an English explanation; the reconstructor then converts that explanation back into an activation, and the explanation is considered successful if the round trip brings back the original activation. We claim that this criterion evaluates whether the information is sufficient and does not address descriptive faithfulness. A sentence can contain sufficient information to make it possible to identify an activation even if it wrongly explains what the activation does: in the case of reconstruction, the primary aim is to accurately identify the activations and not to describe them. This issue has the same structure as the chain-of-thought faithfulness problem, in which verbal output that has been trained to go with a process is taken as a report on that process, and it parallels the dissociation between an individual's introspective report and the actual governing process as documented in human psychology. Although proposed as interpretability tools for the purpose of alignment, NLAs reintroduce, through unverified explanations, the epistemic problem that they were intended to resolve.
Natural Language Autoencoders (NLAs) are proposed as oversight tools for language models: an Activation Verbaliser (AV) writes an English explanation of an activation vector, and an Activation Reconstructor (AR) maps the explanation back to a vector. Their value for oversight depends on a human monitor being able to tell a true account of an activation from a false one, given the AV's explanation. We formalise NLA training and NLA oversight deployment as two different referential games from Emergent Communication: training is a generative referential game, while oversight is a discriminative, multi-modal, partially-observable referential game played over a many-to-many relation between the monitored language model's activations and the accounts that are true of them. The gap between the two games predicts that AV's explanations will identify which activation they describe without discriminating its true account from closely matched false ones, since training never asks explanations to rule anything out. Using the NLA's own AR as a maximally-sympathetic surrogate monitor model, we experimentally confirm this theoretical insight on one released checkpoint: true accounts beat accounts of other activations in all 320 comparisons, but beat fluent rivals differing in a single claim only 60.2% of the time (standard error 6.7%, chance at 50%, n=35). A meaning-preserving rewrite by an uninvolved model costs only 2.1% of the distance to another activation's account, so the failure is not due to the NLA pair having developed private conventions in the AV's explanations. Our rival accounts test only transcript-level truths, which the monitored language model's conversation settles, so these results bound the easiest case and leave claims about the monitored model's own processing untested. We state conjectures that would make both results consequences of the framework.
Natural Language Autoencoders (NLAs) produce unsupervised text explanations of a model's activations: a verbalizer describes an activation and a reconstructor learns to recover it from this text. Under the established point-reconstruction NLA training recipe, explanations become more useful for predicting model behavior while also increasingly introducing unsupported details and exhibiting writing defects. To assess these changes separately, we introduce a standardized evaluation framework for unstructured NLA explanations, measuring information recoverable from explanations, contextual support for their claims, and writing quality. To address confabulation and writing defects, we move beyond predicting a single activation: explanations can distinguish distributions of possible activations even when their means and optimal point-reconstruction rewards are identical. We introduce Flow-NLA, which models the distribution of activations compatible with an explanation and trains the verbalizer using a diffusion likelihood bound. Across Qwen, Gemma, and Apertus, this richer signal retains the utility gains of point reconstruction while curbing the growth of confabulation and writing defects, opening up a direction for improving activation-derived training to encourage more informative, supported, and readable explanations. Code and evaluation prompts will be made publicly available upon acceptance.
Gert Lek, Zixuan Xia, Pin-Yu Chen +1
Universit´e de Neuchˆatel · Universität Bern · IBM Research +1