Natural Language Autoencoders (NLAs) are proposed as oversight tools for language models: an Activation Verbaliser (AV) writes an English explanation of an activation vector, and an Activation Reconstructor (AR) maps the explanation back to a vector. Their value for oversight depends on a human monitor being able to tell a true account of an activation from a false one, given the AV's explanation. We formalise NLA training and NLA oversight deployment as two different referential games from Emergent Communication: training is a generative referential game, while oversight is a discriminative, multi-modal, partially-observable referential game played over a many-to-many relation between the monitored language model's activations and the accounts that are true of them. The gap between the two games predicts that AV's explanations will identify which activation they describe without discriminating its true account from closely matched false ones, since training never asks explanations to rule anything out. Using the NLA's own AR as a maximally-sympathetic surrogate monitor model, we experimentally confirm this theoretical insight on one released checkpoint: true accounts beat accounts of other activations in all 320 comparisons, but beat fluent rivals differing in a single claim only 60.2% of the time (standard error 6.7%, chance at 50%, n=35). A meaning-preserving rewrite by an uninvolved model costs only 2.1% of the distance to another activation's account, so the failure is not due to the NLA pair having developed private conventions in the AV's explanations. Our rival accounts test only transcript-level truths, which the monitored language model's conversation settles, so these results bound the easiest case and leave claims about the monitored model's own processing untested. We state conjectures that would make both results consequences of the framework.
Figures & tables
Figure 1: Setting 1, the referential game a NLA trains on: the AR rebuilds h from the AV’s explanation z , scored by cosine distortion, but no candidate account is ever ruled out.
Figure 2: Setting 2, the referential game formalising the oversight context: a listener scores a lineup of candidate accounts, one true of h and the rest are distractors, which the AV never sees. The deployed question “is c true?” reduces to the threshold-free round {c,¬c} , which our experiments approximate with rival accounts comparison {c0,ck} .
Figure 3: Which accounts each monitor can settle: A monitor reading X settles the accounts in AX (Definition 3.6 ); one whose input includes the transcript can beat a transcript-only monitor only on the processing-level region AX∖Ainput . Our lineups sample Ainput only.
Figure 4: Identification without discrimination. Each row is one kind of distractor, and its value is the fraction of the 8 distractors that the true account outscores under the surrogate monitor model, averaged over conversations ( n per row in the labels, at most 40), with bars giving 95% intervals from resampling conversations; 0.5 is chance and 1 is perfect. Against accounts of other activations (top row) the true account wins every comparison, but against fluent rivals that falsify 1, 2 or 4 claims (purple) it wins only 0.56 to 0.64 of them, while mechanical word swaps with the same number of changed words (grey) sit at chance.
construction
fraction beaten at 1 changed word
SE
top- p 0.9
0.492
0.017
donor
0.490
0.023
donor + POS + absent
0.480
0.025
donor + POS + absent + distance
0.498
0.023
donor + POS
0.506
0.024
top- k
0.509
0.016
Table 1: Every mechanical construction sits at chance, and only the meaning-changing rival does not. Each row gives the fraction of distractors that the true account outscores when the distractor differs from it by one word, with its standard error over conversations. The six mechanical constructions (Section 4.2 ) replace one content word without guaranteeing a false claim; the last row is the judge-confirmed one-claim rival of Figure 4 .
Figure 5: A small edit moves the reconstruction far less than the AR’s own error. Left: the angle, in degrees on a log scale, between the reconstruction of an original explanation and that of an edited one, for edits replacing 1 to 16 tokens (“dose”), compared with the AR’s standing reconstruction error, the angle between two different conversations, and the angle between two random vectors; dots are means over conversations, bars ±1 standard error and lines the full range. Right: the same displacement against the number of corrupted tokens, with a power-law fit, which reaches the AR’s reconstruction error only at about 62 corrupted tokens.
Figure 6: What the rivals change, as a word cloud of the substitutions made by the rewriter across all conversations, sized by frequency. Two populations appear: genuine claim changes, such as trademark to copyright, hypertension to hypercholesterolemia, or requests.get to requests.post , and near-synonyms that leave the claim true, such as benefits to advantages. Because the second population does not falsify anything, the main results use only rivals that a validated judge confirms changed the meaning.
Figure 7: Rewording by an outside model barely damages the reconstruction, but reordering the same words does. Damage is the angle the reconstruction moves, as a fraction of the angle to the account of a different activation, so 0 is undamaged and 1 is as bad as describing something else; n=60 conversations. Top left: damage against the fraction of the original’s words removed, for each construction, showing that meaning-preserving paraphrases (blue) cost less than meaning-changing edits at matched surface change. Bottom left: the fraction of conversations whose damage stays at or below each value, read at 0.25. Right: per-condition damage, one dot per conversation, with interquartile bars and median ticks, showing that most paraphrases barely move the reconstruction and a few break it.
Natural Language Autoencoders (NLAs) express a model's internal activations in the form of readable text. The verbalizer turns an activation into an English explanation; the reconstructor then converts that explanation back into an activation, and the explanation is considered successful if the round trip brings back the original activation. We claim that this criterion evaluates whether the information is sufficient and does not address descriptive faithfulness. A sentence can contain sufficient information to make it possible to identify an activation even if it wrongly explains what the activation does: in the case of reconstruction, the primary aim is to accurately identify the activations and not to describe them. This issue has the same structure as the chain-of-thought faithfulness problem, in which verbal output that has been trained to go with a process is taken as a report on that process, and it parallels the dissociation between an individual's introspective report and the actual governing process as documented in human psychology. Although proposed as interpretability tools for the purpose of alignment, NLAs reintroduce, through unverified explanations, the epistemic problem that they were intended to resolve.
Activation verbalizers translate the internal representations of large language models into human-readable text. The recently proposed natural language autoencoders (NLAs) learn this by reconstructing activations through a natural language bottleneck. However, it remains unclear how well these methods generalize beyond the setting they were trained for. Here, we study this question for NLAs along four closely related questions: (i) Do NLA verbalizations exhibit compositional structure with respect to their input? (ii) Do NLAs generalize to finetuned versions of the model they were trained for? (iii) Can NLAs recover finetuning objectives in model diffing? (iv) Can NLAs be specialized for model diffing? Across four models from three families, our experiments provide evidence supporting a largely positive answer to all four questions. Specifically, verbalizations of composed analogy activations mostly name the target while rarely mentioning the subtracted word, and an NLA can be reused on lightweight finetunes of its original model without retraining, with reconstruction quality degrading noticeably only for the largest model. Verbalizing activation differences between Taboo model organisms and their original models recovers the concealed secret word for most organisms, and reinforcement learning on activation differences further improves this recovery on held-out organisms. We also find that NLAs can recover the finetuning objective without the finetuned model generating any token, since the most informative token positions precede the user prompt. NLAs do not yet match state-of-the-art model diffing methods on the emergent misalignment detection task, but outperform a blackbox baseline. Our findings suggest that NLAs can meaningfully verbalize differences in activation space, rendering NLAs a promising direction for future work.
Andrea Blasi Núñez, Gianluca Barmina, Federico Torrielli +3
Natural Language Autoencoders (NLAs) produce unsupervised text explanations of a model's activations: a verbalizer describes an activation and a reconstructor learns to recover it from this text. Under the established point-reconstruction NLA training recipe, explanations become more useful for predicting model behavior while also increasingly introducing unsupported details and exhibiting writing defects. To assess these changes separately, we introduce a standardized evaluation framework for unstructured NLA explanations, measuring information recoverable from explanations, contextual support for their claims, and writing quality. To address confabulation and writing defects, we move beyond predicting a single activation: explanations can distinguish distributions of possible activations even when their means and optimal point-reconstruction rewards are identical. We introduce Flow-NLA, which models the distribution of activations compatible with an explanation and trains the verbalizer using a diffusion likelihood bound. Across Qwen, Gemma, and Apertus, this richer signal retains the utility gains of point reconstruction while curbing the growth of confabulation and writing defects, opening up a direction for improving activation-derived training to encourage more informative, supported, and readable explanations. Code and evaluation prompts will be made publicly available upon acceptance.
Gert Lek, Zixuan Xia, Pin-Yu Chen +1
Universit´e de Neuchˆatel · Universität Bern · IBM Research +1