Selecting The Most Informative Tokens in Natural Language Autoencoders

Federico Torrielli; Gianluca Barmina; Andrea Blasi Núñez; Amon Rapp; Luigi Di Caro; Peter Schneider-Kamp; Lukas Galke Poech

cs.CLSep 29, 2026

Selecting The Most Informative Tokens in Natural Language Autoencoders

Authors: Federico Torrielli, Gianluca Barmina, Andrea Blasi Núñez, Amon Rapp, Luigi Di Caro, Peter Schneider-Kamp, Lukas Galke Poech

Organizations: Department of Computer Science, University of Turin, Italy · Department of Mathematics and Computer Science, University of Southern Denmark, Denmark

Abstract

Natural language autoencoders translate a language model's internal activations into readable explanations. Explaining every token position is costly. Which positions should an auditor inspect to understand a potential threat? We study this question across $4.7$ million explanations on prompt injection and concealment. We compare signals from model computation with a ranker trained only on chat structure. Chat structure usually selects more relevant explanations than the computational signals, without requiring a model forward pass for position selection. On three of four datasets, explaining just $5\%$ of positions retains nearly all of the success rate from explaining every position, where success means obtaining an explanation about the threat. The benefit varies with the audit task. We also show that pretrained verbalizers recover words that models have learned to conceal through fine-tuning, without additional verbalizer training. These results identify where auditors can concentrate explanation generation and show that useful explanations can extend beyond the model a verbalizer was trained to describe.

Figures & tables

Signal	Definition	Elevated at positions where
predictive distribution
surprisal	$-\log p_{t}(x_{t})$	the observed token was improbable
entropy	$H(p_{t})$	the continuation was uncertain
varentropy	$\operatorname{Var}_{p_{t}}[-\log p_{t}]$	the uncertainty is concentrated on few alternatives
temporal_kl	$D_{\mathrm{KL}}(p_{t}\,\\|\,p_{t-1})$	the observed token changed the prediction
attention pattern

Table 1: The thirteen signals. Section 2.1 defines the symbols.

Figure 1: On-task explanations by dataset and model. (a) Share of positions the judge calls on-task. (b) On-task rate inside the threat span over the rate outside it.

	OpenPromptInjection				Tensor Trust				Liars’ Bench		Taboo organisms
signal	q7	g12	g27	l70	q7	g12	g27	l70	g27	l70	q7	g12	g27	l70
predictive distribution
surprisal	0.511	0.506	0.495	0.513	0.522	0.629	0.599	0.612	0.599	0.548	0.522	0.504 ∘	0.571	0.491 ∘
entropy	0.372	0.397	0.452	0.451	0.510 ∘	0.598	0.589	0.611	0.700	0.609	0.566	0.557	0.604	0.522
varentropy	0.366	0.391	0.442	0.424	0.498 ∘	0.583	0.582	0.583	0.712	0.596	0.551	0.601	0.622	0.535
temporal_kl	0.606	0.508	0.509	0.482	0.627	0.648	0.601	0.650	0.396	0.375	0.499 ∘	0.500 ∘	0.402	0.515 ∘

Table 2: Pooled AUROC of every signal against the on-task label. Bold is the entry furthest from

0.5

in each column; ∘ fails Benjamini-Hochberg control at

q=0.05

Panel of Figure 2 — Figure 2: Dataset-shared positional relevance. Rows are models, columns are token position, and color is $\bar{P}_{R}$ .

Figure 3: Audit success against the number of explanations read, as a share of verbalizing every position. Top row: at least one on-task explanation. Bottom row: at least three.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Model	Positions	Base rate	Lift, all positions	Lift, within role
Open PromptInjection	q7	$111{,}370$	$0.254$	$+0.163$ $[+0.157,+0.169]$	$\mathbf{+0.328}$ $[+0.322,+0.333]$
	g12	$107{,}750$	$0.170$	$+0.099$ $[+0.094,+0.104]$	$\mathbf{+0.206}$ $[+0.201,+0.210]$
	g27	$107{,}750$	$0.184$	$+0.112$ $[+0.107,+0.118]$	$\mathbf{+0.240}$ $[+0.236,+0.245]$
	l70	$126{,}340$	$0.129$	$+0.169$ $[+0.165,+0.173]$	$\mathbf{+0.227}$ $[+0.223,+0.232]$
Taboo organisms	q7	$7{,}310$	$0.212$	n/a	n/a
	g12	$5{,}260$	$0.276$	n/a	n/a

Appendix

Table 3: Base rate and localization of on-task explanations.

Strategy	Qwen2.5-7B	Gemma-3-12B	Gemma-3-27B	Llama-3.3-70B
naive	$+0.292$ $[+0.282,+0.303]$	$+0.173$ $[+0.165,+0.182]$	$+0.207$ $[+0.199,+0.215]$	$+0.212$ $[+0.203,+0.220]$
escape	$+0.306$ $[+0.294,+0.318]$	$+0.171$ $[+0.163,+0.180]$	$+0.209$ $[+0.201,+0.218]$	$+0.201$ $[+0.193,+0.210]$
fake_comp	$+0.316$ $[+0.305,+0.327]$	$+0.203$ $[+0.194,+0.212]$	$+0.239$ $[+0.230,+0.247]$	$+0.208$ $[+0.199,+0.216]$
ignore	$+0.357$ $[+0.346,+0.368]$	$\mathbf{+0.243}$ $[+0.235,+0.253]$	$+0.254$ $[+0.245,+0.263]$	$+0.247$ $[+0.239,+0.256]$
combine	$\mathbf{+0.361}$ $[+0.351,+0.372]$	$+0.231$ $[+0.223,+0.240]$	$\mathbf{+0.286}$ $[+0.277,+0.295]$	$\mathbf{+0.263}$ $[+0.254,+0.271]$

Appendix

Table 4: OpenPromptInjection lift by attack strategy, within the user message. Largest per model in bold.

			All positions		Shared prompt		Own word, by concealed word
Model	Positions	On-task	Own word	Other	Own	Other	moon	ship	snow
q7	$7{,}310$	$0.212$	$\mathbf{0.198}$ $[0.170,0.227]$	$0.0192$	$0.122$	$0.0282$	$0.262$	$0.150$	$0.186$
g12	$5{,}260$	$0.276$	$\mathbf{0.258}$ $[0.221,0.295]$	$0.0015$	$0.222$	$0.0034$	$0.249$	$0.262$	$0.262$
g27	$5{,}531$	$0.300$	$\mathbf{0.268}$ $[0.227,0.310]$	$0.0002$	$0.253$	$0.0000$	$0.263$	$0.194$	$0.343$
l70	$8{,}583$	$0.162$	$\mathbf{0.143}$ $[0.120,0.167]$	$0.0015$	$0.135$	$0.0027$	$0.162$	$0.100$	$0.166$

Appendix

Table 5: Secret-word recovery per model and per concealed word.

Figure 4: Secret-word recovery on the twelve taboo organisms.

OpenPromptInjection

Tensor Trust

Liars’ Bench

Taboo organisms

signal

g12

g27

l70

g12

g27

l70

g27

l70

g12

g27

l70

Agreeing

Minority

predictive distribution

surprisal

2/4

entropy

4/4

varentropy

3/4

temporal_kl

2/4

Appendix

Table 6: Direction of every signal in every cell. H means larger values select on-task positions and L means smaller values do.

signal	Correlation	Pooled	Within structure
predictive distribution
surprisal	$-0.50$ to $+0.07$	$0.546$	$0.581$
entropy	$-0.49$ to $+0.07$	$0.585$	$0.607$
varentropy	$-0.59$ to $+0.05$	$0.589$	$0.609$
temporal_kl	$-0.12$ to $+0.35$	$0.572$	$0.564$
attention pattern

Appendix

Table 7: Sequence position against prediction, over the fourteen cells.

Dataset	Model	Signal	Direction	Pooled [95% CI]	Case-macro	Within structure	Free structure
OpenPromptInjection	q7	peak_ratio	lower	$0.761$ $[0.757,0.764]$	$0.765$	$0.723$	$0.786$
	g12	resid_jump_nla	higher	$0.685$ $[0.681,0.689]$	$0.683$	$0.668$	$0.759$
	g27	resid_jump_nla	higher	$0.635$ $[0.630,0.639]$	$0.625$	$0.612$	$0.774$
	l70	sink_drain	higher	$0.673$ $[0.669,0.678]$	$0.675$	$0.598$	$0.819$
Tensor Trust	q7	resid_jump	higher	$0.639$ $[0.631,0.647]$	$0.539$	$0.603$	$0.649$
	g12	norm_ratio	higher	$0.719$ $[0.709,0.729]$	$0.523$	$0.697$	$0.709$

Appendix

Table 8: The strongest signal of every dataset and model, beside the two free rankers.

			Signal		Ensemble		Structure
Dataset	Model	Base	$1$	$8$	$1$	$8$	$1$	$8$
OpenPromptInjection	q7	$0.254$	$0.532$	$0.524$	$0.388$	$0.482$	$0.620$	$0.571$
	g12	$0.170$	$0.000$	$0.208$	$0.400$	$0.360$	$0.501$	$0.491$
	g27	$0.184$	$0.455$	$0.299$	$0.588$	$0.533$	$0.286$	$0.570$
	l70	$0.129$	$0.263$	$0.207$	$0.242$	$0.278$	$0.544$	$0.433$
Taboo organisms	q7	$0.212$	$0.000$	$0.542$	$0.594$	$0.618$	$0.542$	$0.602$

Appendix

Table 9: Precision at a budget of one and of eight explanations per transcript.

Dataset	Model	Signal 1	Signal 2	Held-out $\Delta$	$95\%$ CI
OpenPromptInjection	q7	lookback_ratio	sink_drain	$0.0303$	$[0.0251,0.0356]$
	g12	lookback_ratio	sink_drain	$0.1337$	$[0.1287,0.1389]$
	g27	lookback_ratio	sink_drain	$0.1941$	$[0.1875,0.2012]$
	l70	peak_ratio	sink_drain	$0.0595$	$[0.0567,0.0623]$
Tensor Trust	q7	resid_jump	w	$0.1116$	$[0.1061,0.1174]$
	g12	norm_ratio	resid_jump	$0.0413$	$[0.0377,0.0450]$

Appendix

Table 10: Model-best ensembles.

			Complete-data pooled		Held-out pooled		Held-out case-macro
Dataset	Model	Ensemble	Ens.	Single	Ens.	Single	Ens.	Single
Open Prompt- Injection	q7	lookback_ratio / sink_drain $(0.50/0.50)$	$0.7856$	$0.7606$	$0.7856$	$0.7605$	$0.7948$	$0.7645$
	g12	lookback_ratio / sink_drain $(0.50/0.50)$	$0.8098$	$0.6850$	$0.8098$	$0.6850$	$0.8162$	$0.6825$
	g27	lookback_ratio / sink_drain $(0.50/0.50)$	$0.8101$	$0.6347$	$0.8101$	$0.6347$	$0.8196$	$0.6255$
	l70	peak_ratio / sink_drain $(0.50/0.50)$	$0.7292$	$0.6734$	$0.7292$	$0.6734$	$0.7343$	$0.6749$
Tensor Trust	q7	resid_jump / w $(0.50/0.50)$	$0.6754$	$0.6389$	$0.6754$	$0.6390$	$0.6504$	$0.5388$

Appendix

Table 11: Every AUROC estimate for the model-best ensembles.

Dataset	Shared ensemble	Full-data pooled	Held-out pooled	Held-out case-macro
OpenPromptInjection	lookback_ratio + sink_drain $50/50$	$0.7540$	$0.7540$	$0.7613$
Tensor Trust	resid_jump_nla + w $75/25$	$0.6824$	$0.6824$	$0.5669$
Liars’ Bench	dominant_mass + head_disagreement $50/50$	$0.7216$	$0.7205$	$0.5407$
Taboo organisms	dominant_mass + norm_ratio $50/50$	$0.7496$	$0.7497$	$0.7752$

Appendix

Table 12: Every AUROC estimate for the dataset-shared ensembles.

Panel of Figure 5 — Figure 5: Segment-level relevance under model-best selection.

Panel of Figure 6 — Figure 6: Model-best positional relevance profiles, one ensemble selected per dataset and model.

Panel of Figure 7 — Figure 7: Segment-level relevance of the dataset-shared selectors, one line per model.

Panel of Figure 8 — Figure 8: Dataset-shared positional relevance profiles, the curve form of Figure 2 .

Explore similar work

Understanding Confabulation and Rethinking Reconstruction in Activation Explanations
Sep 27, 2026Gert Lek, Zixuan Xia, Pin-Yu Chen +1Model Activations Explainable AI Methods
Universit´e de Neuchˆatel · Universität Bern · IBM Research +1
Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations
Jul 22, 2026Hiskias DingetoVerbalization Interpretability
StackOne Technologies
Measurement Under Selection: Decoy-Calibrated Failure Audits for Language Models
Jun 8, 2026Vyzantinos Repantis, Ameya Gawde, Harshvardhan SinghLarge Language Models Fail Model Auditing
The Educational Approaches to Virtual Reality Technologies Lab, University of Ioannina, Greece · Meta Platforms

Sep 27, 2026, cs.CLJ/K move · Enter open · S save

Understanding Confabulation and Rethinking Reconstruction in Activation Explanations

Gert Lek, Zixuan Xia, Pin-Yu Chen, Lydia Y. Chen

Universit´e de Neuchˆatel · Universität Bern · IBM Research+1

Model Activations Explainable AI Methods Autoencoder Architectures Confabulation

Open paper AskPDF arXiv

Natural Language Autoencoders (NLAs) produce unsupervised text explanations of a model's activations: a verbalizer describes an activation and a reconstructor learns to recover it from this text. Under the established point-reconstruction NLA training recipe, explanations become more useful for predicting model behavior while also increasingly introducing unsupported details and exhibiting writing defects. To assess these changes separately, we introduce a standardized evaluation framework for unstructured NLA explanations, measuring information recoverable from explanations, contextual support for their claims, and writing quality. To address confabulation and writing defects, we move beyond predicting a single activation: explanations can distinguish distributions of possible activations even when their means and optimal point-reconstruction rewards are identical. We introduce Flow-NLA, which models the distribution of activations compatible with an explanation and trains the verbalizer using a diffusion likelihood bound. Across Qwen, Gemma, and Apertus, this richer signal retains the utility gains of point reconstruction while curbing the growth of confabulation and writing defects, opening up a direction for improving activation-derived training to encourage more informative, supported, and readable explanations. Code and evaluation prompts will be made publicly available upon acceptance.

Abstract

Explore similar work

Understanding Confabulation and Rethinking Reconstruction in Activation Explanations

Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations

Measurement Under Selection: Decoy-Calibrated Failure Audits for Language Models