Building Better Activation Oracles

Jan Bauer; Celeste De Schamphelaere; Adam Karvonen; Niclas Luick; Neel Nanda

cs.LGMay 23, 2026

Building Better Activation Oracles

Authors: Jan Bauer, Celeste De Schamphelaere, Adam Karvonen, Niclas Luick, Neel Nanda

Organizations: MATS · Gatsby Unit, UCL · Ghent University · Independent · University of Hamburg

Abstract

Activation Oracles (AOs) are promising methods for interpreting residual stream activations. However, current AOs face important issues, such as hallucinations and vagueness. Additionally, text-inversion confounds make them hard to evaluate. To this end, we improve the Activation Oracle (AO) training regime in four ways: training on on-policy rollouts, improving the conversational dataset, feeding more layers and an improvement to the injection formula. The capability improvements are marginal, but quality of life improvements are quite substantial. In addition, we open source the first comprehensive evaluation suite for AO quality, which we call AObench. Overall, we hope that our work sets a foundation that helps improve AOs and other models in the paradigm of scalable, end-to-end interpretability.

Explore similar work

When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles
Jul 25, 2026Tobias Bersia, Tatiana GaintsevaModel Activations Knowledge Gaps
Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals
May 25, 2026Federico Torrielli, Peter Schneider-Kamp, Lukas Galke PoechModel Activations Frequency
Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agent
Jul 17, 2026Sriram Balasubramanian, Soheil FeiziModel Interpretability Model Activations

Jul 25, 2026 · cs.CLJ/K move · Enter open · S save

When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles

Tobias Bersia, Tatiana Gaintseva

BAISH · Queen Mary University of London

Activation Oracles (AOs) are language models trained to answer natural-language questions about another model's internal activations. They offer a flexible interface for reading hidden information from model states, especially when relevant information is internally represented but absent or incomplete in visible behavior. However, AOs are themselves learned systems: their answers are shaped by training data, objectives, and learned reporting behavior, rather than being neutral readouts of represented information. We study this in a controlled Taboo Word Guessing setting, where subject models are fine-tuned to internally use a hidden concept while avoiding direct disclosure. Contrary to the expectation that an AO trained on such a subject becomes a specialist reader, we find that fine-tuned AOs can become concept-specific anti-readers: they selectively fail to recover the concept persistently present during their own training. This failure is not simply explained by absence of the concept from the subject or oracle representations: the target remains decodable inside the oracle, while LogitLens and layer-ablation analyses indicate that the failure arises in the AO readout pathway. Our results show that behavioral leakage, representation-level decodability, and AO-verbalizability can come apart, raising a reliability concern for learned interpretability interfaces.

Model Activations Knowledge Gaps Representation-Level Certification Reading Verbalization Blind Spots

Open paper Ask about it PDF

Abstract

Explore similar work

When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles

Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals

Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agent