cs.CLNov 6, 2026

Calibrated concept profiles: making the reading of a model's neurons a measurement

Authors: Piotr Wilam

Organizations: University College London

Abstract

Interpretability has largely moved from reading a model's neurons directly to reading a second model trained on the first: a probe, a sparse autoencoder, a crosscoder. The move answered real faults -- mixed meanings, thresholds without a unit, readings that did not transfer from one study to the next -- but gave up what a direct reading alone provides: nothing trained in between, the neurons themselves named, and a reading that can be repeated and compared. Here the direct reading is made a measurement. A neuron's activation varies with the prompt, so its reading is a distribution over a population of stimuli. We declare that population: every factor around a target construct is drawn at its rate in a reference corpus and logged. From the declared distribution we generate a battery of prompts, and draw it twice. From the battery we compute the calibrated concept profile, one number per neuron, stated in units of the neuron's own spread on ordinary input; the two draws give every number a noise unit. Before the profile is read, a fixed set of checks licenses the battery on the model, and the smallest change the profile can resolve across training checkpoints is measured. Realised for Python on OpenCoder-1.5B and followed across checkpoints spanning two trillion training tokens, the profile gives what the alternatives do not report: a unit on every number, a chance level for every comparison, and which neurons carry a construct at every checkpoint. The battery, the code and the data behind every result are released.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Select, Hypothesize and Verify: Towards Verified Neuron Concept Interpretation

    Mar 26, 2026ZeBin Ji, Yang Hu, Xiuli Bi +2Neuron-Level InterpretabilityNeural Network Interpretability

  2. Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agent

    Jul 17, 2026Sriram Balasubramanian, Soheil FeiziMechanistic InterpretabilityNeural Network Interpretability