Calibrated concept profiles: making the reading of a model's neurons a measurement
Organizations: University College London
Abstract
Interpretability has largely moved from reading a model's neurons directly to reading a second model trained on the first: a probe, a sparse autoencoder, a crosscoder. The move answered real faults -- mixed meanings, thresholds without a unit, readings that did not transfer from one study to the next -- but gave up what a direct reading alone provides: nothing trained in between, the neurons themselves named, and a reading that can be repeated and compared. Here the direct reading is made a measurement. A neuron's activation varies with the prompt, so its reading is a distribution over a population of stimuli. We declare that population: every factor around a target construct is drawn at its rate in a reference corpus and logged. From the declared distribution we generate a battery of prompts, and draw it twice. From the battery we compute the calibrated concept profile, one number per neuron, stated in units of the neuron's own spread on ordinary input; the two draws give every number a noise unit. Before the profile is read, a fixed set of checks licenses the battery on the model, and the smallest change the profile can resolve across training checkpoints is measured. Realised for Python on OpenCoder-1.5B and followed across checkpoints spanning two trillion training tokens, the profile gives what the alternatives do not report: a unit on every number, a chance level for every comparison, and which neurons carry a construct at every checkpoint. The battery, the code and the data behind every result are released.
Figures & tables
| factor of a stimulus | template | corpus sample | declared distribution |
|---|---|---|---|
| the target construct in its role | fixed | present, uncontrolled | fixed (set A) |
| the target’s token out of role | rarely provided | uncontrolled | fixed (set B) |
| identifiers and literals | fixed by the template | uncontrolled | drawn at corpus rates, logged |
| scope and nesting | fixed | uncontrolled | drawn at corpus rates, logged |
| companion constructs | fixed or absent | uncontrolled | drawn at corpus rates, matched between A and B, logged |
| length and surrounding code | fixed | uncontrolled | drawn at corpus rates, logged |
| check | what it asks | what its outcome restricts |
|---|---|---|
| 1. twin agreement | do two independently drawn batteries give the same profile? | any statement about which neurons carry the construct |
| 2. convergence | after how many prompts does the profile stop changing? | readings from fewer prompts than the convergence point |
| 3. length | does lengthening a prompt change the profile? | comparisons across prompts of different length, for the sensitive constructs |
| 4. re-weighting | does re-weighting the logged factors to other proportions change the profile? | comparisons across different distributions |
| 5. calibration | do the stated intervals cover the twin difference at their nominal rate? | the width of intervals: inflated by a stated factor |
| 6. companion | do A and B carry the same companion constructs, within a cap? | statements about the construct alone: its profile includes the companion difference |
| choice | what depends on it | how far it is measured |
|---|---|---|
| 1. Two twins : two draws of the battery, sharing their cores | every statement "beyond noise"; the twin ceiling; the resolution | twin agreement (check 1); intervals from the core bootstrap (§4.8). Not measured: more draws, or draws on independent cores |
| 2. The generator : cores and axes, menus of names, how B is built, companion matching | what the profile is a profile of | the mixture (check 4: median 2.34 twin units); companions (check 6: not matched for compound statements). Not measured: generated prompts against real code |
| 3. The yardstick : token class fifth of the prompt; set C; the floor on 0 | which neurons are named; size in standardised units | coarser yardsticks name different neurons (§4.3); the floor (check 8); length (check 3). Not measured: other splits, computed exactly |
| 4. The reference corpus : public Python, not the training data | what "ordinary" and "at corpus rate" mean | not measured: how far public Python differs from the model’s 607-language mixture |
| 5. Naming rules : the inventory, the read site, the gate, the top-5% carrier set | which constructs are read; which neurons are the carriers | the gate’s failures (19 of 188, §3.1); the cut does not change carrier turnover ( Wilam, 2026 ) . Not measured: another read site |
| 6. The read point : MLP neurons, one position, short prompts (median 55 tokens) | everything read | not measured: longer contexts, attention, the residual stream |
| # | check | ran on | outcome |
|---|---|---|---|
| 1 | twin agreement | 188 | median cosine 0.986 between the twins’ profiles |
| 2 | convergence | 31 | cosine 0.99 with the full set reached by 56 of 720 prompts (median) |
| 3 | length | 31 | a median 2.4% of a construct’s neurons length-sensitive; more than 5% for eight statements ( With 55%, For 49%) |
| 4 | re-weighting | 188 | designed arm only; the re-weighted profile sits a median 2.34 twin units from the base, within one twin unit for 4 of 188 — the profile moves with the mixture |
| 5 | calibration | 188 | calibrated at the median; intervals optimistic for 33 constructs, by a stated factor |
| 6 | companion | 8 | the generator’s matching is not met: the largest difference between A’s and B’s share of a companion is 0.19–0.68 (median 0.37) against the design’s cap of 0.15; removing the difference, where it can be removed, moves the profile by 2.95 twin units ( For ) and 5.50 ( With ) |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| object | count |
|---|---|
| inventory: grammar entries + builtin names | 63 + 143 = 206 constructs |
| built / not built | 188 / 18 |
| pass the read-site gate (P 1 ) | 169 of 188 |
| gate failures: structural / keyword tokens | 14 / 5 |
| reference corpus | 253,398 parsed Python files |
| prompts per A and per B set, per twin | 720 (8 cores 90), all 188 constructs |
| check | what is read | ran on | median | lowest | highest |
|---|---|---|---|---|---|
| gate | share of prompts whose A and B read sites land on the same token piece (pass: 0.80 or above) | 188 | 169 pass, 19 fail (B.2) | 0.250 ( BinOp , Call , Compare , List ); among those that pass, 0.873 ( NamedExpr ) | 1.000 |
| 1 twin agreement | cosine between the two twins’ profiles | 188 | 0.9861 (0.9889) | 0.7419 ( UnaryOp ); among the 169, 0.8771 ( None ) | 0.9980 ( NamedExpr ) |
| 2 convergence | prompts needed for a cosine of 0.99 with the profile on all 720, read on a grid of 8, 24, 56, 120 and 176 | 31 | 56 | 24 ( Delete , NotImplemented , With , ord ) | 120 ( For , If , ImportFrom , ResourceWarning ) |
| 3 length index | the length index of the profile | 31 | 0.75 | 0.41 ( IfExp , ListComp ) | 3.36 ( With ) |
| 3 flagged share | share of the construct’s neurons flagged as length-sensitive | 31 | 0.024 | 0.002 ( IfExp , SetComp ) | 0.554 ( With ) |
| 4 re-weighting, cosine | cosine between the profile with the logged factors re-weighted and the base profile | 188 | 0.9095 (0.9135) | 0.6923 ( BinOp ); among the 169, 0.7852 ( Pass ) | 0.9943 ( Global ) |
| construct | concept norm (layer 12) | L2 whole | L2 pad-0 | max whole | max pad-0 | companions |
|---|---|---|---|---|---|---|
| ClassDef | 53.06 | 1.064 | 1.311 | 0.388 | 0.646 | 33 |
| For | 50.54 | 0.644 | 0.780 | 0.228 | 0.358 | 36 |
| FunctionDef | 36.12 | 0.812 | 0.863 | 0.530 | 0.526 | 34 |
| If | 44.93 | 0.919 | 0.989 | 0.678 | 0.688 | 35 |
| Match | 47.65 | 0.395 | 0.464 | 0.186 | 0.261 | 32 |
| Try | 70.75 | 0.572 | 0.729 | 0.266 | 0.524 | 33 |
| reference | 20-cell split | one pooled cell |
|---|---|---|
| 4.19 B tokens | +0.08 (13) | +0.25 (15) |
| 83.9 B | +0.27 (14) | +0.51 (14) |
| 2,034 B | +0.40 (14) | +0.73 (14) |
| category | constructs | purity |
|---|---|---|
| grammar element against builtin name | — | recovered |
| exception classes | 69 | 1.000 |
| warning classes | 12 | 1.000 |
| builtin functions | 43 | 0.86 |
| builtin types | 26 | 0.73 |
| comprehensions | 4 | 1.000 (p = 0.004) |
| in the paper | file | script |
|---|---|---|
| Tables 1–3 (the factors, the nine checks, the six choices) | — (text) | — |
| Table 4 (the licence as run) | LICENCE_TABLE.json , APPENDIX_B1_checks.md | licence_table.py , licence_tables.py |
| Appendix B.1–B.2, the gate and checks 1, 4, 5, 8 | TABLE_B1.csv | licence_tables.py |
| Appendix B.1 and D, check 4 in twin units | CHECK4_RELATIVE.{csv,json} | check4_relative.py |
| Appendix B.1–B.2, checks 2, 3, 7a, 7b | TABLE_B2.csv ; RECHECK_PER_PROMPT.json (check 2’s prompt counts) | licence_tables.py |
| Appendix B.1, the two null ratios | NULL_RATIOS.json (the layer-12 real norm, each null’s p95 and the ratio) | licence_tables.py |
| stage | script | output |
|---|---|---|
| 1. inventory | split_p1.py | inventory/inventory_v2.json , validation/battery/SPLIT.json |
| 2. corpus calibration (the declared list ) | build_calibration.py | calibration/*.json |
| 3. battery | build_battery.py | phase_battery/ , CONTENT_HASHES.json |
| 4. recording | run_s15.py , run_s16.py ( run_colab.py for a hosted GPU) | record/<ref>/ |
| 5. licence | licence_table.py , licence_tables.py | LICENCE_TABLE.json , Tables B.1–B.3 |
LLM Interpretability
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
See all 824 papers