cs.LGAug 19, 2026

Mechanistic Tomography: Designed Measurement for Control-Oriented Interpretability

Authors: Vijay Erramilli

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Mechanistic interpretability seeks quantities that models do not expose directly: represented states, component effects, interactions, and responses to interventions. Patching, gradients, Hessian-vector products, and subset interventions provide different measurements under different access assumptions and may target different quantities. We formulate their shared measurement structure as mechanistic tomography: designed measurement for recovering internal mechanisms and intervention effects. For a chosen basis and intervention family, measurements take the form y = Ax + w, where A describes the interventions, x is the target map, and w contains nonlinear response, sampling error, and basis misspecification. This language gives a practical procedure: start with the least costly measurements, test on held-out interventions at the intended scale, calibrate simple mismatch, and expand the measurement family when structured residuals remain. Control provides a demanding validation setting because an estimate that guides an intervention acts as an observer. In a two-HMM model, control error rises with observer error, while target improvement can hide nuisance-state movement. Under forward-only access, sparse aggregate measurements recover a finite-effect map with fewer interventions than coordinate patching. With gradient access, finite probes improve a local attribution map. Lifted measurements and Hessian-vector products recover interactions missed by first-order maps, while Tracr shows that the required family depends on the basis. On GPT-2-small IOI, the Name Mover-Negative Name Mover interaction is the largest held-out predictive term among three tested cross-group pairs. On Qwen-2.5-7B, finite calibration makes an additive refusal-response map adequate, so held-out error does not support pairwise lifting.

Figures & tables

Method and accessDeclared targetInduced measurement
Coordinate patching; forwardfinite singleton effectsone-coordinate rows
Subset recovery; forwardadditive finite mapmulti-coordinate rows; joint fit
AtP; backwardlocal first-order map xxgradient returns coordinates
Lifted recovery; forwardfinite main and pair effectscolumns include aia_{i} and aiaja_{i}a_{j}
HVP; white-box second orderlocal curvature H0H_{0}directions give curvature projections
Table 1. Mechanistic tomography maps each method to a target, access path, and measurement design.
Figure 1. With the controller and actuator fixed, closed-loop target error rises with observer error across ten observer variants. Oracle and probe lie low-left; coarse, noisy, proxy, and entangled estimates move up and right.
Figure 2. Aggregate recovery of the 32-coordinate finite-effect map. OMP scores Pearson r=0.989r=0.989 and held-out R2=0.935R^{2}=0.935 with 12 aggregate measurements. Ridge reaches comparable held-out prediction with 32 measurements, the exhaustive coordinate count.
Scaleraw AtP R2R^{2}calibrated AtP R2R^{2}OMP R2R^{2}fitted gain g^\widehat{g}
ε=5\varepsilon=50.943 [0.920, 0.967]0.963 [0.933, 0.983]0.965 [0.952, 0.976]0.874 [0.799, 0.915]
ε=8\varepsilon=80.818 [0.751, 0.903]0.960 [0.925, 0.978]0.969 [0.960, 0.977]0.742 [0.677, 0.780]
Table 2. Finite-scale calibration on one trained model whose intervention directions are estimated once. We cross three independently generated 256-sequence evaluation batches with three mask-and-split designs. Entries are means [min, max] across the nine cells.
Figure 3. Finite-scale calibration across the fixed-direction 3×33\times 3 design. Left: fitted gain at each scale for the nine evaluation-batch/design cells. Right: the median held-out R2R^{2} gap between OMP and calibrated AtP as the calibration budget grows; bands show the 10th–90th percentiles over cells and subset draws.
Figure 4. Interaction-aware recovery at ε=8\varepsilon=8 with a generous fixed budget. Lifted OMP recovers the planted pair-bearing response, while methods without pair features remain below 0.50.5 held-out R2R^{2} .
Figure 5. Measurement budget for lifted recovery. With four planted nonzero terms among 2,080 main-and-pair features, lifted OMP reaches the prediction and pair-recall thresholds with 64–72 aggregate measurements, compared with 2,016 exhaustive pair probes. First-order OMP has no pair features.
Figure 6. White-box interaction recovery. Designed HVPs recover the local pair map, and the quadratic harness supplies the known conversion to finite-scale pair coefficients. Accurate prediction also requires the AtP main-effect map.
Figure 7. Noise robustness with 96 aggregate measurements. Pair support can remain recoverable after held-out finite-response prediction degrades, separating support identification from coefficient accuracy.
Figure 8. Basis comparison in Tracr circuits. The current-token control is additive in both bases. The AB detectors are additive in the native compiler basis because it contains an explicit detector coordinate. In the restricted pre-conjunction basis, additive features explain only about half the variance, while lifted pair features recover the intended conjunction and restore held-out R2=1.0R^{2}=1.0 .
Figure 9. Detector-group writeback in Tracr circuits. Left: ablating the final-residual detector group destroys output prediction for the AB programs, while writing back the lifted reconstruction restores the original held-out R2R^{2} . Right: additive features only partially reconstruct the detector groups, whereas lifted features reconstruct them completely.
Pair (heads)mean ablation II [95%]zero ablation II [95%]II /pair (mean/zero)
P×BP\times B (24)1.080 [.912, 1.262]1.268 [1.013, 1.558]0.045/0.053
P×EP\times E (6)1.830 [1.569, 2.076]1.681 [1.474, 1.889]0.305/0.28
B×EB\times E (16).502 [0.429, 0.576]0.518 [0.416, 0.618]0.031/0.032
Table 3. Direct IOI group interactions across four lexical frames in ABBA and BABA order, with 128 shared matched name pairs. Intervals use a crossed bootstrap that resamples lexical frames and one shared name-pair vector while retaining both orders. The last column divides by the number of possible cross-group head pairs; it does not assign the same effect to every individual pair.
Figure 10. IOI interaction measurements. Left: direct group interactions on the original-template mean-ablation surface. Right: held-out gains from adding one count-product term to the count-additive model. P×EP\times E gives the largest of the three tested gains; the interval for B×EB\times E crosses zero.
Response mapparametersMAERMSER2R^{2}
Calibrated additive20.003790.004699.9829
Lifted pairwise48.003801.004613.9835
Table 4. Held-out prediction on the Qwen-2.5-7B refusal-response surface. The test contains 128 actions at the unseen scale 0.750.75 and 224 prompts.
RegimeAvailable accessStart withEscalate when
Additive, white-boxgradients and finite evaluationsAtP, then a few finite probesheld-out error does not reduce to a small gain correction
Additive, forward-onlyscalar intervention responsessparse subset measurementsthe map is dense or held-out prediction fails
Interactional, white-boxgradients, HVP queries, and finite evaluationsAtP plus HVPs for local curvaturelocal support fails, or finite-scale probes show that calibration is needed
Interactional, forward-onlyscalar intervention responseslifted subset measurementslifted coverage or conditioning fails
Uncertain basiscandidate coordinates and interventionssimplest declared basisresiduals persist or writeback fails
Downstream usefixed controller and actuatorlowest-dimensional validated observertarget, collateral, or transfer tests fail
Table 5. Observer selection as a measurement decision.
More in this topic

Mechanistic Interpretability

Mechanistic interpretability studies how neural networks implement behaviors through internal features, circuits, and causal information flow.

See all 391 papers

Explore similar work

CardsList
Sep 2, 2026

ObserverBench: Testing Mechanistic Estimates for Intervention and Control

Mechanistic interpretability is increasingly used to guide interventions such as activation steering, circuit removal, and safety monitoring. Yet an internal estimate that is accurate on average can still choose a poor action. We present ObserverBench, a benchmark framework for testing whether an internal estimator---an observer---is adequate for the intervention, control, or safety task it directs. Each task fixes the model, information boundary, allowed actions, decision rule, held-out cases, and loss. The benchmark reports estimation accuracy separately from the loss caused by the chosen action. Theory and experiments show why both are needed. In closed-loop control, observer errors matter at the starting point and along directions the allowed intervention can reach. On circuit-intervention tasks in GPT-2-small and Qwen2.5-7B, pairwise observers predict unseen effects more accurately without always choosing better actions; observers trained on action loss choose lower-loss actions. In safety triage, a score that perfectly separates violations can allocate a fixed intervention budget poorly when violations have different costs. Across Qwen2.5-7B, Gemma-2-9B-it, and prospectively frozen Qwen3.5-9B APPS tasks, AUROC can rank monitors differently from deployment loss, and the best information source changes across models. Sparse SAE readouts also trail their layer-matched dense controls on the reported Qwen panels, under disclosed activation-density or checkpoint mismatches. ObserverBench provides fixed task contracts, runnable baselines, and table-based submissions for evaluating interpretability methods through the actions they enable.

Vijay Erramilli

Mechanistic InterpretabilityAI Safety EvaluationInference-Time InterventionExplainability Evaluation
AskPDF
Jul 9, 2026

Certified Interventional Fidelity: Anytime-Valid, Adaptive Evaluation of Causal Claims in Mechanistic Interpretability

Mechanistic interpretability often evaluates explanations by intervening on a model: swapping hidden states, patching activations, ablating components, or comparing a compressed model to the original one. These experiments are usually summarized by a point estimate, even though the evaluation may be monitored while it runs or adapted toward suspected failures. This makes it hard to tell whether a reported fidelity or patching effect is a stable causal claim or a consequence of finite sampling and evaluation choices. We introduce Certified Interventional Fidelity (CIF), a statistical layer for interventional interpretability evaluations. CIF first writes the quantity being reported as a causal estimand: an expectation of a bounded score over a stated input distribution and a stated intervention distribution. It then provides confidence intervals and anytime-valid confidence sequences for this estimand, including under adaptive intervention sampling via bounded mixture importance weighting. We instantiate CIF with Hoeffding-style sequences and variance-adaptive betting sequences, the latter reducing certification cost by 10-30x in our experiments. On MNIST abstractions and GPT-2 Small IOI circuits, CIF certifies high-fidelity claims, shows when apparent method differences are not statistically supported, and makes sensitivity to the intervention distribution explicit.

Amir Asiaee

Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN 37232, USA

Mechanistic InterpretabilityCausal Interventions in Language ModelsExplainability EvaluationCausal Intervention
AskPDF
Jun 30, 2026

Representation as a Bottleneck for Mechanistic Interpretability: The Manifestation Unit Protocol

Mechanistic interpretability has produced a rich inventory of component-level analyses that characterise what neural-network components encode and how they interact. Their outputs, however, are not easily reusable: selectivity tables, circuit diagrams, and feature lists remain locked in per-study notebooks - non-composable, not queryable in natural language, and not directly actionable for downstream audit or intervention. We study the representation layer that sits between these analyses and downstream use as a bottleneck that can be evaluated independently, and introduce Manifestation Units, a typed tuple protocol (E, S, R, D, G) extended with attention-head primitives (T) for transformer architectures, organising per-component statistics into structured fields populated automatically and queried through hybrid retrieval. Instantiated across generative vision (beta-VAE), discriminative vision (CNN), and language (GPT-2), the protocol supports two findings: typed structure substantially outperforms unstructured baselines on retrieval, and CNN filters retrieved by the schema satisfy causal sufficiency and necessity criteria under matched-budget controls. The schema absorbs attention-head primitives without modification, set-recovers known IOI circuit members under retrieval-budget-matched controls, and reveals an irreducible two-field core (S+R) with remaining fields either redundant or actively interfering. We present this as schema infrastructure for mechanistic interpretability rather than frontier-scale validation.

Hussein Chouman, Wataru Sasaki, Tomokazu Matsui +2

Ubiquitous Computing Systems Laboratory, Nara Institute of Science and Technology, Nara, Japan.

Transformer InterpretabilityMechanistic InterpretabilityExplainability Evaluation
AskPDF