How Do Linear Probes Emerge? A Circuit-Tracing Framework with Concept-Targeted Attribution

Vedant Palit; Florent Draye; Terry Jingchen Zhang; Bernhard Schölkopf; Zhijing Jin

cs.CLAug 27, 2026

How Do Linear Probes Emerge? A Circuit-Tracing Framework with Concept-Targeted Attribution

Authors: Vedant Palit, Florent Draye, Terry Jingchen Zhang, Bernhard Schölkopf, Zhijing Jin

Organizations: Jinesis Lab, University of Toronto & Vector Institute · EuroSafeAI · Max Planck Institute for Intelligent Systems, Tübingen, Germany · Hector Foundation ·

Abstract

Transcoder attribution graphs are usually trained to explain why a model assigns high probability to a particular next token. We introduce Concept-Targeted Attribution (CTA), which instead trains attribution graphs with respect to a linear probe direction. CTA therefore yields probe-specific circuits that explain why an internal concept representation arises in a prompt, independently of whether it is expressed in the generated token. Using Cross-Layer Transcoders, we show that these probe-targeted graphs contain predictive structure: graph-level features predict probe accuracy across four widely studied concept categories ( $ρ= 0.91$ , $R^2 = 0.84$ ), while local features identify the sparse components driving per-prompt classification. This connects probe performance to interpretable circuit structure, allowing us to ask not only whether a probe works, but which internal computations make it work. Causal ablations further show that probe-targeted and logit-targeted graphs capture functionally distinct mechanisms. Removing probe-relevant features reduces internal concept scores while largely preserving generated tokens, whereas removing logit-relevant features changes the generated token in 92% to 100% of cases with near-zero effect on probe scores. CTA provides a framework for moving from behavioral probe accuracy to mechanistic explanations of probe performance, enabling more detailed audits of internal concept representations, including safety-critical ones. Our code is available at https://github.com/vedantpalit/concept-targeted-attribution

Figures & tables

Figure 1: The CTA pipeline: CLT replaces MLP modules for interpretability; attribution graphs capture causal feature flow toward the probe; graph-structural features predict probe accuracy; CLT feature activations predict the probe margin.

Figure 2: Probe score trajectories across layers for all four concepts. Shaded bands mark the steepest single-step rise, indicating the belief point

L_{b}

where the residual stream most sharply aligns with the concept direction.

	Gemma-2-2B						Llama-3.2-1B
	Toxicity (L22)			Truthfulness (L14)			Toxicity (L14)			Truthfulness (L1)
Subspace	Post- $S$	$\boldsymbol{\Delta S}$	Flip	Post- $S$	$\boldsymbol{\Delta S}$	Flip	Post- $S$	$\boldsymbol{\Delta S}$	Flip	Post- $S$	$\boldsymbol{\Delta S}$	Flip
Clean	102.52	—	0.0%	41.87	—	0.0%	0.43	—	0.0%	0.11	—	0.0%
$G_{\text{probe}}\setminus G_{\text{logit}}$	41.28	$-$ 61.24	0.0%	29.84	$-$ 12.03	25.0%	$-$ 0.10	$-$ 0.53	0.0%	0.11	$\approx$ 0	0.0%
$G_{\text{logit}}\setminus G_{\text{probe}}$	101.82	0.70	100.0%	41.84	0.03	100.0%	0.43	$\approx$ 0	100.0%	0.09	$-$ 0.02	100.0%
$G_{\text{probe}}\cap G_{\text{logit}}$	$+$ 36.50	$-$ 66.02	38.0%	$+$ 2.93	$-$ 38.94	67.0%	$+$ 0.67	$+$ 0.24	78.0%	$+$ 0.08	$-$ 0.03	17.0%

Table 1: Causal dissociation for Toxicity and Truthfulness depicted at their respective

L_{b}

, implemented across Gemma-2-2B and Llama-3.2-1B. Ablating probe-exclusive nodes collapses

\Delta S

with near-zero behavioural flip while ablating logit-exclusive nodes flips generation in 100% of cases with negligible impact on

S

\Delta S

magnitudes differ between models due to scale, however the dissociation effects remain consistent across both. Extended results in Appendices 7 and D .

Figure 3: Distributions of five selected graph-structural features for high- versus low-accuracy probe layers, shown for Toxicity. The separation is statistically significant for each feature (

p<0.05

, independent

t

-test) and corresponds to medium-to-large effect sizes, with Cohen’s

d

ranging from

+0.88

+1.68

. These results indicate that CTA graphs contain a separable structural signal of a layer’s concept-encoding quality.

Model	Spearman $\rho$	$R^{2}$
Layer baseline	$0.08\pm 0.15$	$-0.19\pm 0.29$
Ridge	$0.65\pm 0.11$	$0.33\pm 0.27$
GB	$\mathbf{0.91}\pm 0.05$	$\mathbf{0.84}\pm 0.14$

Table 2: Predicting probe accuracy from graph-structural features using 5-fold group-aware cross-validation. Bootstrap 95% confidence intervals for GB are

\rho\in[0.724,0.964]

, with

p<0.005

under group-aware permutation.

Figure 4: Top activated examples for L6/F4508. This feature is present in CTAs of more than 50% of positive prompts from the SST-2 sentiment dataset, specifically activating in positive superlatives.

Figure 5: Top predictive features appear 6–20

\times

more frequently than a random feature at

L_{b}

, revealing a stable mechanistic core despite prompt-level circuit variation.

Figure 6: Consecutive-layer Jaccard

J(L,L-1)

for a representative prompt per concept, showing rapid circuit evolution in early layers followed by stabilization at

J\approx 0.9

by intermediate layers.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Probe validation accuracy across layers for Gemma-2-2B (left) and Llama-3.2-1B (centre). Toxicity is linearly decodable from early layers in both models; Sentiment and Truthfulness show weaker trajectories in Llama-3.2-1B, consistent with limited concept encoding at the 1B scale. Right: normalised probe score trajectories for Llama-3.2-1B, with shaded bands marking the belief layer

L_{b}

per concept.

	Gemma-2-2B		Llama-3.2-1B
Concept	$L_{b}$	Val acc.	$L_{b}$	Val acc.
Toxicity	22	90.0%	14	90.9%
Sentiment	21	80.1%	15	63.0%
Reasoning	15	87.6%	9	76.5%
Truthfulness	14	88.2%	1	55.2%

Appendix

Table 3: Probe accuracy at each concept’s belief layer

L_{b}

for both models. Sentiment and Truthfulness show lower ceilings in Llama-3.2-1B, consistent with weaker encoding of affective and factual concepts at the 1B scale.

	Toxicity (L22)				Sentiment (L21)				Reasoning (L15)				Truthfulness (L14)
Subspace	N	$S^{\prime}$	$\boldsymbol{\Delta S}$	Flip	N	$S^{\prime}$	$\boldsymbol{\Delta S}$	Flip	N	$S^{\prime}$	$\boldsymbol{\Delta S}$	Flip	N	$S^{\prime}$	$\boldsymbol{\Delta S}$	Flip
Control	—	$+$ 102.52	—	0%	—	$+$ 28.31	—	0%	—	$-$ 21.44	—	0%	—	$+$ 41.87	—	0%
$G_{\text{probe}}\setminus G_{\text{logit}}$	${\sim}$ 190	$+$ 41.28	$-$ 61.24	0%	${\sim}$ 203	$-$ 12.11	$-$ 40.42	0%	${\sim}$ 130	$-$ 16.39	$+$ 5.05	0%	${\sim}$ 168	$+$ 29.84	$-$ 12.03	25%
$G_{\text{logit}}\setminus G_{\text{probe}}$	${\sim}$ 347	$+$ 101.82	$-$ 0.70	100%	${\sim}$ 394	$+$ 28.70	$+$ 0.39	100%	${\sim}$ 539	$-$ 20.91	$+$ 0.53	92%	${\sim}$ 297	$+$ 41.84	$-$ 0.03	100%
$G_{\text{probe}}\cap G_{\text{logit}}$	${\sim}$ 166	$+$ 36.50	$-$ 66.02	38%	${\sim}$ 174	$+$ 0.57	$-$ 27.74	30%	${\sim}$ 195	$-$ 11.24	$+$ 10.20	17%	${\sim}$ 190	$+$ 2.93	$-$ 38.94	67%

Appendix

Table 4: Full causal ablation results across all four concepts on Gemma-2-2B (see Section 4.2 for the main study).

S^{\prime}

= post-ablation probe score;

\Delta S

= change from control. Reasoning’s

\Delta S

signs are inverted because the L15 probe is anti-aligned with all test prompts (baseline

S=-21.44

); ablation reduces magnitude rather than score, but the dissociation pattern holds: probe-exclusive ablation yields

0\%

flip, logit-exclusive yields

92\%

flip with

\Delta S\approx 0

. The logit-exclusive subgraph is

1.77

–

4.15\times

larger than the probe-exclusive subgraph across concepts (Reasoning’s

4.15\times

ratio reflects sparse probe encoding at the anti-aligned token position).

Model	Raw		De-meaned
	$\rho$	$R^{2}$	$\rho$	$R^{2}$
Layer baseline	$+0.083$	$-0.197$	$+0.587$	$+0.240$
Ridge	$+0.655$	$+0.325$	$+0.819$	$+0.648$
GB	$+0.905$	$+0.835$	$+0.925$	$+0.850$

Appendix

Table 5: 5-fold group-aware CV results for Channel 2 (Gemma-2-2B).

	Raw	De-meaned
Observed $\rho$	$+0.890$	$+0.908$
95% CI on $\rho$	$[+0.724,\,+0.964]$	$[+0.817,\,+0.962]$
Null mean $\rho$	$-0.005\pm 0.128$	$-0.001\pm 0.122$
$p(\rho)$	$<0.005$	$<0.005$

Appendix

Table 6: Bootstrap 95% CI (500 resamples) and permutation test (200 shuffles) for Channel 2 GB model (Gemma-2-2B).

Figure 8: Spearman

\rho

between predicted and actual probe score margins across layers for all four concepts. Toxicity maintains consistently high predictability across all layers; Truthfulness peaks at mid-depth before decaying; Sentiment and Reasoning show weak and noisy signals throughout, reflecting distributed or geometrically encoded concept representations that individual CLT features cannot recover.

Figure 9: Top predictive features appear 8–21

\times

more frequently than a random feature at highest validation accuracy layer, revealing a stable mechanistic core despite prompt-level circuit variation.

Figure 10: Mean per-prompt Jaccard similarity $J(T{@}L_{T},\,R{@}L_{R})$ between Truthfulness and Reasoning attribution graphs across all layer-pair combinations. Cross-concept overlap is non-negligible only when both layers are in the same depth regime.

Panel of Figure 11 — Figure 11: Representative CLT features identified by the probe attribution pipeline, visualized via Neuronpedia. Each panel shows the feature’s layer, index, mean activation, and top/bottom predicted tokens, with activating contexts highlighted. The features encode semantically coherent concepts: L9/F3223 fires on content keywords in structured titles; L9/F2578 and L9/F14198 specialize in geographic place names and city tokens respectively; L1/F3007 captures French proper names and French-language text; and L9/F15693 activates on common English function words across diverse domains. The semantic breadth of these features illustrates that probe-predictive circuits draw on general linguistic primitives alongside concept-specific representations.

Toxicity (L14)

Sentiment (L15)

Reasoning (L9)

Truthfulness (L1)

Subspace

S^{\prime}

\boldsymbol{\Delta S}

Flip

S^{\prime}

\boldsymbol{\Delta S}

Flip

S^{\prime}

\boldsymbol{\Delta S}

Flip

S^{\prime}

\boldsymbol{\Delta S}

Flip

Control

—

+0.43

—

-0.47

—

+0.60

—

+0.11

—

G_{\text{probe}}\setminus G_{\text{logit}}

{\sim}

244

-0.10

\mathbf{-0.53}

{\sim}

168

+2.40

\mathbf{+2.87}

{\sim}

115

+0.39

\mathbf{-0.21}

{\sim}

+0.11

\mathbf{-0.001}

G_{\text{logit}}\setminus G_{\text{probe}}

{\sim}

203

+0.43

+0.002

100%

{\sim}

289

-0.48

-0.011

100%

{\sim}

372

+0.57

-0.028

100%

{\sim}

359

+0.09

-0.020

100%

G_{\text{probe}}\cap G_{\text{logit}}

{\sim}

169

+0.67

+0.24

78%

{\sim}

112

+0.45

+0.92

25%

{\sim}

206

-0.33

-0.93

58%

{\sim}

+0.08

-0.026

17%

Appendix

Table 7: Full causal ablation results across all four concepts on Llama-3.2-1B.

S^{\prime}

= post-ablation probe score;

\Delta S

= change from control. Sentiment baseline is negative as the probe encodes positive sentiment and all prompts are negative-sentiment text; ablating probe-exclusive features correctly reduces the magnitude. Truthfulness at

L_{1}

shows a near-zero internal effect due to the weak probe circuit (

\sim

11 nodes,

55.2\%

accuracy) at that layer; the logit dissociation holds cleanly (

100\%

flip).

Model	$\rho$	$R^{2}$
Layer baseline	$-0.437$	$-0.660$
Ridge	$+0.423$	$+0.069$
GB	$+0.808$	$+0.850$

Appendix

Table 8: 5-fold group-aware CV results for Channel 2 (Llama-3.2-1B).

	Raw
Observed $\rho$	$+0.780$
95% CI on $\rho$	$[+0.368,\,+0.955]$
Null mean $\rho$	$-0.011\pm 0.179$
$p(\rho)$	$<0.005$

Appendix

Table 9: Bootstrap 95% CI (500 resamples) and permutation test (200 shuffles) for Channel 2 GB model (Llama-3.2-1B).

Panel of Figure 12 — Figure 12: Gemma-2-2B : Distributions of five selected graph-structural features for high- versus low-accuracy probe layers, shown for sentiment. The separation is statistically significant for each feature ( $p<0.05$ , independent $t$ -test) and corresponds to medium-to-large effect sizes, with Cohen’s $d$ ranging from $+0.93$ to $+2.59$ . These results indicate that CTA graphs contain a separable structural signal of a layer’s concept-encoding quality.

Panel of Figure 13 — Figure 13: Gemma-2-2B : Distributions of five selected graph-structural features for high- versus low-accuracy probe layers, shown for reasoning. The separation is statistically significant for each feature ( $p<0.05$ , independent $t$ -test) except the influence concentration and corresponds to medium to large effect sizes, with Cohen’s $d$ ranging from $+0.69$ to $+2.00$ . These results indicate that CTA graphs contain a separable structural signal of a layer’s concept-encoding quality.

Panel of Figure 14 — Figure 14: Gemma-2-2B : Distributions of five selected graph-structural features for high- versus low-accuracy probe layers, shown for truthfulness. The separation is not purely significant for features ( $p<0.05$ , independent $t$ -test) except graph density and corresponds to low to medium effect sizes, with Cohen’s $d$ ranging from $-0.84$ to $+0.59$ . These results indicate that CTA graphs contain a separable structural signal of a layer’s concept-encoding quality.

Panel of Figure 15 — Figure 15: Llama-3.2-1B : Distributions of five selected graph-structural features for high- versus low-accuracy probe layers, shown for Sentiment. No feature reaches significance at $p<0.05$ , and effect sizes are small, with Cohen’s $d$ ranging from $-0.98$ to $+0.58$ , consistent with the weak within-concept structural separation observed for Sentiment in Llama-3.2-1B.

Panel of Figure 16 — Figure 16: Llama-3.2-1B : Distributions of five selected graph-structural features for high- versus low-accuracy probe layers, shown for Sentiment. No feature reaches significance at $p<0.05$ , and effect sizes are small, with Cohen’s $d$ ranging from $-0.98$ to $+0.58$ , consistent with the weak within-concept structural separation observed for Sentiment in Llama-3.2-1B.

Panel of Figure 17 — Figure 17: Llama-3.2-1B : Distributions of five selected graph-structural features for high- versus low-accuracy probe layers, shown for Reasoning. Maximum logit edge and graph density reach significance ( $p<0.005$ ), with large effect sizes ( $d=-2.00$ and $d=-1.73$ respectively); the remaining features do not. Cohen’s $d$ ranges from $-2.00$ to $+0.84$ . Negative $d$ values indicate that lower-accuracy layers exhibit higher feature values, a pattern inverted relative to Gemma-2-2B.

Panel of Figure 18 — Figure 18: Llama-3.2-1B : Distributions of five selected graph-structural features for high- versus low-accuracy probe layers, shown for Truthfulness. No feature reaches significance at $p<0.05$ , and effect sizes are small, with Cohen’s $d$ ranging from $-0.40$ to $+0.71$ , reflecting the near-flat val_acc trajectory and weak concept encoding of Truthfulness at the 1B scale.

Mechanistic Interpretability

Mechanistic interpretability studies how neural networks implement behaviors through internal features, circuits, and causal information flow.

See all 391 papers

Oct 14, 2025

Attribution Graphs and Causal Probing for Mechanistic Discovery and Bias Repair in Multimodal Generative Learning

We treat the internals of generative models as mechanistic objects rather than black boxes. We introduce Attribution Graphs (AGs), which extend GradCAM++ to circuit-level representations, and Causal Probing, a do-calculus intervention method for identifying causal latent structures, enabling detection and correction of spurious correlations, demographic biases, and misaligned decision circuits during training. We further propose the Cognitive Alignment Score (CAS), quantifying agreement between model-internal representations and human concepts, a saliency-first privacy mechanism sharing only thresholded attribution nodes, a bias-aware regularizer aligning subgroup statistics, and a Reveal-to-Revise loop integrating attribution signals into parameter updates without separate fine-tuning. Evaluated on CelebA, FairFace, Jigsaw, and HateXplain, our method achieves 94.1% accuracy, 92.3% macro F1, 79.4% IoU-XAI, and 12.7 FID at 72--76% adversarial robustness, while reducing subgroup disparity

Δ_{\mathrm{bias}}

by 41%, demonstrating that mechanistic interpretability, fairness, and generative performance can be jointly optimized.

Noor Islam S. Mohammad, Uluğ Bayazıt

1,2Istanbul Technical University

Gradient-Based Attribution Causal Representation Learning Algorithmic Fairness Causal Attribution Mechanistic Interpretability Algorithmic Bias+2

Ask PDF

Apr 8, 2026

ADAG: Automatically Describing Attribution Graphs

In language model interpretability research, circuit tracing aims to identify which internal features causally contributed to a particular output and how they affected each other, with the goal of explaining the computations underlying some behaviour. However, all prior circuit tracing work has relied on ad-hoc human interpretation of the role that each feature in the circuit plays, via manual inspection of data artifacts such as the dataset examples the component activates on. We introduce ADAG, an end-to-end pipeline for describing these attribution graphs which is fully automated. To achieve this, we introduce attribution profiles which quantify the functional role of a feature via its input and output gradient effects. We then introduce a novel clustering algorithm for grouping features, and an LLM explainer--simulator setup which generates and scores natural-language explanations of the functional role of these feature groups. We run our system on known human-analysed circuit-tracing tasks and recover interpretable circuits, and further show ADAG can find steerable clusters which are responsible for a harmful advice jailbreak in Llama 3.1 8B Instruct.

Aryaman Arora, Zhengxuan Wu, Jacob Steinhardt +1

Stanford University · Transluce

Mechanistic Interpretability Circuits in Language Models Feature Attribution

Ask PDF

Sep 22, 2026

Matryoshka attribution: Learning to attribute language model outputs to representations and weights

Attributing language model outputs to their internal computations is an open problem in interpretability. Existing methods, which use causal interventions, gradients, or learnable masks, either are infeasibly expensive or struggle to identify actual causally-important internal computations. We propose framing attribution as the problem of identifying nested subsets of internal components which minimise a downstream loss. To learn this task, we introduce Matryoshka Attribution (MAttr), a mask learning method that parametrises the mask with a simple differentiable sigmoid top-

k

operator. We supervise training over all sparsities simultaneously by randomising

k

over training, resulting in a learned ordering of components by attribution score. MAttr achieves number 1 on the official leaderboard of the Mechanistic Interpretability Benchmark (Mueller et al., 2025); our method identifies sparse and task-transferrable circuits across varying circuit bases. As a practical application, we show that MAttr can be trained with reinforcement learning to identify weight changes responsible for downstream behaviours in LLM finetuning. We train MAttr on refusal judge scores and find that restoring

1\%

of Llama 3.1 8B Instruct's weights to their base model state is sufficient to remove refusals while maintaining capabilities. We view MAttr as a successful formulation of interpretability into a learnable objective that we can tackle with gradient descent, and encourage future work along these lines.

Aryaman Arora, Kirill Acharya, Nathan Hu +4

Stanford University

LLM Interpretability Circuits in Language Models

Ask PDF

Abstract

Mechanistic Interpretability

Explore similar work

Attribution Graphs and Causal Probing for Mechanistic Discovery and Bias Repair in Multimodal Generative Learning

ADAG: Automatically Describing Attribution Graphs

Matryoshka attribution: Learning to attribute language model outputs to representations and weights