Localizing behavior to individual components of a language model is a central goal of mechanistic interpretability. However, scoring components one at a time misses context-dependent effects: a primary component can inhibit the activation of a backup, leading to issues with ranking components. Actual causality studies the structure of such interactions via witnesses: variables that provide contextual information to resolve interaction terms. However, estimation with witnesses typically requires combinatorial enumeration and is infeasible in practice. We introduce the witness-integrated set effect (WISE), a family of causal estimands that build on the witness mechanism while taking expectations over sets of causes and witnesses to remain computationally feasible. Building on this approach, we introduce JuntaLearner, a gradient-based circuit discovery method that learns to rank components by their causal impact across varying-sized sets of components and witnesses. Alongside faithfulness metrics, we introduce measures of necessity and task specificity, and the circuit recognition score (CRS) to summarize each metric across circuit sizes while emphasizing effects achieved by small circuits. Across tasks and models of increasing size, JuntaLearner achieves higher mean CRS compared to attribution baselines on all metrics. Since its cost does not grow with the number of candidate components, JuntaLearner scales to large models while accounting for set-level interactions and avoiding first-order approximations.
Figures & tables
Figure 1: Interactions and evaluation on GPT-2/IOI. Left: performance of JuntaLearner , compared to the signed per-head NIE and MAttr, across a combination of our metrics. Center: as some task (IOI) is successfully “killed” (by removing the estimated 10 -head IOI circuit), other tasks may also take collateral damage; this is captured in our “net” metric. Right: MAttr faithfulness on GPT-2/IOI incurs significant “overshoot”; CPR rewards this overshoot, while CMD penalizes it. Meanwhile, CRS rewards early discovery by measuring area under the log-curve.
Figure 2: Witness interactions in the GPT-2 IOI circuit. The Name Mover group plays a primary role in the GPT-2 IOI circuit, but its backups obscure this role under the standard NIE; witness-integration recovers the intuitive result that the primaries (NM) rank above the backups (BU).
Figure 3: Two faithfulness curves on Qwen3-1.7B-Base, SVA, at MIB’s ten log-spaced proportions. CPR and 1−CMD prefer the late, overshooting curve; CRS prefers the early one.
Figure 4: Every (model, task) cell, scored by CRS. Per-model default is one configuration per model, chosen by median per-cell score; Tuned is, for each cell and each goal, the JuntaLearner configuration with the highest CRS on that goal (mean over seeds)
Kill ( ↑ )
Faithfulness ( ↑ )
Net ( ↑ )
Method
Mean
Worst
Mean
Worst
Mean
Worst
JuntaLearner (Ours; Tuned)
0.67
0.38
0.61
0.38
0.54
0.28
JuntaLearner (Ours; Default)
0.65
0.38
0.59
0.32
0.48
0.21
Per-head NIE, signed
0.64
0.37
0.59
0.31
0.47
0.12
EAP, signed
0.56
0.31
0.53
0.28
0.45
0.22
MAttr
0.55
0.24
0.58
0.42
0.36
-0.03
Table 1: CRS across the fifteen model-task cells. Worst is the lowest score across all cells; higher is better in every column. Best results are bold and second-best results are underlined, ranked before rounding. Baselines marked "signed" order heads by signed score instead of magnitude. Figure 4 shows the per-cell variation in scores.
Figure 5: Each model’s default against signed per-head \NIE , pooled over its five tasks, with a 95% interval across tasks. Vertical rules mark the other baselines’ margins on the same axis. Arrows denote that random order is lies significantly off-axes, and would distort the scale to be included.
Figure 6: Training time for each model–task–regime run (seconds, log scales). The dashed line marks equal time; points below it favor JuntaLearner .
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Agreement of three score orderings with the learned JuntaLearner ordering for GPT-2/IOI (seed 0). Plots the free, half-held, and held witness regimes, for the first 50 out of 144 attention heads. Each curve gives the fraction of the top k learned heads also in a score ordering’s top k . The gray dashed curve is the random-set expectation k/∣P∣ ; the dotted vertical line marks k=26 . The Qwen held cell was not run.
Figure 8: One training run on Llama-3.1-8B: arithmetic, the free witness regime, seed 0 , ladder (142,71,43,21,7) . Top: every candidate’s inclusion log-prob across steps, a point marking a step that drew it, with vertical rules where a rung ends and why. Bottom left: the separation margin, climbing again once a new rung starts. Bottom right: the mean objective per member of the drawn set, smoothed; a jump at a boundary is the change in k . Bottom: the share of the top k carried over from the previous step, which is the quantity a rung has to hold above 0.9 to end early.
Task
GPT-2 small
Qwen3-1.7B
Llama-3.1-8B
IOI
51
32
44
SVA
51
36
24
Arithmetic
76
31
32
Induction
51
31
32
Entity tracking
26
29
26
Total
255
159
158
Appendix
Table 2: Distinct trained configurations by model and task.
Knob
Values tried
Meaning
Witness regime
free, half held, held
share of the eligible witness pool pinned
k schedule
ladder, k=1 , reverted
decreasing, one flat k , or rising
Ladder shape
top rung 1 to 512 , 1 to 7 rungs
compared as a share of the pool
Measurement stage
on, off
measures the leading candidates one at a time in the final steps
Objective
\NIE , \NIE+\pie , bottleneck
\pie alone is out of scope
Prompt pairs
100 , 150 , 200
150 on Llama only
Appendix
Table 3: The knobs varied across the trained configurations.
Setting
GPT-2 small
Qwen3-1.7B
Llama-3.1-8B
Candidate pool
144 head writes
448
1024
Prompt pairs
100 or 200
100 or 200
100 or 150
Default ladder
20 - 10 - 6 - 3 - 1
150 - 67 - 33 - 8 - 1
142 - 71 - 43 - 21 - 7
Steps per run
30 to 90
30 to 90
30 to 150
Configurations
255
159
158
Seeds per default cell
5 to 35
5 to 10
5 to 10
Appendix
Table 4: Settings per model. The candidate pool and the default ladder describe each model’s default; the other rows summarize all of its runs.
Task
Factual prompt → answer
Alternative prompt
IOI
When Mary and John went to the store, John gave a drink to → Mary
Second subject mention replaced by a third name
SVA
The friends → have
Subject number flipped ( The friend → has)
Arithmetic
Q: How much is 27 plus 64? A: → 91
Both operands resampled
Induction
A random word block, repeated: oak map … rat hat oak map … rat → hat
Boundary word changed ( hat → cow )
Entity tracking
The apple is in Box C. The key is in Box A. … Box C contains the → apple
Same box queried, different object placed in it
Appendix
Table 5: Example factual prompts and the single change that produces each alternative prompt.
Figure 9: Kill curves for each cell’s best-scoring trained configuration (best configuration, for brevity) against all eight baselines. The x axis is the number of attention heads patched, log-scaled.
Figure 10: Faithfulness curves for each cell’s best configuration, selected on Sfaith and so not always the same configuration as Figure 9 .
Figure 11: Net curves for each cell’s own best configuration. Net regularly picks a different winner from kill and faithfulness.
Figure 12: GPT-2 small: three best settings against the baselines on all three goals, with the CRS ranking at the right of each row.
Figure 13: Qwen3-1.7B, drawn as Figure 12 . Its ladder shapes score within a few thousandths of each other on kill.
Figure 14: Llama-3.1-8B, drawn as Figure 12 .
Figure 15: Per-cell margins over signed per-head \NIE for model-level default configurations.
Figure 16: Kill margins over signed per-head \NIE for each cell’s own winner, drawn as Figure 15 .
Figure 17: GP-estimated effects of each knob, grouped by goal and model. Rows compare each setting while holding other settings at their observed values. Cells give the posterior mean difference divided by its standard deviation.
Figure 18: Permutation importance over the fitted model: the increase in error when one column is shuffled.
Figure 19: Task-specific knob contrasts across the three goals (rows) and models (columns). Each cell is the mean GP posterior difference divided by its standard deviation. The signed color scale is shared across panels and clipped at ±3 ; labels denote the first setting minus the second.
Figure 20: The fifteen cells of the main experiments: ours, tuned , ours, default , signed per-head \NIE and MAttr with its authors’ published MIB recipe. Note that for readability each triangle has its own scale and that values between goals are not directly comparable. Ours, tuned is chosen per cell and goal by CRS, as in Table 1 , and scored under all three summaries. Each axis shows a method’s summed score relative to the best, so outward is better on every axis; CMD appears as 1−CMD .
Figure 21: Faithfulness for fifteen cells of the main experiments. Left: each method’s faithfulness curve, averaged over cells; shading marks where it rises above 1 . Right: each method’s faithfulness CPR, split in two components: area under the curve capped at 1 with area above one stacked 13above.
Figure 22: The twelve MIB cells, public test split: ours, default in one untuned pass, per-node \NIE and MAttr’s released circuits, drawn as Figure 20 , with faithfulness from MIB’s evaluator. CMD appears as 1−CMD .
Figure 23: Twelve MIB cells, drawn and computed as Figure 21 .
Figure 24: MIB public test, the four IOI cells and two of the three MCQA cells. Columns: MIB faithfulness over all nodes and faithfulness over heads (share of the pool kept), then kill and net (top k heads patched). Dashed grey marks f=1 and kill d=1 ( d is the share of the full-pool effect removed), the ideal values, and net =0 , where a circuit’s damage to the target equals its mean damage to the controls; the ideal net is 1 .
Figure 25: MIB public test, the remaining cells, drawn as Figure 24 .
The circuits framework in mechanistic interpretability aims to identify sparse subgraphs of model components that are causally responsible for a behavior, typically evaluated by measuring necessity and sufficiency. But these criteria say little about whether a circuit consistently captures how a model performs a task, or if it is specific to that task. We study these two properties, consistency and specificity, across six tasks and five models, extracting circuits at the component level (attention heads and MLP blocks) and at the level of individual MLP neurons. We find that component-level circuits are highly consistent and causally important for most tasks, but they are not specific: for a given task, ablating its own circuit drops accuracy by about as much as ablating another task's circuit. Neuron-level circuits, on the other hand, exhibit higher specificity across tasks, but far less consistency within tasks. This is explained by circuit overlap: component-level circuits share most of their components across tasks, related or not, while neuron-level circuits overlap only between closely related tasks. In a case study of the components shared by the task circuits of Llama-3.2-3B, we show that they consist mostly of MLP blocks, while the few attention heads within turn out to have general-purpose roles. Overall, our findings raise questions about the degree to which circuits can support targeted understanding of, and intervention on, model behavior.
Michael Li, Nishant Subramani
Northeastern University · Carnegie Mellon University – Language Technologies Institute
Causal tracing systematically intervenes on a large language model's (LLM's) internal representations to uncover and quantify the causal pathways linking specific inputs or computations to specific metrics of interest, quantifying the LLM's behavior. Building on previous single-component or single-layer studies, this paper presents a unified framework for causally tracing multiple components simultaneously. This framework systematically identifies the subsets of components (e.g., attention heads and multi-layer perceptron neurons) most critical to a desired target performance metric (e.g., accuracy and fairness). This is achieved by incorporating flexible interventions applied to a wide range of desired metrics. To address the combinatorial complexity of the multi-component problem, an efficient algorithm is designed that leverages soft interventions and a carefully designed metric transformation, converting the combinatorial search problem into a continuous one that can be solved efficiently under proper constraints, thereby generating proper binary decisions for selecting components. Experimental results demonstrate that the proposed method efficiently identifies subsets of the model's components that have a high impact on the target metric, outperforming existing baseline approaches. Our code is available at https://github.com/ZiruiYan/multi-component-causal-tracing.
A prominent research direction in mechanistic interpretability is learning sparse circuits over LLM components to reveal how they jointly produce model behavior. However, raw neurons are polysemantic, making learned circuits hard to interpret. Sparse autoencoder (SAE) features alleviate this, but their high dimensionality makes existing intervention-based circuit learning methods computationally prohibitive. We propose CircuitLasso, a scalable circuit learning approach based on sparse linear regression. CircuitLasso recovers circuits whose structural accuracy matches that of state-of-the-art intervention-based methods on the benchmark data, at a fraction of the computational cost. For interpretability, CircuitLasso efficiently uncovers relationships among SAE features, showing how human-interpretable semantic features propagate through the model and influence its predictions. Finally, we validate the utility of our learned circuits by leveraging their insights to achieve comparable performance at substantially lower cost on a domain-generalization task.
Naiyu Yin, Dennis Wei, Tian Gao +3
Lehigh University, Mathematics Department, Bethlehem, USA · IBM Research, Yorktown Heights, USA