cs.CLAug 26, 2026

MechaTerp-TRACE: A Novel Approach for Component Ablation Analysis in Language Models

Authors: Brandon Colelough, Davis Bartels, Madeline Bittner, Dina Demner-Fushman

Organizations: National Institutes of Health, National Library of Medicine Bethesda, Maryland, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Interpretability research on large language models has produced accounts of factual recall in feed-forward layers and of token relationships in self-attention, but little work offers a unified way to compare the causal contribution of different architecture components to a model's output. We introduce MechaTerp (the Mechanistic Interpretability suite) -TRACE (subset for Teacher-forced Registry of Ablated Component Effects), an architecture and study that measures how much each registered component of a language model supports the production of a named entity. TRACE ablates one component at a time and measures the resulting change in the output distribution at a fixed answer token, so component types from whole transformer blocks down to individual neurons and output logits can be compared on a common scale. We apply it to thirteen instruction-tuned dense decoder models spanning five families and one to thirty billion parameters, ablating 49,656 components across 48 medical and 42 general-knowledge prompts. We find that the components carrying the most effect are the same few, positionally fixed components in every model, regardless of which entity a prompt asks about, and that once these are removed, the remaining support is close to evenly spread in eleven of the thirteen models. Apparent localisation of entity knowledge is therefore largely attributable to generic generation machinery, which has direct consequences for methods that assume entity knowledge sits in a findable place, including targeted knowledge editing.

Figures & tables

 GraniteLlamaMistralPhiQwen
MetricGr-3BGr-8BGr-30BL-1BL-3BL-8BMi-7BMn-3B †MPPhQ-1.7BQ3.5-9BQ3.5-2B
Identity
Params3B8B30B1B3B8B7B3B3.8B3.8B1.7B9B2B
BB (block-lane size)40406416283232143232283224
Correctness
Correct (of 90)6375774671687207164355755
Table 1: MechaTerp-TRACE per-model results (13 models; Gemma excluded from this cohort), combining localisation (Eq. 4/5, hard-floor-removed – see notes), cross-prompt reuse (Eq. 6), and domain-common/generation-floor components with their soft LLI correction (Eq. 9, 11, 12), transposed with models as columns. Short labels (family-Params): Gr-3B=Granite-4.1-3B, Gr-8B=Granite-4.1-8B, Gr-30B=Granite-4.1-30B, L-1B=Llama-3.2-1B, L-3B=Llama-3.2-3B, L-8B=Llama-3.1-8B-Instruct, Mi-7B=Mistral-7B-Instruct-v0.2, Mn-3B=Ministral-3B, MP=MediPhi-Clinical, Ph=Phi-4-mini, Q-1.7B=Qwen3-1.7B, Q3.5-9B=Qwen3.5-9B, Q3.5-2B=Qwen3.5-2B.
Figure 1: Median block/lane LLI (Eq. 4 ) by model and component lane, before floor removal (raw), split by domain. Shown as a cross-lane diagnostic: floor removal is validated and applied only at the block lane (Fig. 2 and Table 1 ), so the raw values here illustrate where effect concentrates architecturally prior to correction and should not be read as the corrected localization result. Blank cells denote lanes absent from that model’s architecture. Sub-figure A shows results for medical prompt subset and sub-figure B shows results for general-knowledge prompt subset.
Figure 2: Block-lane effective support ( Neff/B′N_{\mathrm{eff}}/B^{\prime} , Eq. 4 ) after hard removal of generation-floor components (Eq. 10 ), ranked across the 13-model cohort. Lower values indicate more localized entity support (few components carry the effect); higher values indicate distributed support. Once structurally-fixed always-on components are removed, all models but MediPhi-Clinical and Qwen3-1.7B fall toward the distributed pole, indicating that raw concentration is largely attributable to generic generation machinery rather than entity-specific processing. Filled markers: medical prompts; open markers: general prompts.
More in this topic

Concept Localization in Language Models

See all 10 papers

Explore similar work

CardsList
Jun 2, 2026

Multi-component Causal Tracing in Large Language Models

Causal tracing systematically intervenes on a large language model's (LLM's) internal representations to uncover and quantify the causal pathways linking specific inputs or computations to specific metrics of interest, quantifying the LLM's behavior. Building on previous single-component or single-layer studies, this paper presents a unified framework for causally tracing multiple components simultaneously. This framework systematically identifies the subsets of components (e.g., attention heads and multi-layer perceptron neurons) most critical to a desired target performance metric (e.g., accuracy and fairness). This is achieved by incorporating flexible interventions applied to a wide range of desired metrics. To address the combinatorial complexity of the multi-component problem, an efficient algorithm is designed that leverages soft interventions and a carefully designed metric transformation, converting the combinatorial search problem into a continuous one that can be solved efficiently under proper constraints, thereby generating proper binary decisions for selecting components. Experimental results demonstrate that the proposed method efficiently identifies subsets of the model's components that have a high impact on the target metric, outperforming existing baseline approaches. Our code is available at https://github.com/ZiruiYan/multi-component-causal-tracing.

Zirui Yan, Dennis Wei, Dmitriy A. Katz +2

Rensselaer Polytechnic Institute · IBM Research

LLM InterpretabilityCausal Reasoning in Language ModelsCausal Interventions in Language Models
AskPDF
Mar 1, 2026

Truth as a Trajectory: What Internal Representations Reveal About Large Language Model Reasoning

Existing explainability methods for Large Language Models (LLMs) typically treat hidden states as static points in activation space, assuming that correct and incorrect inferences can be separated using representations from an individual layer. However, these activations are saturated with polysemantic features, leading to linear probes learning surface-level lexical patterns rather than underlying reasoning structures. We introduce Truth as a Trajectory (TaT), which models the transformer inference as an unfolded trajectory of iterative refinements, shifting analysis from static activations to layer-wise geometric displacement. By analyzing displacement of representations across layers, TaT uncovers geometric invariants that distinguish valid reasoning from spurious behavior. We evaluate TaT across dense and Mixture-of-Experts (MoE) architectures on benchmarks spanning commonsense reasoning, question answering, and toxicity detection. Without access to the activations themselves and using only changes in activations across layers, we show that TaT effectively mitigates reliance on static lexical confounds, outperforming conventional probing, and establishes trajectory analysis as a complementary perspective on LLM explainability.

Hamed Damirchi, Ignacio Meza De la Jara, Ehsan Abbasnejad +3

Australian Institute for Machine Learning, Adelaide University · Monash University · Concordia University

LLM InterpretabilityRepresentation Geometry in Language Models
AskPDF
Sep 22, 2026

Matryoshka attribution: Learning to attribute language model outputs to representations and weights

Attributing language model outputs to their internal computations is an open problem in interpretability. Existing methods, which use causal interventions, gradients, or learnable masks, either are infeasibly expensive or struggle to identify actual causally-important internal computations. We propose framing attribution as the problem of identifying nested subsets of internal components which minimise a downstream loss. To learn this task, we introduce Matryoshka Attribution (MAttr), a mask learning method that parametrises the mask with a simple differentiable sigmoid top-kk operator. We supervise training over all sparsities simultaneously by randomising kk over training, resulting in a learned ordering of components by attribution score. MAttr achieves number 1 on the official leaderboard of the Mechanistic Interpretability Benchmark (Mueller et al., 2025); our method identifies sparse and task-transferrable circuits across varying circuit bases. As a practical application, we show that MAttr can be trained with reinforcement learning to identify weight changes responsible for downstream behaviours in LLM finetuning. We train MAttr on refusal judge scores and find that restoring 1%1\% of Llama 3.1 8B Instruct's weights to their base model state is sufficient to remove refusals while maintaining capabilities. We view MAttr as a successful formulation of interpretability into a learnable objective that we can tackle with gradient descent, and encourage future work along these lines.

Aryaman Arora, Kirill Acharya, Nathan Hu +4

Stanford University

LLM InterpretabilityCircuits in Language Models
AskPDF