Interpreting Language Model Parameters

Lucius Bushnaq; Dan Braun; Oliver Clive-Griffin; Bart Bussmann; Nathan Hu; Michael Ivanitskiy; Linda Linsefors; Lee Sharkey

cs.LGOct 14, 2026

Interpreting Language Model Parameters

Authors: Lucius Bushnaq, Dan Braun, Oliver Clive-Griffin, Bart Bussmann, Nathan Hu, Michael Ivanitskiy, Linda Linsefors, Lee Sharkey

Organizations: Goodfire · MATS · Independent

Abstract

Mechanistic interpretability aims to uncover how neural networks use their parameters to implement their impressive neural algorithms. Although previous work has uncovered substantial structure in the intermediate representations that networks use, little progress has been made to understand how the parameters and nonlinearities of networks perform computations on those representations. In this work, we present a method that brings us closer to this understanding by decomposing a language model's parameters into subcomponents that each implement only a small part of the model's learned algorithm, while simultaneously requiring only a small fraction of those subcomponents to account for the network's behavior on any input. The method, adVersarial Parameter Decomposition (VPD), optimizes for decompositions of neural network parameters into simple subcomponents that preserve the network's input-output behavior even when many subcomponents are ablated, including under ablations that are adversarially selected to destroy behavior. This encourages learning subcomponents that provide short, mechanistically faithful descriptions of the network's behavior that should aggregate appropriately into more global descriptions of the network's learned algorithm. We study how sequences of interactions between these parameter subcomponents produce the network's output on particular inputs, enabling a new kind of 'circuit' analysis. While more work remains to be done to deepen our understanding of how neural networks use their parameters to compute their behavior, our work suggests an approach to identify a small set of simple, mechanistically faithful subcomponents on which further mechanistic analysis can be based.

Figures & tables

Figure 1: Parameter decomposition methods decompose target model parameters into vectors in parameter space (parameter components) that are optimized to approximate the model’s mechanisms.

Figure 2: Our target model is a standard decoder-only transformer language model.

Attributes of our target model
Layers	4
Residual stream dimension	768
MLP intermediate dimension	3072
Attention heads	6
Attention head dimension	128
Context length	512

Table 1:

Layer	$C$	Alive	Mean L0	L0/Alive
Layer 0	$9728$	$3709$	$44.6$	$0.012$
Layer 1	$9728$	$848$	$18.9$	$0.022$
Layer 2	$9728$	$1943$	$49.5$	$0.025$
Layer 3	$9728$	$3472$	$92.0$	$0.026$
Total	$38912$	$9972$	$205.0$	$0.021$

Table 2: Table: Per-layer decomposition summary statistics: Subcomponent dictionary sizes

C

; alive subcomponents (subcomponents with mean causal importances above

10^{-6}

at the end of training); average

L_{0}

scores of subcomponents with causal importance

>0

per batch and sequence position; and fraction of all subcomponents with causal importance

>0

per batch and sequence position.

Masking mode (excluding $\Delta$ -components)	Validation CE Loss	Pretraining Compute Recovered (%)
Target Model	2.71	100%
Unmasked (All masks(=)1)	2.72	82.4%
Stochastic Masks	2.84	26.9%
Rounded Masks (Mask(=)1 if CI(>)0)	2.94	11.8%
Rounded Masks (Mask(=)1 if CI(>)0.1)	2.95	11.3%
Causal Importance values (CIs) used as Masks	2.99	9.4%

Table 3:

Adversarial optimization steps $n^{\text{adv}}$	KL divergence to target model
20	0.8280
40	1.3539
80	3.8381
160	25.2560
320	40.2200

Table 4: While the decomposition is somewhat robust to approximately

20

steps of adversarial optimization, it is clearly not at all robust to

160

steps or more. 15 15 15 To provide some sense of scale, zero-ablating all of the target model’s weight matrices gives a KL-divergence of ca.

67.2449

Figure 3: Side-by-side generation comparisons across masking strategies. Static snapshot of an interactive figure; explore it at the web version .

Figure 4: CE degradation when simultaneously replacing all 4 MLP layers with sparse reconstructions from each method. (a) Active subcomponents per module (raw L0). (b) Active subcomponents per MLP reconstruction, adjusting for CLT’s cross-layer writes and VPD’s paired modules. (c) Total active parameters. VPD (purple markers) Pareto-dominates the activation-based methods under all three sparsity measures. The dashed line indicates zero-ablation (all MLP outputs set to zero). Lower is better.

Figure 5: Browse all VPD parameter subcomponents by weight matrix. Green highlights indicate causal importances; colored underlines show subcomponent activations. Static snapshot of an interactive figure; explore it at the web version .

Figure 6: Intruder detection scores for various CLT and PLT latents, and VPD subcomponents at different CI thresholds. Error bars are 95% bootstrap CIs on the mean. Dashed line is random chance accuracy (20%). Higher is better.

Figure 7: Number of alive subcomponents as a function of total subcomponent capacity. PLTs and CLTs scale roughly linearly with dictionary size, staying close to the $y=x$ line. VPD (purple) remains flat at 6,500-7,000 alive subcomponents regardless of capacity, indicating that additional capacity is not used for feature splitting. Dashed line: $y=x$ (all subcomponents alive).

Figure 8: The norm of the weights of each $W_{Q}$ and $W_{K}$ subcomponent in each head. No parameter subcomponent is exclusively localized in a single head, suggestive of computations that are distributed across attention heads.

Figure 9: The Standardized Static Interaction Strengths of pairs of parameter subcomponents in the $Q$ and $K$ projection matrices in each head (bottom grid) and all heads (top). The ten pairs with the largest interaction strengths at any offset are shown in color, with the rest in grey. The 1.attn.q:316 and 1.attn.k:329 pair exhibit strong positive Static Interaction Strength at early offsets, indicating this pair’s involvement in cross-head previous token behavior (and, more generally, ’recent token behavior’.

Figure 10: Attention scores $Z$ illustrated as a sum of Data Dependent Interaction Strengths between pairs of subcomponents. Static snapshot of an interactive figure; explore it at the web version .

Figure 11: The attention score consists of a sum of Data Dependent Interaction Strengths. This panel shows the same prompt as the figure above, but here you can control which pairs of subcomponents to include in the sum, allowing you to study their individual effects on the reconstructed attention score and attention pattern. Static snapshot of an interactive figure; explore it at the web version .

Figure 12: Identifying the previous token head: Mean attention across multiple inputs on offset $\tau=1$ , i.e. the previous token. Left : Average over sequences of random tokens, as per [ 66 ] . Right : Average over sequences sampled from the dataset. The plots reveal L1H1 is the most canonical “previous token head”. But note other heads place substantial average attention at offset $\tau=1$ .

Figure 13: Effect of ablations: Ablating 1.attn.q:316 very strongly reduces attention to tokens in the recent past across all heads that otherwise attended there strongly. The effects of ablating other W_Q components has no distinguishable effect compared with the baseline and are therefore not shown. Here the baseline is the unablated average attention pattern.

Figure 14: Data-weighted cosine similarities between each head’s $W_{OV}^{h}$ read and write matrices, and the cosine similarity between each head’s raw $W_{OV}^{h}$ . Here, data-weighting uses data where subcomponent 1.attn.k:329 is causally important.

Figure 15: Data Dependent Interaction Strength for the 1.attn.k:119 - 1.attn.q:316 interaction only, and the corresponding attention pattern if only that interaction contributed to the attention pattern. On the right hand side, the ground truth attention pattern is shown for comparison. Static snapshot of an interactive figure; explore it at the web version .

Figure 16: Static snapshot of an interactive figure; explore it at the web version .

Figure 17: Static snapshot of an interactive figure; explore it at the web version .

Figure 18: Static snapshot of an interactive figure; explore it at the web version .

Figure 19: Static snapshot of an interactive figure; explore it at the web version .

Figure 20: Static snapshot of an interactive figure; explore it at the web version .

Figure 21: Static snapshot of an interactive figure; explore it at the web version .

Figure 22: Static snapshot of an interactive figure; explore it at the web version .

Figure 23: Static snapshot of an interactive figure; explore it at the web version .

Figure 24: To exclude indirect effects (i.e. effects that one parameter subcomponent has on another that are mediated by intermediate parameter subcomponents), we stop the gradients flowing through all subcomponents that are not the source subcomponent.

Figure 25: Attribution graph for predicting ·her on the prompt ‘‘The princess lost her crown.’’, pruned with adversarial sampling. 25 25 25 Coefficient $0.5$ for cross-entropy reconstruction with stochastic sampling, coefficient $0.5$ for cross entropy with $4$ steps of PGD, lr $1$ , importance minimality coefficient $0.09$ , $p=0.3$ , $2000$ optimization steps. There are 150 subcomponents in the graph. Static snapshot of an interactive figure; explore it at the web version .

Figure 26: Attribution graph for predicting ·her on the prompt ‘‘The princess lost her crown.’’, pruned with causal importance masking. 29 29 29 Coefficient $1.0$ for cross-entropy reconstruction with causal importance masking, importance minimality coefficient $1.0$ , $p=0.3$ , $2000$ optimization steps. Static snapshot of an interactive figure; explore it at the web version .

Figure 27: Attribution graph for predicting ·his on the prompt ‘‘The prince lost his crown.’’, pruned with adversarial sampling. 35 35 35 Coefficient $0.5$ for cross-entropy reconstruction with stochastic sampling, coefficient $0.5$ for cross entropy with $4$ steps of PGD, lr $1$ , importance minimality coefficient $0.05$ , $p=0.3$ , $2000$ optimization steps. There are 160 subcomponents in the graph. The target model assigns probability $0.512$ to ·his . Static snapshot of an interactive figure; explore it at the web version .

Figure 28: Attribution graph for predicting ·his on the prompt “The prince lost his crown.”, pruned with causal importance masking. Static snapshot of an interactive figure; explore it at the web version .

Figure 29: Attribution graph for predicting > after v on the prompt < u , v > , pruned with adversarial sampling. Static snapshot of an interactive figure; explore it at the web version .

Figure 30: Attribution graph for predicting > on the prompt < u , v > after u , pruned with adversarial sampling. 41 41 41 Coefficient $0.5$ for cross-entropy reconstruction with stochastic sampling, coefficient $0.5$ for cross entropy with $4$ steps of PGD, lr $1$ , importance minimality coefficient $0.1$ , $p=0.3$ , $4000$ optimization steps. There are 162 subcomponents in the graph. The target model assigns probability $0.119$ to > . Static snapshot of an interactive figure; explore it at the web version .

Figure 31: Attribution graph for predicting > on the prompt < u , v > , pruned with causal importance masking. Static snapshot of an interactive figure; explore it at the web version .

Figure 32: Comparing VPD-based edits with a LoRA-augmented target model trained to complete emoticons with o . The orange background shows the per-token KL divergence between the edited model and the target model. The purple underline shows the probability assigned to o . The left panel shows the direct VPD-based manual edit, obtained by adding the unembedding vector for o to the emoticon subcomponent’s $\vec{U}$ vector (scaled by prefactor $\alpha=3$ ). The right panel shows a LoRA-augmented target model trained on $n=947$ examples, each consisting of a token the emoticon subcomponent was causally important on and the $20$ tokens either side of it. Static snapshot of an interactive figure; explore it at the web version .

Figure 33: Model editing for emoticon completions, LoRA vs. manual subcomponent edit. Manual edits were performed by adding the unembedding vector for o to the $\vec{U}$ vector of the emoticon subcomponent with different prefactors $\alpha$ . LoRAs were trained on $n=10$ and $n=947$ examples, each consisting of a token the emoticon subcomponent was causally important on, and the $20$ tokens immediately preceding and following it, with a KL-regularisation term weighted by $\lambda$ . The y-axis shows the average probability the edited model assigns to o on tokens the emoticon subcomponent is active on. The x-axis in the left plot shows $D_{\text{KL},\text{Surrounding}}$ , the KL divergence between the edited model and the target model on the $20$ other tokens immediately preceding and following tokens the emoticon subcomponent is causally important on, across a holdout set of $50$ examples. The x-axis in the right plot shows $D_{\text{KL},\text{Global}}$ , the average KL-divergence between the edited model and the target model on all other tokens across samples from the whole dataset.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
CI model dimension ( $d_{\mathrm{model}}$ )	2048
Transformer layers ( $N$ )	8
Attention heads	16
Head dimension	128
FFN hidden dimension ( $d_{\mathrm{ff}}$ )	8192
Positional encoding	RoPE (base $=10,000$ , max length $=512$ )

Appendix

Table 5: Table: Hyperparameters for the causal importance function

\Gamma

48 48 48 Recall that the target model is a 4-layer Llama-style transformer with

d_{\mathrm{model}}=768

and

d_{\mathrm{intermediate}}=3072

, decomposed across

L=24

weight matrices (6 per layer: c_fc , down_proj , q_proj , k_proj , v_proj , o_proj ), yielding a total of

C_{\mathrm{total}}=38,912

subcomponents and an input dimension of

D=27,648

Loss term	Reference	Coefficient
$\mathcal{L}_{\text{Delta-L2}}$ (auxiliary loss; a.k.a. parameter-faithfulness)	Equation 1	$10^{7}$
$\mathcal{L}_{\mathrm{stochastic\text{-}recon\text{-}subset}}$ (stochastic KL)	Equation 2	$0.5$
$\mathcal{L}_{\mathrm{adversarial\text{-}recon\text{-}subset}}$ (persistent PGD KL)	Equation 8	$0.5$
$\mathcal{L}_{\mathrm{importance\text{-}minimality}}$ ( $\ell_{p}$ on CI values)	Section 2.3	$2\times 10^{-4}$
$\mathcal{L}_{\mathrm{frequency\text{-}minimality}}$ (superlinear CI frequency penalty)	Equation 3	$1\times 10^{-4}$

Appendix

Table 6: VPD loss terms and their coefficients. The importance minimality loss uses

p

-annealing from

2.0

0.4

. In practice we implement

\mathcal{L}_{\mathrm{importance\text{-}minimality}}

and

\mathcal{L}_{\mathrm{frequency\text{-}minimality}}

as a single fused term with an inner weight

\beta=0.5

on the frequency part (see above); this is functionally equivalent to the two losses with the coefficients shown. All reconstruction losses use KL divergence.

Module type	Subcomponents ( $C$ ) per layer
c_fc (MLP Up-projection, $768\times 3072$ )	3072
down_proj (MLP Down-projection, $3072\times 768$ )	3584
q_proj (query projection, $768\times 768$ )	512
k_proj (key projection, $768\times 768$ )	512
v_proj (value projection, $768\times 768$ )	1024
o_proj (output projection, $768\times 768$ )	1024

Appendix

Table 7: Number of subcomponents per module type at initialization.

Loss	Value
Total	$24.62$
Delta-L2 (MSE)	$0.00000240$
StochasticReconSubsetLoss (KL)	$0.2419$
PersistentPGDReconLoss (KL)	$0.5733$
ImportanceMinimalityLoss	$1102.0$

Appendix

Table 8: Training losses (Measured at final step).

Loss	Value
StochasticReconSubsetLoss (KL)	$0.2381$
PGDReconLoss (KL)	$0.9268$
StochasticHiddenActsReconLoss (MSE)	$0.4130$
CIHiddenActsReconLoss (MSE)	$0.8464$

Appendix

Table 9: Evaluation reconstruction losses.

Used in	What	WandB link
Figure 4 ; Figure 35	PLT/CLT, local-MSE objective, $k\in\{8,16,32,64\}$	dict_4k , dict_32k
Figure 34	PLT/CLT, end-to-end KL objective, $k\in\{8,16,32,64\}$ , three training modes ( cascading = error-propagating, parallel = clean-input, independent = single-layer)	dict_4k , dict_32k
Table 11 — PLT/CLT seed runs	5 seeds $\times$ {PLT local-MSE, PLT e2e-independent, CLT local-MSE, CLT e2e-parallel}, $k=16$ , 4k dict	multiseed
Table 11 — VPD seed runs	5 VPD seed runs (otherwise identical to the main decomposition)	VPD multiseed
Table 11 — hidden-activation aux-loss VPD	VPD trained with an auxiliary stochastic-forward-pass hidden-activation MSE loss	VPD hidden-act run
Figure 7 ; Figure 35	VPD capacity sweep ( $0.5\times$ , $1\times$ , $2\times$ , $4\times$ subcomponents); $1\times$ is the main run above	capacity_sweep

Appendix

Table 43

Figure 34: CE degradation vs. L0 (active features per module) for end-to-end KL-trained methods under three evaluation protocols. (a) Error-propagating: each encoder sees the modified residual stream. (b) Clean-input: each encoder sees the clean residual stream. (c) Single-layer replacement, averaged over layers. PLTs (blue) and CLTs (orange) perform well in their training mode but degrade by 5-20x in the mismatched mode. VPD (purple markers) is relatively stable across all three protocols. Linestyle indicates training mode: solid = error-propagating, dashed = clean-input, dotted = single-layer.

Figure 35: Cross-model latent/subcomponent splitting (decoder vector cosine similarity $>0.5$ ). Each cell shows the percentage of alive subcomponents or latents in the source model (row) that have more than one match in the target model (column). VPD shows low cosine similarity across models. PLTs and CLTs show high mutual cosine similarity, suggesting substantial splitting.

Method	Cross-seed Mean Max Cos Sim
VPD U vectors	0.4808
VPD V vectors	0.5156
PLT (e2e)	0.4390
CLT (e2e, parallel)	0.3468
TC (local MSE)	0.8063
CLT (local MSE)	0.6078

Appendix

Table 11:

Figure 36: Comparison between a decomposition with and without adversarial loss. The training configuration is otherwise identical. The CE loss is especially improved for small values of $r$ .

Figure 37: The norm of the weights of each $W_{V}$ and $W_{O}$ subcomponent in each head.

Figure 38: The probability of each K subcomponent being active when a given V subcomponent is active, and vice versa. This tells us what $W_{K}$ subcomponents are responsible for moving information from particular kinds of $W_{V}$ subcomponents. The 1.attn.k:329 subcomponent is always active, and therefore moves all kinds of $W_{V}$ subcomponents.

Figure 39: Data-weighted cosine similarities between each head’s $W_{OV}^{h}$ read- and write matrices, and the cosine similarity between each head’s raw $W_{OV}^{h}$ . Here, data-weighting uses data where subcomponent 1.attn.k:119 is causally important.

Rank	Comp	Alignment	Label
1	v.22	74.9453	punctuation, syntax, and formatting tokens
2	v.984	68.3169	fires on punctuation and symbols
3	v.1000	64.4708	fires on punctuation, delimiters, and structural boundaries
4	v.346	62.6574	distinguishes function words (positive) and content words (negative)
5	v.568	59.1917	fires on word prefixes and partial words

Appendix

Table 51

Rank	Comp	Alignment	Label
1	o.923	231.5484	first token of the sequence
2	o.411	203.4490	code, markup, and technical formatting syntax
3	o.630	191.1107	punctuation, symbols, and syntax in technical text
4	o.753	181.5993	closing parentheses and brackets in code and math
5	o.300	173.9110	code and structured text syntax/indentation

Appendix

Table 52

Rank	Comp	Alignment	Label
1	v.984	59.9101	fires on punctuation and symbols
2	v.346	56.6177	distinguishes function words (positive) and content words (negative)
3	v.22	52.5358	punctuation, syntax, and formatting tokens
4	v.1000	52.4579	fires on punctuation, delimiters, and structural boundaries
5	v.946	45.8153	distinguishes content words from function words/symbols

Appendix

Table 53

Rank	Comp	Alignment	Label
1	o.362	167.0992	fires on names, citations, proper nouns and formatting tokens
2	o.490	164.5913	line start and indentation tokens
3	o.337	149.7576	fires inside parentheses or mathematical formulas
4	o.895	132.7422	variables, math symbols, and syntax in technical text
5	o.986	116.9061	sentence/paragraph boundaries and transition words

Appendix

Table 54

Rank	Comp	Alignment	Label
1	v.984	83.9813	fires on punctuation and symbols
2	v.22	79.5923	punctuation, syntax, and formatting tokens
3	v.346	69.2600	distinguishes function words (positive) and content words (negative)
4	v.1000	68.2968	fires on punctuation, delimiters, and structural boundaries
5	v.531	66.0043	opening parentheses, brackets, braces, and quotes

Appendix

Table 55

Rank	Comp	Alignment	Label
1	o.923	432.9877	first token of the sequence
2	o.578	249.4896	heterogeneous subcomponent / lack of clear pattern
3	o.180	227.2081	diffuse firing on tokens within words/phrases
4	o.866	208.1991	predicts newlines and separators at line ends
5	o.336	205.5378	diverges between function words and complex technical terms

Appendix

Table 56

Rank	Comp	Alignment	Label
1	v.984	70.1569	fires on punctuation and symbols
2	v.745	57.7404	formatting symbols, operators, and spatial alignment
3	v.946	53.8508	distinguishes content words from function words/symbols
4	v.494	53.3743	predicts line breaks or indentation in formatted text
5	v.346	52.9415	distinguishes function words (positive) and content words (negative)

Appendix

Table 57

Rank	Comp	Alignment	Label
1	o.311	333.3696	fires universally on most tokens
2	o.630	263.6867	punctuation, symbols, and syntax in technical text
3	o.37	257.2931	continuations of multi-token entities and compound words
4	o.300	251.2589	code and structured text syntax/indentation
5	o.180	177.4386	diffuse firing on tokens within words/phrases

Appendix

Table 58

Rank	Comp	Alignment	Label
1	v.346	146.1802	distinguishes function words (positive) and content words (negative)
2	v.22	145.5668	punctuation, syntax, and formatting tokens
3	v.984	142.4020	fires on punctuation and symbols
4	v.745	129.9418	formatting symbols, operators, and spatial alignment
5	v.1000	116.1570	fires on punctuation, delimiters, and structural boundaries

Appendix

Table 59

Rank	Comp	Alignment	Label
1	o.753	832.6620	closing parentheses and brackets in code and math
2	o.630	591.0163	punctuation, symbols, and syntax in technical text
3	o.411	538.0296	code, markup, and technical formatting syntax
4	o.860	522.9536	structural and formatting markers vs content words
5	o.292	503.4616	fires broadly on various tokens, promoting line breaks and punctuation

Appendix

Table 60

Rank	Comp	Alignment	Label
1	v.1000	57.2500	fires on punctuation, delimiters, and structural boundaries
2	v.946	56.3107	distinguishes content words from function words/symbols
3	v.346	53.8432	distinguishes function words (positive) and content words (negative)
4	v.428	51.4916	fragments of proper nouns, foreign, and technical words
5	v.984	50.2268	fires on punctuation and symbols

Appendix

Table 61

Rank	Comp	Alignment	Label
1	o.923	227.1903	first token of the sequence
2	o.311	178.6259	fires universally on most tokens
3	o.630	160.5296	punctuation, symbols, and syntax in technical text
4	o.180	156.3924	diffuse firing on tokens within words/phrases
5	o.300	144.0436	code and structured text syntax/indentation

Appendix

Table 62

Figure 40: Entries of the interaction matrix $I_{c,c^{\prime}}$ for the layer $0$ MLP Up projection matrix subcomponents. Entries greater than $1.0$ are clamped to $1$ . Indices are sorted by the components that each subcomponent belong to. Entries $\geq 1.0$ in a row indicate that the nonlinear interaction between subcomponents $c^{\prime}$ and $c$ is large compared to the self-interaction of $c$ on the diagonal. Many interactions are either block-diagonal, indicating they take place inside higher-rank components, or arranged along vertical and horizontal lines, indicating they are caused by a relatively small number of highly interacting components.

Figure 41: Histogram of matrix entries of the interaction matrix $I_{c,c^{\prime}}$ for the layer $0$ up projection subcomponents. Most entries are much smaller than the self-interaction $1.0$ .

Figure 42: Entries of the interaction matrix $I_{c,c^{\prime}}$ for the layer 1 MLP Up projection subcomponents. Indices are sorted by the components subcomponents belong to.

Figure 43: Histogram of the entries of the interaction matrix $I_{c,c^{\prime}}$ for the layer 1 MLP Up projection subcomponents.

Figure 44: Entries of the interaction matrix $I_{c,c^{\prime}}$ for the layer 2 MLP Up projection subcomponents. Indices are sorted by the components subcomponents belong to.

Figure 45: Histogram of the entries of the interaction matrix $I_{c,c^{\prime}}$ for the layer 2 MLP Up projection subcomponents.

Figure 46: Entries of the interaction matrix $I_{c,c^{\prime}}$ for the layer 3 MLP Up projection subcomponents. Indices are sorted by the components subcomponents belong to.

Figure 47: Histogram of the entries of the interaction matrix $I_{c,c^{\prime}}$ for the layer 3 MLP Up projection subcomponents.

Mechanistic Interpretability

Mechanistic interpretability studies how neural networks implement behaviors through internal features, circuits, and causal information flow.

See all 388 papers

Weights Read and Write Features: Scalable Parameter Decomposition Grounded in Activation Space
Sep 29, 2026Tue M. Cao, Lisiane Pruinelli, My T. ThaiMechanistic Interpretability Circuits in Language Models
Department of Computer and Information Science and Engineering, University of Florida, Gainesville, FL 32611, USA · College of Nursing, University of Florida, Gainesville, FL 32611, USA
Targeted Recovery of Weight-Space Mechanisms From Neural Networks
Jun 19, 2026Antoine Vigouroux, Lee SharkeyTransformer Interpretability Circuit Discovery
1MATS · 2Goodfire
DV-Lens: Revealing the Functional Organization of Language Model Parameters
Oct 3, 2026Chenhang Cui, Jian Yu, Shuyi Miao +5LLM Interpretability
National University of Singapore · Nanjing University of Science and Technology · Beihang University +2

Sep 29, 2026J/K move · Enter open · S save

Weights Read and Write Features: Scalable Parameter Decomposition Grounded in Activation Space

Tue M. Cao, Lisiane Pruinelli, My T. Thai

Department of Computer and Information Science and Engineering, University of Florida, Gainesville, FL 32611, USA · College of Nursing, University of Florida, Gainesville, FL 32611, USA

Mechanistic Interpretability Circuits in Language Models

Open paper AskPDF arXiv

Activation space and parameter space provide complementary views of model computation. Activations represent information, while weights read, transform, and write that information. Yet existing interpretability methods largely study the two spaces separately, leaving the connection between represented information and parameter-level computation underexplored. We introduce Activation-Supported Parameter Decomposition (ASPD), which jointly decomposes activation and parameter spaces and grounds each learned weight component in the activation features it reads or writes. This grounding constrains otherwise non-unique parameter decompositions using the model's internal activations, while an internal reconstruction objective provides a local learning signal at the weight matrix being analyzed. Together, these properties enable scalable, interpretable, and causally editable parameter decomposition in pretrained large language models, demonstrated on Qwen-3-8B. The learned read--write components can also be composed into parameter-level mechanism circuits. We use ASPD to recover mechanisms underlying the classic IOI circuit and trace semantic transformations through model weights.

Abstract

Mechanistic Interpretability

Explore similar work

Weights Read and Write Features: Scalable Parameter Decomposition Grounded in Activation Space

Targeted Recovery of Weight-Space Mechanisms From Neural Networks

DV-Lens: Revealing the Functional Organization of Language Model Parameters