Weights Read and Write Features: Scalable Parameter Decomposition Grounded in Activation Space

Tue M. Cao; Lisiane Pruinelli; My T. Thai

cs.LGSep 29, 2026

Weights Read and Write Features: Scalable Parameter Decomposition Grounded in Activation Space

Authors: Tue M. Cao, Lisiane Pruinelli, My T. Thai

Organizations: Department of Computer and Information Science and Engineering, University of Florida, Gainesville, FL 32611, USA · College of Nursing, University of Florida, Gainesville, FL 32611, USA

Abstract

Activation space and parameter space provide complementary views of model computation. Activations represent information, while weights read, transform, and write that information. Yet existing interpretability methods largely study the two spaces separately, leaving the connection between represented information and parameter-level computation underexplored. We introduce Activation-Supported Parameter Decomposition (ASPD), which jointly decomposes activation and parameter spaces and grounds each learned weight component in the activation features it reads or writes. This grounding constrains otherwise non-unique parameter decompositions using the model's internal activations, while an internal reconstruction objective provides a local learning signal at the weight matrix being analyzed. Together, these properties enable scalable, interpretable, and causally editable parameter decomposition in pretrained large language models, demonstrated on Qwen-3-8B. The learned read--write components can also be composed into parameter-level mechanism circuits. We use ASPD to recover mechanisms underlying the classic IOI circuit and trace semantic transformations through model weights.

Figures & tables

Figure 1: Parameter-level mechanisms recovered for the IOI circuit ( Wang et al., 2022 ) . ASPD identifies weight components implementing known head-level computations and connects them through read–write interactions across heads. For example, the duplicate-name component $224O$ of H3.0 and the induction component $228O$ of H5.5 provide inputs to the downstream S-inhibition mechanism in H8.6.

	GPT2		Gemma-2-2b		Qwen-3-8b
Method	Interp $\uparrow$	Sim $\downarrow$	Interp $\uparrow$	Sim $\downarrow$	Interp $\uparrow$	Sim $\downarrow$
ASPD (Ours)	0.68 $\pm$ 0.04	0.01 $\pm$ 0.00	0.62 $\pm$ 0.03	0.03 $\pm$ 0.00	0.57 $\pm$ 0.04	0.03 $\pm$ 0.00
PD Transcoder (Ours)	0.54 $\pm$ 0.04	0.00 $\pm$ 0.00	0.29 $\pm$ 0.03	0.10 $\pm$ 0.01	0.60 $\pm$ 0.04	0.05 $\pm$ 0.00
VPD	0.37 $\pm$ 0.03	0.01 $\pm$ 0.00	0.20 $\pm$ 0.02	0.44 $\pm$ 0.01	0.22 $\pm$ 0.02	0.18 $\pm$ 0.00
VPD + internal	0.43 $\pm$ 0.03	0.00 $\pm$ 0.00	0.25 $\pm$ 0.02	0.14 $\pm$ 0.00	0.24 $\pm$ 0.02	0.24 $\pm$ 0.01

Table 1: Interpretability and Diversity experiment results. Interp

=0.2

means random chance.

	GPT2	Gemma-2-2B	Qwen-3-8b
Method	Matching $\uparrow$	Matching $\uparrow$	Matching $\uparrow$
ASPD (Ours)	1.02 $\pm$ 0.14	0.30 $\pm$ 0.11	0.61 $\pm$ 0.14
PD Transcoder (Ours)	0.94 $\pm$ 0.15	0.27 $\pm$ 0.17	0.64 $\pm$ 0.15
VPD	0.31 $\pm$ 0.11	-0.02 $\pm$ 0.10	0.17 $\pm$ 0.11
VPD + internal	0.45 $\pm$ 0.13	0.07 $\pm$ 0.12	0.10 $\pm$ 0.14

Table 2: Meaning localization experiment results. Matching

=0

means random chance.

Single
	GPT2		Gemma-2-2b		Qwen-3-8b
Method	$ratio$ $\uparrow$	$localization$ $\uparrow$	$ratio$ $\uparrow$	$localization$ $\uparrow$	$ratio$ $\uparrow$	$localization$ $\uparrow$
ASPD (Ours)	3.1 $\pm$ 0.5	0.040 $\pm$ 0.005	2.7 $\pm$ 0.8	0.013 $\pm$ 0.004	6.7 $\pm$ 0.9	0.053 $\pm$ 0.006
PD Transcoder (Ours)	3.0 $\pm$ 0.7	0.049 $\pm$ 0.010	4.3 $\pm$ 1.1	0.012 $\pm$ 0.004	2.5 $\pm$ 0.4	0.050 $\pm$ 0.005
VPD	0.9 $\pm$ 0.2	0.017 $\pm$ 0.002	0.5 $\pm$ 0.1	0.004 $\pm$ 0.001	0.9 $\pm$ 0.2	0.020 $\pm$ 0.003
VPD + internal	0.3 $\pm$ 0.2	0.007 $\pm$ 0.004	0.7 $\pm$ 0.2	0.006 $\pm$ 0.001	1.1 $\pm$ 0.2	0.021 $\pm$ 0.003

Table 3: Weight Editing Localization experiment results.

ratio<1

is poorer than random chance.

Figure 2: Weight editing on the S-inhibition mechanism in H8.6. The left panel shows the original attention pattern. The right panel shows the change after removing the identified $QK$ component pair from the weights. The edit substantially suppresses attention at the S-inhibition position.

Figure 3: Tracing a semantic transformation through the layer-7 MLP. $MLP_{\mathrm{in}}$ component 2535, associated with contexts involving NBC, ABC, and Fox, reads news-organization features from the residual stream and interacts strongly with $MLP_{\mathrm{out}}$ component 3885, which writes to downstream features associated with broader news-related context.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Model	dataset
GPT2s	apollo-research/Skylion007-openwebtext-tokenizer-gpt2 ( Apollo-research, )
Gemma-2-2b	monology/pile-uncopyrighted ( Gao et al., 2020 )
Qwen-3-8b	monology/pile-uncopyrighted ( Gao et al., 2020 )

Appendix

Table 4: Datasets used for the paper.

Loss coefficients	$\lambda_{sparse}$	$\lambda_{freq}$	$\lambda_{ablate\_stoch}$	$\lambda_{ablate\_adv}$	$\lambda_{param}$	$\lambda_{internal}$	$\lambda_{auxiliary}$
GPT-2 small
VPD	1e-5	0.5	0.5	0.5	1000
VPD + internal	1e-5	0.5	0.5	0.5	1000	0.5
VPD + internal + no param	1e-5	0.5	0.5	0.5		0.5
VPD + internal + no ablate	1e-5	0.5			1000	0.5
PD Transcoder						1.0	0.03125

Appendix

Table 5: Training loss coefficients.

	GPT2		Gemma-2-2b		Qwen-3-8b
Method	Interp $\uparrow$	Sim $\downarrow$	Interp $\uparrow$	Sim $\downarrow$	Interp $\uparrow$	Sim $\downarrow$
PD Transcoder (Ours)	0.54 $\pm$ 0.04	0.00 $\pm$ 0.00	0.29 $\pm$ 0.03	0.10 $\pm$ 0.01	0.60 $\pm$ 0.04	0.05 $\pm$ 0.00
PD Transcoder + param	0.45 $\pm$ 0.04	0.00 $\pm$ 0.00	0.24 $\pm$ 0.02	0.07 $\pm$ 0.01	0.36 $\pm$ 0.03	0.07 $\pm$ 0.00
PD Transcoder + ablate	0.18 $\pm$ 0.03	0.00 $\pm$ 0.00	0.24 $\pm$ 0.02	0.13 $\pm$ 0.00	0.22 $\pm$ 0.02	0.12 $\pm$ 0.00
VPD + internal	0.43 $\pm$ 0.03	0.00 $\pm$ 0.00	0.25 $\pm$ 0.02	0.14 $\pm$ 0.00	0.24 $\pm$ 0.02	0.24 $\pm$ 0.01
VPD + internal + no param	0.42 $\pm$ 0.03	0.00 $\pm$ 0.00	0.24 $\pm$ 0.02	0.16 $\pm$ 0.00	0.28 $\pm$ 0.02	0.21 $\pm$ 0.00

Appendix

Table 6: Interpretability and Diversity experiment results of adding or removing

\mathcal{L}_{param},\mathcal{L}_{ablate}

Single
	GPT2		Gemma-2-2b		Qwen-3-8b
Method	$ratio$ $\uparrow$	$localization$ $\uparrow$	$ratio$ $\uparrow$	$localization$ $\uparrow$	$ratio$ $\uparrow$	$localization$ $\uparrow$
PD Transcoder (Ours)	3.0 $\pm$ 0.7	0.049 $\pm$ 0.010	4.3 $\pm$ 1.1	0.012 $\pm$ 0.004	2.5 $\pm$ 0.4	0.050 $\pm$ 0.005
PD Transcoder + param	1.6 $\pm$ 0.034	0.046 $\pm$ 0.008	2.2 $\pm$ 0.7	0.013 $\pm$ 0.004	2.6 $\pm$ 0.4	0.047 $\pm$ 0.006
PD Transcoder + ablate	0.6 $\pm$ 0.2	0.016 $\pm$ 0.004	2.6 $\pm$ 0.6	0.012 $\pm$ 0.003	1.0 $\pm$ 0.2	0.018 $\pm$ 0.003
VPD + internal	0.3 $\pm$ 0.2	0.007 $\pm$ 0.004	0.7 $\pm$ 0.2	0.006 $\pm$ 0.001	1.1 $\pm$ 0.2	0.021 $\pm$ 0.003

Appendix

Table 7: Weight Editing Localization experiment results of adding or removing

\mathcal{L}_{param},\mathcal{L}_{ablate}

	GPT2	Gemma-2-2B	Qwen-3-8b
Method	Matching $\uparrow$	Matching $\uparrow$	Matching $\uparrow$
PD Transcoder (Ours)	0.94 $\pm$ 0.15	0.27 $\pm$ 0.17	0.64 $\pm$ 0.15
PD Transcoder + param	1.14 $\pm$ 0.14	0.17 $\pm$ 0.15	0.22 $\pm$ 0.15
PD Transcoder + ablate	0.33 $\pm$ 0.16	-0.01 $\pm$ 0.14	0.12 $\pm$ 0.14
VPD + internal	0.45 $\pm$ 0.13	0.07 $\pm$ 0.12	0.10 $\pm$ 0.14
VPD + internal + no param	0.72 $\pm$ 0.14	-0.06 $\pm$ 0.14	0.20 $\pm$ 0.16

Appendix

Table 8: Meaning Localization results of adding or removing

\mathcal{L}_{param},\mathcal{L}_{ablate}

	GPT2		Gemma-2-2b		Qwen-3-8b
Method	Interp $\uparrow$	Sim $\downarrow$	Interp $\uparrow$	Sim $\downarrow$	Interp $\uparrow$	Sim $\downarrow$
ASPD (Ours)	0.68 $\pm$ 0.04	0.01 $\pm$ 0.00	0.62 $\pm$ 0.03	0.03 $\pm$ 0.00	0.57 $\pm$ 0.04	0.03 $\pm$ 0.00
PD Transcoder (Ours)	0.54 $\pm$ 0.04	0.00 $\pm$ 0.00	0.29 $\pm$ 0.03	0.07 $\pm$ 0.01	0.60 $\pm$ 0.04	0.05 $\pm$ 0.00
VPD + ( $g_{t,c}>0.01$ )	0.41 $\pm$ 0.03	0.00 $\pm$ 0.00	0.22 $\pm$ 0.02	0.46 $\pm$ 0.01	0.24 $\pm$ 0.02	0.19 $\pm$ 0.02
VPD + internal + ( $g_{t,c}>0.01$ )	0.36 $\pm$ 0.03	0.00 $\pm$ 0.00	0.32 $\pm$ 0.03	0.08 $\pm$ 0.01	0.26 $\pm$ 0.02	0.13 $\pm$ 0.02
VPD + ( $g_{t,c}>0.1$ )	0.53 $\pm$ 0.03	0.00 $\pm$ 0.00	0.22 $\pm$ 0.02	0.46 $\pm$ 0.01	0.26 $\pm$ 0.02	0.22 $\pm$ 0.01

Appendix

Table 9: Interpretability and Diversity experiment results of filtering

g_{t,c}>\tau

	GPT2		Gemma-2-2b		Qwen-3-8b
Method	Interp $\uparrow$	Sim $\downarrow$	Interp $\uparrow$	Sim $\downarrow$	Interp $\uparrow$	Sim $\downarrow$
VPD with adaptive $L_{0}$	0.34 $\pm$ 0.03	0.00 $\pm$ 0.00	0.21 $\pm$ 0.02	0.24 $\pm$ 0.00	0.20 $\pm$ 0.02	0.34 $\pm$ 0.01
VPD without adaptive $L_{0}$	0.37 $\pm$ 0.03	0.01 $\pm$ 0.00	0.20 $\pm$ 0.02	0.44 $\pm$ 0.01	0.22 $\pm$ 0.02	0.18 $\pm$ 0.00

Appendix

Table 10: Interpretability and Diversity experiment results comparing VPD with and without adaptive sparisty loss.

Single
	GPT2		Gemma-2-2b		Qwen-3-8b
Method	$ratio$ $\uparrow$	$localization$ $\uparrow$	$ratio$ $\uparrow$	$localization$ $\uparrow$	$ratio$ $\uparrow$	$localization$ $\uparrow$
VPD with adaptive $L_{0}$	0.2 $\pm$ 0.1	0.004 $\pm$ 0.002	0.6 $\pm$ 0.1	0.005 $\pm$ 0.001	0.9 $\pm$ 0.2	0.020 $\pm$ 0.003
VPD without adaptive $L_{0}$	0.9 $\pm$ 0.2	0.020 $\pm$ 0.002	0.5 $\pm$ 0.1	0.004 $\pm$ 0.001	0.9 $\pm$ 0.2	0.020 $\pm$ 0.003
Multiple
VPD with adaptive $L_{0}$	0.2 $\pm$ 0.0	0.003 $\pm$ 0.000	0.4 $\pm$ 0.0	0.004 $\pm$ 0.000	0.8 $\pm$ 0.0	0.017 $\pm$ 0.001

Appendix

Table 11: Weight Editing Localization experiment results comparing VPD with and without adaptive sparisty loss.

	GPT2	Gemma-2-2B	Qwen-3-8b
Method	Matching $\uparrow$	Matching $\uparrow$	Matching $\uparrow$
VPD with adaptive $L_{0}$	0.18 $\pm$ 0.10	-0.02 $\pm$ 0.12	0.05 $\pm$ 0.14
VPD without adaptive $L_{0}$	0.31 $\pm$ 0.11	-0.02 $\pm$ 0.10	0.17 $\pm$ 0.11

Appendix

Table 12: Meaning localization experiment results comparing VPD with and without adaptive sparisty loss.

Figure 4: $QK$ circuit at position $Q:9(John)-K:3(John)$ of $H3.0$ (“duplicate token head”). We observed that components $1178QK$ (“John”) contribute strongly to the attention pattern, and ablating those components suppresses the “duplicate token” attention pattern.

Figure 5: $OV$ circuit of $H3.0$ (“duplicate token head”). The components $1178V$ (“John”) and $224O$ (“duplicate name”) have a strong causal effect on each other. The component $224O$ writes the “duplicate token” signal to the activation space, which is the main role of $H3.0$ .

Figure 6: $QK$ circuit at position $Q:4(went)-K:3(John)$ of $H4.11$ (“previous token head”). We observed that components $12QK,70QK$ (dense components), $5740Q,440Q$ (“follow a name”), $341K$ (“name”) contribute strongly to the attention pattern, and ablating those components suppresses the “previous token” attention pattern.

Figure 7: $OV$ circuit of $H4.11$ (“previous token head”). The components $341V$ (“names”), $2774V$ (“John”) have a strong causal effect on $5690O$ (“follow a name”), $5608O$ (“follow names like John, James, Joan, etc.”). The components $5690O,5608O$ write the “follow a name (John)” signal to the activation space, which is the main role of $H4.11$ .

Figure 8: Connection between $H4.11$ (“previous token head”) and $5.5$ (“induction head”). Component $5690O\,H4.11$ (“tokens followed a name”), which is the component that writes the main functionality of $H4.11$ to the activation space, contributes strongly to the component $5897V\,H5.5$ (“tokens followed a name”).

Figure 9: $QK$ circuit at position $Q:9(John)-K:4(went)$ of $H5.5$ (“induction head”). We observed that components $164Q$ (“names”), $5897K$ (“token followed a name”) contribute strongly to the attention pattern and ablating those components suppress the “induction” attention pattern.

Figure 10: $OV$ circuit of $H5.5$ (“induction head”). The components $4897V$ (“tokens followed a name”) have a strong causal effect on $228O,665O,7290O$ (“name induction”). The components $228O,665O,7290O$ write the “induction” signal to the activation space, which is the main role of $H5.5$ .

Figure 11: Connection between $H3.0$ (“duplicate token head”), $5.5$ (“induction head”) and $H8.6$ (“S-inhibition head”). Components $224O\,H3.0$ (“duplicate name”) and $228O\,H5.5$ (“name induction”), which are the components that write the main functionality of $H3.0$ and $H5.5$ to the activation space, contribute strongly to the component $2861V,101V\,H8.6$ (“fire on pronouns, names - but fire on second names/pronouns stronger ”).

Figure 12: $QK$ circuit at position $Q:13(to)-K:9(John)$ of $H8.6$ (“S-inhibition head”). We observed that components $14Q$ (“preposition”), $101K$ (“fire on pronouns, names - but fire on second names/pronouns stronger ”) contribute strongly to the attention pattern, and ablating those components suppresses the “S-inhibition” attention pattern.

Figure 13: $OV$ circuit of $H8.6$ (“S-inhibition head”). The components $2861V,101V$ (“fire on pronouns, names - but fire on second names/pronouns stronger ”) have a strong causal effect with $223O$ (“suppress certain names”), $919O$ (“promote female pronouns”), $6080O$ (“next token is a name”), $260O$ (“promote male pronouns”). The components $223O,919O,6080O,260O$ suppress/promote “male/female pronouns ” and write the information to the activation space, which is the main role of $H8.6$ .

Figure 14: Connection between $H8.6$ (“S-inhibition head”) and $H9.6$ (“name mover head”). Components $223O\,H8.6$ (“suppress a certain names”), $260O\,H8.6$ (“promote male pronouns”), $6080O\,H8.6$ (“next token is a name”), which are the components that write the main functionality of $H8.6$ to the activation space, contribute strongly to the component $122Q\,H9.6$ (“contexts around a certain names”) and $254Q\,H9.6$ (“preposition, verbs connected to a person”).

Figure 15: $QK$ circuit at position $Q:13(to)-K:1(Mary)$ of $H9.6$ (“name mover head”). We observed that components $254Q$ (“prepositions, verbs connected to a person”), $122Q$ (“contexts around a certain name”), $293K$ (“names”) contribute strongly to the attention pattern, and ablating those components suppresses the “name mover” attention pattern.

Figure 16: $OV$ circuit of $H9.6$ (“name mover head”). The components $740V$ (“female, woman”) have a strong causal effect on $3796O$ (“predict next token is a woman/female name”), $17O$ (“predict next token is a male name”). The components $3796O,17O$ promote “male/female pronouns ” and write the information to the activation space, which is the main role of $H9.6$ .

Figure 17: Connection between $H8.6$ (“S-inhibition head”) and $H11.10$ (“negative name mover head”). Components $223O\,H8.6$ (“suppress certain names”), $260O\,H8.6$ (“promote male pronouns”), $6080O\,H8.6$ (“next token is a name”), which are the components that write the main functionality of $H8.6$ to the activation space, contribute strongly to the components $238Q,48Q,263Q\,H11.10$ (“prepositions, verbs”).

Figure 18: $QK$ circuit at position $Q:13(to)-K:1(Mary)$ of $H11.10$ (“negative name mover head”). We observed that components $238Q,48Q,263Q$ (“preposition, verbs”), $267K$ (“names”) contribute strongly to the attention pattern, and ablating those components suppresses the “negative name mover” attention pattern.

Figure 19: $OV$ circuit of $H11.10$ (“negative name mover head”). The components $1323V$ (“female pronouns”), $267V$ (“names”) have a strong causal effect on $60O$ (“suppressing female pronouns”), $3278O$ (“predict John”), $30O$ (“predict male names”). The components $60O,3278O,30O$ promote/suppress “male/female names/pronouns” and write the information to the activation space, which is the main role of $H11.10$ .

Figure 20: The image shows the component $5342\,Q$ (“tech, car, airplane companies”) at layer 9 reads the features $1964,13889,19129$ of residual-stream-pre which activate on “tech, car companies” context. $5342\,Q$ interacts with components $1518\,K$ that fires on “device and tech-related” and read the related features $18090,2403,2823$ from the residual-stream-pre, forming attention pattern.

Figure 21: The image shows the component $5306\,V$ (“hack, hacking”) at layer 5 reads the feature $16372$ of residual-stream-pre which activates on “hack” context. $5306\,V$ interacts with components $1454\,O$ that fires on “release, disclose” and write out news related features $31334$ at the output of the attention module.

Mechanistic Interpretability

Mechanistic interpretability studies how neural networks implement behaviors through internal features, circuits, and causal information flow.

See all 391 papers

Interpreting Language Model Parameters
Oct 14, 2026Lucius Bushnaq, Dan Braun, Oliver Clive-Griffin +5Mechanistic Interpretability Transformer Interpretability
Goodfire · MATS · Independent
InverseScope: Scalable Activation Inversion for Interpreting Large Language Models
Jun 9, 2025Yifan Luo, Zhennan Zhou, Bin DongRepresentation Geometry LLM Interpretability
School of Mathematical Science, Peking University, Beijing, China · School of Science, Westlake University, Hangzhou, China · Beijing International Center for Mathematical Research and the New Cornerstone Science Laboratory, Peking University, Beijing, China +1
Sparse Weight Decomposition for Efficient Circuit Extraction
Aug 4, 2026Chuanhao Yan, Xuhan Huang, Yawen Duan +4Transformer Interpretability Mechanistic Interpretability
IQuest Research · Safe AI Forum · University of Oxford +2

Oct 14, 2026J/K move · Enter open · S save

Interpreting Language Model Parameters

Lucius Bushnaq, Dan Braun, Oliver Clive-Griffin, Bart Bussmann+4

Goodfire · MATS · Independent

Mechanistic Interpretability Transformer Interpretability Circuits in Language Models

Open paper AskPDF arXiv

Mechanistic interpretability aims to uncover how neural networks use their parameters to implement their impressive neural algorithms. Although previous work has uncovered substantial structure in the intermediate representations that networks use, little progress has been made to understand how the parameters and nonlinearities of networks perform computations on those representations. In this work, we present a method that brings us closer to this understanding by decomposing a language model's parameters into subcomponents that each implement only a small part of the model's learned algorithm, while simultaneously requiring only a small fraction of those subcomponents to account for the network's behavior on any input. The method, adVersarial Parameter Decomposition (VPD), optimizes for decompositions of neural network parameters into simple subcomponents that preserve the network's input-output behavior even when many subcomponents are ablated, including under ablations that are adversarially selected to destroy behavior. This encourages learning subcomponents that provide short, mechanistically faithful descriptions of the network's behavior that should aggregate appropriately into more global descriptions of the network's learned algorithm. We study how sequences of interactions between these parameter subcomponents produce the network's output on particular inputs, enabling a new kind of 'circuit' analysis. While more work remains to be done to deepen our understanding of how neural networks use their parameters to compute their behavior, our work suggests an approach to identify a small set of simple, mechanistically faithful subcomponents on which further mechanistic analysis can be based.

Abstract

Mechanistic Interpretability

Explore similar work

Interpreting Language Model Parameters

InverseScope: Scalable Activation Inversion for Interpreting Large Language Models

Sparse Weight Decomposition for Efficient Circuit Extraction