Beyond Static Interpretability: Anticipating Post-SFT Mechanisms from Pre-SFT Parameters for Better Tuning
Authors: Hang Chen, Jiaying Zhu, Wenya Wang
Organizations: College of Computing and Data Science Nanyang Technological University · School of Computer Science and Engineering The Chinese University of Hong Kong
Mechanistic Localization bridges mechanistic interpretability and post-training optimization by isolating critical parameters via interpretative approaches and then guiding parameter-efficient Supervised Fine-Tuning (SFT) in a ``locating-then-tuning'' paradigm. However, due to the retrospective nature of mechanistic interpretability, directly interpreting pre-SFT models introduces misleading conclusions. Specifically for novel tasks, initially identified neurons differ drastically from those governing the final model, introducing biases that actively disrupt SFT. To address this, we propose a forward-looking localization framework that accurately estimates the post-SFT interpretability state using only pre-SFT parameters and the target dataset. Theoretically, we model SFT as a continuous parameter evolution, leveraging Taylor expansion to rigorously bridge the post-tuning mechanistic objective with the pre-SFT model's dynamic gradients. Practically, we design dual-granularity (neuron- and component-level) localization pipelines. Extensive experiments demonstrate that our approach not only provides superior SFT guidance but also exhibits robust performance and temporal scalability across increasing model sizes. This work transcends the fundamental limitation of traditional interpretability-its inability to identify task-critical mechanisms before they are trained-pioneering a predictive frontier that unites mechanistic interpretability with targeted optimization.
Figures & tables
Figure 1: Extrapolation of the ideal state via the step scalar K . Different values of K yield distinct sensitivity estimations S(θ+K⋅Δθ) , reflecting that the ideal parameter state θI consists of an infinite set of valid configurations rather than a single deterministic solution.
Method
LR
NLU
MR
TTA
PTA
TTA
PTA
TTA
PTA
Graft
82.76 ±0.42
33.27 ±0.04
84.96 ±0.08
31.24 ±0.14
24.19 ±2.13
44.31 ±0.11
FLU
88.57 ±0.35
42.57 ±0.09
84.27 ±0.02
36.72 ±0.13
87.16 ±1.07
49.52 ±0.01
WAGLE
89.55 ±0.28
44.31 ±0.06
85.28 ±0.11
43.57 ±0.04
66.39 ±1.15
51.23 ±0.09
CLUE
90.49 ±0.44
45.42 ±0.02
85.17 ±0.07
45.61 ±0.10
52.78 ±1.55
53.21 ±0.12
CircuitLoRA
97.67 ±0.13
60.62 ±0.11
85.09 ±0.06
61.55 ±0.09
98.60 ±0.12
59.66 ±0.14
Table 1: Performance comparison of Locating-then-tuning methods.
Figure 2: The scalability of performance and time cost with mode size increasing.
Strategy
Neuron-Level
Component-Level
LR
NLU
MR
LR
NLU
MR
Full-Param
69.27
94.03
15.66
99.73
96.33
71.83
Random
88.31
95.03
43.77
99.64
96.86
76.41
Static
87.49
95.52
55.83
99.75
96.33
83.52
Probing
92.76
96.10
87.16
99.96
96.21
99.60
Ours
98.51
96.31
98.79
100.00
97.55
100.00
Table 2: Ablation study across datasets at neuron and component levels.
Figure 3: Ablation study results on the arithmetic task. (a) Performance variations across different digits. (b) Performance variations across different reasoning steps.
Figure 4: Top 50 overlap with Full-Parameter fine-tuning model across different probing SFT setups.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Train
Test
Max Len
Output
Input Prompt Case
Label
IOI
6,000
600
50
Correct name
Then, Aaron and Amber were working at the house. Aaron decided to give a drink to
Amber
Gender
3,000
3,000
50
he / she
So Kelly is always up for an adventure, isn’t
she
BOOL
10,000
4,000
100
True / False
Evaluate the following boolean expression as either ’True’ or ’False’. ( not ( True or True ) ) and ( False or ( not True ) )
False
SST-2
67,349
872
200
positive / negative
Is the sentiment of following sentence positive or negative? that ’s far too tragic to merit such superficial treatment. Answer: It is
negative
MRPC
3,670
1,730
200
Yes / No
Are the meanings of the following two sentences equivalent? 1. And if both apply, they are essentially possible. 2. And if both apply, they are essentially impossible.
No
QQP
364,000
391,000
200
Yes / No
Is Question “How do you control your horniness?” a duplicate of Question “How do I control my horny emotions?”?
Yes
Appendix
Table 3: Dataset statistics, configurations, and prompt construction examples.
Figure 5: Performance in multi-task accuracy. The solid line represents the mean value, and the dashed line represents the lower bound of the value variation. The shaded areas represent the variances of Our_C and Our_N, respectively.
Figure 6: Evolution of conflict component in fine-tuning.
Figure 7: Overlap rate of the top 50 components with the full-parameter circuit ( CFull-Param ) across tasks with varying initial accuracies.
Method / Strategy
Bool
SST-2
Arithmetic
K
Top@50
TTA
K
Top@50
TTA
K
Top@50
TTA
1-Sample Uniform
35.6
28
99.64
2.1
26
96.34
71.5
30
99.51
10-Sample Average
[10, 80]
30.4
100.00
[1, 10]
29.6
97.55
[10, 80]
31.4
100.00
Probe (First Layer)
15.3
24
98.82
8.73
28
95.17
24.8
26
99.17
Probe (Middle Layer)
67.2
32
99.67
1.2
34
96.72
72.4
30
99.88
Probe (Last Layer)
58.5
34
99.83
1.2
30
97.26
77.2
32
100.00
Appendix
Table 4: Ablation study on determining the extrapolation scalar K via linear probing across different residual stream depths on the Mistral-7B backbone, evaluated against standard sampling baselines.
Figure 8: Robustness analysis of K-value sampling.
Figure 9: ρ and CV across different stochastic factors. Each set is randomly repeated 10 times on Ours_C in the Mistral-7B model.
Method
Induction
Gender
IOI
Docstring
Top@50
KL
Top@50
KL
Top@50
KL
Top@50
KL
CFull-Param
100
0
100
0
100
0
100
0
CStatic
22
1.37
18
1.57
20
1.72
24
1.31
CProbing
18
1.29
22
2.23
22
1.25
22
1.04
COurs
28
1.03
36
1.31
32
0.93
28
1.01
Appendix
Table 5: Comparison of circuit overlap (Top@50) and KL divergence against the full-parameter finetuned model across interpretability datasets in component level.
Method
Induction
Gender
IOI
Docstring
Top@50
KL
Top@50
KL
Top@50
KL
Top@50
KL
CFull-Param
100
0
100
0
100
0
100
0
CStatic
20
1.55
24
1.61
26
1.49
20
1.52
CProbing
20
1.51
18
1.72
20
1.62
18
1.27
COurs
32
1.11
34
1.13
34
1.21
30
1.15
Appendix
Table 6: Comparison of circuit overlap (Top@50) and KL divergence against the full-parameter finetuned model across interpretability datasets in neuron level.
Figure 10: Ablation on Second-Order Approximation across Expanding Edge Capacities
Figure 11: Circuit graphs corresponding to the three types of parameters.
Figure 12: Top 50 overlap with Full-Parameter fine-tuning model across different probing SFT setups.
The "Locate-then-Update" paradigm has become a predominant approach in the post-training of large language models (LLMs), identifying critical components via mechanistic interpretability for targeted parameter updates. However, this paradigm rests on a fundamental yet unverified assumption: can mechanisms derived from current static parameters reliably guide future dynamic parameter updates? To investigate this, we systematically track the structural evolution of Transformer circuits throughout the supervised fine-tuning (SFT) process, revealing the underlying dynamics of task mechanisms. We introduce three novel metrics-Circuit Distance, Circuit Stability, and Circuit Conflict-to analyze circuit evolution across three dimensions: neural migration, semantic stability, and cross-task interference. Our empirical results reveal that circuits inherently exhibit "Free Evolution" during parameter updates. Consequently, static mechanisms extracted from current states inevitably suffer from temporal latency, making them fundamentally inadequate for guiding future states. Moreover, by deconstructing the "illusion of effectiveness" in existing methods, this work underscores the necessity of "foresight" in mechanistic localization and proposes a predictive framework for future research.
Hang Chen, Jiaying Zhu, Hongyang Chen +3
School of Computer Science and Technology Xi’an Jiaotong University · School of Computer Science and Engineering The Chinese University of Hong Kong · Shaanxi Co., Ltd(Xi’an 710077, China) China Mobile Group +1
While critical for alignment, Supervised Fine-Tuning (SFT) incurs the risk of catastrophic forgetting, yet the layer-wise emergence of instruction-following capabilities remains elusive. We investigate this mechanism via a comprehensive analysis utilizing information-theoretic, geometric, and optimization metrics across model scales (1B-32B). Our experiments reveal a distinct depth-dependent pattern: middle layers (20%-80%) are stable, whereas final layers exhibit high sensitivity. Leveraging this insight, we propose Mid-Block Efficient Tuning, which selectively updates these critical intermediate layers. Empirically, our method outperforms standard LoRA up to 10.2% on GSM8K (OLMo2-7B) with reduced parameter overhead, demonstrating that effective alignment is architecturally localized rather than distributed. The code is publicly available at https://anonymous.4open.science/r/base_sft.
The cosine similarity between a large language model's hidden activations before and after Supervised Fine-Tuning (SFT) remains very high. This, at first glance, suggests that SFT leaves the model's activation geometry largely undisturbed. However, projecting both sets of activations through a Sparse Autoencoder (SAE) pretrained on the base model reveals that the underlying sparse latents diverge significantly. We introduce a novel investigative pipeline which utilizes these pretrained SAEs as a high-resolution diagnostic tool to mechanistically investigate the drivers of this representational divergence. Through our analytical pipeline, we discover task-specific and layer-specific distributions of the precise semantic features that are systematically altered during supervised fine-tuning. We additionally identify a layer-wise update profile specific to safety alignment. All code, experimental scripts, and analysis files associated with this work are publicly available at: https://github.com/ruhzi/sae-investigation.