Decoupling Internal Representational Changes and Causal Importance in Fine-Tuned Large Language Models

Lingfang Li; Procheta Sen; Shubham Das; Danushka Bollegala

cs.AISep 17, 2026

Decoupling Internal Representational Changes and Causal Importance in Fine-Tuned Large Language Models

Authors: Lingfang Li, Procheta Sen, Shubham Das, Danushka Bollegala

Organizations: University of Liverpool, United Kingdom · IIEST, Shibpur, India · Amazon

Abstract

Fine-tuning has emerged as a widely adopted approach for adapting LLMs to a variety of downstream tasks. However, how it reshapes their internal mechanisms remains poorly understood. To address this, we investigate how fine-tuning alters internal representations in LLMs, including attention patterns and layer-wise activations, and examine whether these changes are linked to task-relevant components identified by EAP (e.g., attention heads and logit-level activations) that drive task performance. We find that EAP-identified components are concentrated within specific layers, indicating a degree of functional localisation in how models internalise task-specific behavior. Notably, the distribution of these components across layers is largely uncorrelated with the layers undergoing the most substantial representational changes during fine-tuning. Furthermore, we observe that overlap in EAP-identified components across tasks does not translate into cross-task performance transfer if the tasks are different in nature (e.g. classification vs. generative tasks). More specifically, fine-tuning on one task can lead to a degradation of performance on another when the two tasks exhibit a high degree of overlap in their EAP-identified components.

Figures & tables

Figure 1: Examples of task-specific EAP -identified components (e.g., attention heads, MLPs) for sentiment classification (top), question answering (middle), and machine translation (bottom). Notations are described in Appendix A .

Panel of Figure 2 — Figure 2: Layer-wise distribution of average KL divergence in the attention matrices before and after FT , and summed EAP scores across Fine-tuned Qwen2 and llama-3.2-1B. For GPT-2 Small and Llama-2-7B refer to Figure 12 in Appendix L ; results for Qwen2 and Llama-3.2-1B on the selected remaining datasets are shown in Figure 13 in Appendix L . Values in parentheses are the normalised layer-wise entropy $(\tilde{H}_{\text{attn}},\,\tilde{H}_{\text{EAP}})$ ; the consistently lower $\tilde{H}_{\text{EAP}}$ indicates that EAP importance is more localised across layers. See Appendix E for the entropy definition.

Figure 3: Probing results of pretrained model vs. fine-tuned model across all the layers of Llama-3.2-1B, Qwen2-0.5B, Llama-2-7B for Sentiment Classification (SST-2), Question Answering (SQuAD), and Machine Translation (KDE4). Probing performance for each task is measured by the corresponding task metric as described in Section 3.1 . Higher values indicate that the correct task output is more readily recoverable from that layer’s representation via the logit lens. The complete layer-wise probing results for all four models across all six datasets are shown in Figure 14 in Appendix L . (X-axis: layer index, Y-axis: binary accuracy for SST-2, SQuAD F1 Score for SQuAD, BERTScore for KDE4).

Panel of Figure 4 — Figure 4: Correlation between Cross-task Performance and Important Component Overlap for Llama-3.2-1B, Qwen2-0.5B and Llama-2-7B. $\rho$ is correlation coefficient and p-values show the significance of the correlation coefficient. The result for GPT-2 Small is provided in Figure 10 , Appendix L .

Figure 5: Distribution percentage of important components among the top-400 edges in Llama-3.2-1B across three tasks: Sentiment Classification (SST-2), Question Answering (SQuAD). Machine Translation (KDE4). The full results for four models and six datasets are given in Figure 8 in Appendix G .

Model	Task	#Induction Heads	#Attention Heads	Overlap(%)
GPT-2 Small	Sentiment (Yelp)	26	101	8.54%
	Sentiment (SST-2)	24	80	1.96%
	QA (SQuAD)	24	98	12.96%
	QA (CoQA)	24	109	16.66%
	MT (KDE4)	24	99	7.89%
	MT (Tatoeba)	24	93	7.33%

Table 1: Overlap of Induction head vs. Attention heads derived from the top-400 EAP-ranked edges across all models for Sentiment Analysis (Sent), Question Answering (QA) and Machine Translation (MT). For each model the highest overlap is boldfaced and the highest overlap percentage across all the models is underlined.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Task Information	Original Input Sequence	Corrupted Input Sequence
Task 1: Sentiment Classification (SST-2/Yelp)
Info Column: Sentiment Label; Input Column: Review Text
Label: Negative	a sometimes tedious film.	a sometimes X film.
Label: Positive	among the year’s most intriguing explorations of alienation.	among the year’s most X explorations of alienation.
Task 2: Machine Translation (Tatoeba/KDE4)
Info Column: Target Translation (French); Input Column: Source English

Appendix

Table 2: Examples of corrupted samples across three downstream tasks.

Panel of Figure 6 — Figure 6: The few-shot prompt template used to extract sentiment-sensitive words via Llama-3-8B-Instruct. We dynamically insert the specific sentiment label (positive/negative) and the review text into the placeholders (marked in blue) for each query.

Panel of Figure 7 — Figure 7: GPT-2 Small, Llama-3.2-1B, Qwen2-0.5B and Llama-2-7B performance with using only top-K edges obtained from EAP. The first row shows the performance on Yelp data, the second row shows the performance on SQuAD data, the third row shows the performance on KDE4 data. Orange line shows the performance of the finetuned model, blue line shows the performance with variation of top-k edges and green line shows the performance using randomly selected k edges.

Model	Task	EAP	Random	$\Delta$
GPT-2 Small	Sent (Yelp)	85%	61%	24
	QA (SQuAD)	80%	55%	25
	MT (KDE4)	80%	67%	13
Llama-2-7B	Sent (Yelp)	81%	58%	23
	QA (SQuAD)	84%	52%	32
	MT (KDE4)	79%	54%	25

Appendix

Table 3: Performance drop caused by intervening on the top-400 EAP-ranked edges versus the same number of randomly selected edges. Random results are averaged over 20 seeds.

\Delta

is the absolute difference in percentage points. The difference is significant for every model–task pair (two-sided Wilcoxon signed-rank test,

p<0.05

Corruption token	Spearman $\rho$ vs. <mask>
neutral	0.71
blank	0.72
abc	0.71

Appendix

Table 4: Spearman rank correlation between edge-importance scores obtained with alternative corruption tokens and those obtained with <mask> , computed over all edges of GPT-2 Small on the sentiment classification task.

Parameter	Llama-2 (7B)	Llama-3.2 (1B)	Qwen2 (0.5B)	GPT-2 (Small)
Core Optimization
Optimizer	AdamW (fused)	AdamW (fused)	AdamW (fused)	AdamW (fused)
Learning Rate	$2\times 10^{-5}$	$2\times 10^{-5}$	$2\times 10^{-5}$	$5\times 10^{-5}$ $\dagger$
Weight Decay	0.0	0.0	0.0	0.0
Warmup Ratio	0.03	0.03	0.03	0.03
LR Scheduler	Cosine	Cosine	Cosine	Cosine

Appendix

Table 5: Complete hyperparameter settings across models and tasks. All four models are fully fine-tuned. Effective batch size is per-device batch size

\times

gradient-accumulation steps.

\dagger

GPT-2 SQuAD/CoQA used

2\times 10^{-5}

Similar Task Analysis
Task Pair		GPT2		Llama3.21B		Qwen2.05B		Llama-2-7B
Fine-tuning Task	Evaluation Task	Perf $\Delta$	Overlap $\Delta$	Perf $\Delta$	Overlap $\Delta$	Perf $\Delta$	Overlap $\Delta$	Perf $\Delta$	Overlap $\Delta$
Yelp	Yelp	34% $\uparrow$	100 %	33.81% $\uparrow$	100%	25.1% $\uparrow$	100%	91% $\uparrow$	100%
Yelp	SST-2	47% $\uparrow$	45.25%	52.2% $\uparrow$	39.75%	23.51% $\uparrow$	33.50%	94.5% $\uparrow$	16.75%
SST-2	SST-2	52.81% $\uparrow$	100%	21.92% $\uparrow$	100%	31.77% $\uparrow$	100%	96.44% $\uparrow$	100%
SST-2	Yelp	29.39% $\uparrow$	37.25%	8.41% $\uparrow$	40.5%	16.7% $\uparrow$	35.00 %	89.2% $\uparrow$	8.00 %

Appendix

Table 6: Cross-task Performance and Important Component Overlap Analysis for GPT-2 Small, Llama-3.2-1B, Qwen2-0.5B and Llama-2-7B.

Figure 8: Important component distribution percentage over top-400 EAP edges, broken down by model(columns) and dataset(rows).The Llama-3.2-1B column on SST-2 / SQuAD / KDE4 reproduces the main-text Figure 5 .

Panel of Figure 9 — Figure 9: Architecture-normalised component composition of the top-400 EAP edges. For each model (a-d), bars show the scaling factor $r_{c}=f_{c}^{\text{top}}/f_{c}^{\text{model}}$ , i.e., the fraction of top-400 edges of component type $c$ divided by the fraction of type $c$ in the full model. Components are grouped along the $x$ -axis (attention heads, MLP, logit, embedding) and the six bars per group correspond to the six tasks. A value above one indicates that the component type is selected more often than its baseline abundance would predict. See Appendix G for the definition.

Model	FT task	Test task	Change
Llama-3.2-1B	MT	QA	$\uparrow$ 3.54%
	QA	MT	$\uparrow$ 45.74%
	Sent	MT	$\uparrow$ 7.91%
	Sent	QA	$\uparrow$ 41.93%
Qwen2-0.5B	QA	MT	$\uparrow$ 84.34%
	MT	QA	$\uparrow$ 44.47%

Appendix

Table 7: Effect of freezing the overlapping EAP-identified components during fine-tuning. The model is fine-tuned on Task 1 with the components shared between Task 1 and Task 2 held fixed, and then evaluated on Task 2. The last column reports the relative change in Task 2 performance compared to standard full fine-tuning on Task 1. Performance on Task 2 improves in every configuration.

Figure 11: Token-level CKA between the pre-trained and the fine-tuned model at each layer, for the four models (lines) across the six datasets (panels). The

x

-axis is relative depth (layer index divided by the number of layers), which allows models of different depths to be compared on a common scale. A CKA of

1

means the representation at that layer is unchanged by fine-tuning; lower values indicate larger representational change.

Panel of Figure 12 — Figure 12: Layer-wise distribution of average attention KL divergence and EAP scores for GPT-2 Small and Llama-2-7B across Sentiment Classification, Question Answering, and Machine Translation.

Panel of Figure 13 — Figure 13: Layer-wise distribution of average attention KL divergence and EAP scores for Qwen2 and Llama-3.2-1B across Sentiment Classification (Yelp), Question Answering (CoQA), and Machine Translation (Tatoeba). Counterpart of Figure 2 .

Figure 14: Layer-wise probing of pre-trained vs. fine-tuned models, full version of Figure 3 . Rows are datasets (Yelp, SST-2, SQuAD, CoQA, KDE4, Tatoeba); columns are models.(GPT-2 Small, Llama-3.2-1B, Qwen2-0.5B, Llama-2-7B). Higher = more task-relevant information is already recoverable at that layer. (X-axis: layer index; Y-axis: binary accuracy for SST-2, SQuAD F1 Score for SQuAD, BERTScore for KDE4)

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 758 papers

Where to Adapt Matters: Layer-Selective Fine-Tuning for Capability Retention
Oct 8, 2026Zhiqiang Pang, Zihong Sun, Qi Xie +3LLM Fine-Tuning Selective Fine-Tuning
School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Shaanxi, China
Analyzing the Effect of Noise in LLM Fine-tuning
Apr 14, 2026Lingfang Li, Procheta SenLLM Fine-Tuning Language Model Robustness
Representation Collapse in Sequential Post-Training of Large Language Models
May 28, 2026Yichen Liu, Mingyu Chen, Hao Wang +7Continual Learning for LLMs Representation Collapse
Hangzhou Dianzi University · Zhejiang Gongshang University · Ningbo University +1

Oct 8, 2026J/K move · Enter open · S save

Where to Adapt Matters: Layer-Selective Fine-Tuning for Capability Retention

Zhiqiang Pang, Zihong Sun, Qi Xie, Jun Shu+2

School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Shaanxi, China

LLM Fine-Tuning Selective Fine-Tuning Parameter-Efficient Fine-Tuning Catastrophic Forgetting Low-Rank Adaptation

Open paper AskPDF arXiv

Parameter-efficient fine-tuning (PEFT) enables large language models (LLMs) to adapt to specialized tasks, but often at the cost of degrading general capabilities acquired during pretraining. Existing approaches primarily mitigate this trade-off through data replay or regularization, relying on additional data or explicit optimization constraints. We instead focus on a different question: where should adaptation be applied? We find that fine-tuning different Transformer layers produces different target-task gains and degrees of capability degradation, suggesting that not all layers are equally suitable for adaptation. To characterize this difference, we use layer-wise empirical Fisher information to measure target-task sensitivity. However, computing Fisher scores requires backward computation and becomes increasingly expensive for large models. We therefore introduce input--output cosine similarity as a lightweight, forward-only proxy for ranking layer sensitivity. Across models and tasks, layers with lower input--output similarity consistently exhibit higher empirical Fisher scores. Building on this observation, we propose Layer-Selective LoRA (LS-LoRA), which places trainable LoRA adapters only in layers with low input--output similarity. Experiments on mathematical reasoning and code generation show that LS-LoRA improves average target-task performance while retaining substantially more commonsense reasoning capability than standard all-layer LoRA, demonstrating that carefully choosing where to adapt can provide a simple and effective way to balance target-task adaptation and general capability retention.

Abstract

LLM Fine-Tuning

Explore similar work

Where to Adapt Matters: Layer-Selective Fine-Tuning for Capability Retention

Analyzing the Effect of Noise in LLM Fine-tuning

Representation Collapse in Sequential Post-Training of Large Language Models