Fine-tuning has emerged as a widely adopted approach for adapting LLMs to a variety of downstream tasks. However, how it reshapes their internal mechanisms remains poorly understood. To address this, we investigate how fine-tuning alters internal representations in LLMs, including attention patterns and layer-wise activations, and examine whether these changes are linked to task-relevant components identified by EAP (e.g., attention heads and logit-level activations) that drive task performance. We find that EAP-identified components are concentrated within specific layers, indicating a degree of functional localisation in how models internalise task-specific behavior. Notably, the distribution of these components across layers is largely uncorrelated with the layers undergoing the most substantial representational changes during fine-tuning. Furthermore, we observe that overlap in EAP-identified components across tasks does not translate into cross-task performance transfer if the tasks are different in nature (e.g. classification vs. generative tasks). More specifically, fine-tuning on one task can lead to a degradation of performance on another when the two tasks exhibit a high degree of overlap in their EAP-identified components.
Figures & tables
Figure 1: Examples of task-specific EAP -identified components (e.g., attention heads, MLPs) for sentiment classification (top), question answering (middle), and machine translation (bottom). Notations are described in Appendix A .
Figure 2: Layer-wise distribution of average KL divergence in the attention matrices before and after FT , and summed EAP scores across Fine-tuned Qwen2 and llama-3.2-1B. For GPT-2 Small and Llama-2-7B refer to Figure 12 in Appendix L ; results for Qwen2 and Llama-3.2-1B on the selected remaining datasets are shown in Figure 13 in Appendix L . Values in parentheses are the normalised layer-wise entropy (H~attn,H~EAP) ; the consistently lower H~EAP indicates that EAP importance is more localised across layers. See Appendix E for the entropy definition.
Figure 3: Probing results of pretrained model vs. fine-tuned model across all the layers of Llama-3.2-1B, Qwen2-0.5B, Llama-2-7B for Sentiment Classification (SST-2), Question Answering (SQuAD), and Machine Translation (KDE4). Probing performance for each task is measured by the corresponding task metric as described in Section 3.1 . Higher values indicate that the correct task output is more readily recoverable from that layer’s representation via the logit lens. The complete layer-wise probing results for all four models across all six datasets are shown in Figure 14 in Appendix L . (X-axis: layer index, Y-axis: binary accuracy for SST-2, SQuAD F1 Score for SQuAD, BERTScore for KDE4).
Figure 4: Correlation between Cross-task Performance and Important Component Overlap for Llama-3.2-1B, Qwen2-0.5B and Llama-2-7B. ρ is correlation coefficient and p-values show the significance of the correlation coefficient. The result for GPT-2 Small is provided in Figure 10 , Appendix L .
Figure 5: Distribution percentage of important components among the top-400 edges in Llama-3.2-1B across three tasks: Sentiment Classification (SST-2), Question Answering (SQuAD). Machine Translation (KDE4). The full results for four models and six datasets are given in Figure 8 in Appendix G .
Model
Task
#Induction Heads
#Attention Heads
Overlap(%)
GPT-2 Small
Sentiment (Yelp)
26
101
8.54%
Sentiment (SST-2)
24
80
1.96%
QA (SQuAD)
24
98
12.96%
QA (CoQA)
24
109
16.66%
MT (KDE4)
24
99
7.89%
MT (Tatoeba)
24
93
7.33%
Table 1: Overlap of Induction head vs. Attention heads derived from the top-400 EAP-ranked edges across all models for Sentiment Analysis (Sent), Question Answering (QA) and Machine Translation (MT). For each model the highest overlap is boldfaced and the highest overlap percentage across all the models is underlined.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Task Information
Original Input Sequence
Corrupted Input Sequence
Task 1: Sentiment Classification (SST-2/Yelp)
Info Column: Sentiment Label; Input Column: Review Text
Label: Negative
a sometimes tedious film.
a sometimes X film.
Label: Positive
among the year’s most intriguing explorations of alienation.
among the year’s most X explorations of alienation.
Task 2: Machine Translation (Tatoeba/KDE4)
Info Column: Target Translation (French); Input Column: Source English
Appendix
Table 2: Examples of corrupted samples across three downstream tasks.
Figure 6: The few-shot prompt template used to extract sentiment-sensitive words via Llama-3-8B-Instruct. We dynamically insert the specific sentiment label (positive/negative) and the review text into the placeholders (marked in blue) for each query.
Figure 7: GPT-2 Small, Llama-3.2-1B, Qwen2-0.5B and Llama-2-7B performance with using only top-K edges obtained from EAP. The first row shows the performance on Yelp data, the second row shows the performance on SQuAD data, the third row shows the performance on KDE4 data. Orange line shows the performance of the finetuned model, blue line shows the performance with variation of top-k edges and green line shows the performance using randomly selected k edges.
Model
Task
EAP
Random
Δ
GPT-2 Small
Sent (Yelp)
85%
61%
24
QA (SQuAD)
80%
55%
25
MT (KDE4)
80%
67%
13
Llama-2-7B
Sent (Yelp)
81%
58%
23
QA (SQuAD)
84%
52%
32
MT (KDE4)
79%
54%
25
Appendix
Table 3: Performance drop caused by intervening on the top-400 EAP-ranked edges versus the same number of randomly selected edges. Random results are averaged over 20 seeds. Δ is the absolute difference in percentage points. The difference is significant for every model–task pair (two-sided Wilcoxon signed-rank test, p<0.05 ).
Corruption token
Spearman ρ vs. <mask>
neutral
0.71
blank
0.72
abc
0.71
Appendix
Table 4: Spearman rank correlation between edge-importance scores obtained with alternative corruption tokens and those obtained with <mask> , computed over all edges of GPT-2 Small on the sentiment classification task.
Parameter
Llama-2 (7B)
Llama-3.2 (1B)
Qwen2 (0.5B)
GPT-2 (Small)
Core Optimization
Optimizer
AdamW (fused)
AdamW (fused)
AdamW (fused)
AdamW (fused)
Learning Rate
2×10−5
2×10−5
2×10−5
5×10−5†
Weight Decay
0.0
0.0
0.0
0.0
Warmup Ratio
0.03
0.03
0.03
0.03
LR Scheduler
Cosine
Cosine
Cosine
Cosine
Appendix
Table 5: Complete hyperparameter settings across models and tasks. All four models are fully fine-tuned. Effective batch size is per-device batch size × gradient-accumulation steps. † GPT-2 SQuAD/CoQA used 2×10−5 .
Similar Task Analysis
Task Pair
GPT2
Llama3.21B
Qwen2.05B
Llama-2-7B
Fine-tuning Task
Evaluation Task
Perf Δ
Overlap Δ
Perf Δ
Overlap Δ
Perf Δ
Overlap Δ
Perf Δ
Overlap Δ
Yelp
Yelp
34% ↑
100 %
33.81% ↑
100%
25.1% ↑
100%
91% ↑
100%
Yelp
SST-2
47% ↑
45.25%
52.2% ↑
39.75%
23.51% ↑
33.50%
94.5% ↑
16.75%
SST-2
SST-2
52.81% ↑
100%
21.92% ↑
100%
31.77% ↑
100%
96.44% ↑
100%
SST-2
Yelp
29.39% ↑
37.25%
8.41% ↑
40.5%
16.7% ↑
35.00 %
89.2% ↑
8.00 %
Appendix
Table 6: Cross-task Performance and Important Component Overlap Analysis for GPT-2 Small, Llama-3.2-1B, Qwen2-0.5B and Llama-2-7B.
Figure 8: Important component distribution percentage over top-400 EAP edges, broken down by model(columns) and dataset(rows).The Llama-3.2-1B column on SST-2 / SQuAD / KDE4 reproduces the main-text Figure 5 .
Figure 9: Architecture-normalised component composition of the top-400 EAP edges. For each model (a-d), bars show the scaling factor rc=fctop/fcmodel , i.e., the fraction of top-400 edges of component type c divided by the fraction of type c in the full model. Components are grouped along the x -axis (attention heads, MLP, logit, embedding) and the six bars per group correspond to the six tasks. A value above one indicates that the component type is selected more often than its baseline abundance would predict. See Appendix G for the definition.
Figure 10: GPT2, ρ=0.5
Model
FT task
Test task
Change
Llama-3.2-1B
MT
QA
↑ 3.54%
QA
MT
↑ 45.74%
Sent
MT
↑ 7.91%
Sent
QA
↑ 41.93%
Qwen2-0.5B
QA
MT
↑ 84.34%
MT
QA
↑ 44.47%
Appendix
Table 7: Effect of freezing the overlapping EAP-identified components during fine-tuning. The model is fine-tuned on Task 1 with the components shared between Task 1 and Task 2 held fixed, and then evaluated on Task 2. The last column reports the relative change in Task 2 performance compared to standard full fine-tuning on Task 1. Performance on Task 2 improves in every configuration.
Figure 11: Token-level CKA between the pre-trained and the fine-tuned model at each layer, for the four models (lines) across the six datasets (panels). The x -axis is relative depth (layer index divided by the number of layers), which allows models of different depths to be compared on a common scale. A CKA of 1 means the representation at that layer is unchanged by fine-tuning; lower values indicate larger representational change.
Figure 12: Layer-wise distribution of average attention KL divergence and EAP scores for GPT-2 Small and Llama-2-7B across Sentiment Classification, Question Answering, and Machine Translation.
Figure 13: Layer-wise distribution of average attention KL divergence and EAP scores for Qwen2 and Llama-3.2-1B across Sentiment Classification (Yelp), Question Answering (CoQA), and Machine Translation (Tatoeba). Counterpart of Figure 2 .
Figure 14: Layer-wise probing of pre-trained vs. fine-tuned models, full version of Figure 3 . Rows are datasets (Yelp, SST-2, SQuAD, CoQA, KDE4, Tatoeba); columns are models.(GPT-2 Small, Llama-3.2-1B, Qwen2-0.5B, Llama-2-7B). Higher = more task-relevant information is already recoverable at that layer. (X-axis: layer index; Y-axis: binary accuracy for SST-2, SQuAD F1 Score for SQuAD, BERTScore for KDE4)
Parameter-efficient fine-tuning (PEFT) enables large language models (LLMs) to adapt to specialized tasks, but often at the cost of degrading general capabilities acquired during pretraining. Existing approaches primarily mitigate this trade-off through data replay or regularization, relying on additional data or explicit optimization constraints. We instead focus on a different question: where should adaptation be applied? We find that fine-tuning different Transformer layers produces different target-task gains and degrees of capability degradation, suggesting that not all layers are equally suitable for adaptation. To characterize this difference, we use layer-wise empirical Fisher information to measure target-task sensitivity. However, computing Fisher scores requires backward computation and becomes increasingly expensive for large models. We therefore introduce input--output cosine similarity as a lightweight, forward-only proxy for ranking layer sensitivity. Across models and tasks, layers with lower input--output similarity consistently exhibit higher empirical Fisher scores. Building on this observation, we propose Layer-Selective LoRA (LS-LoRA), which places trainable LoRA adapters only in layers with low input--output similarity. Experiments on mathematical reasoning and code generation show that LS-LoRA improves average target-task performance while retaining substantially more commonsense reasoning capability than standard all-layer LoRA, demonstrating that carefully choosing where to adapt can provide a simple and effective way to balance target-task adaptation and general capability retention.
Zhiqiang Pang, Zihong Sun, Qi Xie +3
School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Shaanxi, China
Fine-tuning is the dominant paradigm for adapting pretrained large language models (LLMs) to downstream NLP tasks. In practice, fine-tuning datasets may contain various forms of noise arising from annotation errors, preprocessing artifacts, or automated data collection. While prior work has focused on designing robust learning algorithms to mitigate performance degradation under noisy conditions, comparatively little is known about how different types of noise affect the internal learning dynamics of LLMs during fine-tuning. In this work, we systematically study the impact of noise on model behavior across three pretrained model families (GPT-2, Qwen2 and Llama-2) and three diverse NLP tasks. We introduce controlled perturbations corresponding to three common real-world noise types: label noise, grammatical noise, and typographical noise. Beyond task-level performance, we analyze layer-wise representation changes and attention patterns to understand how noise propagates through the network. Our results show that corrupting labels (i.e. label noise) consistently causes the largest performance degradation, whereas grammatical noise and typographical noise can occasionally yield mild regularization benefits. We further find that noise effects are localized primarily to task-specific layers, while attention structures remain comparatively stable.
Large language models are now adapted through chains of post-training stages rather than through a single instruction-tuning pass. This paper studies whether such sequential post-training gradually compresses internal representations into low-rank, anisotropic, and homogeneous feature spaces. We define a measurement suite for hidden states, logits, token trajectories, and LoRA updates, and we use it to analyze supervised fine-tuning, preference optimization, safety/refusal tuning, math and code specialization, and long chain-of-thought tuning under controlled stage orderings. The central hypothesis is that excessive representation concentration is not merely a geometric curiosity: it predicts reduced plasticity during later adaptation, weaker out-of-domain generalization, and poorer calibration. We further evaluate lightweight interventions, including mixed-domain replay, feature refresh, representation diversity regularization, and LoRA update decorrelation, as ways to preserve future learnability without giving up the behavioral gains of post-training.
Yichen Liu, Mingyu Chen, Hao Wang +7
Hangzhou Dianzi University · Zhejiang Gongshang University · Ningbo University +1