Analyzing the Effect of Noise in LLM Fine-tuning

Lingfang Li; Procheta Sen

cs.LGApr 14, 2026

Analyzing the Effect of Noise in LLM Fine-tuning

Authors: Lingfang Li, Procheta Sen

Abstract

Fine-tuning is the dominant paradigm for adapting pretrained large language models (LLMs) to downstream NLP tasks. In practice, fine-tuning datasets may contain various forms of noise arising from annotation errors, preprocessing artifacts, or automated data collection. While prior work has focused on designing robust learning algorithms to mitigate performance degradation under noisy conditions, comparatively little is known about how different types of noise affect the internal learning dynamics of LLMs during fine-tuning. In this work, we systematically study the impact of noise on model behavior across three pretrained model families (GPT-2, Qwen2 and Llama-2) and three diverse NLP tasks. We introduce controlled perturbations corresponding to three common real-world noise types: label noise, grammatical noise, and typographical noise. Beyond task-level performance, we analyze layer-wise representation changes and attention patterns to understand how noise propagates through the network. Our results show that corrupting labels (i.e. label noise) consistently causes the largest performance degradation, whereas grammatical noise and typographical noise can occasionally yield mild regularization benefits. We further find that noise effects are localized primarily to task-specific layers, while attention structures remain comparatively stable.

Figures & tables

Figure 1: Examples of prediction changes in Llama-2 7B under different noise types at 40% corruption ratio: label-flip (SC), typographical (QA), and grammatical (MT).

Metric	Analysis Aspect
$\overline{D}_{KL}(\ell)$	Attention value divergence
$\bar{\rho}_{k}(\ell)$	Attention priority order
$\mathrm{Acc}_{\ell}$	Task-aligned information (classification)
$\mathrm{MRR}_{\ell}$	Task-aligned information (generation)
$\mathrm{Cos}_{s,\ell}$	Per-sample directional shift
$\mathrm{CKA}(\mathbf{{H}_{\ell}},\tilde{\mathbf{H}}_{\ell})$	Inter-sample structural change

Table 1: Summary of different metrics used to analyze the effect of noise. All metrics are computed at every layer

\ell

		Baselines		Label Flip			Typo Error			Gramm. Error
Task	Model	Pretrained	Clean FT	20%	30%	40%	20%	30%	40%	20%	30%	40%
Sentiment	GPT-2	0.12	91.33	90.72	88.82	75.51	91.72	91.32	90.91	91.43	91.32	91.41
	Qwen2	70.00	94.41	90.71	81.62	57.72	94.21	94.02	94.21	94.22	94.51	94.33
	Llama-2	1.13	94.12	91.62	95.12	85.13	94.00	94.42	94.11	94.42	94.21	93.81
QA	GPT-2	8.11	35.51	33.68	32.64	31.58	35.28	34.93	43.97	35.84	35.72	35.72
	Qwen2	41.02	68.00	60.91	51.9	49.72	70.01	69.63	69.62	70.71	70.51	70.63

Table 2: Supervised fine-tuning task-related performance of different models under different noise ratios ( SC : Accuracy; QA : F1 score; MT : BLEU score)

Figure 2: Layer-wise attention pattern divergence ( KL divergence) between clean and noise-trained models. Rows correspond to GPT-2, Qwen2, and Llama-2; columns correspond to SC, QA, and MT tasks. The x-axis denotes noise type and corruption ratio; the y-axis indicates the layer index of each model. Each cell shows the KL divergence averaged across all attention heads at a given layer. Each row uses an independent colour scale due to differing divergence magnitudes across architectures.

Figure 3: Layer-wise attention pattern stability measured by Spearman rank correlation ( $\rho$ ) between clean and noise-trained models.

Panel of Figure 4 — Figure 4: Layer-wise task information analysis for GPT-2 (124M), Qwen2-0.5B and Llama2-7B under all noise conditions. (a,d,g) Probing accuracy for SC . (b,c,e,f,h,i) Logit Lens based MRR (first token) for QA and MT.

Panel of Figure 5 — Figure 5: Layer-wise Linear CKA similarity between clean and noise-trained model representations across three tasks. Each row corresponds to a different backbone: (a–c) GPT-2 Small (12 layers), (d–f) Qwen2-0.5B (24 layers), and (g–i) LLaMA-2-7B (32 layers).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Source	Train	Val	Test
Sentiment	Yelp Polarity	10 000	1 000	1 000
QA	SQuAD v1.1	10 000	1 000	1 000
MT	Tatoeba EN–FR	20 000	1 000	1 000

Appendix

Table 3: Dataset statistics for each task.

Task	Model	Template
Sentiment	All	Review: {text}\nSentiment: {label}
QA	GPT-2	Context: {c}\nQuestion: {q}\nAnswer: {a}
QA	LLaMA-2, Qwen-2	### Context:\n{c}\n\n### Question:\n{q}\n\n### Answer: {a}
MT	GPT-2, LLaMA-2	English: {eng}\nFrench: {fra}
MT	Qwen-2	Translate English to French.\n\n### English:\n{eng}\n\n### French: {fra}

Appendix

Table 4: Prompt templates by model and task. The completion begins after the final colon in each template. \n denotes a newline character.

Task	Original	After Label Flip
Sentiment	Text: “The food was terrible and the service was even worse.” Label: Negative	Text: “The food was terrible and the service was even worse.” Label: Positive
QA	Context: “The Eiffel Tower was built in 1889 for the World’s Fair. It is located in Paris, France.” Question: “When was the Eiffel Tower built?” Answer: “1889”	Context: (unchanged) Question: (unchanged) Answer: “the 10th century”
MT	English: “Swimming at night is dangerous.” French: “Il est dangereux de nager de nuit.”	English: (unchanged) French: “C’est la saison des fraises.”

Appendix

Table 5: Label flip noise examples. The input text remains unchanged; only the target label/output is replaced.

Task	Original	After Typo Injection
Sentiment	“The food was terrible and the service was even worse.”	“The fodo was terrible and the service was even wrse .”
QA	Context: “The Eiffel Tower was built in 1889 for the World’s Fair.”	Context: “The Eiffel Towr was built in 1889 for the World’s Fair.”
MT	English: “Swimming at night is dangerous.”	English: “ Swimmiing at night is dangerous.”

Appendix

Table 6: Typo noise examples. Character-level perturbations (deletion, swap, insertion, or substitution) are applied to randomly selected words at a rate of 10% of words per sample.

Task	Original	After Grammatical Errors
Sentiment	“The food was terrible and the service was even worse.”	“The food were terrible and the service were even worse.”
QA	Context: “The Eiffel Tower was built in 1889. It is located in Paris.”	Context: “The Eiffel Tower were built in 1889. It are located in Paris.”
MT	English: “Swimming at night is dangerous.”	English: “Swimming at night are dangerous.”

Appendix

Table 7: Grammatical noise examples. Rule-based substitutions target verb conjugation ( is

\leftrightarrow

are , was

\leftrightarrow

were , has

\leftrightarrow

have ) and article usage ( a

\leftrightarrow

an ) at a rate of 15% of words per sample.

Model	Task	$r$	$\alpha$	Dropout	Target Modules
Qwen-2	Sentiment	8	32	0.10	All 7
	QA	16	32	0.05	All 7
	MT	32	64	0.05	All 7
LLaMA-2	Sentiment	32	64	0.05	All 7
	QA	16	32	0.05	All 7
	MT	128	256	0.05	All 7

Appendix

Table 8: QLoRA adapter configuration. “All 7” denotes q_proj , k_proj , v_proj , o_proj , gate_proj , up_proj , down_proj .

Model	Task	LR	Eff. Batch	Epochs	Warmup	Max Len
GPT-2	Sentiment	5e-5	32	3	0.03	256
	QA	2e-5	32	3	0.03	1024
	MT	5e-5	32	30	0.03	256
Qwen-2	Sentiment	5e-5	80	3	0.03	256
	QA	2e-5	32	2	0.03	1024
	MT	2e-4	32	3	0.03	256

Appendix

Table 9: Training hyperparameters. All configurations use cosine LR scheduling. Weight decay is 0 except where noted.

Panel of Figure 6 — Figure 6: Layer-wise centered cosine similarity between clean and noise-trained model representations across all nine model–task combinations. Rows correspond to models (GPT-2, Qwen-2, LLaMA-2); columns correspond to tasks (SC, QA, MT). Centered cosine removes the shared mean direction before computing similarity, correcting for the anisotropy of contextualised representations.

Panel of Figure 7 — Figure 7: Robust vs. vulnerable stratification: centered cosine similarity for SC , QA and MT under label-flip noise. Centered cosine removes the shared mean direction before computing similarity, correcting for anisotropy.

Panel of Figure 8 — Figure 8: Robust vs. vulnerable stratification: Linear CKA for SC , QA and MT under label-flip noise. CKA captures inter-sample relational structure.

Panel of Figure 9 — Figure 9: Robust vs. vulnerable stratification: first-token Logit Lens MRR under label-flip noise. MRR measures how well each layer’s representation predicts the correct answer token when projected through the language model head. LLaMA-2 vulnerable MRR collapses to 0.365 at the final layer under 40% noise (vs. 0.740 for robust samples), the largest functional gap observed across all conditions. GPT-2 MRR is uniformly low ( $<$ 0.20) for both groups, reflecting its limited QA capability.

Seed	Clean	Label-flip 40%
1	94.5%	95.7% (+1.2%)
22	94.5%	83.5% ( $-$ 11.0%)
7	94.0%	27.2% (collapse)
42	(default; main experiments)

Appendix

Table 10: LLaMA-2 sentiment accuracy across random seeds under 40% label-flip noise. The three seeds produce qualitatively different outcomes: no degradation (seed 1), moderate degradation (seed 22), and catastrophic collapse (seed 7).

Panel of Figure 10 — Figure 10: Layer-wise top-5 MRR for GPT-2 Small (124M), Qwen-2 (0.5B) and Llama-2 (7B).

Panel of Figure 11 — Figure 11: Layer-wise top 5 token accuracy for the 3 models on (a) question answering and (b) machine translation under all noise conditions. At each layer, hidden states are projected through the LM head, and accuracy is computed as the fraction of the first 5 generated tokens matching the reference.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 754 papers

Feb 16, 2026

Explainable Token-level Noise Filtering for LLM Fine-tuning Datasets

Large Language Models (LLMs) have seen remarkable advancements, achieving state-of-the-art results in diverse applications. Fine-tuning, an important step for adapting LLMs to specific downstream tasks, typically involves further training on corresponding datasets. However, a fundamental discrepancy exists between current fine-tuning datasets and the token-level optimization mechanism of LLMs: most datasets are designed at the sentence-level, which introduces token-level noise, causing negative influence to final performance. In this paper, we propose XTF, an explainable token-level noise filtering framework. XTF decomposes the complex and subtle contributions of token-level data to the fine-tuning process into three distinct and explicit attributes (reasoning importance, knowledge novelty, and task relevance), which can be assessed using scoring methods, and then masks the gradients of selected noisy tokens accordingly to optimize the performance of fine-tuned LLMs. We conduct extensive experiments on three representative downstream tasks (math, code and medicine) across 7 mainstream LLMs. The results demonstrate that XTF can significantly improve downstream performance by up to 13.7% compared to regular fine-tuning. Our work highlights the importance of token-level dataset optimization, and demonstrates the potential of strategies based on attribute decomposition for explaining complex training mechanisms.

Yuchen Yang, Wenze Lin, Enhao Huang +6

1The State Key Laboratory of Blockchain and Data Security, Zhejiang University · 2Hangzhou HighTech Zone (Binjiang) Blockchain and Data Security Research Institute, China · 3Qiuzhen College, Tsinghua University +2

LLM Fine-Tuning Training Data Curation Training Data Selection

Ask PDF

May 22, 2026

Understanding and Improving Noisy Embedding Techniques in Instruction Finetuning

Recent advancements in instructional fine-tuning have injected noise into embeddings, with NEFTune (Jain et al., 2024) setting benchmarks using uniform noise. Despite NEFTune's empirical findings that uniform noise outperforms Gaussian noise, the reasons for this remain unclear. This paper aims to clarify this by offering a thorough analysis, both theoretical and empirical, indicating comparable performance among these noise types. Additionally, we introduce a new fine-tuning method for language models, utilizing symmetric noise in embeddings. This method aims to enhance the model's function by more stringently regulating its local curvature, demonstrating superior performance over the current method, NEFTune. When fine-tuning the LLaMA-2-7B model using Alpaca, standard techniques yield a 29.79% score on AlpacaEval. However, our approach, SymNoise, increases this score significantly to 69.04%, using symmetric noisy embeddings. This is a 6.7% improvement over the state-of-the-art method, NEFTune (64.69%). Furthermore, when tested on various models and stronger baseline instruction datasets, such as Evol-Instruct, ShareGPT, OpenPlatypus, SymNoise consistently outperforms NEFTune. The current literature, including NEFTune, has underscored the importance of more in-depth research into the application of noise-based strategies in the fine-tuning of language models. Our approach, SymNoise, is another significant step towards this direction, showing notable improvement over the existing state-of-the-art method.

Abhay Yadav

Johns Hopkins University Baltimore, MD, USA

LLM Fine-Tuning Instruction Tuning

Ask PDF

May 11, 2026

Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data

Large language models (LLMs) rely on web-scale corpora for pre-training. The noise inherent in these datasets tends to obscure meaningful patterns and ultimately degrade model performance. Data curation mitigates but cannot eliminate such noise, so pre-training corpora remain noisy in practice. We therefore study whether a lightweight pre-pre-training (PPT) stage based on synthetic data with learnable temporal structure helps resist noisy data during the pre-training (PT) stage. Across various corruption settings, our method consistently improves robustness to noise during PT, with larger relative gains at higher noise levels. For a 1B-parameter model, a synthetic PPT stage with only 65M tokens achieves the same final loss as the baseline while using up to 49% fewer natural-text PT tokens across different noise levels. Mechanistic analyses suggest PPT does not immediately suppress attention to noisy tokens. Rather, PPT-initialized models gradually downweight attention between corrupted tokens during noisy PT. This indicates that synthetic PPT inhibits noise self-modeling and shapes the subsequent optimization trajectory. Code is available at https://github.com/guox18/formal-language-prepretraining.

Xu Guo, Runyu Peng, Jian Tong +4

Shanghai AI Laboratory · Fudan University · Shanghai Innovation Institute

Language Model Pretraining Synthetic Data Pretraining Neural Network Robustness Noise Robustness Language Model Robustness

Ask PDF

Abstract

LLM Fine-Tuning

Explore similar work

Explainable Token-level Noise Filtering for LLM Fine-tuning Datasets

Understanding and Improving Noisy Embedding Techniques in Instruction Finetuning

Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data