Noisy training data can significantly degrade the performance of language-model-based classifiers, particularly in non-topical classification tasks. In this study we designed a methodological framework to assess the impact of denoising. More specifically, we explored a range of denoising strategies for sentence-level difficulty detection, using training data derived from document-level difficulty annotations obtained through noisy crowdsourcing. Beyond monolingual settings, we also address cross-lingual transfer, where a multilingual language model is trained in one language and tested in another. We evaluate several noise reduction techniques, including Gaussian Mixture Models (GMM), Co-Teaching, Noise Transition Matrices, and Label Smoothing. Our results indicate that while BERT-based models exhibit inherent robustness to noise, incorporating explicit noise detection can further enhance performance. For our smaller dataset, GMM-based noise filtering proves particularly effective in improving prediction quality by raising the Area-Under-the-Curve score from 0.52 to 0.92, or to 0.93 when de-noising methods are combined. However, for our larger dataset, the intrinsic regularisation of pre-trained language models provides a strong baseline, with denoising methods yielding only marginal gains (from 0.92 to 0.94, while a combination of two denoising methods made no contribution). Nonetheless, removing noisy sentences (about 20% of the dataset) helps in producing a cleaner corpus with fewer infelicities. As a result we have released the largest multilingual corpus for sentence difficulty prediction: see https://github.com/Nouran-Khallaf/denoising-difficulty
Figures & tables
Language
#Docs
Wikipedia
Vikidia
#Words
#Sentences
IQR
#Words
#Sentences
IQR
Catalan
179
396,277
16,813
(15, 29)
19,394
1,000
(13, 23)
English
2585
8,281,625
340,924
(16, 29)
424,306
22,462
(13, 22)
Spanish
3875
7,946,169
301,241
(16, 33)
607,990
27,825
(14, 26)
French
33438
46,618,143
1,945,046
(15, 29)
6,643,567
320,372
(14, 25)
Italian
3902
6,790,163
263,271
(16, 32)
537,723
25,202
(14, 26)
Table 1: Statistics for paired Wikipedia–Vikidia corpora. #Docs is the document count (paired Wikipedia–Vikidia articles by topic). #Words and #Sentences are totals across all documents in each subset (restricted to paired topics). IQR gives the inter-quartile range of sentence lengths (in words) in each subset.
Table 2: Comparative performance of noise reduction methods (measured via ROC-AUC) with models trained on English (top) and French (bottom). “Average” is the macro-average ROC-AUC over transfer languages (excluding the training language). “Baseline” refers to standard fine-tuning with no noise filtering. # Noisy is the count of sentences detected as noisy by the respective method. G-B stands for GMM-Bert, G-S stands for GMM-SBert. The underlined values are not significantly different from the best ones.
Figure 1: Comparison of noisy data intersections across denoising techniques for English and French. Each vertical bar represents the proportion of sentences identified as noisy by the combination of methods in the connected dots below. The small bar chart at the bottom left indicates the contribution of each method to the total pool of sentences identified as noisy.
Category
English
French
Total sentences
230
237
Not noisy
92
78
Noisy
138
159
Table 3: Accuracy analysis at the intersection
Category
English
French
Structural : Total
185
186
SF:Structural fragment
59
83
MS:Markup or symbolic artifact
46
36
EN:Enumeration or list artifact
44
57
LA:Language switch or encoding
17
2
CI:Citation or reference fragment
19
8
Table 4: Manual analysis of noise for English and French. Counts reflect annotation assignments (multiple labels per sentence allowed). Totals may exceed the number of noisy segments.
Figure 2: Pairwise associations between noise categories measured using the Matthews Correlation Coefficient (MCC) for English (left) and French (right).
Data quality is a critical factor in the effectiveness of machine learning models. Label errors, present even in widely used benchmarks, introduce noise into training data and reduce model generalization. In this work, we conduct a comparative analysis of two automatic label error detection methods - Confident Learning and Dataset Cartography - on three Russian text classification corpora of varying size, number of classes, and domain: ru_emotion_e-culture (49,123 examples, emotion classification), RuCoLA (8,524 examples, linguistic acceptability), and TERRa (2,337 examples, textual entailment recognition). We use the pre-trained rubert-base-cased model fine-tuned on each corpus. To verify the meaningfulness of filtering, we conduct control experiments with random removal of an equivalent number of examples. Results show that the effectiveness of both methods depends strongly on dataset characteristics: on large corpora with low noise levels, filtering does not improve performance, while on small datasets with high noise, Confident Learning achieves a significant F1-macro improvement. Dataset Cartography demonstrates more conservative behavior, removing fewer examples. Across all corpora, targeted removal by both methods outperforms random removal, confirming the meaningfulness of the approaches.
Large language models have achieved strong performance across many NLP tasks, yet Urdu remains comparatively underexplored due to limited resources and fragmented evaluation settings. To address this gap, we introduce DunbaaBERT, a family of Urdu RoBERTa-base models trained from scratch with Byte-BPE vocabularies of 32k, 52k, and 96k tokens on a deduplicated 17GB Urdu corpus. We evaluate DunbaaBERT across intrinsic and downstream Urdu NLP benchmarks covering linguistic acceptability, news classification, offensive language detection, and sentiment analysis while analyzing vocabulary-size effects on performance and efficiency trade-offs. Across benchmarks, the DunbaaBERT variants achieve competitive performance against strong multilingual baselines while consistently maintaining favorable efficiency trade-offs. Interestingly, larger vocabularies do not consistently improve downstream effectiveness, with DunbaaBERT32k repeatedly providing the strongest overall efficiency profile. Overall, our results demonstrate that carefully curated Urdu-specific encoder models can remain highly competitive despite comparatively compact model and training scales. All models are released under the MIT license.
Iffat Maab, Waleed Jamil, Raphael Schmitt
Research and Development Center for Large Language Models (LLMC), National Institute of Informatics, Tokyo · Independent Researcher, Edinburgh, United Kingdom · School of Computation, Information and Technology, Technical University of Munich, Germany +1
The performance of textual neural models often degrades when their inputs are corrupted by noise such as typos, OCR errors, or dropped words. We study the degradation rate across neural models, both sentence embeddings and decoder-only LLMs, and find that how consistent it is depends on the scale of the noise: under word-level noise, models with very different architectures decline along nearly the same curve, while under character-level noise they separate. We further identify the determining factor to be the training objective, not the architecture: eight encoders spanning six pretraining paradigms are scattered initially, and collapse onto a common curve after a short contrastive training recipe. We trace the word/character split to tokenization: a single character edit forces the tokenizer to re-segment the surrounding word, disturbing the token sequence far more than dropping a whole word does. This finding and its underlying mechanism provide a practical means to predict a model's robustness to noise without any noisy evaluation, and to install robustness at a chosen noise scale through noise-augmented training.