cs.CLMar 7, 2026

To Predict or Not to Predict? Towards reliable uncertainty estimation in the presence of noise

Authors: Nouran Khallaf, Serge Sharoff

Organizations: Centre for Translation, Localisation and Interpreting Studies, School of Languages, Cultures and Societies, University of Leeds, UK

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This study examines the role of uncertainty estimation (UE) methods in multilingual text classification under noisy and non-topical conditions. Using a complex-vs-simple sentence classification task across several languages, we evaluate a range of UE techniques against a range of metrics to assess their contribution to making more robust predictions. Results indicate that while methods relying on softmax outputs remain competitive in high-resource in-domain settings, their reliability declines in low-resource or domain-shift scenarios. In contrast, Monte Carlo dropout approaches demonstrate consistently strong performance across all languages, offering more robust calibration, stable decision thresholds, and greater discriminative power even under adverse conditions. We further demonstrate the positive impact of UE on non-topical classification: abstaining from predicting the 10% most uncertain instances increases the macro F1 score from 0.81 to 0.85 in the Readme task. By integrating UE with trustworthiness metrics, this study provides actionable insights for developing more reliable NLP systems in real-world multilingual environments. See https://github.com/Nouran-Khallaf/To-Predict-or-Not-to-Predict

Figures & tables

 SimpleComplex
Language#Ex’sIQR#Ex’sIQR
ReadMe
Arabic7669–2053312–33
English1,2969–2044813–32
French81010–2336714–35
Hindi5699–2039611–29
Table 1: Statistics of the datasets.
MethodFormulationKey Characteristics
Class probability-based methods
SRTop-class softmax probability.Directly available, but calibration is questionable.
SMPMC Dropout-averaged top-class probability.Captures disagreement across stochastic passes.
ENTEntropy over softmax probabilities.Captures prediction ambiguity.
ENT-MCMC Dropout-averaged entropy over probabilities.Captures ambiguity across stochastic passes.
PVMC Dropout-averaged variance in probabilities.Captures disagreement across stochastic passes.
Table 2: Summary of Uncertainty Estimation methods, grouped by the underlying approaches.
MetricRangeInterpretation
Uncertainty Discrimination
ROC-AUC[0, 1]Indicates whether incorrect predictions have higher uncertainty. Higher is better.
AU-PRC[0, 1]Identifies incorrect predictions via uncertainty. Higher is better.
Calibration Metrics
C-Slope[0, ∞\infty ]Regression of accuracy on confidence. << 1 = underconfident; >> 1 = overconfident.
CITL[−1,1][-1,1]Measures the difference between confidence and accuracy. The best value is 0.
Table 3: Summary of UE quality metrics grouped by the evaluation perspective.
Readme Language Class P ±\pm Std R ±\pm Std F1 ±\pm Std Arabic complex 0.77 ±\pm 0.050 0.86 ±\pm 0.058 0.81 ±\pm 0.035 simple 0.90 ±\pm 0.023 0.81 ±\pm 0.067 0.85 ±\pm 0.034 English complex 0.80 ±\pm 0.056 0.79 ±\pm 0.080 0.79 ±\pm 0.036 simple 0.93 ±\pm 0.022 0.93 ±\pm 0.029 0.93 ±\pm 0.013 French complex 0.79 ±\pm 0.047 0.85 ±\pm 0.084 0.81 ±\pm 0.037 simple 0.93 ±\pm 0.032 0.89 ±\pm 0.047 0.91 ±\pm 0.021 Hindi complex 0.79 ±\pm 0.048 0.74 ±\pm 0.128 0.76 ±\pm 0.075 simple 0.83 ±\pm 0.069 0.86 ±\pm 0.044 0.84 ±\pm 0.033 Russian complex 0.75 ±\pm 0.083 0.78 ±\pm 0.113 0.75 ±\pm 0.057 simple 0.90 ±\pm 0.037 0.87 ±\pm 0.067 0.88 ±\pm 0.028 Macro averaged 0.84 ±\pm 0.013 0.84 ±\pm 0.016 0.83 ±\pm 0.014Vikidia/Wikipedia Language Class P ±\pm Std R ±\pm Std F1 ±\pm Std Catalan complex 0.72 ±\pm 0.044 0.49 ±\pm 0.137 0.57 ±\pm 0.101 simple 0.62 ±\pm 0.044 0.80 ±\pm 0.075 0.69 ±\pm 0.017 English complex 0.79 ±\pm 0.020 0.62 ±\pm 0.026 0.69 ±\pm 0.021 simple 0.68 ±\pm 0.016 0.83 ±\pm 0.017 0.75 ±\pm 0.015 French complex 0.69 ±\pm 0.006 0.60 ±\pm 0.008 0.64 ±\pm 0.006 simple 0.64 ±\pm 0.005 0.73 ±\pm 0.008 0.68 ±\pm 0.005 Italian complex 0.66 ±\pm 0.017 0.64 ±\pm 0.102 0.65 ±\pm 0.058 simple 0.66 ±\pm 0.049 0.68 ±\pm 0.048 0.66 ±\pm 0.012 Spanish complex 0.68 ±\pm 0.017 0.72 ±\pm 0.007 0.70 ±\pm 0.011 simple 0.71 ±\pm 0.011 0.66 ±\pm 0.024 0.68 ±\pm 0.017 Macro averaged 0.69 ±\pm 0.035 0.68 ±\pm 0.036 0.68 ±\pm 0.040
Table 4: Macro F1 with standard deviations across the folds. The source dataset for training the classifier is Readme. The domain shift tested on Vikidia/Wikipedia and Simplext (Spanish only)
Figure 1: Cross-language average zz -scores after applying a direction-aware benefit transform across languages. Cell colors indicate relative performance (better →\rightarrow warmer), and horizontal whiskers show the standard deviation of zz across languages.
Metric PairARENFRHIRU
Uncertainty Discrimination
AU-PRC vs ROC-AUC0.520.610.650.530.58
Calibration Metrics
CITL vs ECE-0.39-0.61-0.55-0.27-0.32
C-Slope vs CITL0.310.420.33-0.030.11
C-Slope vs ECE-0.37-0.46-0.45-0.28-0.29
Table 5: Kendall’s τ\tau within metric perspectives.
Metric PairARENFRHIRU
Discrimination vs Calibration
AU-PRC vs C-Slope0.450.470.310.510.57
AU-PRC vs CITL-0.020.05-0.16-0.22-0.09
AU-PRC vs ECE-0.03-0.070.030.00-0.03
C-Slope vs ROC-AUC0.380.400.310.590.46
CITL vs ROC-AUC-0.14-0.01-0.14-0.24-0.13
Table 6: Kendall’s τ\tau across metric perspectives.
Figure 2: Improvement measured via Δ\Delta F1 over the baseline from Table 4 across rejection thresholds (Columns for 1, 5, 10, 15%) per language per UE score. A solid black box marks the best UE score per column and a dash–dot box marks the second best. The UE scores are labelled on the right.
More in this topic

Multilingual Text Classification

See all 20 papers

Explore similar work

CardsList

  1. Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs

    Jul 7, 2026Andrea Bacciu, Andrea Alfarano, Saab Mansour +2Multilingual Language Model EvaluationSelective Prediction

    Amazon · INSAIT, Sofia

  2. Uncertainty Estimation for the Open-Set Text Classification systems

    Mar 17, 2026Leonid Erlygin, Alexey ZaytsevUncertainty QuantificationOpen-Set Recognition

    Skolkovo Institute of Science and Technology (Skoltech), Moscow, Russia · Risk Management, Sber, Moscow, Russia

  3. How Much Noise Can BERT Handle? Insights from Multilingual Sentence Difficulty Detection

    Mar 7, 2026Nouran Khallaf, Serge SharoffNoisy-Label LearningLanguage Model Robustness

    Centre for Translation, Localisation and Interpreting Studies School of Languages, Cultures and Societies University of Leeds, UK