Accurate uncertainty estimation is essential for building robust and trustworthy recognition systems. In this paper, we consider the open-set text classification (OSTC) task - and uncertainty estimation for it. For OSTC a text sample should be classified as one of the existing classes or rejected as unknown. To account for the different uncertainty types encountered in OSTC, we adapt the Holistic Uncertainty Estimation (HolUE) method for the text domain. Our approach addresses two major causes of prediction errors in text recognition systems: text uncertainty that stems from ill formulated queries and gallery uncertainty that is related the ambiguity of data distribution. By capturing these sources, it becomes possible to predict when the system will make a recognition error. We propose a new OSTC benchmark and conduct extensive experiments on a wide range of data, utilizing the authorship attribution, intent and topic classification datasets. HolUE achieves 40-365% improvement in Prediction Rejection Ratio (PRR) over the quality-based SCF baseline across datasets: 365% on Yahoo Answers (0.79 vs 0.17 at FPIR 0.1), 347% on DBPedia (0.85 vs 0.19), 240% on PAN authorship attribution (0.51 vs 0.15 at FPIR 0.5), and 40% on CLINC150 intent classification (0.73 vs~0.52). We make public our code and protocols https://github.com/Leonid-Erlygin/text_uncertainty.git
Figures & tables
Рис. 1: An illustrative example in a two-dimensional embedding space demonstrating uncertainty estimation for intent classification. The gallery consists of four known intent classes (Uber, Traffic, Weather, Balance), each marked with a distinct color, while the pink color indicates embeddings of the unknown class. Squares represent individual text query embeddings, while circles denote class centers. Dashed yellow lines depict the decision boundaries among the classes. The intensity of the blue shading reflects uncertainty due to class ambiguity and is proportional to the entropy of p(c∣z) , where z∈R2 is the embedding of a text query and c is the intent label. Entropy is highest near decision boundaries between different intents. The ⧫ symbol lies between the Uber and Traffic classes, exemplifying high uncertainty and highlighting the risk of misclassification for semantically overlapping queries. The ★ symbol resides near the boundary of the Balance class, exhibiting high uncertainty and representing a potential false acceptance of an out-of-domain query. The × symbol represents a noisy or stylistically distinct in-gallery sample whose embedding is displaced from its true class center. Text annotations provide concrete examples of natural language queries mapped to their respective embedding locations. The visualization demonstrates how spatial uncertainty helps distinguish between confident classifications, ambiguous queries, and unknown intents.
Task
Gallery Uncertainty
Embedding Uncertainty
Intent Classification
Semantic overlap between known intent classes (e.g., functionally similar requests lying near decision boundaries).
Ambiguity in user phrasing, slang, or noisy input causing high variance in the embedding position relative to the class center.
Authorship Attribution
Stylistic similarity between known authors and impostors, leading to ambiguous decision regions in the embedding space.
Intra-author stylistic variation due to topic shifts or context changes, displacing the embedding from the author’s class center.
Topic Classification
Semantic proximity between known topics (e.g., Politics vs. Economy), creating dense regions with high class ambiguity.
Multi-topic documents or vague content leading to dispersed embedding distributions that do not align confidently with any single topic center.
Таблица 1: Manifestation of Uncertainty Types in Open-Set Text Classification Tasks
Рис. 2: Architecture of the probabilistic text embedding training. The pipeline consists of two main components: Top: Feature Extraction . Input texts are encoded using a pre-trained BERT Transformer, producing CLS token embeddings ( c1,…,cN ). These embeddings are projected through an MLP into bottleneck features ( h1,…,hN ) that serve as the input for uncertainty estimation. Bottom: Probabilistic Embedding & Loss Computation . Each bottleneck feature hi is processed through two parallel heads: (1) an MLP head that predicts the mean embedding μi , and (2) an SCF head that predicts the concentration parameter κi , which characterizes embedding uncertainty. Together, μi and κi define a von Mises-Fisher distribution p(z∣μi,κi) over the hypersphere. Stop-gradient (sg) operations are applied at three critical points: (i) between hi and the SCF head to prevent uncertainty estimation from affecting feature extraction, (ii) between μi and the probabilistic sampling to isolate the embedding loss Lembi , and (iii) between class centers wk and the SCF loss LSCFi to ensure stable gallery representation. The framework computes two complementary losses: Lembi optimizes embedding discriminability by pulling μi toward learnable class centers, while LSCFi trains the concentration predictor to assign high variance to ambiguous samples (e.g., whose embedding is far from its class center).
Method
FPIR
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
Yahoo Answers
AGNews
DBPedia
AccScr
-0.18
0.26
0.42
0.47
0.49
-0.02
0.42
0.55
0.57
0.58
0.36
0.78
0.84
0.84
0.79
SCF
0.17
0.28
0.39
0.49
0.56
-0.04
-0.02
0.02
0.0
0.06
0.19
0.31
0.43
0.49
0.57
GalUE
-0.18
0.26
0.42
0.47
0.49
-0.02
0.42
0.55
0.58
0.59
0.48
0.78
0.84
0.84
0.8
HolUE
0.79
0.73
0.73
0.75
0.77
0.52
0.41
0.56
0.68
0.75
0.85
0.92
0.93
0.94
0.95
Таблица 2: Prediction Rejection Ratios (PRR, ↑ ) for F1 filtering curve for three topic classification datasets.
Method
FPIR
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
PAN-20-AV
CLINC150
AccScr
-0.1
0.38
0.5
0.36
0.39
0.15
0.39
0.49
0.56
0.59
SCF
0.1
0.13
0.15
0.18
0.15
0.52
0.49
0.48
0.47
0.48
GalUE
-0.09
0.42
0.52
0.44
0.47
0.14
0.35
0.4
0.39
0.38
HolUE
0.36
0.5
0.55
0.5
0.51
0.73
0.64
0.59
0.58
0.6
Таблица 3: Prediction Rejection Ratios (PRR, ↑ ) for F1 filtering curve for PAN authorship attribution and CLINC150 intent classification datasets.
Рис. 3: Figure 3. Risk-Controlled Open-set Text Classification on PAN dataset starting with FPIR=0.2 . Prediction Rejection Ratio is reported. Subplots show:(a) F1 score,(b) FNIR,(c) FPIR, and(d) False Identification Count versus filter out rate. Better to view in zoom.
In open-set recognition (OSR), a probe must either be identified as one of the known gallery classes or rejected as unknown, so three error types coexist: false acceptance, false rejection, and misidentification. An uncertainty score for selective recognition should rank probes by the risk of the decision the system has made. Bayesian gallery-aware models such as Holistic Uncertainty Estimation (HolUE) summarize the posterior over known and unknown classes by Kullback--Leibler (KL) divergence components and map them to an uncertainty score with a supervised nonlinear calibrator. We show that the KL summary is not generally monotone in decision risk: linear fusion of the KL components tuned on validation data yields negative filtering quality on several benchmarks. We propose MPRisk, a mixed-prior posterior decision-risk score that keeps the same Bayesian posterior but directly scores the error events associated with the selected decision: false-acceptance, misidentification, and false-rejection risks, plus a non-specificity penalty for rejections, enabled by modeling unknown identities as a continuous component. Four nonnegative weights tuned on a validation set suffice for ranking; no nonlinear supervised model is required. Across nine image, audio, and text benchmarks, MPRisk achieves the best or tied-best Prediction Rejection Ratio at every operating point on the image and audio benchmarks and on most text operating points, with bootstrap-confirmed gains over HolUE on five benchmarks (up to +0.19 PRR) at comparable or lower runtime.
L. A. Erlygin, P. D. Proskura, A. A. Zaytsev
Skolkovo Institute of Science and Technology (Skoltech), Moscow, Russia · Risk Management, Sber, Moscow, Russia · The Institute for Information Transmission Problems (IITP RAS), Moscow, Russia
This study examines the role of uncertainty estimation (UE) methods in multilingual text classification under noisy and non-topical conditions. Using a complex-vs-simple sentence classification task across several languages, we evaluate a range of UE techniques against a range of metrics to assess their contribution to making more robust predictions. Results indicate that while methods relying on softmax outputs remain competitive in high-resource in-domain settings, their reliability declines in low-resource or domain-shift scenarios. In contrast, Monte Carlo dropout approaches demonstrate consistently strong performance across all languages, offering more robust calibration, stable decision thresholds, and greater discriminative power even under adverse conditions. We further demonstrate the positive impact of UE on non-topical classification: abstaining from predicting the 10% most uncertain instances increases the macro F1 score from 0.81 to 0.85 in the Readme task. By integrating UE with trustworthiness metrics, this study provides actionable insights for developing more reliable NLP systems in real-world multilingual environments. See https://github.com/Nouran-Khallaf/To-Predict-or-Not-to-Predict
Nouran Khallaf, Serge Sharoff
Centre for Translation, Localisation and Interpreting Studies, School of Languages, Cultures and Societies, University of Leeds, UK
Researchers increasingly use text classification--supervised models or large language models--to measure constructs from natural language, providing metrics such as recall and precision as evidence of their validity. Yet, though these metrics are point estimates subject to sampling variation, measures of uncertainty are inconsistently reported alongside them. Further, when they are reported, they are often estimated with methods that are not appropriate when relevant labelled datasets are small or performance is high. To increase and improve confidence interval reporting in the field, this paper evaluates confidence interval methods for performance metrics under conditions typical of social science text classification: small to moderate sample sizes, infrequent constructs, and texts nested within individuals. Across simulations, default methods such as the Wald interval and the basic percentile bootstrap are the least accurate, with coverage sometimes far below the nominal 95% level. Accuracy is improved with the use of Agresti-Coull, Wilson, Clopper-Pearson, and a novel pseudo-count regularized bootstrap (which is particularly relevant to the calculation of F1). When texts are nested within individuals, we demonstrate that adjustment for both effective N and the appropriate degrees of freedom is necessary for producing accurate analytic intervals. Among bootstrap intervals, the hierarchical bootstrap is more accurate than the cluster bootstrap when individuals produce a moderate number of texts but overly conservative when individuals produce only a few. By providing guidance to the field on appropriate interval estimation, we aim to improve the transparency of machine learning applications, and to encourage greater attention to the validation sample size at the design stage.
Kylie Anglin
Department of Educational Psychology, Neag School of Education, University of Connecticut