Uncertainty Estimation for the Open-Set Text Classification systems

Leonid Erlygin; Alexey Zaytsev

cs.CLMar 17, 2026

Uncertainty Estimation for the Open-Set Text Classification systems

Authors: Leonid Erlygin, Alexey Zaytsev

Organizations: Skolkovo Institute of Science and Technology (Skoltech), Moscow, Russia · Risk Management, Sber, Moscow, Russia

Abstract

Accurate uncertainty estimation is essential for building robust and trustworthy recognition systems. In this paper, we consider the open-set text classification (OSTC) task - and uncertainty estimation for it. For OSTC a text sample should be classified as one of the existing classes or rejected as unknown. To account for the different uncertainty types encountered in OSTC, we adapt the Holistic Uncertainty Estimation (HolUE) method for the text domain. Our approach addresses two major causes of prediction errors in text recognition systems: text uncertainty that stems from ill formulated queries and gallery uncertainty that is related the ambiguity of data distribution. By capturing these sources, it becomes possible to predict when the system will make a recognition error. We propose a new OSTC benchmark and conduct extensive experiments on a wide range of data, utilizing the authorship attribution, intent and topic classification datasets. HolUE achieves 40-365% improvement in Prediction Rejection Ratio (PRR) over the quality-based SCF baseline across datasets: 365% on Yahoo Answers (0.79 vs 0.17 at FPIR 0.1), 347% on DBPedia (0.85 vs 0.19), 240% on PAN authorship attribution (0.51 vs 0.15 at FPIR 0.5), and 40% on CLINC150 intent classification (0.73 vs~0.52). We make public our code and protocols https://github.com/Leonid-Erlygin/text_uncertainty.git

Figures & tables

Рис. 1: An illustrative example in a two-dimensional embedding space demonstrating uncertainty estimation for intent classification. The gallery consists of four known intent classes (Uber, Traffic, Weather, Balance), each marked with a distinct color, while the pink color indicates embeddings of the unknown class. Squares represent individual text query embeddings, while circles denote class centers. Dashed yellow lines depict the decision boundaries among the classes. The intensity of the blue shading reflects uncertainty due to class ambiguity and is proportional to the entropy of $p(c|\mathbf{z})$ , where $\mathbf{z}\in\mathbb{R}^{2}$ is the embedding of a text query and $c$ is the intent label. Entropy is highest near decision boundaries between different intents. The $\blacklozenge$ symbol lies between the Uber and Traffic classes, exemplifying high uncertainty and highlighting the risk of misclassification for semantically overlapping queries. The $\bigstar$ symbol resides near the boundary of the Balance class, exhibiting high uncertainty and representing a potential false acceptance of an out-of-domain query. The $\boldsymbol{\times}$ symbol represents a noisy or stylistically distinct in-gallery sample whose embedding is displaced from its true class center. Text annotations provide concrete examples of natural language queries mapped to their respective embedding locations. The visualization demonstrates how spatial uncertainty helps distinguish between confident classifications, ambiguous queries, and unknown intents.

Task	Gallery Uncertainty	Embedding Uncertainty
Intent Classification	Semantic overlap between known intent classes (e.g., functionally similar requests lying near decision boundaries).	Ambiguity in user phrasing, slang, or noisy input causing high variance in the embedding position relative to the class center.
Authorship Attribution	Stylistic similarity between known authors and impostors, leading to ambiguous decision regions in the embedding space.	Intra-author stylistic variation due to topic shifts or context changes, displacing the embedding from the author’s class center.
Topic Classification	Semantic proximity between known topics (e.g., Politics vs. Economy), creating dense regions with high class ambiguity.	Multi-topic documents or vague content leading to dispersed embedding distributions that do not align confidently with any single topic center.

Таблица 1: Manifestation of Uncertainty Types in Open-Set Text Classification Tasks

Рис. 2: Architecture of the probabilistic text embedding training. The pipeline consists of two main components: Top: Feature Extraction . Input texts are encoded using a pre-trained BERT Transformer, producing CLS token embeddings (

c_{1},\ldots,c_{N}

). These embeddings are projected through an MLP into bottleneck features (

h_{1},\ldots,h_{N}

) that serve as the input for uncertainty estimation. Bottom: Probabilistic Embedding & Loss Computation . Each bottleneck feature

h_{i}

is processed through two parallel heads: (1) an MLP head that predicts the mean embedding

\mu_{i}

, and (2) an SCF head that predicts the concentration parameter

\kappa_{i}

, which characterizes embedding uncertainty. Together,

\mu_{i}

and

\kappa_{i}

define a von Mises-Fisher distribution

p(z|\mu_{i},\kappa_{i})

over the hypersphere. Stop-gradient (sg) operations are applied at three critical points: (i) between

h_{i}

and the SCF head to prevent uncertainty estimation from affecting feature extraction, (ii) between

\mu_{i}

and the probabilistic sampling to isolate the embedding loss

\mathcal{L}^{i}_{\text{emb}}

, and (iii) between class centers

w_{k}

and the SCF loss

\mathcal{L}^{i}_{\text{SCF}}

to ensure stable gallery representation. The framework computes two complementary losses:

\mathcal{L}^{i}_{\text{emb}}

optimizes embedding discriminability by pulling

\mu_{i}

toward learnable class centers, while

\mathcal{L}^{i}_{\text{SCF}}

trains the concentration predictor to assign high variance to ambiguous samples (e.g., whose embedding is far from its class center).

Method	$\mathrm{FPIR}$
	$0.1$	$0.2$	$0.3$	$0.4$	$0.5$	$0.1$	$0.2$	$0.3$	$0.4$	$0.5$	$0.1$	$0.2$	$0.3$	$0.4$	$0.5$
	Yahoo Answers					AGNews					DBPedia
AccScr	-0.18	0.26	0.42	0.47	0.49	-0.02	0.42	0.55	0.57	0.58	0.36	0.78	0.84	0.84	0.79
SCF	0.17	0.28	0.39	0.49	0.56	-0.04	-0.02	0.02	0.0	0.06	0.19	0.31	0.43	0.49	0.57
GalUE	-0.18	0.26	0.42	0.47	0.49	-0.02	0.42	0.55	0.58	0.59	0.48	0.78	0.84	0.84	0.8
HolUE	0.79	0.73	0.73	0.75	0.77	0.52	0.41	0.56	0.68	0.75	0.85	0.92	0.93	0.94	0.95

Таблица 2: Prediction Rejection Ratios (PRR,

\uparrow

) for

F_{1}

filtering curve for three topic classification datasets.

Method	$\mathrm{FPIR}$
	$0.1$	$0.2$	$0.3$	$0.4$	$0.5$	$0.1$	$0.2$	$0.3$	$0.4$	$0.5$
	PAN-20-AV					CLINC150
AccScr	-0.1	0.38	0.5	0.36	0.39	0.15	0.39	0.49	0.56	0.59
SCF	0.1	0.13	0.15	0.18	0.15	0.52	0.49	0.48	0.47	0.48
GalUE	-0.09	0.42	0.52	0.44	0.47	0.14	0.35	0.4	0.39	0.38
HolUE	0.36	0.5	0.55	0.5	0.51	0.73	0.64	0.59	0.58	0.6

Таблица 3: Prediction Rejection Ratios (PRR,

\uparrow

) for

F_{1}

filtering curve for PAN authorship attribution and CLINC150 intent classification datasets.

Рис. 3: Figure 3. Risk-Controlled Open-set Text Classification on PAN dataset starting with

\mathrm{FPIR}=0.2

. Prediction Rejection Ratio is reported. Subplots show:(a) F1 score,(b) FNIR,(c) FPIR, and(d) False Identification Count versus filter out rate. Better to view in zoom.

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 636 papers

Mixed-Prior Decision Risk for Open-Set Recognition
Sep 28, 2026L. A. Erlygin, P. D. Proskura, A. A. ZaytsevSelective Prediction Open-Set Recognition
Skolkovo Institute of Science and Technology (Skoltech), Moscow, Russia · Risk Management, Sber, Moscow, Russia · The Institute for Information Transmission Problems (IITP RAS), Moscow, Russia
To Predict or Not to Predict? Towards reliable uncertainty estimation in the presence of noise
Mar 7, 2026Nouran Khallaf, Serge SharoffMultilingual Text Classification Uncertainty Calibration
Centre for Translation, Localisation and Interpreting Studies, School of Languages, Cultures and Societies, University of Leeds, UK
Estimating Uncertainty in Classifier Performance with Applications to Large Language Models and Nested Data
Jun 24, 2026Kylie AnglinLLM Evaluation Confidence Region Estimation
Department of Educational Psychology, Neag School of Education, University of Connecticut

Sep 28, 2026J/K move · Enter open · S save

Mixed-Prior Decision Risk for Open-Set Recognition

L. A. Erlygin, P. D. Proskura, A. A. Zaytsev

Skolkovo Institute of Science and Technology (Skoltech), Moscow, Russia · Risk Management, Sber, Moscow, Russia · The Institute for Information Transmission Problems (IITP RAS), Moscow, Russia

Selective Prediction Open-Set Recognition Classification Bayesian Decision Theory

Open paper AskPDF arXiv

In open-set recognition (OSR), a probe must either be identified as one of the known gallery classes or rejected as unknown, so three error types coexist: false acceptance, false rejection, and misidentification. An uncertainty score for selective recognition should rank probes by the risk of the decision the system has made. Bayesian gallery-aware models such as Holistic Uncertainty Estimation (HolUE) summarize the posterior over known and unknown classes by Kullback--Leibler (KL) divergence components and map them to an uncertainty score with a supervised nonlinear calibrator. We show that the KL summary is not generally monotone in decision risk: linear fusion of the KL components tuned on validation data yields negative filtering quality on several benchmarks. We propose MPRisk, a mixed-prior posterior decision-risk score that keeps the same Bayesian posterior but directly scores the error events associated with the selected decision: false-acceptance, misidentification, and false-rejection risks, plus a non-specificity penalty for rejections, enabled by modeling unknown identities as a continuous component. Four nonnegative weights tuned on a validation set suffice for ranking; no nonlinear supervised model is required. Across nine image, audio, and text benchmarks, MPRisk achieves the best or tied-best Prediction Rejection Ratio at every operating point on the image and audio benchmarks and on most text operating points, with bootstrap-confirmed gains over HolUE on five benchmarks (up to

+0.19

PRR) at comparable or lower runtime.

Abstract

Uncertainty Quantification

Explore similar work

Mixed-Prior Decision Risk for Open-Set Recognition

To Predict or Not to Predict? Towards reliable uncertainty estimation in the presence of noise

Estimating Uncertainty in Classifier Performance with Applications to Large Language Models and Nested Data