Partial AUC Maximization from Positive-unlabeled Data

Atsutoshi Kumagai; Tomoharu Iwata; Taishi Nishiyama; Hiroshi Takahashi; Kazuki Adachi; Yasuhiro Fujiwara

cs.LGSep 25, 2026

Partial AUC Maximization from Positive-unlabeled Data

Authors: Atsutoshi Kumagai, Tomoharu Iwata, Taishi Nishiyama, Hiroshi Takahashi, Kazuki Adachi, Yasuhiro Fujiwara

Organizations: NTT, Inc.

Abstract

The partial area under the receiver operating characteristic curve (pAUC) is an important performance metric for binary classification that summarizes true positive rates within a specific range of false positive rates (FPRs). Classifiers that achieve high pAUC need to be obtained in many real-world applications such as cybersecurity, medical care, and advertising. Although many methods for maximizing the pAUC have been proposed, they typically require both labeled positive and negative data for training. However, in practice, labeled negative data are often difficult to collect due to privacy concerns or the need for high expertise to annotate them. In this paper, we propose a method for maximizing the pAUC from positive and unlabeled (PU) data without negative data. Within an empirical risk minimization framework, we show that the pAUC, including its FPR-dependent thresholds, can be represented using only the positive and marginal densities, and derive an empirical estimator from PU data. A classifier is then trained by maximizing the derived smoothed empirical pAUC estimator. We experimentally demonstrate the effectiveness of the proposed method with ten real-world datasets.

Figures & tables

Figure 1: The comparison between the AUC and pAUC: (a) AUC. (b) pAUC within FPR range $[\alpha,\beta]$ .

Figure 2: A visual explanation of determining threshold ${\hat{t}}_{\alpha}$ : The vertical and horizontal axes represent ${\widehat{\rm FPR}}(t)$ and threshold $t$ , respectively. $s^{(k)}$ denotes the $k$ -th smallest score within the scores of all training data $\{s({\bf x})\}$ , where ${\bf x}\in X^{{\rm p}}\cup X$ . From Eq. ( 11 ), ${\widehat{\rm FPR}}(t)=1$ for any $t$ smaller than $s^{(1)}$ and ${\widehat{\rm FPR}}(t)=0$ for any $t$ greater than or equal to $s^{(N^{{\rm p}}+N)}$ . ${\widehat{\rm FPR}}(t)$ is a discrete function where its value changes only when $t$ crosses one of the score $s^{(k)}$ . When threshold $t$ exceeds the score of the unlabeled (positive) instance, ${\widehat{\rm FPR}}(t)$ decreases (increases). By checking $s^{(1)},s^{(2)},\cdots$ in ascending order and identifying the first (minimum) score at which ${\widehat{\rm FPR}}(t)\leq\alpha$ , we can take the score ( $s^{(6)}$ in the figure) as the threshold ${\hat{t}}_{\alpha}$ .

Data	$\pi$	Ours	NNPU	PUAUC	PAUC	PUpAUCh	2step
MNIST	0.1	0.683 $\pm$ 0.074	0.522 $\pm$ 0.131	0.627 $\pm$ 0.056	0.597 $\pm$ 0.070	0.675 $\pm$ 0.073	0.633 $\pm$ 0.058
	0.2	0.633 $\pm$ 0.071	0.577 $\pm$ 0.068	0.592 $\pm$ 0.054	0.473 $\pm$ 0.058	0.624 $\pm$ 0.064	0.592 $\pm$ 0.054
FMNIST	0.1	0.924 $\pm$ 0.025	0.756 $\pm$ 0.022	0.895 $\pm$ 0.037	0.824 $\pm$ 0.064	0.925 $\pm$ 0.028	0.907 $\pm$ 0.037
	0.2	0.914 $\pm$ 0.028	0.770 $\pm$ 0.035	0.867 $\pm$ 0.047	0.649 $\pm$ 0.119	0.912 $\pm$ 0.036	0.873 $\pm$ 0.047
SVHN	0.1	0.247 $\pm$ 0.081	0.055 $\pm$ 0.004	0.208 $\pm$ 0.070	0.201 $\pm$ 0.053	0.233 $\pm$ 0.078	0.202 $\pm$ 0.075
	0.2	0.191 $\pm$ 0.073	0.054 $\pm$ 0.004	0.171 $\pm$ 0.053	0.161 $\pm$ 0.032	0.152 $\pm$ 0.065	0.157 $\pm$ 0.053

Table 1: Test average pAUCs and their standard deviations with

\alpha=0

\beta=0.1

over different labeled positive data sizes

N^{\rm p}=30

50

, and

100

. Values in bold are not statistically different at the

5\%

level from the best performing method in each row according to a paired t-test. ‘# Best’ row represents the number of results of each method that are the best or comparable to it.

Data	$\pi$	Ours	NNPU	PUAUC	PAUC	PUpAUCh	2step
MNIST	0.1	0.755 $\pm$ 0.056	0.610 $\pm$ 0.127	0.725 $\pm$ 0.048	0.736 $\pm$ 0.055	0.749 $\pm$ 0.055	0.730 $\pm$ 0.050
	0.2	0.723 $\pm$ 0.065	0.677 $\pm$ 0.062	0.696 $\pm$ 0.049	0.652 $\pm$ 0.062	0.717 $\pm$ 0.056	0.697 $\pm$ 0.049
FMNIST	0.1	0.943 $\pm$ 0.022	0.775 $\pm$ 0.016	0.925 $\pm$ 0.028	0.926 $\pm$ 0.032	0.940 $\pm$ 0.023	0.934 $\pm$ 0.027
	0.2	0.934 $\pm$ 0.023	0.789 $\pm$ 0.033	0.904 $\pm$ 0.034	0.853 $\pm$ 0.059	0.931 $\pm$ 0.026	0.911 $\pm$ 0.033
SVHN	0.1	0.342 $\pm$ 0.092	0.105 $\pm$ 0.005	0.305 $\pm$ 0.081	0.334 $\pm$ 0.086	0.329 $\pm$ 0.089	0.313 $\pm$ 0.084
	0.2	0.289 $\pm$ 0.090	0.104 $\pm$ 0.005	0.262 $\pm$ 0.063	0.264 $\pm$ 0.052	0.221 $\pm$ 0.067	0.259 $\pm$ 0.064

Table 2: Test average pAUCs and their standard deviations with

\alpha=0

\beta=0.2

over different labeled positive data sizes

N^{\rm p}=30

50

, and

100

. Values in bold are not statistically different at the

5\%

level from the best performing method in each row according to a paired t-test. ‘# Best’ row represents the number of results of each method that are the best or comparable to it.

Panel of Figure 3 — Figure 3: The average test pAUCs and their standard errors with $\alpha=0$ , $\beta=0.2$ over different class-priors $\pi\in\{0.1,0.2\}$ when changing the number of labeled positive data $N^{{\rm p}}$ .

Panel of Figure 4 — Figure 4: The average test pAUCs and their standard errors with $\alpha=0$ , $\beta=0.2$ when true class-prior $\pi=0.2$ . ‘Input $\pi$ ’ means the class-prior value used for training.

Data	$\pi$	Ours	NNPU	PUAUC	PAUC	PUpAUCh	2step
MNIST	0.2	0.721 $\pm$ 0.061	0.676 $\pm$ 0.065	0.696 $\pm$ 0.049	0.652 $\pm$ 0.062	0.716 $\pm$ 0.053	0.697 $\pm$ 0.049
FMNIST	0.2	0.933 $\pm$ 0.024	0.808 $\pm$ 0.055	0.904 $\pm$ 0.034	0.853 $\pm$ 0.059	0.932 $\pm$ 0.026	0.910 $\pm$ 0.033
SVHN	0.2	0.301 $\pm$ 0.073	0.104 $\pm$ 0.005	0.262 $\pm$ 0.063	0.264 $\pm$ 0.052	0.292 $\pm$ 0.083	0.266 $\pm$ 0.063
CIFAR10	0.2	0.571 $\pm$ 0.064	0.494 $\pm$ 0.078	0.559 $\pm$ 0.078	0.459 $\pm$ 0.070	0.539 $\pm$ 0.081	0.556 $\pm$ 0.078
SUSY	0.2	0.523 $\pm$ 0.044	0.488 $\pm$ 0.051	0.499 $\pm$ 0.047	0.434 $\pm$ 0.056	0.505 $\pm$ 0.039	0.498 $\pm$ 0.046
CODRNA	0.2	0.821 $\pm$ 0.043	0.639 $\pm$ 0.167	0.827 $\pm$ 0.048	0.705 $\pm$ 0.072	0.824 $\pm$ 0.043	0.829 $\pm$ 0.047

Table 3: The results using estimated class-prior when the true class-prior is

\pi=0.2

: test average pAUCs and their standard deviations with

\alpha=0

\beta=0.2

over different labeled positive data sizes

N^{\rm p}=30

50

, and

100

. Values in bold are not statistically different at the

5\%

level from the best performing method in each row according to a paired t-test. ‘# Best’ row represents the number of results of each method that are the best or comparable to it.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Data	$\pi$	Ours	NNPU	PUAUC	PAUC	PUpAUCh	2step
MNIST	0.1	0.838 $\pm$ 0.043	0.698 $\pm$ 0.126	0.823 $\pm$ 0.041	0.839 $\pm$ 0.048	0.839 $\pm$ 0.042	0.826 $\pm$ 0.036
	0.2	0.821 $\pm$ 0.048	0.776 $\pm$ 0.063	0.800 $\pm$ 0.046	0.776 $\pm$ 0.041	0.818 $\pm$ 0.050	0.808 $\pm$ 0.050
FMNIST	0.1	0.958 $\pm$ 0.021	0.795 $\pm$ 0.011	0.955 $\pm$ 0.020	0.964 $\pm$ 0.019	0.958 $\pm$ 0.021	0.963 $\pm$ 0.018
	0.2	0.954 $\pm$ 0.024	0.806 $\pm$ 0.034	0.943 $\pm$ 0.022	0.910 $\pm$ 0.037	0.951 $\pm$ 0.025	0.958 $\pm$ 0.018
SVHN	0.1	0.423 $\pm$ 0.098	0.154 $\pm$ 0.006	0.402 $\pm$ 0.092	0.451 $\pm$ 0.104	0.377 $\pm$ 0.092	0.410 $\pm$ 0.097
	0.2	0.360 $\pm$ 0.104	0.154 $\pm$ 0.007	0.353 $\pm$ 0.074	0.391 $\pm$ 0.085	0.258 $\pm$ 0.063	0.351 $\pm$ 0.081

Appendix

Table 4: Test average pAUCs and their standard deviations with

\alpha=0.1

\beta=0.2

over different labeled positive data sizes

N^{\rm p}=30

50

, and

100

. Values in bold are not statistically different at the

5\%

level from the best performing method in each row according to a paired t-test. ‘# Best’ row represents the number of results of each method that are the best or comparable to it.

Data	$\pi$	Ours	NNPU	PUAUC	PAUC	PUpAUCh	2step
MNIST	0.01	0.799 $\pm$ 0.054	0.338 $\pm$ 0.009	0.768 $\pm$ 0.055	0.794 $\pm$ 0.052	0.799 $\pm$ 0.053	0.792 $\pm$ 0.055
FMNIST	0.01	0.949 $\pm$ 0.021	0.731 $\pm$ 0.112	0.945 $\pm$ 0.021	0.947 $\pm$ 0.022	0.949 $\pm$ 0.021	0.954 $\pm$ 0.017
SVHN	0.01	0.411 $\pm$ 0.089	0.105 $\pm$ 0.005	0.363 $\pm$ 0.089	0.410 $\pm$ 0.089	0.412 $\pm$ 0.083	0.398 $\pm$ 0.087
CIFAR10	0.01	0.661 $\pm$ 0.067	0.081 $\pm$ 0.053	0.634 $\pm$ 0.079	0.662 $\pm$ 0.072	0.656 $\pm$ 0.070	0.679 $\pm$ 0.063
SUSY	0.01	0.571 $\pm$ 0.022	0.295 $\pm$ 0.038	0.528 $\pm$ 0.041	0.570 $\pm$ 0.022	0.569 $\pm$ 0.022	0.552 $\pm$ 0.033
CODRNA	0.01	0.882 $\pm$ 0.025	0.124 $\pm$ 0.031	0.875 $\pm$ 0.032	0.883 $\pm$ 0.024	0.882 $\pm$ 0.024	0.881 $\pm$ 0.027

Appendix

Table 5: The results with

\pi=0.01

and

\pi=0.3

: test average pAUCs and their standard deviations with

\alpha=0

\beta=0.2

over different labeled positive data sizes

N^{\rm p}=30

50

, and

100

. Values in bold are not statistically different at the

5\%

level from the best performing method in each row according to a paired t-test. ‘# Best’ row represents the number of results of each method that are the best or comparable to it.

Data	$\pi$	Ours	PUAUC	PAUC	PUpAUCh	2step
MNIST	0.1	0.759 $\pm$ 0.058	0.727 $\pm$ 0.047	0.743 $\pm$ 0.057	0.752 $\pm$ 0.055	0.732 $\pm$ 0.048
	0.2	0.727 $\pm$ 0.066	0.699 $\pm$ 0.050	0.652 $\pm$ 0.060	0.719 $\pm$ 0.057	0.700 $\pm$ 0.049
FMNIST	0.1	0.945 $\pm$ 0.023	0.915 $\pm$ 0.034	0.933 $\pm$ 0.027	0.944 $\pm$ 0.024	0.929 $\pm$ 0.029
	0.2	0.939 $\pm$ 0.024	0.907 $\pm$ 0.036	0.854 $\pm$ 0.057	0.936 $\pm$ 0.026	0.915 $\pm$ 0.034
SVHN	0.1	0.346 $\pm$ 0.095	0.302 $\pm$ 0.070	0.328 $\pm$ 0.071	0.330 $\pm$ 0.096	0.308 $\pm$ 0.073
	0.2	0.285 $\pm$ 0.091	0.262 $\pm$ 0.061	0.268 $\pm$ 0.050	0.227 $\pm$ 0.069	0.259 $\pm$ 0.062

Appendix

Table 6: The results using the logistic loss: test average pAUCs and their standard deviations with

\alpha=0

\beta=0.2

over different labeled positive data sizes

N^{\rm p}=30

50

, and

100

. Values in bold are not statistically different at the

5\%

level from the best performing method in each row according to a paired t-test. ‘# Best’ row represents the number of results of each method that are the best or comparable to it.

Panel of Figure 5 — Figure 5: The average test pAUCs and their standard errors with $\alpha=0$ , $\beta=0.2$ over different class-priors $\pi\in\{0.1,0.2\}$ and positive data sizes $N^{{\rm p}}\in\{30,50,100\}$ when changing the number of unlabeled data $N$ .

Data	$N$	Ours	NNPU	PUAUC	PAUC	PUpAUCh	2step
MNIST	5,000	90.460	90.054	90.036	90.978	89.618	184.040
CIFAR10	9,000	305.871	304.184	302.897	305.471	302.469	608.849

Appendix

Table 7: Training time [sec] of each method on the MNIST and CIFAR10 datasets, where

(\alpha,\beta)=(0,0.2)

\pi=0.2

N^{{\rm p}}=100

Data	$\beta$	Ours	NNPU	PUAUC	PAUC	PUpAUCh	2step
Mammography	0.1	0.784 $\pm$ 0.041	0.001 $\pm$ 0.001	0.754 $\pm$ 0.041	0.780 $\pm$ 0.045	0.786 $\pm$ 0.037	0.784 $\pm$ 0.037
Mammography	0.2	0.821 $\pm$ 0.039	0.004 $\pm$ 0.004	0.810 $\pm$ 0.032	0.831 $\pm$ 0.039	0.827 $\pm$ 0.036	0.834 $\pm$ 0.029
Online-shoppers	0.1	0.329 $\pm$ 0.081	0.222 $\pm$ 0.072	0.259 $\pm$ 0.069	0.226 $\pm$ 0.032	0.308 $\pm$ 0.085	0.263 $\pm$ 0.071
Online-shoppers	0.2	0.442 $\pm$ 0.079	0.331 $\pm$ 0.079	0.371 $\pm$ 0.073	0.386 $\pm$ 0.063	0.425 $\pm$ 0.079	0.379 $\pm$ 0.073

Appendix

Table 8: The results on Mammography and Online-shoppers: test average pAUCs and their standard deviations with

\alpha=0

\beta=0.1

and

\alpha=0

\beta=0.2

. Values in bold are not statistically different at the

5\%

level from the best performing method in each row according to a paired t-test.

Data	$\beta$	$N^{p}$	FPR	Epoch 1	Epoch 10	Epoch 50	Epoch 100
FMNIST	0.1	30	$\mathrm{FPR}(\hat{t}_{\alpha})$	0.025	0.031	0.031	0.031
	0.1	100	$\mathrm{FPR}(\hat{t}_{\alpha})$	0.017	0.020	0.022	0.023
	0.1	30	$\mathrm{FPR}(\hat{t}_{\beta})$	0.118	0.115	0.115	0.116
	0.1	100	$\mathrm{FPR}(\hat{t}_{\beta})$	0.107	0.109	0.108	0.109
	0.2	30	$\mathrm{FPR}(\hat{t}_{\alpha})$	0.022	0.030	0.031	0.031
	0.2	100	$\mathrm{FPR}(\hat{t}_{\alpha})$	0.013	0.021	0.021	0.022

Appendix

Table 9: FPRs induced by the estimated thresholds during training.

Data	$c$	Oracle $-$ PUpAUCh
MNIST	1.5421	0.08379
FMNIST	0.6666	0.01786
SVHN	1.5432	0.09771
CIFAR10	1.5502	0.09186
SUSY	1.1720	0.04063
CODRNA	2.2437	0.04952

Appendix

Table 10: Pseudo-negative selection quality of PUpAUCh ( Iwata et al., 2020 ) and the test pAUC gap between Oracle and PUpAUCh. Larger

c

indicates greater positive contamination in the selected pseudo-negatives relative to random selection.

Explore similar work

AUC Maximization from Biased Positive-unlabeled Data with Confidence
Sep 10, 2026Atsutoshi Kumagai, Tomoharu Iwata, Hiroshi Takahashi +3Class-Imbalanced Learning Positive-Unlabeled Learning
NTT, Inc.
Focused PU learning from imbalanced data
May 14, 2026Elias Zavitsanos, Georgios PaliourasClass-Imbalanced Learning Binary Classification
Institute of Informatics and Telecommunications, NCSR “Demokritos”, Patriarhou Gregoriou E and 27 Neapoleos St., Agia Paraskevi, 15310, Attica, Greece.
PUe: Biased Positive-Unlabeled Learning Enhancement by Causal Inference
Jul 15, 2026Xutao Wang, Hanting Chen, Tianyu Guo +1Inverse Probability Weighting Positive-Unlabeled Learning
Huawei Noah’s Ark Lab

Sep 10, 2026J/K move · Enter open · S save

AUC Maximization from Biased Positive-unlabeled Data with Confidence

Atsutoshi Kumagai, Tomoharu Iwata, Hiroshi Takahashi, Taishi Nishiyama+2

NTT, Inc.

Class-Imbalanced Learning Positive-Unlabeled Learning

Open paper AskPDF arXiv

Maximizing the area under the receiver operating characteristic curve (AUC) is a standard approach to imbalanced binary classification. Although positive and negative data are required for maximizing the AUC, negative data are often difficult to collect in some real-world applications due to privacy concerns or the need for specialized expertise to annotate them. Thus, AUC maximization from positive and unlabeled (PU) data has been attracting attention. Existing methods assume that labeled positive data are unbiased samples from the true positive distribution. However, this ideal assumption is often violated in practice. In this paper, we propose a method to maximize the AUC from biased PU data. To address the bias, our key idea is to exploit {\it confidence}, i.e., the probability that an instance is positive, associated with the small number of labeled positive data. We derive an estimator of the AUC risk using biased PU data with confidence, enabling AUC maximization under such bias. We further show that the rewritten AUC risk induces a Bayes-optimal AUC ranking even when the available confidence is any strictly increasing transformation of the true posterior probability. We experimentally show the effectiveness of our method on eight real-world datasets.

Abstract

Explore similar work

AUC Maximization from Biased Positive-unlabeled Data with Confidence

Focused PU learning from imbalanced data

PUe: Biased Positive-Unlabeled Learning Enhancement by Causal Inference