This study addresses the PU classification problem under violations of the SCAR assumption. We investigate logistic regression-based approaches, namely the cluster method and its extensions with strict and non-strict Lasso regularization. The primary contribution of this work is the integration of the SMOTE technique to alleviate class imbalance and systematically assess its impact on the performance of the considered algorithms. SMOTE is first applied to rebalance the training dataset. Next, cleaning labels are derived via 2-means clustering. Logistic regression is then trained on the cleaned data, where identified positive instances are augmented with additional true positives and the remaining observations are treated as negative. The experimental evaluation is conducted on 13 real benchmark datasets and one synthetic dataset. For comparison, we include the naive approach and the Spy-EM method. The results demonstrate that incorporating SMOTE improves classification performance when the SCAR condition is violated and indicate moderate robustness of the LassoJoint method in this setting.
Figures & tables
Figure 1: AUC; SMOTE scheme; nonSCAR
Table 1: Number of experiments by SMOTE effect (AUC and Accuracy)
Method
Δ AUC (scar)
Δ AUC (nonscar)
Δ ACC (scar)
Δ ACC (nonscar)
calc_c = 0.3
naive
0.1%
-1.3%
82.4%
10.3%
clust
7.7%
2.1%
16.3%
7.3%
strict-lassclust
7.5%
1.2%
15.8%
6.1%
non-strict-lassclust
7.4%
1.4%
16.3%
7.8%
lassojoint
-0.5%
-0.3%
2.6%
9.1%
Table 2: SMOTE vs non-SMOTE — % diff. for SCAR and nonSCAR scheme
Method
Δ AUC (scar)
Δ AUC (nonscar)
Δ ACC (scar)
Δ ACC (nonscar)
calc_c = 0.3
clust
-5.9%
7.4%
15.2%
30.5%
lassojoint
-0.3%
5.1%
-
-16.0%
non-strict-lassclust
-5.0%
9.6%
14.8%
30.0%
strict-lassclust
-5.0%
9.7%
15.0%
30.3%
calc_c = 0.5
Table 3: % diff. methods (with SMOTE enhancement) vs naive in SCAR and nonSCAR scheme
The present study aims to investigate a cluster cleaning algorithm that is both computationally simple and capable of solving the PU classification when the SCAR condition is unsatisfied. A secondary objective of this study is to determine the robustness of the LassoJoint method to perturbations of the SCAR condition. In the first step of our algorithm, we obtain cleaning labels from 2-means clustering. Subsequently, we perform logistic regression on the cleaned data, assigning positive labels from the cleaning algorithm with additional true positive observations. The remaining observations are assigned the negative label. The proposed algorithm is evaluated by comparing 11 real data sets from machine learning repositories and a synthetic set. The findings obtained from this study demonstrate the efficacy of the clustering algorithm in scenarios where the SCAR condition is violated and further underscore the moderate robustness of the LassoJoint algorithm in this context.
Konrad Furmanczyk, Kacper Paczutkowski
Institute of Information Technology, Warsaw University of Life Sciences, Warsaw, Poland
We propose a new method of learning from positive and unlabeled (PU) examples in highly imbalanced datasets. Many real-world problems, such as disease gene identification, targeted marketing, fraud detection, and recommender systems, are hard to address with machine learning methods, due to limited labeled data. Often, training data comprises positive and unlabeled instances, the latter typically being dominated by negative, but including also several positive instances. While PU learning is well-studied, few methods address imbalanced settings or hard-to-detect positive examples that resemble negative ones. Our approach uses a focused empirical risk estimator, incorporating both positive and unlabeled examples to train binary classifiers. Empirical evaluations demonstrate state-of-the-art performance on imbalanced datasets under two labeling mechanisms - selecting positives completely at random (SCAR) and selecting at random (SAR). Beyond these controlled experiments, we demonstrate the value of the proposed method in the real-world application of financial misstatement detection.
Elias Zavitsanos, Georgios Paliouras
Institute of Informatics and Telecommunications, NCSR “Demokritos”, Patriarhou Gregoriou E and 27 Neapoleos St., Agia Paraskevi, 15310, Attica, Greece.
The partial area under the receiver operating characteristic curve (pAUC) is an important performance metric for binary classification that summarizes true positive rates within a specific range of false positive rates (FPRs). Classifiers that achieve high pAUC need to be obtained in many real-world applications such as cybersecurity, medical care, and advertising. Although many methods for maximizing the pAUC have been proposed, they typically require both labeled positive and negative data for training. However, in practice, labeled negative data are often difficult to collect due to privacy concerns or the need for high expertise to annotate them. In this paper, we propose a method for maximizing the pAUC from positive and unlabeled (PU) data without negative data. Within an empirical risk minimization framework, we show that the pAUC, including its FPR-dependent thresholds, can be represented using only the positive and marginal densities, and derive an empirical estimator from PU data. A classifier is then trained by maximizing the derived smoothed empirical pAUC estimator. We experimentally demonstrate the effectiveness of the proposed method with ten real-world datasets.