Robust performance metrics for imbalanced classification problems

Hajo Holzmann; Bernhard Klar

stat.MLApr 11, 2024

Robust performance metrics for imbalanced classification problems

Authors: Hajo Holzmann, Bernhard Klar

Organizations: Fachbereich Mathematik und Informatik, Philipps-Universität Marburg, Hans-Meerweinstr. 6, 35043 Marburg, Germany · Institut für Stochastik, Karlsruher Institut für Technologie (KIT)

Abstract

We show that established performance metrics in binary classification, such as Matthews' correlation coefficient (MCC), Cohen's $κ$ , the F-score or the Jaccard similarity coefficient are not robust to class imbalance in the sense that if the proportion of the minority class tends to $0$ , the true positive rate (TPR) of the Bayes classifier under these metrics {becomes very small or even tends to $0$ }. Thus, in imbalanced classification problems, these metrics favour classifiers which ignore the minority class. To alleviate this issue we introduce robustified modifications of the MCC, of Cohen's $κ$ and of the F-score with an additional tuning parameter which allows to adapt the amount of robustness against class imbalance. As theoretical guarantee we show that the Bayes-optimal classifier for these robustified performance metrics, when expressed in terms of the density ratio $f_1/f_0$ of the class-conditional densities $f_i$ , has a threshold parameter which is upper-bounded in terms of the tuning parameters. Therefore, even in strongly imbalanced settings, the TPR associated to this classifier will be bounded away from $0$ {, provided that $f_1/f_0$ exceeds this bound with positive probability under the positive class}. We numerically illustrate the behaviour of the various performance metrics and the effect of the tuning parameters in simulations as well as on a credit default data set. We also discuss connections to the receiver operating characteristic and precision-recall curves, which provide an alternative perspective on the proposed notion of robustness, and give recommendations on how to combine their usage with performance metrics.

Figures & tables

		$\phi_{1}$
		1	0
$Y$	1	2640	360
$Y$	0	1352	5648

Table 1: Empirical confusion matrices for the classifiers in Example 1 .

	$F_{1.5}$	MCC	$F_{0.5}$
$\phi_{1}$	0.80	0.64	0.70
$\phi_{2}$	0.77	0.67	0.77
$\phi_{3}$	0.68	0.64	0.79

Table 2: Values of different performance measures for the three classifiers in Example 1 .

$\hat{\pi}$		$F_{1.5}$	MCC	$F_{0.5}$	$F_{\text{rb}}$	$\text{MCC}_{\text{rb}}^{0.1}$	$\text{MCC}_{\text{rb}}^{0.05}$
	value	0.794	0.665	0.797	0.788	0.667	0.665
	$\tilde{\delta}$	0.250	0.426	0.644	0.257	0.362	0.362
0.3	$\hat{\pi}_{1\|1}$	0.871	0.767	0.612	0.867	0.807	0.807
	$\hat{\pi}_{0\|0}$	0.810	0.899	0.958	0.814	0.873	0.873
	$\delta$	0.778	1.732	4.221	0.807	1.324	1.324
	value	0.640	0.570	0.667	0.680	0.597	0.584

Table 3: Values of metrics, values of optimal cutoff

\tilde{\delta}

(in terms of regression function) and

\delta

(in terms of density ratio) and class-conditional probabilities of classifiers that empirically maximize the various performance measures used in Example 2 .

		$\phi(X)$
		1	0	Total
$Y$	1	$\pi_{11}(\phi)$	$\pi_{10}(\phi)$	$\pi$
$Y$	0	$\pi_{01}(\phi)$	$\pi_{00}(\phi)$	$1-\pi$
	Total	$\gamma$	$1-\gamma$	1

Table 4: Population confusion matrix of a classifier.

		$\pi={\mathbb{P}}(Y=1)$
$\Delta$	$10^{-10}$	$10^{-9}$	$10^{-8}$	$10^{-7}$	$10^{-6}$	$10^{-5}$	$10^{-4}$	$10^{-3}$	$0.01$	$0.1$
1.00	269.43	186.43	126.15	83.11	52.99	32.42	18.79	10.11	4.85	1.86
1.50	3.4e3	2.0e3	1.1e3	602.13	311.30	152.04	69.04	28.43	10.17	2.82
2.00	3.5e4	1.7e4	8.0e3	3.6e3	1.5e3	595.58	214.56	68.95	18.93	3.97
3.00	2.0e6	6.8e5	2.3e5	7.0e4	2.0e4	5.3e3	1.2e3	258.47	45.80	6.33

Table 5: Limiting behavior of the optimal threshold

\delta^{*}

of JAC and

F_{1}

-score for

\pi\to 0

and different values of

\Delta

	$\pi={\mathbb{P}}(Y=1)$
	$0.0001$		$0.001$		$0.01$		$0.1$
JAC	18.80		10.10		4.85		1.87
	0.01	1.00	0.03	1.00	0.14	0.98	0.45	0.87
$F_{1.5}$	15.30		7.93		3.61		1.26
	0.01	1.00	0.06	0.99	0.22	0.96	0.61	0.77
$F_{0.5}$	26.30		14.80		7.66		3.34

Table 6: Lines 1,3,5: Optimal threshold

\delta^{*}

for JAC,

F_{1.5}

and

F_{0.5}

, using

\Delta=1

. Lines 2,4,6: Corresponding conditional probabilities

\pi_{1|1}

and

\pi_{0|0}

Figure 1: Plots of the optimal threshold

\delta^{\ast}

as a function of

\pi

for the different performance metrics used in Example 4

Figure 2: Plots of the optimal threshold

\delta^{\ast}

as a function of

\pi

for the different performance metrics used in Example 5 .

Figure 3: Plots of the optimal threshold

\delta^{\ast}

as a function of

\pi

for

\text{MCC}_{\text{rb}}

proposed in subsection 5.1 , using different choices of the parameter

d

Figure 4: Plots of

\pi_{1|1}

associated to the optimal threshold

\delta^{\ast}

as a function of

\pi

for

\text{MCC}_{\text{rb}}

proposed in subsection 5.1 , using different choices of the parameter

d

Figure 5: Plots of the optimal threshold

\delta^{\ast}

as a function of

\pi

for

\kappa_{\text{rb}}

proposed in subsection 5.2 , using different choices of the parameter

d

Figure 6: Plots of the optimal threshold

\delta^{\ast}

as a function of

\pi

for

\text{F}_{\text{rb}}

-score proposed in subsection 5.3 with different choices of the parameters.

Figure 7: Plots of

\pi_{1|1}

for the optimal threshold

\delta^{\ast}

as a function of

\pi

for

\text{F}_{\text{rb}}

-score proposed in subsection 5.3 with different choices of the parameters.

Figure 8: Plots of the optimal threshold

\delta^{*}

as a function of

\pi

for the standardized

t_{5}

location-shift model (left) and the standardized skew-

t_{20}

location-shift model with shape parameter

\alpha=2

(right). Solid lines represent the classical metrics and dashed lines their robustified versions. The robustness parameters are calibrated to the common bound

c_{\mathrm{r}}=6

$n$		MCC	$\mathrm{MCC}_{\mathrm{rb}}^{0.1}$
500	value	$0.525\pm 0.108$	$0.610\pm 0.094$
	$\tilde{\delta}$	$0.271\pm 0.139$ (0.51)	$0.082\pm 0.054$ (0.66)
	$\delta$	$17.566\pm 13.453$ (0.77)	$3.822\pm 2.990$ (0.78)
2000	value	$0.464\pm 0.055$	$0.562\pm 0.050$
	$\tilde{\delta}$	$0.272\pm 0.115$ (0.42)	$0.069\pm 0.028$ (0.41)
	$\delta$	$16.135\pm 10.422$ (0.65)	$2.920\pm 1.330$ (0.46)

Table 7: Finite-sample behavior of the classical MCC and the robust metric

\mathrm{MCC}_{\mathrm{rb}}^{0.1}

for varying sample sizes

n

at class proportion

\hat{\pi}=0.025

. Reported are the mean and standard deviation (over 1000 repetitions) of the maximized metric value, the corresponding regression-function threshold

\tilde{\delta}

, and the density-ratio threshold

\delta

. In parentheses: coefficient of variation of the thresholds.

Figure 9: Population ROC curve for Example 4 in black, solid line. Plot of recall against 1-precision in color with different line styles for varying

\pi

. Circles show the corresponding MCC optimal points; triangles show the points optimal with respect to the robust MCC with

d=0.15

		JAC	MCC	$F_{1.5}$	$F_{0.5}$	Kappa
test	value	0.183	0.262	0.318	0.341	0.261
$\hat{\pi}=0.067$	$\tilde{\delta}$	0.123	0.135	0.104	0.169	0.135
	$\hat{\pi}_{1\|1}$	0.312	0.288	0.383	0.209	0.288
	$\hat{\pi}_{0\|0}$	0.949	0.958	0.909	0.978	0.958
subset.test	value	0.066	0.123	0.149	0.131	0.105
$\hat{\pi}=0.014$	$\tilde{\delta}$	0.043	0.036	0.035	0.061	0.050

Table 8: Test data: Maximal value of performance measure, optimal cutoff

\tilde{\delta}

, TPR

\hat{\pi}_{1|1}

and TNR

\hat{\pi}_{0|0}

of classifiers that empirically maximize the various performance measures used in Section 7 . Upper panel: Logistic regression. Middle panel: Random forests. Lower panel: Gradient boosting.

	$(d_{0},d_{1})$	$(0,1)$	$(0.1,1)$	$(0.2,1)$	$(0.5,1)$	$(0.8,1)$	$(0.2,2)$
test	value	0.309	0.323	0.350	0.436	0.500	0.371
$\hat{\pi}=0.067$	$\tilde{\delta}$	0.123	0.103	0.088	0.057	0.048	0.080
	$\hat{\pi}_{1\|1}$	0.312	0.388	0.471	0.699	0.767	0.523
	$\hat{\pi}_{0\|0}$	0.949	0.906	0.839	0.570	0.449	0.790
subset.test	value	0.125	0.227	0.277	0.406	0.492	0.284
$\hat{\pi}=0.014$	$\tilde{\delta}$	0.043	0.025	0.019	0.014	0.011	0.019

Table 9: Test data: Maximal value of performance measure, optimal cutoff

\tilde{\delta}

, TPR

\hat{\pi}_{1|1}

and TNR

\hat{\pi}_{0|0}

of classifiers that empirically maximize the robust F-score with

c=0

and different choices of

(d_{0},d_{1})

used in Section 7 . Upper panel: Logistic regression. Middle panel: Random forests. Lower panel: Gradient boosting.

	$d$	$0$	$0.01$	$0.05$	$0.1$	$0.5$	$1.0$
test	value	0.262	0.260	0.260	0.264	0.288	0.298
$\hat{\pi}=0.067$	$\tilde{\delta}$	0.135	0.127	0.123	0.104	0.088	0.088
	$\hat{\pi}_{1\|1}$	0.288	0.303	0.312	0.383	0.471	0.471
	$\hat{\pi}_{0\|0}$	0.958	0.953	0.949	0.909	0.839	0.839
subset.test	value	0.123	0.153	0.194	0.216	0.269	0.282
$\hat{\pi}=0.014$	$\tilde{\delta}$	0.036	0.029	0.025	0.025	0.019	0.019

Table 10: Test data: Maximal value of performance measure, optimal cutoff

\tilde{\delta}

, TPR

\hat{\pi}_{1|1}

and TNR

\hat{\pi}_{0|0}

of classifiers that empirically maximize the robust MCC with different choices of

d

used in Section 7 . Upper panel: Logistic regression. Middle panel: Random forests. Lower panel: Gradient boosting.

Figure 10: Empirical true positive rates

\hat{\pi}_{1|1}

(solid black) and true negative rates

\hat{\pi}_{0|0}

(dashed gray) of classifiers that maximize the respective performance measures for the full dataset (

\hat{\pi}=0.067

), shown for logistic regression, random forest, and gradient boosting. The left block displays classical metrics (JAC, MCC,

F_{1.5}

F_{0.5}

, Kappa), while the middle and right blocks correspond to the robust

F

-score and robust MCC with varying parameter choices.

Figure 11: Same plot as Figure 10 for the imbalanced subset (

\hat{\pi}=0.014

Figure 12: Solid lines: ROC curves of the gradient boosting classifier for the complete test data (black) and the subset of test data (grey) used in Section 7 . Dotted lines: Recall against 1-precision for the complete test data (black) and the subset of test data (grey). Circles show the corresponding MCC optimal points; points optimal with respect to the robust MCC with

d=0.01,0.05,0.1

are shown as triangles, diamonds and squares.

Figure 13: Left panel: Solid lines show ROC curves of logistic regression (black), random forest (dark gray) and gradient boosting classifier (light gray) for the complete test data used in Section 7 . Dotted lines show recall against 1-precision curve of logistic regression (black), random forest (dark gray) and gradient boosting classifier (light gray) for this data set. Circles show the corresponding MCC optimal points; points optimal with respect to the robust MCC with

d=0.01,0.05,0.1

are shown as triangles, diamonds and squares. Right panel: Same plot for the subset of test data used in Section 7 .

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

		$\pi={\mathbb{P}}(Y=1)$
$\Delta$		$0.0001$		$0.001$		$0.01$		$0.1$		$0.9$		$0.99$		$0.999$		$0.9999$
	JAC	4.41		3.16		2.11		1.19		0.10		0.01		0.00		0.00
		0.00	1.00	0.02	0.99	0.11	0.96	0.46	0.73	1.00	0.00	1.00	0.00	1.00	0.00	1.00	0.00
	$F_{1.5}$	3.94		2.77		1.78		0.92		0.05		0.00		0.00		0.00
0.5		0.01	1.00	0.04	0.99	0.18	0.92	0.66	0.54	1.00	0.00	1.00	0.00	1.00	0.00	1.00	0.00
	$F_{0.5}$	5.26		3.88		2.72		1.70		0.31		0.04		0.00		0.00

Appendix

Table 11: Lines 1,3,…: Optimal threshold

\delta^{*}

for standard metrics and several values of

\Delta

. Lines 2,4,…: Corresponding conditional probabilities

\pi_{1|1}

and

\pi_{0|0}

		$\pi={\mathbb{P}}(Y=1)$
$\Delta$	$10^{-10}$	$10^{-9}$	$10^{-8}$	$10^{-7}$	$10^{-6}$	$10^{-5}$	$10^{-4}$	$10^{-3}$	$0.01$	$0.1$
1.00	237.21	162.92	109.27	71.22	44.80	26.94	15.25	7.93	3.61	1.26
1.50	2825.82	1622.27	900.43	480.03	243.61	116.22	51.16	20.19	6.79	1.70
2.00	27356.75	13175.23	6079.02	2665.88	1099.51	420.39	146.00	44.70	11.49	2.19
3.00	1354744.38	464321.86	150125.64	45357.40	12653.67	3210.71	727.02	143.52	23.96	3.10

Appendix

Table 12: Limiting behavior of the optimal threshold

\delta^{\ast}

F_{1.5}

for

\pi\to 0

and several values of

\Delta

		$\pi={\mathbb{P}}(Y=1)$
$\Delta$	$10^{-10}$	$10^{-9}$	$10^{-8}$	$10^{-7}$	$10^{-6}$	$10^{-5}$	$10^{-4}$	$10^{-3}$	$0.01$	$0.1$
1.00	333.17	233.27	160.06	107.22	69.78	43.82	26.29	14.83	7.66	3.35
1.50	4671.72	2756.78	1580.39	875.71	465.90	235.84	112.15	49.15	19.21	6.12
2.00	53081.39	26479.61	12729.65	5860.90	2563.68	1054.07	401.43	138.66	41.98	10.11
3.00	3589564.48	1291462.21	441531.99	142346.77	42862.63	11910.11	3007.53	676.76	131.88	20.34

Appendix

Table 13: Limiting behavior of the optimal threshold

\delta^{\ast}

F_{0.5}

for

\pi\to 0

and several values of

\Delta

		$\pi={\mathbb{P}}(Y=1)$
$\Delta$	$10^{-10}$	$10^{-9}$	$10^{-8}$	$10^{-7}$	$10^{-6}$	$10^{-5}$	$10^{-4}$	$10^{-3}$	$0.01$	$0.1$
1.00	269.58	186.60	126.33	83.30	53.19	32.65	19.05	10.40	5.21	2.29
1.50	3411.85	1979.74	1112.83	602.41	311.60	152.37	69.40	28.84	10.62	3.26
2.00	35067.15	17119.42	8025.62	3587.06	1514.13	596.00	215.02	69.43	19.43	4.38
3.00	1950933.76	681242.81	225050.90	69722.95	20036.29	5267.30	1244.92	259.00	46.22	6.59

Appendix

Table 14: Limiting behavior of the optimal threshold

\delta^{\ast}

of Kappa for

\pi\to 0

and several values of

\Delta

		$\pi={\mathbb{P}}(Y=1)$
$\Delta$	$10^{-10}$	$10^{-9}$	$10^{-8}$	$10^{-7}$	$10^{-6}$	$10^{-5}$	$10^{-4}$	$10^{-3}$	$0.01$	$0.1$
1	2.58	2.58	2.58	2.58	2.58	2.58	2.57	2.56	2.46	1.85
1.5	10.51	10.51	10.51	10.51	10.51	10.51	10.46	10.01	7.46	3.08
2	110.26	110.26	110.26	110.25	110.15	109.13	100.38	62.89	20.92	4.55
3	178272.20	177701.27	172284.45	136127.76	58364.38	14096.18	2541.43	394.69	55.79	6.93

Appendix

Table 15: Limiting behavior of the optimal threshold

\delta^{\ast}

of MCC for

\pi\to 0

and several values of

\Delta

		$\pi={\mathbb{P}}(Y=1)$
$\Delta$	$d_{0}$	$0.0001$		$0.001$		$0.01$		$0.1$		$0.9$		$0.99$		$0.999$		$0.9999$
	0.00	4.41		3.16		2.11		1.19		0.10		0.01		0.00		0.00
		0.00	1.00	0.02	0.99	0.11	0.96	0.46	0.73	1.00	0.00	1.00	0.00	1.00	0.00	1.00	0.00
	0.10	1.23		1.23		1.19		0.96		0.09		0.01		0.00		0.00
0.5		0.43	0.75	0.43	0.75	0.46	0.73	0.63	0.57	1.00	0.00	1.00	0.00	1.00	0.00	1.00	0.00
	0.40	0.77		0.77		0.76		0.68		0.07		0.01		0.00		0.00

Appendix

Table 16: Lines 1,3,…: Optimal threshold

\delta^{*}

for

F_{\text{rb}}

with varying values of

d_{0}

and

\Delta

;

c=0

and

d_{1}=1

fixed. Lines 2,4,…: Corresponding conditional probabilities

\pi_{1|1}

and

\pi_{0|0}

		$\pi={\mathbb{P}}(Y=1)$
$\Delta$	$d_{1}$	$0.0001$		$0.001$		$0.01$		$0.1$		$0.9$		$0.99$		$0.999$		$0.9999$
	0.25	1.23		1.23		1.22		1.12		0.23		0.03		0.00		0.00
		0.43	0.75	0.43	0.75	0.44	0.74	0.51	0.68	1.00	0.00	1.00	0.00	1.00	0.00	1.00	0.00
	0.50	1.23		1.23		1.21		1.06		0.15		0.02		0.00		0.00
0.5		0.43	0.75	0.43	0.75	0.45	0.74	0.55	0.64	1.00	0.00	1.00	0.00	1.00	0.00	1.00	0.00
	2.00	1.23		1.22		1.16		0.83		0.05		0.00		0.00		0.00

Appendix

Table 17: Lines 1,3,…: Optimal threshold

\delta^{*}

for

F_{\text{rb}}

with varying values of

d_{1}

and

\Delta

;

c=0

and

d_{0}=0.1

fixed. Lines 2,4,…: Corresponding conditional probabilities

\pi_{1|1}

and

\pi_{0|0}

		$\pi={\mathbb{P}}(Y=1)$
$\Delta$	$d_{0}$	$0.0001$		$0.001$		$0.01$		$0.1$		$0.9$		$0.99$		$0.999$		$0.9999$
	1.00	1.15		1.15		1.14		1.06		0.18		0.02		0.00		0.00
		0.49	0.70	0.49	0.70	0.50	0.70	0.55	0.64	1.00	0.00	1.00	0.00	1.00	0.00	1.00	0.00
	2.00	0.68		0.68		0.68		0.63		0.10		0.01		0.00		0.00
0.5		0.84	0.31	0.84	0.31	0.85	0.30	0.88	0.25	1.00	0.00	1.00	0.00	1.00	0.00	1.00	0.00
	0.50	2.05		2.05		2.03		1.86		0.33		0.04		0.00		0.00

Appendix

Table 18: Lines 1,3,…: Optimal threshold

\delta^{*}

for

\text{F}_{\text{rb}}

with varying values of

d_{0}

and

\Delta

;

c=1

and

d_{1}=1(d_{1}=4\text{ for last entry})

fixed. Lines 2,4,…: Corresponding conditional probabilities

\pi_{1|1}

and

\pi_{0|0}

		$\pi={\mathbb{P}}(Y=1)$
$\Delta$	$d$	$0.0001$		$0.001$		$0.01$		$0.1$		$0.9$		$0.99$		$0.999$		$0.9999$
	0.00	1.25		1.25		1.24		1.19		0.84		0.80		0.80		0.80
		0.42	0.76	0.42	0.76	0.43	0.75	0.46	0.73	0.72	0.46	0.75	0.43	0.76	0.42	0.76	0.42
	0.01	1.22		1.22		1.21		1.16		0.86		0.83		0.82		0.82
		0.44	0.74	0.44	0.74	0.45	0.74	0.48	0.71	0.71	0.48	0.74	0.45	0.74	0.44	0.74	0.44
	0.05	1.15		1.15		1.14		1.11		0.90		0.88		0.87		0.87

Appendix

Table 19: Lines 1,3,…: Optimal threshold

\delta^{*}

for

\text{MCC}_{\text{rb}}

with varying values of

d

and

\Delta

. Lines 2,4,…: Corresponding conditional probabilities

\pi_{1|1}

and

\pi_{0|0}

Figure 14: Plots of the optimal threshold

\delta^{\ast}

as a function of

\pi

for

\text{F}_{\text{rb}}

with different choices of the parameters in the setting of Example 5 .

Figure 15: Plots of the optimal threshold

\delta^{\ast}

as a function of

\pi

for

\text{MCC}_{\text{rb}}

with different choices of the parameter

d

in the setting of Example 5 .

		JAC	MCC	$F_{1.5}$	$F_{0.5}$	Kappa
training	value	0.184	0.264	0.322	0.339	0.263
$\hat{\pi}=0.067$	$\tilde{\delta}$	0.123	0.135	0.104	0.169	0.135
	$\hat{\pi}_{1\|1}$	0.314	0.286	0.385	0.209	0.286
	$\hat{\pi}_{0\|0}$	0.950	0.959	0.911	0.978	0.959
subset.train	value	0.086	0.148	0.179	0.165	0.145
$\hat{\pi}=0.014$	$\tilde{\delta}$	0.043	0.036	0.035	0.061	0.050

Appendix

Table 20: Training data: Maximal value of performance measure, optimal cutoff

\tilde{\delta}

, TPR

\hat{\pi}_{1|1}

and TNR

\hat{\pi}_{0|0}

of classifiers that empirically maximize the various performance measures used in the data example. Upper panel: Logistic regression. Middle panel: Random forests. Lower panel: Gradient boosting.

	$(d_{0},d_{1})$	$(0,1)$	$(0.1,1)$	$(0.2,1)$	$(0.5,1)$	$(0.8,1)$	$(0.2,2)$
training	value	0.311	0.326	0.352	0.436	0.503	0.370
$\hat{\pi}=0.067$	$\tilde{\delta}$	0.123	0.103	0.088	0.057	0.048	0.080
	$\hat{\pi}_{1\|1}$	0.314	0.389	0.473	0.700	0.773	0.523
	$\hat{\pi}_{0\|0}$	0.950	0.908	0.839	0.567	0.445	0.789
subset.train	value	0.158	0.249	0.289	0.387	0.464	0.295
$\hat{\pi}=0.014$	$\tilde{\delta}$	0.043	0.025	0.019	0.014	0.011	0.019

Appendix

Table 21: Training data: Maximal value of performance measure, optimal cutoff

\tilde{\delta}

, TPR

\hat{\pi}_{1|1}

and TNR

\hat{\pi}_{0|0}

of classifiers that empirically maximize the robust F-score with

c=0

and different choices of

(d_{0},d_{1})

used in the data example. Upper panel: Logistic regression. Middle panel: Random forests. Lower panel: Gradient boosting.

	$d$	$0$	$0.01$	$0.05$	$0.1$	$0.5$	$1.0$
training	value	0.264	0.262	0.262	0.268	0.291	0.300
$\hat{\pi}=0.067$	$\tilde{\delta}$	0.135	0.127	0.123	0.104	0.088	0.088
	$\hat{\pi}_{1\|1}$	0.286	0.303	0.314	0.385	0.473	0.473
	$\hat{\pi}_{0\|0}$	0.959	0.954	0.950	0.911	0.839	0.839
subset.train	value	0.148	0.171	0.217	0.241	0.286	0.301
$\hat{\pi}=0.014$	$\tilde{\delta}$	0.036	0.029	0.025	0.025	0.019	0.019

Appendix

Table 22: Training data: Maximal value of performance measure, optimal cutoff

\tilde{\delta}

, TPR

\hat{\pi}_{1|1}

and TNR

\hat{\pi}_{0|0}

of classifiers that empirically maximize the robust MCC with different choices of

d

used in the data example. Upper panel: Logistic regression. Middle panel: Random forests. Lower panel: Gradient boosting.

Figure 16: Empirical true positive rates

\hat{\pi}_{1|1}

of classifiers that maximize the respective performance measures, shown for logistic regression, random forest, and gradient boosting. The left block displays classical metrics (JAC, MCC,

F_{1.5}

F_{0.5}

, Kappa), while the middle and right blocks correspond to the robust

F

-score and robust MCC with varying parameter choices. Solid lines represent the full dataset (

\hat{\pi}=0.067

), dashed lines the imbalanced subset (

\hat{\pi}=0.014

Figure 17: Empirical true negative rates

\hat{\pi}_{0|0}

corresponding to the classifiers in Figure 16 .

	Logistic regression	Regression forest	XGBoost
Model fitting	0.33	64.81	1.73
Prediction on training data	–	–	0.19
Threshold search on training data	51.38	52.80	49.71
Prediction on test data	0	1.22	0.05
Evaluation on the test threshold grid	13.72	13.51	13.48

Appendix

Table 23: Elapsed computation times in seconds for the full credit-default data set.

Class-Imbalanced Learning

Class-imbalanced learning develops classifiers that recognize rare classes without sacrificing performance on common ones.

See all 260 papers

Imbalanced Classification under Capacity Constraints
May 5, 2026Daniel Fraiman, Ricardo FraimanClass-Imbalanced Learning Binary Classification
Departamento de Matem´atica y Ciencias, Universidad de San Andr´es, Buenos Aires, Argentina · CONICET, Argentina. · PEDECIBA, Matem´atica, Uruguay.
When Single-Dataset Conclusions Fail: A 45-Task Study of Threshold Tuning and Resampling for Imbalanced Classification
Aug 17, 2026Diyorbek MusaevClass-Imbalanced Learning Cross-Dataset Generalization
Queensland University of Technology, Brisbane, Australia
Correcting Performance Estimation Bias in Imbalanced Classification with Minority Subconcepts
Apr 28, 2026Taylor Maxson, Roberto Corizzo, Yaning Wu +2Class-Imbalanced Learning

May 5, 2026J/K move · Enter open · S save

Imbalanced Classification under Capacity Constraints

Daniel Fraiman, Ricardo Fraiman

Departamento de Matem´atica y Ciencias, Universidad de San Andr´es, Buenos Aires, Argentina · CONICET, Argentina. · PEDECIBA, Matem´atica, Uruguay.

Class-Imbalanced Learning Binary Classification Classification

Open paper AskPDF arXiv

Detecting observations from a minority class under severe class imbalance is a central challenge in applications such as fraud detection, medical screening, and industrial quality control. In these settings, each positive prediction triggers a costly follow-up action, an MRI scan, a transaction audit, whose execution is subject to real operational constraints. This paper proposes a formal classification framework under capacity constraints: given a user-defined bound limit

b

on the proportion of observations that can be labeled as belonging to the minority class, the goal is to find the classifier that maximizes sensitivity on that class. We characterize the optimal classifier under this constraint and establish its equivalence with the classical Bayes classifier under a reweighting of the prior probabilities. We also introduce a capacity-adjusted performance metric

M

that accounts for the effective detection rate when the capacity constraint is binding. The framework is implemented on top of standard learning methods, k-NN, SVM, random forests, and neural networks, and statistical consistency is established for each. We further show that these methods reduce to post-hoc thresholding when no hyperparameters are oriented toward the capacity-constrained objective, and introduce a capacity-aware support vector machine that exploits the constraint during training and achieves the strongest empirical performance. Experiments on the Taiwanese credit card default dataset confirm that capacity-constrained classifiers substantially outperform both classical approaches and SMOTE under high imbalance regimes. The framework extends naturally to multiclass settings and online environments.

Abstract

Class-Imbalanced Learning

Explore similar work

Imbalanced Classification under Capacity Constraints

When Single-Dataset Conclusions Fail: A 45-Task Study of Threshold Tuning and Resampling for Imbalanced Classification

Correcting Performance Estimation Bias in Imbalanced Classification with Minority Subconcepts