When Single-Dataset Conclusions Fail: A 45-Task Study of Threshold Tuning and Resampling for Imbalanced Classification

Diyorbek Musaev

cs.AIAug 17, 2026

When Single-Dataset Conclusions Fail: A 45-Task Study of Threshold Tuning and Resampling for Imbalanced Classification

Authors: Diyorbek Musaev

Organizations: Queensland University of Technology, Brisbane, Australia

Abstract

Class-imbalance handling is routinely evaluated on a single benchmark dataset, and the resulting conclusions are reported as if they were properties of the method. We show this practice is unsafe. On the public Kaggle credit-card fraud dataset, under a leakage-free nested cross-validation protocol in which the decision threshold is selected on a held-out inner validation fold, a plain Random Forest at the default 0.5 threshold attains F1 = 0.861 +/- 0.021, and threshold tuning yields it no benefit (delta-F1 = -0.002). Read alone, this supports an appealing conclusion: for a well-calibrated ensemble, imbalance handling is unnecessary. We then apply the identical protocol to 45 binary tasks spanning imbalance ratios from 1:1.5 to 1:178 (2,025 model fits, four model families). The conclusion reverses. Random Forest benefits most from threshold tuning across the suite (delta-F1 = +0.101 +/- 0.134), not least, while three other families replicate their fraud-dataset behaviour almost exactly. SMOTE likewise harms the fraud dataset but helps across the suite (mean delta-F1 = +0.076; 138 wins, 39 losses; Wilcoxon p = 2.7e-17). Two further results. Threshold-tuning benefit is non-monotonic in the imbalance ratio: near zero below 1:5, peaking at +0.120 in the 1:15-1:40 band, declining to +0.045 beyond 1:100 - explaining why the fraud dataset, at 1:577, is an unrepresentative place to study the question. And we reject an intuitive heuristic: validation-set calibration error does not predict tuning benefit (expected calibration error r = -0.087; Brier r = +0.137), so calibration diagnostics cannot tell a practitioner whether tuning is worthwhile. We release the protocol, the 45-task harness, and all per-run metrics.

Figures & tables

Model	Strategy	F1	AUPRC	ROC-AUC	R@P $\geq$ 0.90	Threshold
Random Forest	plain	0.861 $\pm$ 0.021	0.847	0.973	0.819	0.500
Random Forest	tuned threshold	0.859 $\pm$ 0.027	0.847	0.973	0.819	0.420 $\pm$ 0.126
Gradient Boosting	SMOTE + tuned	0.842 $\pm$ 0.041	0.792	0.972	0.633	0.984 $\pm$ 0.006
Random Forest	SMOTE + tuned	0.838 $\pm$ 0.028	0.837	0.979	0.776	0.730 $\pm$ 0.040
MLP	SMOTE + tuned	0.820 $\pm$ 0.022	0.823	0.961	0.740	0.984 $\pm$ 0.009
Logistic Regression	tuned threshold	0.767 $\pm$ 0.027	0.754	0.973	0.342	0.124 $\pm$ 0.076

Table 1

Model family	$\Delta$ F1, credit-card fraud	$\Delta$ F1, 45-task suite	Agreement
Logistic Regression	+0.045	+0.044 $\pm$ 0.087	replicates
Histogram Gradient Boosting	+0.057	+0.057 $\pm$ 0.091	replicates
MLP	+0.015	+0.035 $\pm$ 0.071	replicates
Random Forest	-0.002	+0.101 $\pm$ 0.134	reverses

Table 2

Strategy	Mean $\Delta$ F1	Median $\Delta$ F1	Wins	Losses	Wilcoxon p
Tuned threshold	+0.0592	+0.0260	138	31	$4.1\times 10^{-20}$
Class-balanced + tuned	+0.0547	+0.0162	95	34	$3.2\times 10^{-09}$
SMOTE + tuned	+0.0757	+0.0260	138	39	$2.7\times 10^{-17}$

Table 3

Imbalance ratio	Mean $\Delta$ F1	Std	n runs
< 1:5	-0.0088	0.039	48
1:5 – 1:15	+0.0454	0.092	192
1:15 – 1:40	+0.1199	0.131	84
1:40 – 1:100	+0.0996	0.131	72
> 1:100	+0.0447	0.060	144

Table 4

Statistic	Pearson r	p	Spearman $\rho$	p
Expected calibration error (equal-width)	-0.087	$4.4\times 10^{-2}$	-0.075	$8.1\times 10^{-2}$
Expected calibration error (equal-frequency)	-0.060	$1.6\times 10^{-1}$	+0.032	$4.5\times 10^{-1}$
Brier score	+0.137	$1.5\times 10^{-3}$	+0.255	$1.9\times 10^{-9}$

Table 5

Class-Imbalanced Learning

Class-imbalanced learning develops classifiers that recognize rare classes without sacrificing performance on common ones.

See all 257 papers

The Hidden Cost of Resampling: How Imbalance Correction Degrades Probability Calibration in Tree Ensembles
Jun 29, 2026Zewen LiuClass-Imbalanced Learning Post-Hoc Calibration
Qilu Institute of Technology · Tai’an, Shandong, China
Imbalanced Classification under Capacity Constraints
May 5, 2026Daniel Fraiman, Ricardo FraimanClass-Imbalanced Learning Binary Classification
Departamento de Matem´atica y Ciencias, Universidad de San Andr´es, Buenos Aires, Argentina · CONICET, Argentina. · PEDECIBA, Matem´atica, Uruguay.
RUBRIC: Realism--Utility Balanced Ranking for Imbalanced Classification
Jul 10, 2026Yanxuan Yu, Dong Liu, Shu Wang +7Class-Imbalanced Learning Synthetic Data Augmentation
Columbia University · University of California, Los Angeles · University of Illinois Urbana-Champaign +3

Jun 29, 2026J/K move · Enter open · S save

The Hidden Cost of Resampling: How Imbalance Correction Degrades Probability Calibration in Tree Ensembles

Zewen Liu

Qilu Institute of Technology · Tai’an, Shandong, China

Class-Imbalanced Learning Post-Hoc Calibration Probability Calibration

Open paper AskPDF arXiv

Resampling methods such as SMOTE and random under/over-sampling are standard tools for class-imbalanced classification, almost always evaluated by minority-class accuracy or F1. Prior work has established that undersampling degrades probability calibration by distorting the training prior [1]. We extend this lens to synthetic oversampling (SMOTE) and provide a practical, evidence-based guide to when calibration damage matters and how to fix it. Across five public datasets (imbalance ratio 1.9-70) and two ensemble models (random forest, gradient boosting), with ten seeds and paired statistics, we find: (1) SMOTE's calibration cost is real but small (ECE +0.009; Cliff's delta = +0.27, small-to-moderate) across the studied imbalance range (IR 1.9-70) and its discrimination gains typically outweigh the calibration penalty; (2) random undersampling is the genuine danger -- its damage grows sharply with imbalance, inflating ECE from 0.008 to 0.395 on a dataset with ratio 70, largely because the resulting training sets are too small to estimate probabilities reliably; (3) a single post-hoc recalibration step (Platt or isotonic) eliminates the damage, reducing ECE by up to 66% at a negligible ranking-power cost (AUC -0.002, Cliff's delta = -0.07); and (4) the analytic prior-shift correction that repairs undersampling does not transfer to SMOTE, because SMOTE distorts the class-conditional density rather than only the prior -- so data-driven recalibration remains necessary. We recommend that imbalanced-learning studies report calibration alongside discrimination, and that practitioners recalibrate after resampling whenever predicted probabilities drive decisions.

Abstract

Class-Imbalanced Learning

Explore similar work

The Hidden Cost of Resampling: How Imbalance Correction Degrades Probability Calibration in Tree Ensembles

Imbalanced Classification under Capacity Constraints

RUBRIC: Realism--Utility Balanced Ranking for Imbalanced Classification