cs.AIAug 17, 2026

When Single-Dataset Conclusions Fail: A 45-Task Study of Threshold Tuning and Resampling for Imbalanced Classification

Authors: Diyorbek Musaev

Organizations: Queensland University of Technology, Brisbane, Australia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Class-imbalance handling is routinely evaluated on a single benchmark dataset, and the resulting conclusions are reported as if they were properties of the method. We show this practice is unsafe. On the public Kaggle credit-card fraud dataset, under a leakage-free nested cross-validation protocol in which the decision threshold is selected on a held-out inner validation fold, a plain Random Forest at the default 0.5 threshold attains F1 = 0.861 +/- 0.021, and threshold tuning yields it no benefit (delta-F1 = -0.002). Read alone, this supports an appealing conclusion: for a well-calibrated ensemble, imbalance handling is unnecessary. We then apply the identical protocol to 45 binary tasks spanning imbalance ratios from 1:1.5 to 1:178 (2,025 model fits, four model families). The conclusion reverses. Random Forest benefits most from threshold tuning across the suite (delta-F1 = +0.101 +/- 0.134), not least, while three other families replicate their fraud-dataset behaviour almost exactly. SMOTE likewise harms the fraud dataset but helps across the suite (mean delta-F1 = +0.076; 138 wins, 39 losses; Wilcoxon p = 2.7e-17). Two further results. Threshold-tuning benefit is non-monotonic in the imbalance ratio: near zero below 1:5, peaking at +0.120 in the 1:15-1:40 band, declining to +0.045 beyond 1:100 - explaining why the fraud dataset, at 1:577, is an unrepresentative place to study the question. And we reject an intuitive heuristic: validation-set calibration error does not predict tuning benefit (expected calibration error r = -0.087; Brier r = +0.137), so calibration diagnostics cannot tell a practitioner whether tuning is worthwhile. We release the protocol, the 45-task harness, and all per-run metrics.

Figures & tables

ModelStrategyF1AUPRCROC-AUCR@P ≥\geq 0.90Threshold
Random Forestplain0.861 ±\pm 0.0210.8470.9730.8190.500
Random Foresttuned threshold0.859 ±\pm 0.0270.8470.9730.8190.420 ±\pm 0.126
Gradient BoostingSMOTE + tuned0.842 ±\pm 0.0410.7920.9720.6330.984 ±\pm 0.006
Random ForestSMOTE + tuned0.838 ±\pm 0.0280.8370.9790.7760.730 ±\pm 0.040
MLPSMOTE + tuned0.820 ±\pm 0.0220.8230.9610.7400.984 ±\pm 0.009
Logistic Regressiontuned threshold0.767 ±\pm 0.0270.7540.9730.3420.124 ±\pm 0.076
Table 1
Model familyΔ\Delta F1, credit-card fraudΔ\Delta F1, 45-task suiteAgreement
Logistic Regression+0.045+0.044 ±\pm 0.087replicates
Histogram Gradient Boosting+0.057+0.057 ±\pm 0.091replicates
MLP+0.015+0.035 ±\pm 0.071replicates
Random Forest-0.002+0.101 ±\pm 0.134reverses
Table 2
StrategyMean Δ\Delta F1Median Δ\Delta F1WinsLossesWilcoxon p
Tuned threshold+0.0592+0.0260138314.1×10−204.1\times 10^{-20}
Class-balanced + tuned+0.0547+0.016295343.2×10−093.2\times 10^{-09}
SMOTE + tuned+0.0757+0.0260138392.7×10−172.7\times 10^{-17}
Table 3
Imbalance ratioMean Δ\Delta F1Stdn runs
< 1:5-0.00880.03948
1:5 – 1:15+0.04540.092192
1:15 – 1:40+0.11990.13184
1:40 – 1:100+0.09960.13172
> 1:100+0.04470.060144
Table 4
StatisticPearson rpSpearman ρ\rhop
Expected calibration error (equal-width)-0.0874.4×10−24.4\times 10^{-2}-0.0758.1×10−28.1\times 10^{-2}
Expected calibration error (equal-frequency)-0.0601.6×10−11.6\times 10^{-1}+0.0324.5×10−14.5\times 10^{-1}
Brier score+0.1371.5×10−31.5\times 10^{-3}+0.2551.9×10−91.9\times 10^{-9}
Table 5
More in this topic

Class-Imbalanced Learning

Class-imbalanced learning develops classifiers that recognize rare classes without sacrificing performance on common ones.

See all 257 papers

Explore similar work

CardsList
Jun 29, 2026

The Hidden Cost of Resampling: How Imbalance Correction Degrades Probability Calibration in Tree Ensembles

Resampling methods such as SMOTE and random under/over-sampling are standard tools for class-imbalanced classification, almost always evaluated by minority-class accuracy or F1. Prior work has established that undersampling degrades probability calibration by distorting the training prior [1]. We extend this lens to synthetic oversampling (SMOTE) and provide a practical, evidence-based guide to when calibration damage matters and how to fix it. Across five public datasets (imbalance ratio 1.9-70) and two ensemble models (random forest, gradient boosting), with ten seeds and paired statistics, we find: (1) SMOTE's calibration cost is real but small (ECE +0.009; Cliff's delta = +0.27, small-to-moderate) across the studied imbalance range (IR 1.9-70) and its discrimination gains typically outweigh the calibration penalty; (2) random undersampling is the genuine danger -- its damage grows sharply with imbalance, inflating ECE from 0.008 to 0.395 on a dataset with ratio 70, largely because the resulting training sets are too small to estimate probabilities reliably; (3) a single post-hoc recalibration step (Platt or isotonic) eliminates the damage, reducing ECE by up to 66% at a negligible ranking-power cost (AUC -0.002, Cliff's delta = -0.07); and (4) the analytic prior-shift correction that repairs undersampling does not transfer to SMOTE, because SMOTE distorts the class-conditional density rather than only the prior -- so data-driven recalibration remains necessary. We recommend that imbalanced-learning studies report calibration alongside discrimination, and that practitioners recalibrate after resampling whenever predicted probabilities drive decisions.

Zewen Liu

Qilu Institute of Technology · Tai’an, Shandong, China

Class-Imbalanced LearningPost-Hoc CalibrationProbability Calibration
AskPDF
May 5, 2026

Imbalanced Classification under Capacity Constraints

Detecting observations from a minority class under severe class imbalance is a central challenge in applications such as fraud detection, medical screening, and industrial quality control. In these settings, each positive prediction triggers a costly follow-up action, an MRI scan, a transaction audit, whose execution is subject to real operational constraints. This paper proposes a formal classification framework under capacity constraints: given a user-defined bound limit bb on the proportion of observations that can be labeled as belonging to the minority class, the goal is to find the classifier that maximizes sensitivity on that class. We characterize the optimal classifier under this constraint and establish its equivalence with the classical Bayes classifier under a reweighting of the prior probabilities. We also introduce a capacity-adjusted performance metric MM that accounts for the effective detection rate when the capacity constraint is binding. The framework is implemented on top of standard learning methods, k-NN, SVM, random forests, and neural networks, and statistical consistency is established for each. We further show that these methods reduce to post-hoc thresholding when no hyperparameters are oriented toward the capacity-constrained objective, and introduce a capacity-aware support vector machine that exploits the constraint during training and achieves the strongest empirical performance. Experiments on the Taiwanese credit card default dataset confirm that capacity-constrained classifiers substantially outperform both classical approaches and SMOTE under high imbalance regimes. The framework extends naturally to multiclass settings and online environments.

Daniel Fraiman, Ricardo Fraiman

Departamento de Matem´atica y Ciencias, Universidad de San Andr´es, Buenos Aires, Argentina · CONICET, Argentina. · PEDECIBA, Matem´atica, Uruguay.

Class-Imbalanced LearningBinary ClassificationClassification
AskPDF
Jul 10, 2026

RUBRIC: Realism--Utility Balanced Ranking for Imbalanced Classification

Class imbalance poses a fundamental challenge in risk-sensitive applications such as fraud detection and medical diagnosis, where minority-class samples are scarce yet critical for accurate classification. Existing oversampling methods generate synthetic samples to rebalance class distributions; however, they often produce large numbers of low-quality candidates that distort decision boundaries or introduce artifacts, leading to overfitting and degraded generalization. In this work, we introduce RUBRIC, a generator-agnostic filtering framework that formulates synthetic sample selection as a quality-over-quantity optimization problem. RUBRIC ranks candidates using a realism-utility trade-off: realism is estimated via a neural density-ratio discriminator from each candidate's resemblance to real minority samples, while utility captures proximity to the decision boundary through a concave, margin-based scoring function g−(t)=−log⁡(1+e−t/τ)g_-(t)=-\log(1+e^{-t/τ}). The discriminator uses the same architecture and training protocol on every benchmark and is fit independently to that dataset's real minority class versus its synthetic pool. We show that, under mild regularity conditions, the proposed filtering framework monotonically tightens the generalization bound for margin-based classifiers by jointly reducing distribution shift and suppressing near-negative tail contributions. Through extensive experiments on standard public imbalanced-classification benchmarks, we demonstrate that RUBRIC boosts minority-class recall while preserving overall discriminative ability across multiple data generators. Sensitivity analyses in λλ and the selection budget KK further characterize performance trade-offs oriented toward ranking quality.

Yanxuan Yu, Dong Liu, Shu Wang +7

Columbia University · University of California, Los Angeles · University of Illinois Urbana-Champaign +3

Class-Imbalanced LearningSynthetic Data AugmentationSynthetic Data Selection
AskPDF