The Hidden Costs of 99% Accuracy: A Trustworthiness Audit of the Telco Customer Churn Benchmark

Soumyadeep Roy

cs.LGSep 9, 2026

The Hidden Costs of 99% Accuracy: A Trustworthiness Audit of the Telco Customer Churn Benchmark

Authors: Soumyadeep Roy

Organizations: Department of Data Science, St. Xavier’s College (Autonomous), Kolkata, 700016, India.

Abstract

Customer churn prediction on the IBM Telco Customer Churn benchmark (n = 7,043) routinely reports test accuracies above 95%, with the most cited published study reporting 99.01%. We audit this benchmark for four trustworthiness failures invisible to the accuracy- and F1-centred reporting that dominates the literature. First, pre-split SMOTE inflates churn-class F1 by 13.1 percentage points across ten classifiers and fifteen seeds (Wilcoxon p < 10^-4 per classifier); the same leaky pipeline ordering paired with class weighting yields no inflation, isolating the effect to SMOTE's geometric construction. We measure the mechanism directly: approximately 36% of synthetic training points are nearest-neighbour interpolations of test-set instances. Second, the TotalCharges field is approximately determined by tenure multiplied by MonthlyCharges (R2 = 0.999); removing it changes accuracy by less than 0.2 percentage points, yet TreeSHAP ranks it ninth in mean absolute attribution - a pattern that materially corrupts SHAP-based interpretation. We propose an R2 > 0.95 pre-modelling diagnostic. Third, in a 15-seed calibration audit, isotonic regression is the strongest default; temperature scaling fails on class-weighted tree ensembles whose predicted-probability distribution is bimodal. Fourth, the cost-optimal decision threshold (under a 50 USD retention offer and 24-month CLV proxy) is approximately 5-10 times lower than the F1-optimal threshold, saving approximately 77,000 USD per 1,000 customers. We replicate F1 and F2 on Iranian Telecom Churn (within domain) and Bank Customer Churn (across domain): F1 generalises; F2 generalises only within telecom. We synthesise these findings into a four-component reporting checklist - pipeline disclosure, redundancy diagnostic, calibration audit, and cost-sensitive thresholds - and release a reproducible implementation.

Figures & tables

Diagnostic	Value	Note
Class balance
Total rows	7,043
Negatives (No churn)	5,174 (73.46%)
Positives (Churn)	1,869 (26.54%)
Imbalance ratio (neg:pos)	$1:2.77$	mild
Duplicate-row audit

Table 1: Diagnostic audit of the IBM Telco Customer Churn benchmark (

n=7{,}043

). Quirks flagged in the “Note” column are, to our knowledge, not mentioned in any high-accuracy published study on this benchmark.

Pipeline	Accuracy	$F_{1}$ (churn)	Recall	ROC-AUC
P-Clean (correct)	0.7669	0.5744	0.5993	0.8047
P-Drop (drop TC )	0.7651	0.5691	0.5920	0.8024
P-Leak (pre-split Smote )	0.8035	0.7055	0.7139	0.8585
P-Scale (scaler leakage only)	0.7669	0.5745	0.5994	0.8045
$\Delta$ ( P-Leak $-$ P-Clean )	$+3.66$ pp	$+13.11$ pp	$+11.46$ pp	$+5.38$ pp
$\Delta$ ( P-Drop $-$ P-Clean )	$-0.18$ pp	$-0.53$ pp	$-0.73$ pp	$-0.23$ pp

Table 2: Pipeline-aggregated mean test metrics on the IBM Telco Customer Churn benchmark (10 classifiers

\times

15 seeds

\times

2 imbalance strategies = 300 fits per pipeline).

\Delta

rows give the absolute difference of P-Leak and P-Drop relative to P-Clean , in percentage points. Accuracy and

F_{1}

are computed at the default threshold of

0.5

Classifier	P-Clean acc	P-Leak acc	Acc. inflation	F2 drop (acc)
LogReg	0.7514	0.7610	$+0.96$ pp	$+0.36$ pp
XGBoost	0.7689	0.8041	$+3.52$ pp	$-0.39$ pp
CatBoost	0.7745	0.8102	$+3.57$ pp	$-0.03$ pp
LightGBM	0.7717	0.8095	$+3.78$ pp	$+0.04$ pp
GBM	0.7937	0.8315	$+3.78$ pp	$\phantom{+}0.00$ pp
BalancedRF	0.7762	0.8146	$+3.84$ pp	$-0.41$ pp

Table 3: Per-classifier accuracy inflation on Telco Churn. Each cell is the mean over 15 seeds and 2 imbalance strategies (30 fits per cell). Acc. inflation is P-Leak accuracy minus P-Clean accuracy in percentage points; the F2 drop column is P-Drop minus P-Clean accuracy. Sorted by accuracy inflation.

Classifier	P-Clean $F_{1}$	P-Leak $F_{1}$	$\Delta F_{1}$ (pp)	Wilcoxon $p$
LogReg	$0.6288\pm 0.014$	$0.7800\pm 0.011$	$+15.13\pm 1.36$	$3.1\times 10^{-5}$
GBM	$0.6279\pm 0.016$	$0.8580\pm 0.008$	$+23.00\pm 1.69$	$3.1\times 10^{-5}$
CatBoost	$0.6013\pm 0.020$	$0.8572\pm 0.006$	$+25.60\pm 2.10$	$3.1\times 10^{-5}$
RF	$0.5970\pm 0.013$	$0.8554\pm 0.007$	$+25.85\pm 1.34$	$3.1\times 10^{-5}$
BalancedRF	$0.5966\pm 0.015$	$0.8553\pm 0.008$	$+25.87\pm 1.44$	$3.1\times 10^{-5}$
LightGBM	$0.5707\pm 0.022$	$0.8506\pm 0.006$	$+27.99\pm 2.29$	$3.1\times 10^{-5}$

Table 4: Per-classifier SMOTE-specific churn-class

F_{1}

inflation on Telco. Each row is the paired-by-seed mean

\pm

standard deviation across 15 seeds, restricted to the Smote imbalance strategy. The Wilcoxon signed-rank test (one-sided, alternative “ P-Leak

>

P-Clean ”) reaches its lower-bound

p=3.05\cdot 10^{-5}

on every classifier. Sorted by

\Delta F_{1}

Pipeline	Imbalance strategy	Accuracy	$F_{1}$ (churn)
P-Leak	Smote	0.8404	0.8393
P-Leak	Class weight ( Cw )	0.7666	0.5716
P-Clean	Smote	0.7678	0.5785
P-Clean	Class weight ( Cw )	0.7660	0.5703

Table 5: The SMOTE-versus-class-weight contrast on the leaky pipeline. Each cell is the mean over 10 classifiers and 15 seeds (150 fits per cell) at default threshold

0.5

Model	Raw	Platt	Isotonic	Temperature
GBM	0.0297 $\pm$ 0.0062	0.0460 $\pm$ 0.0091	0.0304 $\pm$ 0.0074	0.0300 $\pm$ 0.0059
RF	0.0358 $\pm$ 0.0077	0.0485 $\pm$ 0.0084	0.0352 $\pm$ 0.0109	0.0377 $\pm$ 0.0083
BalancedRF	0.1092 $\pm$ 0.0092	0.0363 $\pm$ 0.0066	0.0324 $\pm$ 0.0076	0.1127 $\pm$ 0.0092
ExtraTrees	0.0629 $\pm$ 0.0125	0.0544 $\pm$ 0.0110	0.0344 $\pm$ 0.0077	0.0861 $\pm$ 0.0183
CatBoost	0.1129 $\pm$ 0.0094	0.0398 $\pm$ 0.0064	0.0332 $\pm$ 0.0042	0.1160 $\pm$ 0.0084

Table 6: Multi-seed Expected Calibration Error (ECE, 15-bin equal-width) on the top five classifiers under four calibration methods, on P-Clean + Cw . Each cell is mean

\pm

standard deviation across 15 random seeds (seeds

0,\ldots,14

). Best per row in bold. ECE is lower-is-better.

Strategy	Threshold	Total cost ($)	Recall	Precision	$F_{1}$
Default	0.50	311,496	0.5053	0.6540	0.5701
$F_{1}$ -optimal	0.38	213,283	0.6791	0.5826	0.6272
Cost-optimal	0.04	59,300	0.9866	0.3531	0.5201

Table 7: Threshold-strategy comparison on the canonical P-Clean + Cw + Gbm +Platt model, seed = 42. Total cost is computed on the full test set (

n_{\text{test}}=1{,}409

customers).

Rank	Feature	Mean $\|$ SHAP $\|$	Direction
1	tenure	0.5733	high $\to$ no churn
2	Contract_Two year	0.4475	true $\to$ no churn
3	InternetService_Fiber optic	0.3952	true $\to$ churn
4	PaymentMethod_Electronic check	0.2136	true $\to$ churn
5	Contract_One year	0.1992	true $\to$ no churn
6	PaperlessBilling_Yes	0.1327	true $\to$ churn

Table 8: Top ten features by mean absolute SHAP value on the canonical P-Clean + Cw + Gbm model, seed = 42. “Direction” indicates whether high feature values push the prediction toward churn or away from it.

Benchmark Auditing

Benchmark auditing checks whether evaluation datasets, metrics, and infrastructure measure the capabilities and risks they claim to assess.

See all 93 papers

Data-Driven Telecom Marketing Optimization: A Machine Learning-Based Churn Prediction and Customer Segmentation Framework
Jul 11, 2026Nada Ali, Lina Ahmed, Tahani Abdalla Attia Gasmalla
Department of Electrical and Electronic Engineering University of Khartoum
CARRE: Counterfactual Action Retrieval and Reason Evaluation for Explainable Churn Prescription
Sep 9, 2026Minjoo Kim, Sangjin Park, Seung Hwan ChoCounterfactual Explanations
Department of Industrial Data Engineering Hanyang University Seoul, Republic of Korea
Customer Churn Prediction on Structured Data Using FT-Transformer and Stacking Ensembles
May 26, 2026Joyjit Roy, Samaresh Kumar Singh, Laxmi ShawTransformer Ensemble Learning
Independent Researcher, Austin, TX, USA · Independent Researcher, Leander, TX · Texas A & M University-Victoria, Victoria, TX

Jul 11, 2026J/K move · Enter open · S save

Data-Driven Telecom Marketing Optimization: A Machine Learning-Based Churn Prediction and Customer Segmentation Framework

Nada Ali, Lina Ahmed, Tahani Abdalla Attia Gasmalla

Department of Electrical and Electronic Engineering University of Khartoum

Open paper AskPDF arXiv

Customer churn is a major challenge for telecommunication companies, directly eroding revenue and long term customer relationships. Traditional retention programs rely on generic, not personalized incentives and lack the precision to identify high risk customers before they leave. This paper presents a data driven marketing optimization framework integrating machine learning based churn prediction, customer segmentation combining churn risk with customer value, and tailored, segment specific marketing and Return on Investment ROI strategies. Using the IBM Telco Customer Churn dataset with 7043 customers and 21 features, three gradient boosting ensembles, XGBoost, LightGBM, and CatBoost, were trained and tuned via randomized search with stratified 5 fold cross validation, class weighting, and F1 score driven decision threshold optimization to counter a class imbalance of 73.4% versus 26.6%. CatBoost was selected as the deployment model, achieving 77.68% accuracy, an F1 score of 0.6366, a PR AUC of 0.6553, and a ROC AUC of 0.8403 on the held out test set. Customers were partitioned with K Means clustering, validated via the Elbow method and visualized with Principal Component Analysis, into High, Medium, and Low Value segments, cross tabulated against churn risk labels to define four actionable clusters. Segment specific retention, upsell, and engagement strategies were designed for each cluster, and a theoretical ROI and CLV framework quantifies the financial impact of the proposed interventions. The pipeline was operationalized in an interactive Streamlit web application allowing marketing teams to upload data, filter by segment, visualize churn drivers via SHAP, and download automated segment reports. Results confirm that combining predictive churn modeling with value aware segmentation yields more actionable and profitable marketing decisions than churn prediction alone.

Abstract

Benchmark Auditing

Explore similar work

Data-Driven Telecom Marketing Optimization: A Machine Learning-Based Churn Prediction and Customer Segmentation Framework

CARRE: Counterfactual Action Retrieval and Reason Evaluation for Explainable Churn Prescription

Customer Churn Prediction on Structured Data Using FT-Transformer and Stacking Ensembles