Tabular Foundation Models vs Tuned Gradient Boosting for Telecom Churn: A Profit- and Calibration-Aware Evaluation
Organizations: Jahangirnagar University
Abstract
Tabular foundation models (TFMs) predict from a labelled context in a single forward pass. A recent benchmark reported that they decisively outperform tree ensembles for customer churn, but it compared them with untuned gradient-boosted trees (GBDTs) and scored only ranking metrics, while a retention campaign is judged in money. We re-evaluate TabPFN-2.5, TabPFN-3.5, TabPFN-3.5-Fast and TabICL v2 against Optuna-tuned GBDTs on five public telecom churn datasets over 10 seeded splits. We separate three questions: whether a model ranks churners well, whether that ranking earns more expected maximum profit (EMP), and whether its probabilities can drive a threshold policy. With a split-aware test and Holm correction over 20 pre-declared comparisons, TabPFN-3.5 earns significantly more EMP than the best tuned GBDT on Cell2Cell (1.9%) and Iranian (7.6%), and these two gains also hold against each of the three tuned GBDTs separately. On Orange it is ahead in all 10 splits (14.3%), but the gain is not significant after correction (Holm p = 0.155). The splits resample the same datasets, so these results do not stand for independent operators. TabICL v2, the earlier benchmark's leader, beats the best tuned GBDT only on Iranian, not all three of them, and underpredicts Orange churn by 2.1 points on a 7.3% base rate. On the datasets and splits where we tested them, a substantially larger tuning budget and a duplicate-aware split leave the main conclusions unchanged. Retuning the baseline cuts the earlier headline gap from 6.1 to 4.0 PR-AUC points without closing it. The licence of the TabPFN-3.5 weights we used (License v1.0), not scoring cost, is what stands between its gains and production use.
Figures & tables
| Study | Domain | TFMs | Tuned | Profit metric | Calibration | Tests |
|---|---|---|---|---|---|---|
| Manzoor et al. [ 18 ] | churn | no | yes | e-Profits, EMP | no | yes |
| Baesens et al. [ 8 ] | credit risk | yes | yes | no | yes | yes |
| Seyedzadeh & Karimi [ 30 ] | churn | yes | no | no | log loss only | no |
| This work | churn | yes | yes | EMP, e-Profits, policy | yes | yes |
| Dataset | Source | Rows | Feat. | Churn % | Dup. | e-Prof. | Role |
|---|---|---|---|---|---|---|---|
| Telco [ 5 ] | IBM sample data (fictional) | 7,003 | 19 | 26.4 | 40 | yes | main |
| Maven [ 3 ] | Maven Analytics (fictional) | 6,589 | 34 | 28.4 | 0 | yes | main |
| Cell2Cell [ 1 ] | US wireless carrier | 51,047 | 56 | 28.8 | 0 | yes | main |
| Iranian [ 6 ] | Iranian telecom operator | 2,836 | 15 | 15.6 | 314 | no | main |
| Orange [ 11 ] | Orange, KDD Cup 2009 | 50,000 | 202 | 7.3 | 0 | no | main |
| E-Commerce [ 2 ] | Kaggle, origin unstated | 5,074 | 18 | 16.6 | 556 | no | replication |
| Dataset | TFM | EMP | % | 95% CI (corrected) | Seeds | Holm | All | |
|---|---|---|---|---|---|---|---|---|
| Telco | TabICL v2 | 9/10 | 0.133 | 1.000 | 1.000 | |||
| TabPFN-2.5 | 8/10 | 0.240 | 1.000 | 1.000 | ||||
| TabPFN-3.5 | 9/10 | 0.078 | 1.000 | 1.000 | ||||
| TabPFN-3.5-Fast | 9/10 | 0.097 | 1.000 | 1.000 | ||||
| Maven | TabICL v2 | 6/10 | 0.589 | 1.000 | 1.000 | |||
| TabPFN-2.5 | 8/10 | 0.270 | 1.000 | 1.000 |
| EMP | e-Profits, top 20% | e-Profits, | ||||
| Dataset | top-1 changed | top-1 changed | top-1 changed | |||
| Telco | 7/10 | 10/10 | 10/10 | |||
| Maven | 7/10 | 5/10 | 7/10 | |||
| Cell2Cell | 0/10 | 1/10 | 1/10 | |||
| Iranian | 6/10 | – | – | – | – | |
| Orange | 1/10 | – | – | – | – | |
| ECE | Policy regret (%) | |||||||
|---|---|---|---|---|---|---|---|---|
| Dataset | Model | CITL | Brier | raw | iso. | raw | iso. | Platt |
| Telco | LightGBM | 0.1414 | 0.051 | 0.029 | 1.8 | 1.3 | 1.4 | |
| TabICL v2 | 0.1338 | 0.030 | 0.028 | 1.5 | 1.5 | 1.1 | ||
| TabPFN-2.5 | 0.1340 | 0.028 | 0.031 | 1.4 | 1.6 | 1.4 | ||
| TabPFN-3.5 | 0.1333 | 0.031 | 0.029 | 1.3 | 1.7 | 1.2 | ||
| TabPFN-3.5-Fast | 0.1333 | 0.029 | 0.030 | 1.5 | 1.5 | 1.2 | ||
| Dataset | Model | Context | AUC | PR-AUC | EMP | EMP | Seeds | CITL | |
|---|---|---|---|---|---|---|---|---|---|
| Orange | TabPFN-3.5 | full (natural rate) | 0.748 | 0.238 | 0.658 | 10/10 | 0.011 | ||
| Orange | TabPFN-3.5 | 1:2 | 0.739 | 0.219 | 0.611 | 5/5 | 0.201 | ||
| Orange | TabPFN-3.5 | 1:1 | 0.736 | 0.208 | 0.589 | 3/5 | 0.541 | ||
| Orange | TabICL v2 | full (natural rate) | 0.727 | 0.210 | 0.550 | 4/10 | 0.457 | ||
| Orange | TabICL v2 | 1:2 | 0.730 | 0.209 | 0.557 | 3/5 | 0.875 | ||
| Orange | TabICL v2 | 1:1 | 0.727 | 0.196 | 0.525 | 0/5 | 0.309 |
| (a) Tuning budget | |||||||
|---|---|---|---|---|---|---|---|
| Dataset | TFM | Ref. (25–40 trials) | EMP | Ref. (incl. 200 trials) | EMP | Seeds | |
| Cell2Cell | TabPFN-3.5 | LightGBM | LightGBM (200 trials) | 5/5 | 0.005 | ||
| Cell2Cell | TabICL v2 | LightGBM | LightGBM (200 trials) | 4/5 | 0.312 | ||
| Iranian | TabPFN-3.5 | LightGBM | LightGBM | 5/5 | 0.011 | ||
| Iranian | TabICL v2 | LightGBM | LightGBM | 5/5 | 0.020 | ||
| Orange | TabPFN-3.5 | CatBoost | CatBoost | 5/5 | 0.005 | ||
| Seyedzadeh & Karimi [ 30 ] | This work | |
|---|---|---|
| Splits | 5-fold stratified CV | 60/20/20 holdout, 5 seeds |
| Duplicates | not stated | removed (556 rows) |
| GBDT baseline | XGBoost, library defaults | tuned LightGBM, XGBoost, CatBoost; default XGBoost |
| TFMs | TabICL v1 and v2, TabPFN v2, TabDPT, Mitra, LimiX, ConTextTab | TabICL v2, TabPFN-2.5, 3.5, 3.5-Fast |
| Metrics | F1, ROC-AUC, PR-AUC, log loss | PR-AUC, EMP, with corrected -test |
| Model | PR-AUC | EMP | PR vs default | PR vs tuned | EMP |
| XGBoost (library default) | 0.906 (0.019) | 8.090 | – | – | – |
| XGBoost | 0.923 (0.019) | 8.119 | – | – | – |
| LightGBM | 0.922 (0.016) | 8.116 | – | – | – |
| CatBoost | 0.927 (0.017) | 8.119 | – | – | – |
| TabICL v2 | 0.967 (0.012) | 8.505 | |||
| TabPFN-2.5 | 0.966 (0.012) | 8.512 |
| Model | Telco | Cell2Cell | Orange |
|---|---|---|---|
| LightGBM (1,000 trees, CPU) | 0.8 | 0.5 | 0.8 |
| TabICL v2 | 2.5 | 18.8 | 89.2 |
| TabICL v2, KV cache | – | – | 16.7 |
| TabPFN-3.5 | 8.8 | 49.4 | 1,686 † |
| TabPFN-3.5, cached context | 8.1 | 30.5 | 44.9 |
| TabPFN-3.5-Fast | 2.7 | 12.4 | 25.9 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Software | tabpfn 9.2.0 (TabPFN-2.5, 3.5, 3.5-Fast); tabicl 2.2.0 (checkpoint 2026-02-12); lightgbm 4.6.0; xgboost 3.4.1; catboost 1.2.10; interpret 0.7.8; optuna 5.0.0; scikit-learn 1.6.1; torch 2.11.0 (CUDA 12.8) |
| Hardware | Kaggle: NVIDIA T4 (16 GB) for TFMs; CPU sessions for all other models |
| LightGBM | 40 trials (200 in Section 4.5 ); learning rate log; leaves log; min child samples log; subsample ; column sample ; L2 log |
| XGBoost | 40 trials; learning rate log; depth ; min child weight log; subsample ; column sample ; L2 log |
| CatBoost | 25 trials (200 in Section 4.5 ); learning rate log; depth ; L2 log; random strength log; bagging temperature |
| EBM, LR | defaults; LR with one-hot encoding (up to 30 categories), median imputation with missing indicators, scaling |
| Telco | Maven | Cell2Cell | Iranian | Orange | ||||||
| vs. XGB / LGBM | vs. CatB / XGB | vs. XGB / LGBM | vs. LGBM / LGBM | vs. CatB / LGBM | ||||||
| TFM | PR | EMP | PR | EMP | PR | EMP | PR | EMP | PR | EMP |
| TabICL v2 | ||||||||||
| TabPFN-2.5 | ||||||||||
| TabPFN-3.5 | ||||||||||
| TabPFN-3.5-Fast | ||||||||||
| Telco | Maven | Cell2Cell | Iranian | Orange | ||||||
| Model | PR-AUC | EMP | PR-AUC | EMP | PR-AUC | EMP | PR-AUC | EMP | PR-AUC | EMP |
| Logistic regression | 0.647 | 10.294 | 0.830 | 12.769 | 0.388 | 8.654 | 0.754 | 6.356 | 0.173 | 0.409 |
| EBM | 0.661 | 10.246 | 0.894 | 13.076 | 0.460 | 9.117 | 0.936 | 7.699 | 0.194 | 0.534 |
| LightGBM | 0.657 | 10.232 | 0.890 | 13.067 | 0.468 | 9.176 | 0.941 | 7.860 | 0.206 | 0.576 |
| XGBoost | 0.659 | 10.241 | 0.891 | 13.059 | 0.466 | 9.175 | 0.936 | 7.791 | 0.200 | 0.565 |
| CatBoost | 0.659 | 10.215 | 0.893 | 13.070 | 0.466 | 9.151 | 0.935 | 7.709 | 0.210 | 0.557 |
| Model | Telco | Maven | Cell2Cell | Iranian | Orange |
|---|---|---|---|---|---|
| Logistic regression | 0.845 | 0.920 | 0.620 | 0.932 | 0.702 |
| EBM | 0.846 | 0.944 | 0.682 | 0.983 | 0.728 |
| LightGBM | 0.844 | 0.942 | 0.685 | 0.987 | 0.734 |
| XGBoost | 0.846 | 0.942 | 0.685 | 0.985 | 0.733 |
| CatBoost | 0.845 | 0.943 | 0.684 | 0.985 | 0.728 |
| TabICL v2 | 0.849 | 0.945 | 0.687 | 0.997 | 0.727 |
Tabular Foundation Models
Tabular foundation models learn reusable representations or in-context prediction skills for structured data such as spreadsheets and databases.
See all 209 papers