Evaluating Uplift Modeling under Structural Biases: Insights into Metric Stability and Model Robustness
Authors: Yuxuan Yang, Dugang Liu, Yiyan Huang
Organizations: College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, Guangdong, China · School of Computing and Information Technology, Great Bay University, Dongguan, Guangdong, China
In personalized marketing, uplift models estimate the incremental effect of an intervention by modeling how customer behavior would change under alternative treatments using counterfactual analysis. However, real-world marketing data often exhibit various biases, such as selection bias, spillover effects, measurement error, and unobserved confounding. These biases can adversely affect both the accuracy of uplift estimation and the validity of evaluation metrics. Despite the importance of bias-aware assessment, there remains a lack of systematic studies evaluating how different models and metrics perform under such biased conditions. To bridge this gap, we design a systematic benchmarking framework. Unlike standard predictive tasks, real-world uplift datasets inherently lack counterfactual ground truth. This limitation renders the direct validation of evaluation metrics infeasible and prevents the precise quantification of biases. Therefore, a semi-synthetic approach serves as a critical enabler for systematic benchmarking. This approach effectively bridges the gap by retaining real-world feature dependencies while providing the ground truth needed to isolate structural biases. Our investigations reveal that (i) uplift targeting and prediction can manifest as distinct objectives, where proficiency in one does not ensure efficacy in the other; (ii) while many models exhibit inconsistent performance under diverse biases, TARNet shows notable robustness, providing insights for subsequent model design; (iii) the stability of evaluation metrics is linked to their mathematical alignment with the ATE, suggesting that ATE-approximating metrics yield more consistent model rankings under structural data imperfections. These findings suggest the need for more robust uplift models and evaluation metrics under real-world data imperfections.
Figures & tables
Model
Parameter
Range
Explanations
LR
C
(0.01,10)
Regularization strength
XGBoost
n_estimators
(3,10)
Number of trees
max_depth
(3,10)
Max depth
learning_rate
(0.01,0.3)
Learning rate
subsample
(0.6,1)
Sampling ratio
colsample_bytree
(0.6,1)
Sampling feature ratio
Table 1. The search space of hyperparameters. LR and XGBoost follow the scikit-learn implementations. TARNet and Dragonnet follow the implementations in ( shalit2017estimating ; shi2019adapting ) .
Model/Setting
Setting A (varying ξ )
Setting B (varying θ0 )
Setting C (varying ω )
Setting D (varying m )
0.8
1.6
2.4
0.4
0.5
0.6
1.2
2.4
3.6
0.1
0.3
0.5
S-learner
2.6860
2.6162
2.6905
2.6694
3.1238
3.6429
2.6694
2.6394
2.6551
2.6694
2.8040
2.9545
T-learner
2.7296
2.7995
2.8449
2.7604
3.2095
3.6946
2.7604
2.7251
2.7434
2.7604
2.9870
3.0597
X-learner
2.7712
2.8247
2.8632
2.7492
3.2435
3.7478
2.7492
2.7512
2.7441
2.7492
2.9827
3.0822
R-learner
2.6032
2.5219
2.4124
2.6417
3.1016
3.5165
2.6417
2.6130
2.6032
2.6417
2.8712
3.0388
U-learner
2.5828
2.4473
2.3895
2.6549
3.0766
3.5295
2.6549
2.6231
2.6266
2.6549
2.8866
3.0466
Table 2. Comparison of model performance (measured by PEHE 30% averaged over 10 runs) under various settings (Table omits the corresponding values of θ1∈{0.8,0.95,1.1} in Setting B). Lower values indicate better performance.
Model/Setting
Setting A (varying ξ )
Setting B (varying θ0 )
Setting C (varying ω )
Setting D (varying m )
0.8
1.6
2.4
0.4
0.5
0.6
1.2
2.4
3.6
0.1
0.3
0.5
S-learner
2.7054
2.6744
2.6295
2.8102
2.8640
3.0842
2.8102
2.8134
2.8096
2.8102
2.7056
2.1225
T-learner
2.8102
2.7429
2.6856
2.8841
3.0120
3.1529
2.8841
2.8746
2.8818
2.8841
2.7606
2.1834
X-learner
2.8181
2.7536
2.6836
2.8765
3.0297
3.1697
2.8765
2.8774
2.8716
2.8765
2.7596
2.1867
R-learner
2.7944
2.6808
2.6557
2.8499
2.9866
3.1343
2.8499
2.8541
2.8550
2.8499
2.7581
2.1871
U-learner
2.7968
2.6824
2.6765
2.8315
2.9834
3.1332
2.8315
2.8409
2.8480
2.8315
2.7567
2.1832
Table 3. Comparison of model performance (measured by ATE 30% averaged over 10 runs) under various settings (Table omits the corresponding values of θ1∈{0.8,0.95,1.1} in Setting B). Higher values indicate better performance.
Figure 1. Radar chart of model performance ranked by PEHE k across different bias settings. Outer points indicate higher ranks.
Figure 2. Radar chart of model performance ranked by ATE k across different bias settings. Outer points indicate higher ranks.
Figure 3. Radar chart of model performance ranked by AUUC k across different bias settings. Outer points indicate higher ranks.
Figure 4. Radar chart of model performance ranked by Qini k across different bias settings. Outer points indicate higher ranks.
Figure 5. Radar chart of model performance ranked by Uplift k across different bias settings. Outer points indicate higher ranks.
Metric/Setting
Setting A (varying ξ )
Setting B (varying θ0 )
Setting C (varying ω )
Setting D (varying m )
0.8
1.6
2.4
0.4
0.5
0.6
1.2
2.4
3.6
0.1
0.3
0.5
Uplift 10%
0.417
0.733
0.383
0.883
0.967
0.667
0.883
0.783
0.550
0.883
0.867
0.333
AUUC 10%
0.333
0.683
0.367
0.800
0.667
0.650
0.800
0.617
0.367
0.800
0.933
0.583
Qini 10%
0.550
0.917
0.467
0.700
0.400
0.567
0.700
0.700
0.600
0.700
0.083
0.467
Uplift 30%
0.950
0.850
0.483
1.000
0.950
0.983
1.000
0.867
0.983
1.000
0.533
0.917
AUUC 30%
0.950
0.917
0.783
0.900
0.933
0.867
0.900
0.817
0.950
0.900
0.800
0.800
Table 4. Comparison of rank correlation for different evaluation metrics (measured by Spearman rank correlation coefficient averaged over 10 runs) under various settings (Table omits the corresponding values of θ1∈{0.8,0.95,1.1} in Setting B). Higher values indicate better performance.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model/Setting
Setting A (varying ξ )
Setting B (varying θ0 )
Setting C (varying ω )
Setting D (varying m )
0.8
1.6
2.4
0.4
0.5
0.6
1.2
2.4
3.6
0.1
0.3
0.5
S-learner
2.2471
2.2272
2.1129
2.2202
2.2815
2.2588
2.2202
2.2202
2.2202
2.2202
2.4187
3.7783
T-learner
1.7829
1.8215
1.8108
1.8589
1.8772
1.8964
1.8589
1.8589
1.8589
1.8589
2.1192
3.4795
X-learner
1.8170
1.7917
1.8111
1.8467
1.8943
1.8845
1.8467
1.8468
1.8471
1.8467
2.1314
3.4734
R-learner
2.6312
2.4902
2.8622
2.7786
2.5291
2.6807
2.7786
2.7297
2.7483
2.7786
2.6756
3.5028
U-learner
2.9636
73.8624
9.2050
2.8027
3.0159
2.8829
2.8027
2.7270
2.7808
2.8027
2.6614
3.5086
Appendix
Table 5. Results on the Bank dataset: Comparison of model performance (measured by PEHE 30% averaged over 10 runs) under various settings (Table omits the corresponding values of θ1∈{0.8,0.95,1.1} in Setting B). Lower values indicate better performance.
Model/Setting
Setting A (varying ξ )
Setting B (varying θ0 )
Setting C (varying ω )
Setting D (varying m )
0.8
1.6
2.4
0.4
0.5
0.6
1.2
2.4
3.6
0.1
0.3
0.5
S-learner
3.0847
3.1177
3.1857
3.1153
3.1205
3.1582
3.1153
3.1153
3.1153
3.1153
2.9675
2.3754
T-learner
3.2149
3.2156
3.2218
3.2097
3.2299
3.2420
3.2097
3.2097
3.2097
3.2097
3.0827
2.5112
X-learner
3.2037
3.2138
3.2231
3.2325
3.2420
3.2589
3.2325
3.2331
3.2328
3.2325
3.0864
2.5099
R-learner
3.1319
3.1332
3.0265
2.8459
2.9489
2.9341
2.8459
2.8629
2.8611
2.8459
2.8831
2.5098
U-learner
3.0855
2.8689
2.5634
2.8195
2.9061
2.9042
2.8195
2.8202
2.8121
2.8195
2.8601
2.5020
Appendix
Table 6. Results on the Bank dataset: Comparison of model performance (measured by ATE 30% averaged over 10 runs) under various settings (Table omits the corresponding values of θ1∈{0.8,0.95,1.1} in Setting B). Higher values indicate better performance.
Uplift modeling (conditional-average-treatment-effect estimation) drives personalized targeting, yet published uplift benchmarks frequently disagree on which estimator performs best; we show the disagreement is substantially about metrics, not models. UpliftBench evaluates 12 uplift estimators under an outer-test-isolated, multi-objective protocol across seven dataset families; its two findings are identified where a reference objective exists -- F1 on the standard continuous benchmark (IHDP), F2 in a within-sample case study on Jobs. On that benchmark, Qini shows no detectable alignment with effect accuracy -- across all 100 IHDP realizations its mean rank correlation with effect accuracy is +0.07, 95% CI [-0.03, +0.16] -- while AUUC is consistently more aligned (paired prefix-mean-AUUC-over-Qini gap +0.49 [+0.40, +0.59]; the shipped cumulative-gain AUUC aligns better still, +0.73). On Jobs, ranking metrics are structurally insufficient for a sign-threshold policy because they discard the score level; empirically, within the released split-rotation analysis direct policy-risk selection yields lower benchmark regret than random model selection while Qini, AUUC, and uplift-at-k do not (14-15% regret). Calibrating the decision threshold removes 81% of the Qini-selection regret. Both findings are bounded, not universal: F1 is not detected on either validation family (the ACIC and Revenue-Synthetic gaps are both indistinguishable from zero), and F2 vanishes under a budgeted-value objective where rank suffices. UpliftBench releases versioned loaders, fixed protocols, result artifacts, and a reproducible living leaderboard; the public repository accompanies the paper.
Estimating Conditional Average Treatment Effects (CATE) at the individual level is central to precision marketing, yet systematic benchmarking of uplift modeling methods at industrial scale remains limited. We present UpliftBench, an empirical evaluation of four CATE estimators: S-Learner, T-Learner, X-Learner (all with LightGBM base learners), and Causal Forest (EconML), applied to the Criteo Uplift v2.1 dataset comprising 13.98 million customer records. The near-random treatment assignment (propensity AUC = 0.509) provides strong internal validity for causal estimation. Evaluated via Qini coefficient and cumulative gain curves, the S-Learner achieves the highest Qini score of 0.376, with the top 20% of customers ranked by predicted CATE capturing 77.7% of all incremental conversions, a 3.9x improvement over random targeting. SHAP analysis identifies f8 as the dominant heterogeneous treatment effect (HTE) driver among the 12 anonymized covariates. Causal Forest uncertainty quantification reveals that 1.9% of customers are confident persuadables (lower 95% CI > 0) and 0.1% are confident sleeping dogs (upper 95% CI < 0). Our results provide practitioners with evidence-based guidance on method selection for large-scale uplift modeling pipelines.
Uplift modeling, crucial for estimating individual treatment effects (ITE), faces dual challenges: flexibly leveraging inter-group similarity to enhance discriminative power and debiasing under unobserved confounding scenarios. In this paper, we propose the Cross-Head Attention Uplift Network (CHAUN) and Robust Adversarial Inverse Propensity Score (RA-IPS) method to address these limitations. CHAUN employs shared feature embeddings and cross-head attention mechanisms to dynamically integrate treatment-specific and control-specific representations, enhancing inter-group correlation modeling. Theoretically, we prove that access to the true propensity scores ensures ITE identifiability even with unobserved confounders. For practical scenarios lacking true propensity scores, RA-IPS adversarially optimizes propensity weights within constrained uncertainty sets to mitigate bias from unobserved variables. Experiments on public datasets (CRITEO-UPLIFT, LAZADA) and a production e-commerce dataset demonstrate CHAUN's superiority over state-of-the-art uplift models, achieving relative improvements of up to 25.6% in QINI scores. RA-IPS further enhances robustness, outperforming standard IPS by 5.4% under unobserved confounding. The results validate the effectiveness of our proposed methods in real-world causal inference tasks.
Haoran Zhang, Chuanpu Li, Yuxin Fu +4
Center for Applied Statistics and School of Statistics, Renmin University of China, Beijing, China · Alibaba Group, Beijing, China