Evaluating Uplift Modeling under Structural Biases: Insights into Metric Stability and Model Robustness

Yuxuan Yang; Dugang Liu; Yiyan Huang

cs.LGMar 21, 2026

Evaluating Uplift Modeling under Structural Biases: Insights into Metric Stability and Model Robustness

Authors: Yuxuan Yang, Dugang Liu, Yiyan Huang

Organizations: College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, Guangdong, China · School of Computing and Information Technology, Great Bay University, Dongguan, Guangdong, China

Abstract

In personalized marketing, uplift models estimate the incremental effect of an intervention by modeling how customer behavior would change under alternative treatments using counterfactual analysis. However, real-world marketing data often exhibit various biases, such as selection bias, spillover effects, measurement error, and unobserved confounding. These biases can adversely affect both the accuracy of uplift estimation and the validity of evaluation metrics. Despite the importance of bias-aware assessment, there remains a lack of systematic studies evaluating how different models and metrics perform under such biased conditions. To bridge this gap, we design a systematic benchmarking framework. Unlike standard predictive tasks, real-world uplift datasets inherently lack counterfactual ground truth. This limitation renders the direct validation of evaluation metrics infeasible and prevents the precise quantification of biases. Therefore, a semi-synthetic approach serves as a critical enabler for systematic benchmarking. This approach effectively bridges the gap by retaining real-world feature dependencies while providing the ground truth needed to isolate structural biases. Our investigations reveal that (i) uplift targeting and prediction can manifest as distinct objectives, where proficiency in one does not ensure efficacy in the other; (ii) while many models exhibit inconsistent performance under diverse biases, TARNet shows notable robustness, providing insights for subsequent model design; (iii) the stability of evaluation metrics is linked to their mathematical alignment with the ATE, suggesting that ATE-approximating metrics yield more consistent model rankings under structural data imperfections. These findings suggest the need for more robust uplift models and evaluation metrics under real-world data imperfections.

Figures & tables

Model	Parameter	Range	Explanations
LR	C	(0.01,10)	Regularization strength
XGBoost	n_estimators	(3,10)	Number of trees
	max_depth	(3,10)	Max depth
	learning_rate	(0.01,0.3)	Learning rate
	subsample	(0.6,1)	Sampling ratio
	colsample_bytree	(0.6,1)	Sampling feature ratio

Table 1. The search space of hyperparameters. LR and XGBoost follow the scikit-learn implementations. TARNet and Dragonnet follow the implementations in ( shalit2017estimating ; shi2019adapting ) .

Model/Setting	Setting A (varying $\xi$ )			Setting B (varying $\theta_{0}$ )			Setting C (varying $\omega$ )			Setting D (varying $m$ )
Model/Setting	0.8	1.6	2.4	0.4	0.5	0.6	1.2	2.4	3.6	0.1	0.3	0.5
S-learner	2.6860	2.6162	2.6905	2.6694	3.1238	3.6429	2.6694	2.6394	2.6551	2.6694	2.8040	2.9545
T-learner	2.7296	2.7995	2.8449	2.7604	3.2095	3.6946	2.7604	2.7251	2.7434	2.7604	2.9870	3.0597
X-learner	2.7712	2.8247	2.8632	2.7492	3.2435	3.7478	2.7492	2.7512	2.7441	2.7492	2.9827	3.0822
R-learner	2.6032	2.5219	2.4124	2.6417	3.1016	3.5165	2.6417	2.6130	2.6032	2.6417	2.8712	3.0388
U-learner	2.5828	2.4473	2.3895	2.6549	3.0766	3.5295	2.6549	2.6231	2.6266	2.6549	2.8866	3.0466

Table 2. Comparison of model performance (measured by PEHE 30% averaged over 10 runs) under various settings (Table omits the corresponding values of

\theta_{1}\in\{0.8,0.95,1.1\}

in Setting B). Lower values indicate better performance.

Model/Setting	Setting A (varying $\xi$ )			Setting B (varying $\theta_{0}$ )			Setting C (varying $\omega$ )			Setting D (varying $m$ )
Model/Setting	0.8	1.6	2.4	0.4	0.5	0.6	1.2	2.4	3.6	0.1	0.3	0.5
S-learner	2.7054	2.6744	2.6295	2.8102	2.8640	3.0842	2.8102	2.8134	2.8096	2.8102	2.7056	2.1225
T-learner	2.8102	2.7429	2.6856	2.8841	3.0120	3.1529	2.8841	2.8746	2.8818	2.8841	2.7606	2.1834
X-learner	2.8181	2.7536	2.6836	2.8765	3.0297	3.1697	2.8765	2.8774	2.8716	2.8765	2.7596	2.1867
R-learner	2.7944	2.6808	2.6557	2.8499	2.9866	3.1343	2.8499	2.8541	2.8550	2.8499	2.7581	2.1871
U-learner	2.7968	2.6824	2.6765	2.8315	2.9834	3.1332	2.8315	2.8409	2.8480	2.8315	2.7567	2.1832

Table 3. Comparison of model performance (measured by ATE 30% averaged over 10 runs) under various settings (Table omits the corresponding values of

\theta_{1}\in\{0.8,0.95,1.1\}

in Setting B). Higher values indicate better performance.

Figure 1. Radar chart of model performance ranked by PEHE k across different bias settings. Outer points indicate higher ranks.

Figure 2. Radar chart of model performance ranked by ATE k across different bias settings. Outer points indicate higher ranks.

Figure 3. Radar chart of model performance ranked by AUUC k across different bias settings. Outer points indicate higher ranks.

Figure 4. Radar chart of model performance ranked by Qini k across different bias settings. Outer points indicate higher ranks.

Figure 5. Radar chart of model performance ranked by Uplift k across different bias settings. Outer points indicate higher ranks.

Metric/Setting	Setting A (varying $\xi$ )			Setting B (varying $\theta_{0}$ )			Setting C (varying $\omega$ )			Setting D (varying $m$ )
Metric/Setting	0.8	1.6	2.4	0.4	0.5	0.6	1.2	2.4	3.6	0.1	0.3	0.5
Uplift 10%	0.417	0.733	0.383	0.883	0.967	0.667	0.883	0.783	0.550	0.883	0.867	0.333
AUUC 10%	0.333	0.683	0.367	0.800	0.667	0.650	0.800	0.617	0.367	0.800	0.933	0.583
Qini 10%	0.550	0.917	0.467	0.700	0.400	0.567	0.700	0.700	0.600	0.700	0.083	0.467
Uplift 30%	0.950	0.850	0.483	1.000	0.950	0.983	1.000	0.867	0.983	1.000	0.533	0.917
AUUC 30%	0.950	0.917	0.783	0.900	0.933	0.867	0.900	0.817	0.950	0.900	0.800	0.800

Table 4. Comparison of rank correlation for different evaluation metrics (measured by Spearman rank correlation coefficient averaged over 10 runs) under various settings (Table omits the corresponding values of

\theta_{1}\in\{0.8,0.95,1.1\}

in Setting B). Higher values indicate better performance.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Model/Setting	Setting A (varying $\xi$ )			Setting B (varying $\theta_{0}$ )			Setting C (varying $\omega$ )			Setting D (varying $m$ )
Model/Setting	0.8	1.6	2.4	0.4	0.5	0.6	1.2	2.4	3.6	0.1	0.3	0.5
S-learner	2.2471	2.2272	2.1129	2.2202	2.2815	2.2588	2.2202	2.2202	2.2202	2.2202	2.4187	3.7783
T-learner	1.7829	1.8215	1.8108	1.8589	1.8772	1.8964	1.8589	1.8589	1.8589	1.8589	2.1192	3.4795
X-learner	1.8170	1.7917	1.8111	1.8467	1.8943	1.8845	1.8467	1.8468	1.8471	1.8467	2.1314	3.4734
R-learner	2.6312	2.4902	2.8622	2.7786	2.5291	2.6807	2.7786	2.7297	2.7483	2.7786	2.6756	3.5028
U-learner	2.9636	73.8624	9.2050	2.8027	3.0159	2.8829	2.8027	2.7270	2.7808	2.8027	2.6614	3.5086

Appendix

Table 5. Results on the Bank dataset: Comparison of model performance (measured by PEHE 30% averaged over 10 runs) under various settings (Table omits the corresponding values of

\theta_{1}\in\{0.8,0.95,1.1\}

in Setting B). Lower values indicate better performance.

Model/Setting	Setting A (varying $\xi$ )			Setting B (varying $\theta_{0}$ )			Setting C (varying $\omega$ )			Setting D (varying $m$ )
Model/Setting	0.8	1.6	2.4	0.4	0.5	0.6	1.2	2.4	3.6	0.1	0.3	0.5
S-learner	3.0847	3.1177	3.1857	3.1153	3.1205	3.1582	3.1153	3.1153	3.1153	3.1153	2.9675	2.3754
T-learner	3.2149	3.2156	3.2218	3.2097	3.2299	3.2420	3.2097	3.2097	3.2097	3.2097	3.0827	2.5112
X-learner	3.2037	3.2138	3.2231	3.2325	3.2420	3.2589	3.2325	3.2331	3.2328	3.2325	3.0864	2.5099
R-learner	3.1319	3.1332	3.0265	2.8459	2.9489	2.9341	2.8459	2.8629	2.8611	2.8459	2.8831	2.5098
U-learner	3.0855	2.8689	2.5634	2.8195	2.9061	2.9042	2.8195	2.8202	2.8121	2.8195	2.8601	2.5020

Appendix

Table 6. Results on the Bank dataset: Comparison of model performance (measured by ATE 30% averaged over 10 runs) under various settings (Table omits the corresponding values of

\theta_{1}\in\{0.8,0.95,1.1\}

in Setting B). Higher values indicate better performance.

Uplift Modeling

See all 12 papers

UpliftBench: Revealing Outcome-Regime and Objective Mismatch in Uplift Evaluation
Aug 2, 2026Binshuang LiConditional Average Treatment Effect Estimation Benchmark Design
Independent Researcher USA
A Large-Scale Empirical Comparison of Meta-Learners and Causal Forests for Heterogeneous Treatment Effect Estimation in Marketing Uplift Modeling
Apr 7, 2026Aman SinghUplift Modeling Conditional Average Treatment Effect Estimation
Cross-Head Attention Uplift Network with Inverse Propensity Score under Unobserved Confounding
Jun 25, 2026Haoran Zhang, Chuanpu Li, Yuxin Fu +4Inverse Probability Weighting Causal Effect Estimation
Center for Applied Statistics and School of Statistics, Renmin University of China, Beijing, China · Alibaba Group, Beijing, China

Aug 2, 2026J/K move · Enter open · S save

UpliftBench: Revealing Outcome-Regime and Objective Mismatch in Uplift Evaluation

Binshuang Li

Independent Researcher USA

Conditional Average Treatment Effect Estimation Benchmark Design Policy Evaluation Uplift Modeling Benchmark Validity Individualized Treatment Effect Estimation

Open paper AskPDF arXiv

Uplift modeling (conditional-average-treatment-effect estimation) drives personalized targeting, yet published uplift benchmarks frequently disagree on which estimator performs best; we show the disagreement is substantially about metrics, not models. UpliftBench evaluates 12 uplift estimators under an outer-test-isolated, multi-objective protocol across seven dataset families; its two findings are identified where a reference objective exists -- F1 on the standard continuous benchmark (IHDP), F2 in a within-sample case study on Jobs. On that benchmark, Qini shows no detectable alignment with effect accuracy -- across all 100 IHDP realizations its mean rank correlation with effect accuracy is +0.07, 95% CI [-0.03, +0.16] -- while AUUC is consistently more aligned (paired prefix-mean-AUUC-over-Qini gap +0.49 [+0.40, +0.59]; the shipped cumulative-gain AUUC aligns better still, +0.73). On Jobs, ranking metrics are structurally insufficient for a sign-threshold policy because they discard the score level; empirically, within the released split-rotation analysis direct policy-risk selection yields lower benchmark regret than random model selection while Qini, AUUC, and uplift-at-

k

do not (14-15% regret). Calibrating the decision threshold removes 81% of the Qini-selection regret. Both findings are bounded, not universal: F1 is not detected on either validation family (the ACIC and Revenue-Synthetic gaps are both indistinguishable from zero), and F2 vanishes under a budgeted-value objective where rank suffices. UpliftBench releases versioned loaders, fixed protocols, result artifacts, and a reproducible living leaderboard; the public repository accompanies the paper.

Abstract

Uplift Modeling

Explore similar work

UpliftBench: Revealing Outcome-Regime and Objective Mismatch in Uplift Evaluation

A Large-Scale Empirical Comparison of Meta-Learners and Causal Forests for Heterogeneous Treatment Effect Estimation in Marketing Uplift Modeling

Cross-Head Attention Uplift Network with Inverse Propensity Score under Unobserved Confounding