Winner's Curse Drives False Promises in Data-Driven Decisions

Hamsa Bastani; Osbert Bastani; Bryce McLaughlin

stat.MLFeb 9, 2026

Winner's Curse Drives False Promises in Data-Driven Decisions

Authors: Hamsa Bastani, Osbert Bastani, Bryce McLaughlin

Organizations: Wharton School · University of Pennsylvania

Abstract

A major challenge in data-driven decision-making is accurate policy evaluation---i.e., guaranteeing that a learned decision-making policy achieves the promised benefits. A popular strategy is model-based policy evaluation, which estimates a model from data to infer counterfactual outcomes. This strategy is known to produce unwarrantedly optimistic estimates of the true benefit due to the winner's curse. We searched the recent literature on data-driven decision-making, identifying a sample of 55 papers published in Management Science in the past decade; all but two relied on this flawed methodology. Several common justifications are provided: (1) the estimated models are accurate, stable, and well-calibrated, (2) the historical data uses random treatment assignment, (3) the model family is well-specified, and (4) the evaluation methodology uses sample splitting. Unfortunately, we show that no combination of these justifications avoids the winner's curse. First, we provide a theoretical analysis demonstrating that the winner's curse can cause large, spurious reported benefits even when all these justifications hold. Second, we perform a simulation study based on the recent and consequential data-driven refugee matching problem. We construct a synthetic refugee matching environment (calibrated to closely match the real setting) but designed so that no assignment policy can improve expected employment compared to random assignment. Model-based methods report large, stable gains of around 60% even when the true effect is zero; these gains are on par with improvements of 22-75% reported in the literature. Our results provide strong evidence against model-based evaluation.

Figures & tables

Search Criteria	Count
(1) Initial Search Results (Keyword match, last 10 years)	875
(2) Potentially Relevant (AI Screened via GPT-5.2)	111
(3) Confirmed Estimate-Then-Optimize (Manual Review)	55
(4) Susceptible to Winner’s Curse (Model-Based Evaluation)	53

Table 1: Survey of Estimate-Then-Optimize Papers in Management Science

	Count (%)
Model-based evaluation
Optim & eval use same model family	53 (100%)
Optim & eval use exact same model	35 (66%)
Corrections made for winner’s curse
Robust optimization	3 (6%)
De-biasing	2 (4%)

Table 2: Survey of model-based counterfactual practices in estimate-then-optimize papers

Figure 1: Illustration of stylized example. The

x

-axis is the treatment

t

, the

y

-axis is the performance outcome

y

, the solid black line is

f^{*}(t)

, the dashed black line is

\hat{f}(t)

, the green star denotes

f^{*}(t^{*})

t^{*}=\operatorname*{\arg\max}_{t\in[0,1]}f^{*}(t)

, the red circle denotes

\hat{f}(\hat{t})

\hat{t}=\operatorname*{\arg\max}_{t\in[0,1]}\hat{f}(t)

, and the red line denotes the optimistic bias

\hat{f}(\hat{t})-f^{*}(\hat{t})

due to the winner’s curse.

Panel of Figure 2 — Figure 2: Histograms comparing 250 evaluations according to the prediction model used to optimize the policy with IPW estimates. All prediction models exhibit biasess due to the winner’s curse while IPW estimation suffers from excessive variance. All evaluations are reported in percent change in employment rate relative to the observed employment rate in the testing dataset.

Panel of Figure 3 — Figure 3: Histograms comparing 250 evaluations according to prediction models built off bootstrapped samples of the training dataset with IPW estimates and direct evaluation by the prediction model which optimized assignments. The stability of the LASSO estimator leads the bootstrapped samples to exhibit a larger winner’s curse than the original prediction model while GBM exhibits a reduced, but still quite large, winner’s curse. Honest Random Forests were able to eliminate most of their winner’s curse bias in this simulation, although in alternate setups we have observed sizable inaccuracies (both positive and negative) indicating the bootstrapping approach cannot generate trustworthy confidence intervals. All evaluations are reported in percent change in employment rate relative to the observed employment rate in the testing dataset.

Covariate	Probability
$\text{Age}\sim\mathrm{Unif}(18,30)$	0.44
$\text{Age}\sim\mathrm{Unif}(30,40)$	0.28
$\text{Age}\sim\mathrm{Unif}(40,50)$	0.16
$\text{Age}\sim\mathrm{Unif}(50,60)$	0.12
Gender = Male	0.53
Education = None	0.18

Table 3: Probability of covariates being assigned (independently) to each refugee, following the marginals reported in the Supplemental Material of Bansak et al. [2018] . For binary covariates only one option is reported.

Figure 4: Probability of each location being assigned to each refugee sample, sorted in ascending order. This distribution follows the empirical distribution reported in Bansak et al. [2018] .

Figure 5: The distribution of employment rates across location have been tuned to resemble those in Bansak et al. [2018] to maximize similarities between the simulation environment and real world data.

Panel of Figure 6 — Figure 6: Calibration curves for each prediction model.

Panel of Figure 7 — Figure 7: ROC curves detailing the performance of each prediction model on the test dataset.

Policy Evaluation

Policy evaluation estimates the outcomes, value, risks, and fairness of decision rules before or after deployment.

See all 56 papers

Dissecting the Impact of Model Misspecification in Data-driven Optimization
Mar 1, 2025Adam N. Elmachtoub, Henry Lam, Haixiang Lan +1Model Misspecification Decision-Focused Learning
Columbia University · Morgan Stanley
Robustness of Refugee-Matching Gains to Off-Policy Evaluation Choices
Apr 25, 2026Kirk Bansak, Elisabeth Paulson, Dominik Rothenhäusler +3Counterfactual Evaluation Causal Effect Estimation
Immigration Policy Lab, Stanford University · Department of Political Science, University of California, Berkeley · Technology and Operations Management Unit, Harvard Business School +3
The Challenger: When Do New Data Sources Justify Switching Machine Learning Models?
Dec 20, 2025Vassilis Digalakis, Christophe Pérignon, Sébastien Saurin +1Sequential Decision Making Learning Curves
Boston University · HEC Paris

Mar 1, 2025J/K move · Enter open · S save

Dissecting the Impact of Model Misspecification in Data-driven Optimization

Adam N. Elmachtoub, Henry Lam, Haixiang Lan, Haofeng Zhang

Columbia University · Morgan Stanley

Model Misspecification Decision-Focused Learning Statistical Learning Theory

Open paper AskPDF arXiv

Data-driven optimization aims to translate a machine learning model into decision-making by optimizing decisions on estimated costs. Such a pipeline can be conducted by fitting a distributional model which is then plugged into the target optimization problem. While this fitting can utilize traditional methods such as maximum likelihood, a more recent approach uses estimation-optimization integration that minimizes decision error instead of estimation error. Although intuitive, the statistical benefit of the latter approach is not well understood yet is important to guide the prescriptive usage of machine learning. In this paper, we dissect the performance comparisons between these approaches in terms of the amount of model misspecification. In particular, we show how the integrated approach offers a ``universal double benefit'' on the top two dominating terms of regret when the underlying model is misspecified, while the traditional approach can be advantageous when the model is nearly well-specified. Our comparison is powered by finite-sample tail regret bounds that are derived via new higher-order expansions of regrets and the leveraging of a recent Berry-Esseen theorem.

Abstract

Policy Evaluation

Explore similar work

Dissecting the Impact of Model Misspecification in Data-driven Optimization

Robustness of Refugee-Matching Gains to Off-Policy Evaluation Choices

The Challenger: When Do New Data Sources Justify Switching Machine Learning Models?