In this paper, we develop an asymptotically normal estimator for the value of policy improvement (VPI) under adaptively collected data. This value, which is defined as the gap between the value of the best treatment policy in a class and that of a baseline policy, gauges whether or not it is worth deploying a personalized treatment policy. Building an estimator for the VPI is naturally more difficult than estimating the value of a fixed policy. This is because, in addition to contending with the correlated data and decaying propensities associated with adaptive experiments, any estimator must also handle the non-regularity, or lack of pathwise differentiability, of the underlying estimand when the optimal policy is non-unique. Our proposed estimator works by sequentially fitting a weighted empirical welfare maximizing policy on historical data and then evaluating it using inverse propensity weighting on future samples. We neither assume that the optimal policy is unique nor make a margin assumption on the underlying heterogeneous treatment effects. A key difficulty in analyzing our estimator is showing that the regret incurred through policy learning is asymptotically smaller than the standard error of the estimator. We accomplish this by showing faster regret rates for policy learning under decaying propensities, which may be of independent interest. We not only evaluate the coverage of our estimator on a mix of synthetic and semi-synthetic data, but also employ it to estimate the VPI in an adaptive experiment evaluating the efficacy of job assistance programs in Jordan due to Caria et al. Although our estimates are generally more conservative than their hierarchical Bayesian analysis, which assumes that the optimal policy is unique, we still corroborate the benefits of personalization amongst Syrian refugees.
Figures & tables
Work
Setting
Target
Result
Key assumptions
Luedtke and Van Der Laan (2016)
i.i.d. data
Unrestricted optimal value V∘
Asymptotically normal estimator
Propensity and regression nuisance convergence rates.
Zhan et al. (2021)
Contextual bandit data
Fixed-policy value V(π)
Asymptotically normal estimator
α<1/2
Bibaut et al. (2021a)
Contextual bandit data
Fixed-policy value V(π)
Asymptotically normal estimator
α≤1/2 ( α≤1/3 for variance convergence)
This paper
Contextual bandit data
Best in-class contrast V∗−V(q)
Asymptotically normal estimator and expected regret o(n(α−1)/2)
Finite disagreement VC dimension; α<1
Table 1: A comparison of our results with the most closely related papers. We let V∗ denote the value of an optimal policy within a class Π and V∘ the optimal value of any policy. An extended comparison is shown in Table 6 in Appendix A
Table 2: A summary of comparators and the design choices that distinguish them. CADR and StabVar are each run in two modes: a sublinear burn-in split consisting of bn=o(n) samples, and a 50/50 split ( bn=n/2 ).
Figure 1: Empirical coverage versus sample size in the synthetic DGP for all six estimators at logging exponents α∈{0,0.1,0.25,0.5} . Error bars are Wilson 95% intervals and the dashed line is the nominal 0.95 coverage target. Double-dipping ( DD , the lowest curve) under-covers and does not recover with n , while AW and the 50/50 estimators obtain nominal coverage except in the α=0.5 case.
Figure 2: The Pareto frontier of empirical coverage (x-axis) and confidence interval width (y-axis) in the synthetic DGP. We examine this frontier at n=15,000 samples and logging exponents α∈{0,0.1,0.25,0.5} . Estimators toward the lower right are better calibrated.
Figure 3: Empirical coverage versus sample size in the semi-synthetic DGP for all six estimators at logging exponents α∈{0,0.1,0.25} . Error bars are Wilson 95% intervals and the dashed line is the nominal 0.95 coverage target.
Figure 4: The Pareto frontier of empirical coverage (x-axis) and confidence interval width (y-axis) in the semi-synthetic DGP. Here, we consider n=20,000 samples and logging exponents α∈{0,0.1,0.25} .
Figure 5: A visualization of scaled regret against n under various logging exponents α . The shaded region gives a 95% pointwise confidence interval over M=100 independent runs. The synthetic panel leverages an exact solver, whereas the semi-synthetic one uses an approximate solution.
Treatment arm
Description
Control
Participants received a small cash infusion of 4 Jordanian dinars (approximately US $5.60) and a basic informational pamphlet.
Cash
Participants received a larger cash transfer of 65 Jordanian dinars (approximately US $92).
Information
Participants received more extensive information about preparing for job interviews, including in-person instruction.
Nudge
Participants received a behavioral job-search intervention consisting of a four-week planning calendar, an instructional video, face-to-face guidance, and optional SMS reminders.
Table 3: Interventions considered in the job assistance program.
Estimator
6 weeks
2 months
4 months
AW
0.26 [-1.98, 2.51]
2.09 [-1.53, 5.71]
-0.07 [-2.65, 2.50]
Bayes
1.70 [0.07, 3.33]
4.42 [1.97, 6.91]
2.73 [0.59, 4.93]
Table 4: Our frequentist AW VPI estimate, and the Bayesian estimates (which have formal properties under the assumption of a unique optimal policy ( Caria et al., 2024 ) ) alongside 95% credible/confidence intervals. Here, the baseline is the “all control” policy. Estimates of AW are produced using roughly 20% of the sample for burn-in.
Group
Estimator
6 weeks
2 months
4 months
Jordanian
AW
-0.05 [-2.99, 2.90]
-1.76 [-6.14, 2.62]
-2.46 [-5.51, 0.60]
Jordanian
Bayes
1.94 [-0.39, 4.27]
1.83 [-0.92, 4.72]
1.01 [-2.06, 4.21]
Syrian
AW
0.33 [-2.69, 3.34]
7.12 [1.25, 12.99]
3.36 [-0.74, 7.46]
Syrian
Bayes
1.49 [-0.73, 3.87]
8.30 [4.08, 12.61]
4.97 [1.46, 8.74]
Table 5: VPI estimates restricted to the Jordanian and Syrian subpopulations alongside 95% credible/confidence intervals.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Work
Setting
Target
Result
Key assumptions
Luedtke and Van Der Laan [2016]
i.i.d. data
Unrestricted optimal value V∘
Asymptotically normal estimator
Propensity and regression nuisance convergence rates.
Li and Brunskill [2026]
i.i.d. data
Best in-class improvement V∗−maxaV(a)
Asymptotically normal estimator
Uniqueness, Nuisance convergence rates.
Zhan et al. [2021]
Contextual bandit data
Fixed-policy value V(π)
Asymptotically normal estimator
α<1/2
Bibaut et al. [2021a]
Contextual bandit data
Fixed-policy value V(π)
Asymptotically normal estimator
α≤1/2 ( α≤1/3 for variance convergence)
Imai and Li [2023]
i.i.d. randomized experiment
Fixed-policy contrast V(π)−V(q)
Finite-sample variance and bias bounds for policies
Binary treatment; complete randomization
Bibaut et al. [2021b]
Contextual bandit data
Best-in-class policy learning
High probability regret Θ(n(α−1)/2)
Sequential bracketing entropy
Appendix
Table 6: An extended comparison of our results with the most closely related papers.
α
n
AW
StabVar (50/50)
CADR (50/50)
0
2000
0.880 [.83, .91]
0.888 [.84, .92]
0.880 [.83, .91]
5000
0.952 [.92, .97]
0.940 [.90, .96]
0.924 [.88, .95]
10000
0.960 [.93, .98]
0.964 [.93, .98]
0.956 [.92, .98]
15000
0.932 [.89, .96]
0.940 [.90, .96]
0.924 [.88, .95]
0.1
2000
0.872 [.82, .91]
0.912 [.87, .94]
0.904 [.86, .93]
5000
0.924 [.88, .95]
0.940 [.90, .96]
0.924 [.88, .95]
Appendix
Table 7: Synthetic DGP: Empirical 95% coverage [with 95% Wilson interval] of the three well-calibrated estimators over 250 independent runs; contrast truth θ∗≈0.350 .
Figure 6: Examples of two depth 2 decision trees with minimum node size of at least 15% that obtain the optimal policy value. Note that these policies just differ in how they assign treatments to units with x1∈[0.4,0.7] .
joke id
mean
std
best arm share
16
-2.595
5.027
0.058
57
-1.515
5.446
0.084
41
0.011
5.484
0.139
81
1.986
5.318
0.293
50
3.822
4.150
0.426
Appendix
Table 8: Marginal diagnostics for the fixed five-joke treatment benchmark. The “best arm share” column gives the marginal fraction of the time that the noted arm wins.
joke id
16
57
41
81
50
16
1.000
0.331
0.141
0.067
0.072
57
0.331
1.000
0.288
0.166
0.121
41
0.141
0.288
1.000
0.256
0.225
81
0.067
0.166
0.256
1.000
0.205
50
0.072
0.121
0.225
0.205
1.000
Appendix
Table 9: Pairwise correlation matrix for user ratings in the fixed five joke treatment set.
step
joke id
variance score
redundancy penalty
objective value
1
71
1.000
0.000
1.000
2
2
0.773
0.142
0.630
3
75
0.764
0.168
0.596
4
7
0.730
0.138
0.592
5
80
0.780
0.243
0.537
6
60
0.684
0.177
0.507
Appendix
Table 10: Results of the covariate selection procedure. The table reports the selected joke ID at each step alongside the normalized variance score, average absolute-correlation penalty relative to the previously selected jokes, and the resulting net objective value.
α
n
AW
StabVar (50/50)
CADR (50/50)
0
2000
0.952 [.92, .97]
0.940 [.90, .96]
0.864 [.82, .90]
5000
0.964 [.93, .98]
0.944 [.91, .97]
0.912 [.87, .94]
8000
0.948 [.91, .97]
0.956 [.92, .98]
0.932 [.89, .96]
10000
0.948 [.91, .97]
0.936 [.90, .96]
0.932 [.89, .96]
15000
0.968 [.94, .98]
0.912 [.87, .94]
0.920 [.88, .95]
20000
0.948 [.91, .97]
0.936 [.90, .96]
0.924 [.88, .95]
Appendix
Table 11: Semi-synthetic DGP: Empirical 95% coverage [with 95% Wilson interval] of the three well-calibrated estimators over 250 independent runs.
px:=∑t∈Tnt∑i,t\mathbbm1{Xi,t=x}.
Appendix
Algorithm 2 Posterior estimate of the value of targeting
Diagnostic
Mean
5%
Median
95%
P(<0.5 pp)
P(<1 pp)
Single stratum
20.52
1.27
16.24
54.33
2.0%
4.0%
Minimum over 16 strata
2.82
0.08
1.25
10.72
26.2%
43.3%
Appendix
Table 12: Prior-predictive Bayesian suboptimality gap diagnostics
Figure 7: Prior-predictive distributions of best-minus-second-best Bayesian arm gaps.
Group
Horizon
n
Burn-in rows
Burn-in share
Burn-in days
First eval date
All
6 weeks
3709
759
20.5%
42
2019-04-17
All
2 months
3631
732
20.2%
38
2019-04-17
All
4 months
3516
724
20.6%
39
2019-04-22
Jordanian
6 weeks
2064
422
20.4%
40
2019-04-17
Jordanian
2 months
2013
405
20.1%
38
2019-04-17
Jordanian
4 months
1941
393
20.2%
38
2019-04-17
Appendix
Table 13: Natural day-boundary burn-in cutoffs after 20 percent of each sample
Policy learning in digital experimentation faces three challenges: weak signal-to-noise ratios, rich covariate spaces, and massive data volumes. We formalize this regime by modeling treatment-effect estimates from increasingly fine covariate partitions as Gaussian observations with bounded signal-to-noise ratios. We establish that, in general, the optimal treatment policy is not learnable in this setting. Even learning the optimal policy value suffers from impractically slow rates. However, when treatment effects vary smoothly, we derive minimax-adaptive policies based on linear smoothers that achieve vanishing welfare regret. We demonstrate the practical value of our framework by applying it to large-scale real-world experiments at Netflix, showing that personalized linear-smoothing policies can dominate unpersonalized policies even in this challenging empirical setting.
Policy learning from observational data is fundamental to applications ranging from personalized medicine to welfare-program targeting, but standard approaches optimize empirical performance and may fail when the deployment distribution differs from the training distribution. We study distributionally-robust policy learning: given observational data, we seek a policy π∈Π that maximizes the worst-case policy value over an f-divergence ball of radius ρ around the data-generating distribution. We propose an estimator built on cross-fitted doubly-robust scores for the conditional treatment effect, paired with a DRO solver that handles KL, χ2, and CVaR ambiguity sets. We establish two convergence guarantees. First, the estimated policy attains the standard O(n−1/2) rate on its DRO regret, and inherits the doubly-robust property -- the rate continues to hold whenever either the outcome model or the propensity model is consistently estimated, even at slow nonparametric rates. We then propose a sample-splitting bias-correction algorithm derived from the Lagrangian dual, and show that under a Tsybakov-style margin condition on the treatment-effect, the estimator attains a fast rate of O(n−(1+α)/(2+α)) , interpolating between O(1/n) and O(1/n) as the margin parameter α ranges over [0,∞). To our knowledge, this is the first work to incorporate a margin condition into the analysis of distributionally-robust policy learning, and the first to obtain fast rates in this setting. Empirically, on synthetic and semi-synthetic experiments calibrated to a welfare-to-work program, distributional robustness yields meaningful improvements over empirical welfare maximization when training and test distributions differ.
Debmalya Mandal
Dept. of Computer Science University of Warwick, UK
Policy learning methods are increasingly used to inform treatment allocation under budget constraints. Most proposed methods assume complete treatment data, yet applications frequently suffer from missingness that can bias estimates and lead to suboptimal policies. We address this gap by extending efficient estimators for average treatment effect (ATE) estimation to policy value and conditional average treatment effect (CATE) estimation under missing at random (MAR) and missing completely conditionally at random (MCCAR) treatment data. Through asymptotic efficiency analysis, we prove that the MAR estimator, which leverages partially-observed units, is both valid and more efficient than the MCCAR estimator when MCCAR assumptions hold. This result provides formal justification for preferring MAR-based estimation in policy learning under both missing data settings. Our comprehensive experiments using synthetic and semi-synthetic datasets confirm that correctly specifying the missingness mechanism is crucial: misspecified estimators remain biased regardless of sample size, while our estimators achieve near-oracle performance when assumptions are satisfied. Our work provides practitioners with theoretically grounded, empirically validated tools for robust policy learning in the presence of missing treatment data.