Inference on the Value of Policy Improvement in Adaptive Experiments

Justin Whitehouse; Emma Brunskill; Vasilis Syrgkanis

stat.MLOct 11, 2026

Inference on the Value of Policy Improvement in Adaptive Experiments

Authors: Justin Whitehouse, Emma Brunskill, Vasilis Syrgkanis

Organizations: Management Science and Engineering, Stanford University · Computer Science, Stanford University

Abstract

In this paper, we develop an asymptotically normal estimator for the value of policy improvement (VPI) under adaptively collected data. This value, which is defined as the gap between the value of the best treatment policy in a class and that of a baseline policy, gauges whether or not it is worth deploying a personalized treatment policy. Building an estimator for the VPI is naturally more difficult than estimating the value of a fixed policy. This is because, in addition to contending with the correlated data and decaying propensities associated with adaptive experiments, any estimator must also handle the non-regularity, or lack of pathwise differentiability, of the underlying estimand when the optimal policy is non-unique. Our proposed estimator works by sequentially fitting a weighted empirical welfare maximizing policy on historical data and then evaluating it using inverse propensity weighting on future samples. We neither assume that the optimal policy is unique nor make a margin assumption on the underlying heterogeneous treatment effects. A key difficulty in analyzing our estimator is showing that the regret incurred through policy learning is asymptotically smaller than the standard error of the estimator. We accomplish this by showing faster regret rates for policy learning under decaying propensities, which may be of independent interest. We not only evaluate the coverage of our estimator on a mix of synthetic and semi-synthetic data, but also employ it to estimate the VPI in an adaptive experiment evaluating the efficacy of job assistance programs in Jordan due to Caria et al. Although our estimates are generally more conservative than their hierarchical Bayesian analysis, which assumes that the optimal policy is unique, we still corroborate the benefits of personalization amongst Syrian refugees.

Figures & tables

Work	Setting	Target	Result	Key assumptions
Luedtke and Van Der Laan (2016)	i.i.d. data	Unrestricted optimal value $V^{\circ}$	Asymptotically normal estimator	Propensity and regression nuisance convergence rates.
Zhan et al. (2021)	Contextual bandit data	Fixed-policy value $V(\pi)$	Asymptotically normal estimator	$\alpha<1/2$
Bibaut et al. (2021a)	Contextual bandit data	Fixed-policy value $V(\pi)$	Asymptotically normal estimator	$\alpha\leq 1/2$ ( $\alpha\leq 1/3$ for variance convergence)
This paper	Contextual bandit data	Best in-class contrast $V^{\ast}-V(q)$	Asymptotically normal estimator and expected regret $o\!\left(n^{(\alpha-1)/2}\right)$	Finite disagreement VC dimension; $\alpha<1$

Table 1: A comparison of our results with the most closely related papers. We let

V^{\ast}

denote the value of an optimal policy within a class

\Pi

and

V^{\circ}

the optimal value of any policy. An extended comparison is shown in Table 6 in Appendix A

\widetilde{\sigma}_{t}^{2}:=\frac{1}{t-1}\sum_{s=1}^{t-1}\frac{g_{\widehat{\pi}_{h}}(A_{s}\mid X_{s})^{2}}{p_{s}(A_{s}\mid X_{s})p_{t}(A_{s}\mid X_{s})}Y_{s}^{2}-\left(\frac{1}{t-1}\sum_{s=1}^{t-1}\frac{g_{\widehat{\pi}_{h}}(A_{s}\mid X_{s})}{p_{s}(A_{s}\mid X_{s})}Y_{s}\right)^{2}

Algorithm 1 Batched Estimator for

\theta^{\ast}

Estimator	Learns policy on	Evaluates on	Inv.-SD wt.	Refits	Out-of-sample
AW (ours)	geometric epochs	future rounds	$\checkmark$	$\checkmark$	$\checkmark$
CADR	prefix ( $b_{n}$ or $n/2$ )	remaining rounds	$\checkmark$	$\times$	$\checkmark$
StabVar	prefix ( $b_{n}$ or $n/2$ )	remaining rounds	$\times$	$\times$	$\checkmark$
DD	all $n$	all $n$ (same data)	$\times$	$\times$	$\times$

Table 2: A summary of comparators and the design choices that distinguish them. CADR and StabVar are each run in two modes: a sublinear burn-in split consisting of

b_{n}=o(n)

samples, and a

50/50

split (

b_{n}=n/2

Figure 1: Empirical coverage versus sample size in the synthetic DGP for all six estimators at logging exponents

\alpha\in\{0,0.1,0.25,0.5\}

. Error bars are Wilson

95\%

intervals and the dashed line is the nominal

0.95

coverage target. Double-dipping ( DD , the lowest curve) under-covers and does not recover with

n

, while AW and the

50/50

estimators obtain nominal coverage except in the

\alpha=0.5

case.

Panel of Figure 2 — Figure 2: The Pareto frontier of empirical coverage (x-axis) and confidence interval width (y-axis) in the synthetic DGP. We examine this frontier at $n=15{,}000$ samples and logging exponents $\alpha\in\{0,0.1,0.25,0.5\}$ . Estimators toward the lower right are better calibrated.

Figure 3: Empirical coverage versus sample size in the semi-synthetic DGP for all six estimators at logging exponents

\alpha\in\{0,0.1,0.25\}

. Error bars are Wilson

95\%

intervals and the dashed line is the nominal

0.95

coverage target.

Panel of Figure 4 — Figure 4: The Pareto frontier of empirical coverage (x-axis) and confidence interval width (y-axis) in the semi-synthetic DGP. Here, we consider $n=20{,}000$ samples and logging exponents $\alpha\in\{0,0.1,0.25\}$ .

Panel of Figure 5 — Figure 5: A visualization of scaled regret against $n$ under various logging exponents $\alpha$ . The shaded region gives a $95\%$ pointwise confidence interval over $M=100$ independent runs. The synthetic panel leverages an exact solver, whereas the semi-synthetic one uses an approximate solution.

Treatment arm	Description
Control	Participants received a small cash infusion of 4 Jordanian dinars (approximately US $5.60) and a basic informational pamphlet.
Cash	Participants received a larger cash transfer of 65 Jordanian dinars (approximately US $92).
Information	Participants received more extensive information about preparing for job interviews, including in-person instruction.
Nudge	Participants received a behavioral job-search intervention consisting of a four-week planning calendar, an instructional video, face-to-face guidance, and optional SMS reminders.

Table 3: Interventions considered in the job assistance program.

Estimator	6 weeks	2 months	4 months
AW	0.26 [-1.98, 2.51]	2.09 [-1.53, 5.71]	-0.07 [-2.65, 2.50]
Bayes	1.70 [0.07, 3.33]	4.42 [1.97, 6.91]	2.73 [0.59, 4.93]

Table 4: Our frequentist AW VPI estimate, and the Bayesian estimates (which have formal properties under the assumption of a unique optimal policy ( Caria et al., 2024 ) ) alongside 95% credible/confidence intervals. Here, the baseline is the “all control” policy. Estimates of AW are produced using roughly 20% of the sample for burn-in.

Group	Estimator	6 weeks	2 months	4 months
Jordanian	AW	-0.05 [-2.99, 2.90]	-1.76 [-6.14, 2.62]	-2.46 [-5.51, 0.60]
Jordanian	Bayes	1.94 [-0.39, 4.27]	1.83 [-0.92, 4.72]	1.01 [-2.06, 4.21]
Syrian	AW	0.33 [-2.69, 3.34]	7.12 [1.25, 12.99]	3.36 [-0.74, 7.46]
Syrian	Bayes	1.49 [-0.73, 3.87]	8.30 [4.08, 12.61]	4.97 [1.46, 8.74]

Table 5: VPI estimates restricted to the Jordanian and Syrian subpopulations alongside 95% credible/confidence intervals.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Work	Setting	Target	Result	Key assumptions
Luedtke and Van Der Laan [2016]	i.i.d. data	Unrestricted optimal value $V^{\circ}$	Asymptotically normal estimator	Propensity and regression nuisance convergence rates.
Li and Brunskill [2026]	i.i.d. data	Best in-class improvement $V^{*}-\max_{a}V(a)$	Asymptotically normal estimator	Uniqueness, Nuisance convergence rates.
Zhan et al. [2021]	Contextual bandit data	Fixed-policy value $V(\pi)$	Asymptotically normal estimator	$\alpha<1/2$
Bibaut et al. [2021a]	Contextual bandit data	Fixed-policy value $V(\pi)$	Asymptotically normal estimator	$\alpha\leq 1/2$ ( $\alpha\leq 1/3$ for variance convergence)
Imai and Li [2023]	i.i.d. randomized experiment	Fixed-policy contrast $V(\pi)-V(q)$	Finite-sample variance and bias bounds for policies	Binary treatment; complete randomization
Bibaut et al. [2021b]	Contextual bandit data	Best-in-class policy learning	High probability regret $\Theta\left(n^{(\alpha-1)/2}\right)$	Sequential bracketing entropy

Appendix

Table 6: An extended comparison of our results with the most closely related papers.

$\alpha$	$n$	AW	StabVar (50/50)	CADR (50/50)
$0$	2000	0.880 [.83, .91]	0.888 [.84, .92]	0.880 [.83, .91]
	5000	0.952 [.92, .97]	0.940 [.90, .96]	0.924 [.88, .95]
	10000	0.960 [.93, .98]	0.964 [.93, .98]	0.956 [.92, .98]
	15000	0.932 [.89, .96]	0.940 [.90, .96]	0.924 [.88, .95]
$0.1$	2000	0.872 [.82, .91]	0.912 [.87, .94]	0.904 [.86, .93]
	5000	0.924 [.88, .95]	0.940 [.90, .96]	0.924 [.88, .95]

Appendix

Table 7: Synthetic DGP: Empirical

95\%

coverage [with

95\%

Wilson interval] of the three well-calibrated estimators over

250

independent runs; contrast truth

\theta^{\ast}\approx 0.350

Panel of Figure 6 — Figure 6: Examples of two depth 2 decision trees with minimum node size of at least 15% that obtain the optimal policy value. Note that these policies just differ in how they assign treatments to units with $x_{1}\in[0.4,0.7]$ .

joke id	mean	std	best arm share
16	-2.595	5.027	0.058
57	-1.515	5.446	0.084
41	0.011	5.484	0.139
81	1.986	5.318	0.293
50	3.822	4.150	0.426

Appendix

Table 8: Marginal diagnostics for the fixed five-joke treatment benchmark. The “best arm share” column gives the marginal fraction of the time that the noted arm wins.

joke id	16	57	41	81	50
16	1.000	0.331	0.141	0.067	0.072
57	0.331	1.000	0.288	0.166	0.121
41	0.141	0.288	1.000	0.256	0.225
81	0.067	0.166	0.256	1.000	0.205
50	0.072	0.121	0.225	0.205	1.000

Appendix

Table 9: Pairwise correlation matrix for user ratings in the fixed five joke treatment set.

step	joke id	variance score	redundancy penalty	objective value
1	71	1.000	0.000	1.000
2	2	0.773	0.142	0.630
3	75	0.764	0.168	0.596
4	7	0.730	0.138	0.592
5	80	0.780	0.243	0.537
6	60	0.684	0.177	0.507

Appendix

Table 10: Results of the covariate selection procedure. The table reports the selected joke ID at each step alongside the normalized variance score, average absolute-correlation penalty relative to the previously selected jokes, and the resulting net objective value.

$\alpha$	$n$	AW	StabVar (50/50)	CADR (50/50)
$0$	2000	0.952 [.92, .97]	0.940 [.90, .96]	0.864 [.82, .90]
	5000	0.964 [.93, .98]	0.944 [.91, .97]	0.912 [.87, .94]
	8000	0.948 [.91, .97]	0.956 [.92, .98]	0.932 [.89, .96]
	10000	0.948 [.91, .97]	0.936 [.90, .96]	0.932 [.89, .96]
	15000	0.968 [.94, .98]	0.912 [.87, .94]	0.920 [.88, .95]
	20000	0.948 [.91, .97]	0.936 [.90, .96]	0.924 [.88, .95]

Appendix

Table 11: Semi-synthetic DGP: Empirical

95\%

coverage [with

95\%

Wilson interval] of the three well-calibrated estimators over

250

independent runs.

p_{x}:=\frac{\sum_{i,t}\mathbbm{1}\left\{X_{i,t}=x\right\}}{\sum_{t\in\mathcal{T}}n_{t}}.

Appendix

Algorithm 2 Posterior estimate of the value of targeting

Diagnostic	Mean	5%	Median	95%	$P(<0.5\text{ pp})$	$P(<1\text{ pp})$
Single stratum	20.52	1.27	16.24	54.33	2.0%	4.0%
Minimum over 16 strata	2.82	0.08	1.25	10.72	26.2%	43.3%

Appendix

Table 12: Prior-predictive Bayesian suboptimality gap diagnostics

Panel of Figure 7 — Figure 7: Prior-predictive distributions of best-minus-second-best Bayesian arm gaps.

Group	Horizon	$n$	Burn-in rows	Burn-in share	Burn-in days	First eval date
All	6 weeks	3709	759	20.5%	42	2019-04-17
All	2 months	3631	732	20.2%	38	2019-04-17
All	4 months	3516	724	20.6%	39	2019-04-22
Jordanian	6 weeks	2064	422	20.4%	40	2019-04-17
Jordanian	2 months	2013	405	20.1%	38	2019-04-17
Jordanian	4 months	1941	393	20.2%	38	2019-04-17

Appendix

Table 13: Natural day-boundary burn-in cutoffs after 20 percent of each sample

Figure 8: Full-sample learned-policy heatmaps, 6-week employment, natural 20 percent burn-in

Figure 9: Full-sample learned-policy heatmaps, 2-month employment, natural 20 percent burn-in

Figure 10: Full-sample learned-policy heatmaps, 4-month employment, natural 20 percent burn-in

Policy Evaluation

Policy evaluation estimates the outcomes, value, risks, and fairness of decision rules before or after deployment.

See all 55 papers

Policy Learning with Weak Signals
Oct 7, 2026Benedikt Koch, Winston Chou, Aurélien Bibaut +1Policy Learning Causal Effect Estimation
Harvard University · Netflix
Distributionally-Robust Policy Learning with Observational Data
Oct 18, 2026Debmalya MandalPolicy Learning Distribution Shift Robustness
Dept. of Computer Science University of Warwick, UK
Learning Who to Treat When Treatment is Missing
Jul 15, 2026Johnna Sundberg, Rayid Ghani, Eli Ben-Michael +1Causal Effect Estimation Conditional Average Treatment Effect Estimation
Carnegie Mellon University, Pittsburgh, Pennsylvania

Oct 7, 2026J/K move · Enter open · S save

Policy Learning with Weak Signals

Benedikt Koch, Winston Chou, Aurélien Bibaut, Nathan Kallus

Harvard University · Netflix

Policy Learning Causal Effect Estimation

Open paper AskPDF arXiv

Policy learning in digital experimentation faces three challenges: weak signal-to-noise ratios, rich covariate spaces, and massive data volumes. We formalize this regime by modeling treatment-effect estimates from increasingly fine covariate partitions as Gaussian observations with bounded signal-to-noise ratios. We establish that, in general, the optimal treatment policy is not learnable in this setting. Even learning the optimal policy value suffers from impractically slow rates. However, when treatment effects vary smoothly, we derive minimax-adaptive policies based on linear smoothers that achieve vanishing welfare regret. We demonstrate the practical value of our framework by applying it to large-scale real-world experiments at Netflix, showing that personalized linear-smoothing policies can dominate unpersonalized policies even in this challenging empirical setting.

Abstract

Policy Evaluation

Explore similar work

Policy Learning with Weak Signals

Distributionally-Robust Policy Learning with Observational Data

Learning Who to Treat When Treatment is Missing