Policy learning in digital experimentation faces three challenges: weak signal-to-noise ratios, rich covariate spaces, and massive data volumes. We formalize this regime by modeling treatment-effect estimates from increasingly fine covariate partitions as Gaussian observations with bounded signal-to-noise ratios. We establish that, in general, the optimal treatment policy is not learnable in this setting. Even learning the optimal policy value suffers from impractically slow rates. However, when treatment effects vary smoothly, we derive minimax-adaptive policies based on linear smoothers that achieve vanishing welfare regret. We demonstrate the practical value of our framework by applying it to large-scale real-world experiments at Netflix, showing that personalized linear-smoothing policies can dominate unpersonalized policies even in this challenging empirical setting.
Figures & tables
Smoother
Weight wr,j(x)
Conditions
Partition pooling
∑ℓ:xℓ∈Ar(x)pℓpj1{xj∈Ar(x)}
Ar a finite partition of X ; Ar(x)∈Ar the cell containing x ; maxA∈Arsupx,y∈A∥x−y∥2≤r ;
Kernel smoothing
∑ℓ=1KK((xℓ−x)/r)K((xj−x)/r)
K:Rd→[0,∞) bounded, supported on B(0,1) , continuous at 0 , K(0)>0 .
Figure 1: The first three panels show the true curved CATE g , one draw of the rescaled estimates Zk=nτk , and the plug-in policy 1{Zk>0} that these estimates imply. The remaining panels show representative linear smoothers. The dashed line is the true frontier at which g=0 , while the solid line represents the learner’s decision boundary, gw=0 .
Figure 2: The top row shows the observed CATEs and full-sample fits of a plane, tensor-product B-spline, regression tree, and local-linear Epanechnikov kernel applied to Experiment 1. Each displayed fit uses the candidate selected most often across outer folds in nested CV. The lower-left panel shows the average held-out policy gains over unpersonalized policies of the family with the lowest estimated policy loss. The remaining panels show the calibration of cross-fitted CATE predictions after centering within each fold.
Figure 3: Panels are laid out as in Figure 2 , but for Experiment 2.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: The first three panels show the true linear CATE g , one draw of the rescaled estimates Zk=nτk , and the plug-in policy 1{Zk>0} that these estimates imply. The remaining panels show representative linear smoothers. The dashed line is the true frontier at which g=0 , while the solid line represents the learner’s decision boundary, gw=0 .
Linear signal
Curved signal
Learner
r~
RK
W/V∗
RK
W/V∗
Plug-in
1
0.085
0.27
0.052
0.14
Pooled
10
0.117
0.00
0.060
0.00
Tree ( J=4 )
5.0
0.035
0.70
0.034
0.43
Tree ( J=9 )
3.3
0.040
0.65
0.034
0.44
Tree ( J=25 )
2.0
0.063
0.46
0.043
0.30
Appendix
Table 2: Exact welfare regret and proportional welfare for representative learners. The resolution r~ is measured in units of K−1/d , so r~=1 corresponds to the plug-in rule and r~=10 corresponds to full pooling, that is, a single treatment decision is made for all cells jointly.
Figure 5: Approximation error Ag(wr) , estimation error γ∗Eσ(wr) , their sum, and exact regret RK as functions of resolution r~ . Rows correspond to signal regimes, while columns correspond to the smoother families considered in the main text.
Figure 6: Every candidate is one point in terms of approximation error Ag(w) and estimation error γ∗Eσ(w) . Note that the axis are on different scales. Better smoothers lie closer to the bottom-left corner.
Adaptive experimentation under unknown network interference requires solving two coupled problems: (i) learning the underlying dynamics of interference among units and (ii) using these dynamics to inform treatment allocation in order to maximize a cumulative outcome of interest (e.g. revenue). Existing adaptive experimentation methods either assume the interference network is fully known or bypass the network by operating on coarse cluster-level randomizations. We develop a Thompson sampling algorithm that jointly learns the interference network and adaptively optimizes individual-level treatment allocations via a Gibbs sampler. The algorithm returns both an optimized treatment policy and an estimate of the interference network; the latter supports downstream causal analyses such as estimation of direct, indirect, and total treatment effects. For additive spillover models, we show that total reward is linear in the treatment vector with coefficients given by an n-dimensional latent score. We prove a Bayesian regret bound of order nT⋅Blog(en/B) for exact posterior sampling; empirically, our Gibbs-based approximate sampler achieves regret consistent with this rate and remains sublinear when the additive spillovers assumption is violated. For general Neighborhood Interference, where this reduction is unavailable, we analyze an explore-then-commit variant with O(n2logT) graph-discovery cost. An information-theoretic Ω(nlogT) lower bound complements both results. Empirically, our method achieves more than an order-of-magnitude reduction in regret in head-to-head comparisons. On two real-world networks, the algorithm achieves sublinear regret and yields downstream effect estimates with small RMSE relative to the truth.
Policy learning has received substantial attention with the goal of learning policies from observational data for decision-making. A majority of work in this space has focused on developing algorithms for computing policies that minimize regret compared to the optimal policy. However, in many practical settings, there is insufficient data to obtain low regret. As a result, recent work has shifted attention to alternative objectives, most notably, studying whether it is possible to learn an improving policy that statistically significantly outperforms baseline policies. We argue that there is substantial merit in studying a broader range of policy learning problems. When there is insufficient data to learn an improving policy, there may still be useful questions that can be answered. To this end, we provide a mathematical framework for studying the relationships between policy learning problems. We formalize three problems within our framework: beyond the optimal policy problem and the improving policy problem, we also propose the policy existence problem, which aims to determine if an improving policy exists. Within our framework, we show that the policy existence problem reduces to the improving policy problem, which in turn reduces to the optimal policy problem; these reductions prove that each problem is at least as easy as the next one (in sample complexity). A key question remains: is this hardness strict? We provide partial answers. First, the gap between the optimal policy and improving policy problems is strict. For the improving policy and policy existence problems, we prove that a sublinear polynomial gap exists under natural conditions on improving policy learning algorithms. Thus, we may be able to answer questions about the existence of an improving policy even when we cannot find one. These results highlight the value in studying a broader range of policy learning problems.
Hamsa Bastani, Osbert Bastani, Shihan Chen
Wharton School, University of Pennsylvania · University of Pennsylvania · Graduate Group in Applied Mathematics and Computational Science, University of Pennsylvania
Offline policy learning has received growing attention in causal inference. The primary objective is to learn a policy (individualized treatment rule) as a mapping from covariates to treatment that maximizes the empirical welfare defined as the mean of scalar-valued potential outcomes. In this paper, we study offline policy learning with distribution-valued outcomes, where each potential outcome is a probability measure on R and the reward is defined through a utility functional applied to the Wasserstein barycenter of induced outcome distributions. We establish statistical guarantees for the policy learning framework based on both Inverse Probability Weighting (IPW) and Doubly Robust (DR) estimators. By handling the challenging uniform deviation over the product of the combinatorial policy class and the infinite-dimensional quantile domain, we prove that the finite-sample regret has leading dependence O(N-dim(Π)/N). In the one-dimensional Wasserstein setting and under the stated regularity conditions, the leading regret rate is still governed by the policy-class complexity. Moreover, we provide a minimax lower bound establishing the sharpness of the leading dependence on N and N-dim(Π).
Yiyan Huang, Cheuk Hang Leung, Qi Wu +1
School of Statistics and Data Science & Institute of Big Data Research, Shanghai University of Finance and Economics, Shanghai, China