We investigate how historical data from fixed randomized experiments (A/B tests) can be used to inform the deployment of adaptive experiments based on contextual bandits. Given data collected under a static allocation, our goal is to assess which adaptive policies, if any, would have outperformed the original design and under what conditions. To this end, we combine off-policy evaluation (OPE) with a controlled warm-start simulation. From logged A/B test data exhibiting heterogeneous treatment effects, we estimate nuisance components and use doubly robust estimators to rank a portfolio of pre-specified adaptive and non-adaptive policies. When ground truth is available, we then deploy the same offline-trained policies in a simulator that reuses the exact data-generating reward probabilities, providing a safe, ground-truth-anchored environment to study the offline-to-online transition under warm starting. Using synthetic randomized controlled trials with known heterogeneity structures and an oracle policy, our results indicate that adaptive, context-aware policies improve upon fixed allocations when meaningful heterogeneity is present, while providing little benefit in its absence. We reinforce our findings on standard open benchmarks (Hillstrom, Criteo Uplift, and LaLonde), reinterpreted through a policy-value and regret perspective. Overall, our results provide a practical methodology for deciding when adaptive experimentation is worth deploying and how to select among competing adaptive policies using existing A/B test data.
Figures & tables
Figure 1: Ground truth probability of effect given the Treatment and Covariate value.
Name
Ground Truth
N
NCov
Treatment/Control
NBoot
Online
TOn
NOn
Allocation
Simulation?
No Contex
Eq. ( 1 )
3000
0
50/50
300
Yes
500
100
Categorical
Eq. ( 2 )
3000
1
50/50
300
Yes
500
100
Linear
Eq. ( 3 )
3000
1
50/50
300
Yes
500
100
Oscillatory
Eq. ( 4 )
3000
1
50/50
300
Yes
500
100
Double Average
Eq. ( 5 )
3000
2
50/50
300
Yes
500
100
Table 1: Summary of Experiments ran for results generation.
Aspect
Synthetic
Open data
Data source
Pre-generated RCT data
RCT or observational dataset
Ground truth
FUNC_TRAT , FUNC_CONT
Not available
Oracle policy
argmax(Ground Truth)
Not defined
Logging propensity
Uniform (RCT)
Uniform (RCT)
or empirical frequency
Outcome model (binary)
Calibrated RF per arm
Calibrated RF per arm
Table 2: Structural differences between the two experiment families.
Figure 2: OPE (DR) and regret. Policies are ranked by DR value (left) and estimated mean regret relative to the best observed policy (right). Whenever contextual heterogeneity is present, adaptive policies substantially outperform fixed baselines; among strong adaptive candidates, differences are comparatively modest.
Figure 3: Policy and value functions by bins. In the continuous settings, the learned policies recover the main decision structure implied by the oracle: a monotone switching threshold in the linear case and alternating treatment advantage in the oscillatory case.
Figure 4: Warm-start online cumulative metrics ( T=500,N=100 ). Left: cumulative reward; right: cumulative regret relative to the oracle. Several adaptive policies remain close to the oracle and far outperform the fixed baselines. The dominant contrast is between adaptive and non-adaptive designs; among strong adaptive policies, realized differences are modest and partly attributable to finite-sample variability.
Figure 5: Two-covariate with average combination policy and value surfaces. Panels show π(x1,x2) (left) and Vπ(x1,x2) (right) for each policy. Adaptive policies concentrate probability mass in regions favored by the oracle, while fixed baselines fail to exploit the two-dimensional heterogeneity structure. Softmax and UCB policies offers plots indistinguishable from the plots for Epsilon-Greedy policy.
Figure 6: Two-covariate with product combination policy and value surfaces.
Table 3: Warm-start online simulations ( T=500 timesteps, average over N=100 simulations). Accuracy is the fraction of oracle-consistent actions; reward and regret are reported both per-step and cumulatively. Across heterogeneous settings, strong adaptive policies remain close to the oracle and far outperform the fixed baselines.
Figure 7: OPE (DR) and regret for open Datasets. For Hillstrom and Lalonde, adaptive policies do not clearly separate from the strongest fixed alternatives. For Criteo, adaptive policies consistently outperform non-adaptive baselines.
Off-policy evaluation (OPE) of ranking policies is challenging be- cause selecting and ordering multiple items from a candidate set makes the number of possible rankings grow combinatorially with the number of candidates and the ranking length. Consequently, Inverse Propensity Scoring (IPS), whose importance weight is the full-ranking probability ratio under the evaluation and logging policies, can have excessive variance. Independent IPS (IIPS) and Reward Interaction IPS (RIPS) reduce variance by imposing fixed assumptions on how users browse rankings, but may introduce bias when those assumptions mismatch actual behavior. Adaptive Inverse Propensity Scoring (AIPS) addresses this trade-off by adap- tively marginalizing importance weights over the actions that affect each position-wise reward. It attains minimum variance within a class of unbiased IPS-based estimators when the true user be- havior model is observed. However, its estimation accuracy may still degrade for longer rankings, and AIPS does not use a reward model for residual correction. We propose Adaptive Doubly Robust (ADR), which combines adaptive importance weighting with re- ward regression through a control-variate correction. We establish its unbiasedness when the true user behavior model is observed and characterize a sufficient condition under which it reduces vari- ance relative to AIPS. Across synthetic experiments with 10,000 simulations per condition, ADR improves mean squared error over AIPS and conventional ranking OPE estimators across a range of logged-data sizes and ranking lengths.
Online controlled experiments are the gold standard for hypothesis testing in online platforms. Notwithstanding their ubiquity, they are notoriously expensive to run, and issues of variance hamper statistical power in assessing treatment effects. While standard variance reduction techniques leverage model-based control variates to reduce outcome noise, they remain agnostic to potential structural relationships between competing policies. In this work, we identify a critical inefficiency in the standard A/B-testing protocol: when a treatment and control policy agree on an action, the resulting outcome contributes noise but no signal regarding the treatment effect -- unnecessarily inflating confidence intervals. We propose a novel experimental protocol that exploits this policy overlap to accelerate experimentation. The key insight is to frame the randomised treatment assignment mechanism as a meta-policy, and leverage Δ-Off-Policy Estimation methods to obtain unbiased estimates for average treatment effects. We prove analytically that our approach recovers standard A/B-testing practices in the general case, but that its variance scales with the divergence between policies rather than raw outcome variance. Hence, we dominate the standard Difference-in-Means estimator whenever policies have common support, and the improvement is strict whenever the overlap region contributes non-zero residual variance. Empirical results corroborate these theoretical insights -- holding promise for significant impact on the real-world evaluation of recommender systems, information retrieval pipelines, and large language model interfaces.
Off-Policy Evaluation and Learning (OPE/L) in contextual bandits is rapidly gaining popularity in real systems because new policies can be evaluated and learned securely using only historical logged data. However, existing methods in OPE/L cannot handle many challenging but prevalent scenarios such as few-shot data, deterministic logging policies, and new actions. In many applications, such as personalized medicine, content recommendations, education, and advertising, we need to evaluate and learn new policies in the presence of these challenges. Existing methods cannot evaluate and optimize effectively in these situations due to the notorious variance issue or limited exploration in the logged data. To enable OPE/L even under these unsolved challenges, we propose a new problem setup of Cross-Domain OPE/L, where we have access not only to the logged data from the target domain in which the new policy will be implemented but also to logged datasets collected from other domains. This novel formulation is widely applicable because we can often use historical data not only from the target hospital, country, device, or user segment but also from other hospitals, countries, devices, or segments. We develop a new estimator and policy gradient method to solve OPE/L by leveraging both target and source datasets, resulting in substantially enhanced OPE/L in the previously unsolved situations in our empirical evaluations.