Always-On Experimentation

Ricardo J. Sandoval; David Arbour; Avi Feller; Michael I. Jordan

stat.MESep 30, 2026

Always-On Experimentation

Authors: Ricardo J. Sandoval, David Arbour, Avi Feller, Michael I. Jordan

Organizations: University of California, Berkeley · Adobe Research · Inria, Paris

Abstract

Generative AI has dramatically accelerated the rate at which new treatments---from novel pharmaceuticals to online marketing campaigns---can be conceived and deployed. As a result, modern experimentation platforms often run continuously, with treatments added as they are ready and removed when they underperform. We formalize this "Always-On" experimental setting, in which treatments can be dynamically generated, added to, and removed from a running experiment, and study the statistical problem of deciding whether to accept or reject each treatment while controlling for the false discovery rate. We develop sequential tests that achieve time-uniform Type-I error control under arbitrary stopping times and "predictable" treatment schedules. Our approach builds on the testing-by-betting framework: we construct test supermartingales for testing the average treatment effect of each treatment, and show that the construction of these test supermartingales is growth-rate optimal in an almost-sure sense.

Figures & tables

Panel of Figure 1 — Figure 1 : (Left) Depiction of an instance of Always-On experimentation in which four treatments were added and removed at different points in the experiment. (Middle & Right) Always-On experimentation achieves the same level of power as platform trials while reducing the number of units needed to perform such discoveries. For information about simulation setup and further simulations, see Section A.1 .

Figure 2 : (a) FDR over window traffic, with

10

concurrent treatments. (b) Mean true discoveries over window traffic, with

10

concurrent treatments. (c) Empirical growth-rates for our proposed process, one that selects the portfolios via the Universal Portfolio algorithm but without restarting the algorithm in each sub-experiment, and a process that has oracle access to the numeraire portfolios.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3 : The true treatment effect of all $30$ treatments under the alternative is equal to $0.25$ . The interim analysis in the platform trials only take place when $50\%$ of the maximum number of units allowed for each treatment have been observed.

Panel of Figure 4 — Figure 4 : The true treatment effect of all $30$ treatments under the alternative is equal to $0.55$ . The interim analysis in the platform trials only take place when $50\%$ of the maximum number of units allowed for each treatment have been observed.

Panel of Figure 5 — Figure 5 : The true treatment effect of all $30$ treatments under the alternative is equal to $0.35$ . In the top row we only perform one interim analysis when $50\%$ of the maximum number of units allowed for each treatment have been observed. In the bottom row we perform interim analyses when $25\%$ , $50\%$ , and $75\%$ of the maximum number of units allowed for each treatment have been observed.

Figure 6 : Validity and power at the primary traffic level (

10

concurrent treatments): mean false discovery rate and mean true discoveries per replication.

Explore similar work

Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap
Jul 16, 2026Olivier JeunenVariance Reduction Heterogeneous Treatment Effects
Independent Researcher Antwerp, Belgium
Robust Sequential Experimental Design for A/B Testing
May 13, 2026Qianglin Wen, Xiangkun Wu, Chengchun Shi +4Bayesian Experimental Design Two-Sample Testing
Yunnan Key Laboratory of Statistical Modeling and Data Analysis, Yunnan University, Kunming, China · School of Mathematical Sciences, Zhejiang University, Hangzhou, China · Department of Statistics, London School of Economics and Political Science, London, United Kingdom +3
When AI Generates Covariates: Causal Typing and Estimand Drift in Sequential Experiments
Sep 15, 2026Takes Fujita, Nobutaka HattoriCausal Inferences Covariates
VRI · Department of Neurology, Juntendo University School of Medicine

Jul 16, 2026, cs.LGJ/K move · Enter open · S save

Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap

Olivier Jeunen

Independent Researcher Antwerp, Belgium

Variance Reduction Heterogeneous Treatment Effects Variance Learning-Augmented Algorithms Counterfactual Learning Evaluation Benchmarks

Open paper AskPDF arXiv

Online controlled experiments are the gold standard for hypothesis testing in online platforms. Notwithstanding their ubiquity, they are notoriously expensive to run, and issues of variance hamper statistical power in assessing treatment effects. While standard variance reduction techniques leverage model-based control variates to reduce outcome noise, they remain agnostic to potential structural relationships between competing policies. In this work, we identify a critical inefficiency in the standard A/B-testing protocol: when a treatment and control policy agree on an action, the resulting outcome contributes noise but no signal regarding the treatment effect -- unnecessarily inflating confidence intervals. We propose a novel experimental protocol that exploits this policy overlap to accelerate experimentation. The key insight is to frame the randomised treatment assignment mechanism as a meta-policy, and leverage

Δ

-Off-Policy Estimation methods to obtain unbiased estimates for average treatment effects. We prove analytically that our approach recovers standard A/B-testing practices in the general case, but that its variance scales with the divergence between policies rather than raw outcome variance. Hence, we dominate the standard Difference-in-Means estimator whenever policies have common support, and the improvement is strict whenever the overlap region contributes non-zero residual variance. Empirical results corroborate these theoretical insights -- holding promise for significant impact on the real-world evaluation of recommender systems, information retrieval pipelines, and large language model interfaces.

Abstract

Explore similar work

Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap

Robust Sequential Experimental Design for A/B Testing

When AI Generates Covariates: Causal Typing and Estimand Drift in Sequential Experiments