stat.MLMay 24, 2026

Choosing Online Experiment Designs under Interference in Ads, Recommendations, and Member-Experience Systems

Authors: Prashant Shekhar, Caroline Howard

Organizations: Department of Mathematics Embry-Riddle Aeronautical University Daytona Beach, FL, USA

Abstract

Online experiments in ads, recommendation, and member-experience systems are often planned before the dominant interference mechanism is known. A treatment may propagate through budgets, inventory, producer exposure, graph spillovers, or temporal carryover, making the randomization design itself a statistical decision. We formulate this problem as robust design selection over uncertain exposure mechanisms. Given a finite catalog of six implementable designs, the selector compares each design by worst-case planning risk over an ambiguity set. The risk combines exposure bias, assignment-unit variance, minimum detectable effect, contamination or carryover, operational cost, and estimand mismatch. For theoretical justification, the paper develops a geometry-aware guarantee, stating that design bias is bounded by Wasserstein distance to the launch exposure distribution, and this penalty is minimax tight under Lipschitz exposure response. We also prove finite-catalog approximation and a robust selector theorem with excess-risk control, exact recovery under separation, and certified shortlists when the risk surface is flat. Empirically, the same selector gives different recommendations across samples from public datasets. It selects user-randomization on Criteo ads with dimensionless robust risk 1.295, switchbacks on Open Bandit-bts/men with risk 2.105, and cluster-randomization on KuaiRand with risk 2.240. The Open Bandit case stresses known but uneven logging support, with propensities from 0.00006 to 0.594 and a 5.17% IPS effective-sample share. Overall, the paper contributes an interference-aware experiment design framework based on mechanism-robust design decisions, where the output is either a justified design choice or an uncertainty shortlist.

Explore similar work

CardsList
Jul 9, 2026cs.LG

Robust Bayesian Decision Making under Adversarial Uncertainty

Scientific experiments are often designed to maximize information gain, yet in many applications the primary objective is to support reliable downstream decision-making. Existing decision-aware experimental design and active learning methods typically assume well-specified outcome models and implicitly rely on the stability of the optimal decision under real-world perturbations. In practice, however, experimental outcomes are frequently influenced by hidden or weakly modeled effects, which can substantially alter decision optimality and lead to misleading conclusions. We study sequential adversarially robust decision-aware experimental design, where data acquisition has to take into account information gain against plausible worst-case unexpected effects, modeled here as variation in adversarial variables. Building on Bayesian decision theory, we formalize an adversarially robust optimal decision under this setting and derive a principled Bayesian experimental design criterion. The criterion explicitly targets decision stability rather than nominal optimality. Experiments on synthetic and real-world scientific datasets show that conventional decision-aware design can converge rapidly to high confidence yet fragile decisions, while our robustness-aware approach yields decisions that are significantly more stable and reliable under adversarial variation.
Haripriya Harikumar, Sammie Katt, Yasir Zubayr Barlas +1
Bayesian Experimental DesignBayesian Inference ApproachesDynamic Regret BoundsAdversariesDecisions
AskPDF
May 13, 2026stat.ML

Robust Sequential Experimental Design for A/B Testing

Experimental design has emerged as a powerful approach for improving the sample efficiency of A/B testing, yet existing designs rely critically on correctly specified models. We study robust sequential experimental design under model misspecification and develop a unified framework that covers both contextual bandit and dynamic settings. Theoretically, we prove that our design bounds the worst-case mean squared error of the estimated treatment effect. Empirically, we demonstrate the effectiveness of the proposed approach using synthetic and real-world datasets from a leading technology company.
Qianglin Wen, Xiangkun Wu, Chengchun Shi +4
Bayesian Experimental DesignTwo-Sample TestingHeterogeneous Treatment EffectsControlled Study
AskPDF
Jul 24, 2026cs.LG

Discrepancy-Rounded Fair Bandits with Static and Time-Varying Exposure Floors

Minimum-exposure constraints arise in recommendation, content curation, and regulated allocation when each provider, arm, or group must receive guaranteed exposure inside a period rather than only in aggregate. We study stochastic bandits with exact exposure floors and show that the right object is a rounding problem: a fractional fair schedule is realized as integral pulls, and the exposure error is exactly a discrepancy vector. The main contribution is a blockwise model with time-varying floors. BDQ-UCB satisfies every block floor deterministically and has fair regret governed by the nonmandatory budget RR, not the horizon TT, with high-probability regret O(KRlog⁡(KT))O(\sqrt{KR\log(KT)}). A MOSS residual variant attains O(KR)O(\sqrt{KR}), and a matching lower bound gives the minimax rate Θ(KR)Θ(\sqrt{KR}), even with positive mandatory exposure; a kl-UCB++^{++} residual rule adds instance-dependent optimality. The formulation becomes essential for overlapping group floors: per-arm rounding can violate a group constraint by Ω(s)Ω(s) in the group size, whereas Beck--Fiala null-space rounding meets every group floor within the block budget with violation below the arm degree tt, and composes with UCB at the same RR-parametrized regret. For learned group plans, we close disjoint systems at Θ~(KT)\widetildeΘ(\sqrt{KT}), give a dual-ledger decomposition explaining why naive index rules fail under overlap, and prove a plan-sampling rule that is pathwise feasible under an initial cover-slack condition and attains a conditional O~(KT)\widetilde O(\sqrt{KT}) guarantee, leaving the condition-free overlap rate open. Experiments on synthetic floors, MovieLens-100k genre exposure, and deployment stress tests show exact feasibility without penalty tuning and regret competitive with tuned Lagrangian baselines.
Ibne Farabi Shihab, Joyanta Jyoti Mondal, Anuj Sharma
BanditsOptimal Comparison AllocationExposureCeiling
AskPDF