stat.MLApr 9, 2026

Policy-Aware Design of Large-Scale Factorial Experiments

Authors: Xin Wen, Xi Chen, Will Wei Sun, Yichen Zhang

Organizations: Stern School of Business, New York University, New York, NY 10012 · Daniels School of Business, Purdue University, West Lafayette, IN 47907

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Digital firms routinely run many online experiments on shared user populations. When product decisions are compositional, such as combinations of interface elements, flows, messages, or incentives, the number of feasible interventions grows combinatorially, while available traffic remains limited. Overlapping experiments can therefore generate interaction effects that are poorly handled by decentralized A/B testing. We study how to design large-scale factorial experiments when the objective is not to estimate every treatment effect, but to identify a high-performing policy under a fixed experimentation budget. We propose a two-stage design that centralizes overlapping experiments into a single factorial problem and models expected outcomes as a low-rank tensor. In the first stage, the platform samples a subset of intervention combinations, uses tensor completion to infer performance on untested combinations, and eliminates weak factor levels using estimated marginal contributions. In the second stage, it applies sequential halving to the surviving combinations to select a final policy. We establish gap-independent simple-regret bounds and gap-dependent identification guarantees showing that the relevant complexity scales with the degrees of freedom of the low-rank tensor and the separation structure across factor levels, rather than the full factorial size. In an offline evaluation based on a product-bundling problem constructed from 100 million Taobao interactions, the proposed method substantially outperforms one-shot tensor completion and unstructured best-arm benchmarks, especially in low-budget and high-noise settings. These results show how centralized, policy-aware experimentation can make combinatorial product design operationally feasible at platform scale.

Figures & tables

Figure 1: Evolution of row-wise maxima during tensor halving. The top row (a-c) displays the unfoldings for Modes 1, 2, and 3 of the original tensor. The bottom row (d-e) tracks the Mode-1 unfolding across the subsequent first and second halving rounds.
 μ^k,i(ℓ)=max⁡j∈∏k′≠kAk′(ℓ)T^(i,j)(ℓ).\widehat{\mu}^{(\ell)}_{k,i}=\max_{\bm{j}\in\prod_{k^{\prime}\neq k}\mathcal{A}_{k^{\prime}}^{(\ell)}}\widehat{\mathcal{T}}^{(\ell)}_{(i,\bm{j})}. 
Algorithm 1 Two-stage algorithm
 Tℓ  ←  ⌊NII∣Sℓ∣ L⌋T_{\ell}\;\leftarrow\;\left\lfloor\frac{N_{\mathrm{II}}}{|\mathcal{S}_{\ell}|\,L}\right\rfloor 
Algorithm 2 Sequential Halving (SH)
Figure 2: Heatmap of the mode-1 matricization M1(X)\mathcal{M}_{1}(\mathcal{X}) (unfolding of the original tensor along mode 1).
Figure 3: Plot of the singular values for each mode unfolding. The rapid decay justifies the low-rank assumption, with the first two components capturing over 95% of the total energy.
Figure 4: Comparative performance across varying noise levels and budgets. The Two-stage algorithm consistently outperforms the One-shot approach, particularly in high-noise and low-budget regimes.
 G=X×1X1⊤×2⋯×mXm⊤.\mathcal{G}=\mathcal{X}\times_{1}X_{1}^{\top}\times_{2}\cdots\times_{m}X_{m}^{\top}. 
Algorithm 3 HOSVD truncation
 G1=p−1Y×1(X11)⊤×2⋯×m(Xm1)⊤.\mathcal{G}^{1}=p^{-1}Y\times_{1}(X_{1}^{1})^{\top}\times_{2}\cdots\times_{m}(X_{m}^{1})^{\top}. 
Algorithm 4 Spectral initialization with diagonal deletion
 Zt=Xt−1p PTXt ⁣(PΩ(Xt)−Y).\mathcal{Z}^{t}=\mathcal{X}^{t}-\frac{1}{p}\,\mathcal{P}_{T_{\mathcal{X}^{t}}}\!\bigl(\mathcal{P}_{\Omega}(\mathcal{X}^{t})-Y\bigr). 
Algorithm 5 Riemannian gradient descent on the Tucker manifold
More in this topic

A/b Testing

A/B testing compares product variants through controlled experiments to estimate causal effects and guide decisions.

See all 29 papers

Explore similar work

CardsList
Jul 16, 2026

Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap

Online controlled experiments are the gold standard for hypothesis testing in online platforms. Notwithstanding their ubiquity, they are notoriously expensive to run, and issues of variance hamper statistical power in assessing treatment effects. While standard variance reduction techniques leverage model-based control variates to reduce outcome noise, they remain agnostic to potential structural relationships between competing policies. In this work, we identify a critical inefficiency in the standard A/B-testing protocol: when a treatment and control policy agree on an action, the resulting outcome contributes noise but no signal regarding the treatment effect -- unnecessarily inflating confidence intervals. We propose a novel experimental protocol that exploits this policy overlap to accelerate experimentation. The key insight is to frame the randomised treatment assignment mechanism as a meta-policy, and leverage ΔΔ-Off-Policy Estimation methods to obtain unbiased estimates for average treatment effects. We prove analytically that our approach recovers standard A/B-testing practices in the general case, but that its variance scales with the divergence between policies rather than raw outcome variance. Hence, we dominate the standard Difference-in-Means estimator whenever policies have common support, and the improvement is strict whenever the overlap region contributes non-zero residual variance. Empirical results corroborate these theoretical insights -- holding promise for significant impact on the real-world evaluation of recommender systems, information retrieval pipelines, and large language model interfaces.

Olivier Jeunen

Independent Researcher Antwerp, Belgium

Causal Effect EstimationA/b TestingVariance Reduction
AskPDF
May 24, 2026

Choosing Online Experiment Designs under Interference in Ads, Recommendations, and Member-Experience Systems

Online experiments in ads, recommendation, and member-experience systems are often planned before the dominant interference mechanism is known. A treatment may propagate through budgets, inventory, producer exposure, graph spillovers, or temporal carryover, making the randomization design itself a statistical decision. We formulate this problem as robust design selection over uncertain exposure mechanisms. Given a finite catalog of six implementable designs, the selector compares each design by worst-case planning risk over an ambiguity set. The risk combines exposure bias, assignment-unit variance, minimum detectable effect, contamination or carryover, operational cost, and estimand mismatch. For theoretical justification, the paper develops a geometry-aware guarantee, stating that design bias is bounded by Wasserstein distance to the launch exposure distribution, and this penalty is minimax tight under Lipschitz exposure response. We also prove finite-catalog approximation and a robust selector theorem with excess-risk control, exact recovery under separation, and certified shortlists when the risk surface is flat. Empirically, the same selector gives different recommendations across samples from public datasets. It selects user-randomization on Criteo ads with dimensionless robust risk 1.295, switchbacks on Open Bandit-bts/men with risk 2.105, and cluster-randomization on KuaiRand with risk 2.240. The Open Bandit case stresses known but uneven logging support, with propensities from 0.00006 to 0.594 and a 5.17% IPS effective-sample share. Overall, the paper contributes an interference-aware experiment design framework based on mechanism-robust design decisions, where the output is either a justified design choice or an uncertainty shortlist.

Prashant Shekhar, Caroline Howard

Department of Mathematics Embry-Riddle Aeronautical University Daytona Beach, FL, USA

Distributionally Robust Optimization
AskPDF
Oct 5, 2026

Incentive Alignment in Online Experimentation

Evaluating the causal effect of new features is a central goal for online platforms. While recent literature addresses limited testing traffic via centralized portfolio optimization, this perspective abstracts away a critical institutional reality: experimentation is operationally decentralized. The experimenters who develop new features also dictate which hypotheses to test, and they are typically rewarded based on empirical average treatment effects that are prone to upward bias. Left unchecked, this principal-agent conflict can severely erode platform value, a structural failure that conventional centralized levers, such as significance thresholds and traffic budgets, cannot resolve. By reframing experimentation as an incentive design problem, we demonstrate that two practical mechanisms, sample splitting and shrinkage, can effectively bridge this gap. Sample splitting aligns incentives perfectly at a bounded traffic cost, while shrinkage consumes no additional traffic and guarantees that interventions with negative expected effects are strictly unprofitable to field.

Ermis Soumalias, Richard Mudd, Abbas Zaidi

Meta Platforms, Inc.

Causal Effect EstimationA/b TestingShrinkage Estimation
AskPDF