Policy-Aware Design of Large-Scale Factorial Experiments
Authors: Xin Wen, Xi Chen, Will Wei Sun, Yichen Zhang
Organizations: Stern School of Business, New York University, New York, NY 10012 · Daniels School of Business, Purdue University, West Lafayette, IN 47907
Digital firms routinely run many online experiments on shared user populations. When product decisions are compositional, such as combinations of interface elements, flows, messages, or incentives, the number of feasible interventions grows combinatorially, while available traffic remains limited. Overlapping experiments can therefore generate interaction effects that are poorly handled by decentralized A/B testing. We study how to design large-scale factorial experiments when the objective is not to estimate every treatment effect, but to identify a high-performing policy under a fixed experimentation budget. We propose a two-stage design that centralizes overlapping experiments into a single factorial problem and models expected outcomes as a low-rank tensor. In the first stage, the platform samples a subset of intervention combinations, uses tensor completion to infer performance on untested combinations, and eliminates weak factor levels using estimated marginal contributions. In the second stage, it applies sequential halving to the surviving combinations to select a final policy. We establish gap-independent simple-regret bounds and gap-dependent identification guarantees showing that the relevant complexity scales with the degrees of freedom of the low-rank tensor and the separation structure across factor levels, rather than the full factorial size. In an offline evaluation based on a product-bundling problem constructed from 100 million Taobao interactions, the proposed method substantially outperforms one-shot tensor completion and unstructured best-arm benchmarks, especially in low-budget and high-noise settings. These results show how centralized, policy-aware experimentation can make combinatorial product design operationally feasible at platform scale.
Figures & tables
Figure 1: Evolution of row-wise maxima during tensor halving. The top row (a-c) displays the unfoldings for Modes 1, 2, and 3 of the original tensor. The bottom row (d-e) tracks the Mode-1 unfolding across the subsequent first and second halving rounds.
μk,i(ℓ)=maxj∈∏k′=kAk′(ℓ)T(i,j)(ℓ).
Algorithm 1 Two-stage algorithm
Tℓ←⌊∣Sℓ∣LNII⌋
Algorithm 2 Sequential Halving (SH)
Figure 2: Heatmap of the mode-1 matricization M1(X) (unfolding of the original tensor along mode 1).
Figure 3: Plot of the singular values for each mode unfolding. The rapid decay justifies the low-rank assumption, with the first two components capturing over 95% of the total energy.
Figure 4: Comparative performance across varying noise levels and budgets. The Two-stage algorithm consistently outperforms the One-shot approach, particularly in high-noise and low-budget regimes.
G=X×1X1⊤×2⋯×mXm⊤.
Algorithm 3 HOSVD truncation
G1=p−1Y×1(X11)⊤×2⋯×m(Xm1)⊤.
Algorithm 4 Spectral initialization with diagonal deletion
Zt=Xt−p1PTXt(PΩ(Xt)−Y).
Algorithm 5 Riemannian gradient descent on the Tucker manifold