cs.AINov 3, 2026

Learnings and Pitfalls from AI-Assisted A/B Experimentation at Scale

Authors: Brett Daley, Ilya Gorbachev, Anubhav Natani, Nicholas Brown, Athena Dai, Rachel Song, Marcin Pekalski, Josip Domazet, 

Organizations: Meta

Abstract

A/B testing underpins product decisions by providing trustworthy, reproducible quantification of proposed changes' effects. Growing large language model (LLM) capabilities have motivated efforts to automate experimentation workflows. Yet LLMs are nondeterministic and prone to hallucination, properties that conflict with the trust experimentation systems must provide. Experiments are also operationally messy: diagnosing anomalies, debugging configurations, and conducting bespoke follow-up analyses require flexible reasoning and manual effort. These tasks let agents accelerate experimentation without replacing its deterministic statistical foundations. We report lessons from building and operating such an agent. Our central finding is that this tension is resolved not by making the model deterministic, but by bounding where its nondeterminism can reach; the model selects and sequences procedures, while deterministic code computes every quantity it reports. Three consequences surprised us: withholding information from the model improved its answers, demand concentrated on mundane retrieval rather than the autonomous decisions we anticipated, and the agent's eagerness to help threatened statistical validity.

Figures & tables

Explore similar work

CardsList
  1. Statistical Foundations of LLM-based A/B Testing: A Surrogacy Framework for Human Causal Inference

    Jun 15, 2026Joel Persson, Mårten Schultzberg, Sebastian AnkargrenSurrogate ModelingA/b Testing

  2. Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research

    Aug 27, 2026Lezhi Yu, Xiaogang Xu, Yuhua Zhou +2AI Agent AuditingAI-Assisted Scientific Research

  3. Augmented Hypothesis Testing with Persona-Based LLM Simulations

    Sep 21, 2026Ziyad Benomar, Aymen Al Marjani, Paul Missault +1A/b TestingLearning-Augmented Algorithms