Overview

A/B testing studies whether changing a product, policy, or interface causes measurable differences in outcomes such as clicks, revenue, or user retention. Researchers use randomized assignment and methods such as CUPED, regression adjustment, and stratification to improve precision, while sequential tests, bandits, and factorial designs help allocate traffic across many options. Work also addresses longer-term effects, marketplace interference, and whether offline or LLM-based predictions can stand in for outcomes measured with real users. The central tension is making experiments faster and cheaper without compromising valid inference, especially when effects vary across users or the experimental setting violates standard assumptions.

Latest papers

CardsList

33 papers4 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

  1. From More to Better: Causal Inference and Digital Experimentation in the Age of AI

    Nov 12, 2026Winston Chou, Mårten Schultzberg, Lo-Hua Yuan +18evidence synthesisCausal Inference

  2. Exploiting Feature-invariant Scaling Laws for Efficient A/B Experimentation in Recommendation Systems

    Nov 10, 2026Zhiliang Chen, Sammy Nammari, Jiahao Su +1Recommender SystemsA/b Testing

  3. Incentive Alignment in Online Experimentation

    Oct 5, 2026Ermis Soumalias, Richard Mudd, Abbas ZaidiCausal Effect EstimationA/b Testing

  4. Always-On Experimentation

    Sep 30, 2026Ricardo J. Sandoval, David Arbour, Avi Feller +1FDR ControlA/b Testing

  5. From Offline Proxies to Online Decisions: A Layered Engagement Evaluation Framework for Conversational AI

    Sep 21, 2026Xuanyi Li, Vaskar Nath, Hossein Amirkhani +2A/b TestingMulti-Turn Dialogue Evaluation

  6. Augmented Hypothesis Testing with Persona-Based LLM Simulations

    Sep 21, 2026Ziyad Benomar, Aymen Al Marjani, Paul Missault +1A/b TestingLearning-Augmented Algorithms

  7. Optimal Multi-way Decision Trees for Stratified Sampling in Online Controlled Experiments

    Sep 20, 2026Tomoka Takei, Shunnosuke Ikeda, Yuichi TakanoStratified SamplingVariance Reduction

  8. BAFF: Bid-Aware Filter Family for Mitigating Training Data Interference in RTB A/B Tests

    Sep 8, 2026Jeonglyul Oh, Ikkyu Choi, Inseop Youn +1Data SelectionA/b Testing

  9. Data-Driven Persona-Conditioned Agents for A/B Test Simulation

    Sep 1, 2026Ziyad Benomar, Weronika Łajewska, Leonardo Perelli +1Human Behavior SimulationA/b Testing

  10. Content Exploration Beyond the Feed: Creator Supply and the Shared Corpus

    Aug 29, 2026Yuanyuan Shen, Yiren Yan, Wenjie Li +1Recommender SystemsSocial Media Analysis

  11. Fast A/B/n Testing: Exact Multi-Policy Comparison via Tree-Coupled Feedback Sharing

    Aug 13, 2026Yuxiao WenPolicy EvaluationA/b Testing

  12. Offline Policy Evaluation as a decision support tool for designing Adaptive Experiments

    Aug 10, 2026João Victor Ferreira Alves, Eduardo Rocha Laurentino, Gustavo de Oliveira Kanno +1Contextual BanditsA/b Testing

  13. The Price of Isolation: Estimating the Ecosystem Cost of Symmetric Two-Sided A/B Testing

    Aug 5, 2026Yuanyuan Shen, Yiren Yan, Wenjie Li +1A/b Testing

  14. Offline-to-Online Creative Optimization with Generative Models and Adaptive Testing

    Jul 26, 2026Kevin Lee, Benjamin Letham, Zhiyuan Jerry Lin +5Online AdvertisingGenerative Modeling

  15. A More Accurate Algorithm Comparison through A/B Testing using Offline Evaluation Methods

    Jul 2, 2026Koki Konishi, Masataka Ushiku, Yuta SaitoAlgorithm SelectionA/b Testing

  16. Statistical Foundations of LLM-based A/B Testing: A Surrogacy Framework for Human Causal Inference

    Jun 15, 2026Joel Persson, Mårten Schultzberg, Sebastian AnkargrenSurrogate ModelingA/b Testing

  17. Variance Reduction for Heavy-Tailed Monetization Metrics in Ranking Experiments via Post-Stratification

    Jun 2, 2026Neeti Pokharna, Olivier Jeunen, Yatharth Saraf +1Causal Effect EstimationA/b Testing

  18. SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents

    May 19, 2026Han Li, Vibhor Malik, Zahra Zanjani Foumani +17Tool-Using Vision-Language AgentsA/b Testing

  19. Robust Sequential Experimental Design for A/B Testing

    May 13, 2026Qianglin Wen, Xiangkun Wu, Chengchun Shi +4Causal Effect EstimationModel Misspecification

  20. Efficient Multi-Cohort Inference for Long-Term Effects and Lifetime Value in A/B Testing with User Learning

    Apr 22, 2026Dario Simionato, Andrea Tonon, Mingxue Wang +3Causal Effect EstimationA/b Testing

  21. Policy-Aware Design of Large-Scale Factorial Experiments

    Apr 9, 2026Xin Wen, Xi Chen, Will Wei Sun +1A/b Testing