SimAB: Simulating A/B Tests with Persona-Conditioned AI Agents for Rapid Design Evaluation
Authors: Tim Rieder, Marian Schneider, Mario Truss, Vitaly Tsaplin, Alina Rublea, Sinem Dere, Francisco Chicharro Sanz, Tobias Reiss, Mustafa Doga Dogan
A/B testing is a standard method for validating design decisions, yet its reliance on real user traffic limits iteration speed and makes certain experiments impractical. We present SimAB, a system that reframes A/B testing as a fast, privacy-preserving simulation using persona-conditioned AI agents. Given design screenshots and a conversion goal, SimAB generates user personas, deploys them as agents that state their preference, aggregates results, and synthesizes rationales. Through a formative study with experimentation practitioners, we identified scenarios where traffic constraints hinder testing, including low-traffic pages, multi-variant comparisons, micro-optimizations, and privacy-sensitive contexts. Our design emphasizes speed, early feedback, actionable rationales, and audience specification. We evaluate SimAB against 47 historical A/B tests with known outcomes, achieving 67% overall accuracy, increasing to 83% for high-confidence cases. Additional experiments show robustness to naming and positional bias and demonstrate accuracy gains from personas. Practitioner feedback suggests that SimAB supports faster evaluation cycles and rapid screening of designs difficult to assess with traditional A/B tests.
Figures & tables
Figure 1. The SimAB pipeline reframes A/B testing as a fast simulation. (1) Input Processing : The system ingests a test specification, containing design variants, conversion goals, target audience definitions, and documents to provide context. (2) Persona Generation : It creates a diverse population of synthetic users (e.g., “Marketer Mia”) conditioned on the specific context. (3) Persona Simulation : Persona-driven agents interact with the designs. (4) Summary Generation : Individual verdicts and rationales are aggregated into actionable insights. This workflow reduces feedback latency from months to minutes. A pipeline diagram showing four stages. 1. Input Processing shows Control and Challenger screenshots with goals. 2. Persona Generation shows a neural network generating persona cards like 'Teacher Theo' and 'Marketer Mia'. 3. Persona Simulation shows the agents interacting with the designs via a counterbalancer and action predictor. 4. Summary Generation shows a funnel aggregating verdicts into pros/cons and actionable insights. The figure illustrates the SimAB pipeline for simulated A/B testing, organized into four stages from left to right. Input Processing: Two design variants, labeled Control and Challenger, are shown as webpage screenshots. Additional inputs include a conversion goal, such as increasing sign-ups, a target audience definition, such as young professionals, and optional context documents, such as analysis results. Persona Generation: A persona generator model creates multiple synthetic user personas. Example persona cards include different roles and backgrounds, such as teacher, marketer, developer, researcher, and others, representing diverse user perspectives. Persona Simulation: Each persona evaluates both design variants. The variants are labeled neutrally as “Image 1” and “Image 2” and passed through a counterbalancer to mitigate ordering bias. Personas interact with the designs through an action predictor that simulates navigation and next-screen behavior. Each persona produces a verdict and a rationale. Summary Generation: Individual verdicts and rationales are aggregated into a summary. The output includes a final verdict, grouped pros for the control and challenger designs, and a list of actionable design insights.
Figure 2. Example Input. Control can be found on the left; Challenger on the right. Together with a primary conversion goal (‘Will users donate to the Wikimedia Foundation?’), this constitutes the mandatory input to start an A/B test using SimAB . The test itself is created for this work, using an old Wikipedia A/B test from 2010 fitted to the current Wikipedia page for A/B testing . They tested 2 different group photo positions to determine the more effective one in terms of Wikimedia Foundation donations. The figure shows two side-by-side screenshots of the Wikipedia article “A/B testing,” illustrating two design variants used in an experiment. Both versions display the same article content, navigation sidebar, and page structure. The primary difference appears at the top of the article body. In the left variant, a banner reads “Please read: An appeal from our staff,” with a compact image and a call-to-action button. In the right variant, the same appeal appears as a full-width banner with a wider group photo.
Figure 3. Comparing the target distribution as provided in audience restrictions from a user to the distribution generated by the SimAB persona generation module. Blue: baseline without providing segment information to the LLM call, orange: using the segments as audience restriction to align generated personas on it. Left: absolute mean squared error (MSE), right: root mean squared error (RMSE). Term Frequency-Inverse Document Frequency (TF-IDF): evaluates a word’s importance in a document relative to the whole corpus; using cosine-similarity to measure the similarity of words in two collections. Both TF-IDF and direct requests to LLMs were used to classify generated personas to the most similar persona segment (the alignment objective provided in audience restrictions ). Note that outliers (RMSE) in particular are reduced significantly. The figure presents two bar charts comparing persona generation accuracy with and without audience alignment. Blue bars show a baseline without audience restrictions, and orange bars show results with audience alignment. The left chart reports mean absolute error, and the right chart reports root mean squared error. Results are shown for TF-IDF and OpenAI embeddings. In both metrics and embedding types, audience alignment reduces error, with a larger reduction visible in root mean squared error, indicating fewer large deviations.
Figure 4. Example Personas. The generated personas include a context, ranging from age, education, and background, to the present context (the main goals in the current browser session). Four example user personas shown as side-by-side cards. The personas are Growth focused Product Manager, Academic Researcher or PhD Student, E commerce Small Business Owner, and UX Designer or Freelance Consultant. Each card lists age range, education level, income level, tech savviness, key interests, and main goals related to A/B testing and experimentation.
Figure 5. Example Winner. After just 31 agents (1-2 minutes, all started in parallel) conclude, statistical significance is reached. The result coincides with the ground truth data gathered from real users. A results screen comparing two variants. Control has 6 votes, and Challenger has 25 votes. The interface highlights Challenger as the preferred option, notes a 95% confidence difference between variants, and states that the AI-based simulation suggests validation before confirmatory testing with real users.
Figure 6. Example Insights. The rationales of all agents are aggregated to show the top reasons why Control / Challenger / None is preferred (for those that preferred Control / Challenger / None ). Additionally, SimAB proposes immediate action items to improve and iterate on the Challenger. An insights dashboard summarizing reasons for preference. Three columns list the top reasons for Control, Challenger, and None, with bullets comparing banner clarity, visual load, human imagery, CTA visibility, and engagement. A section below presents actionable design recommendations to refine the Challenger through adjustments to imagery, headline clarity, CTA contrast, and layout.
Ground Truth
Challenger
Control
Precision
Predicted
Challenger
22 (TP)
9 (FP)
71%
Control
6 (FN)
8 (TN)
Recall
79%
Acc: 67%
Table 1. Confusion matrix SimAB achieved on our corpus of 47 diverse historical A/B tests. Note that two tests did not reach statistical significance and subsequently are not taken into account for the confusion matrix.
Figure 7. Trade-off between agent count and accuracy. We plot the cumulative accuracy for the subset of experiments that concluded with ≤x agents. Experiments that reach statistical significance quickly exhibit substantially higher accuracy compared to those requiring more agents. A line chart shows agent accuracy versus the number of agents. Accuracy is higher with fewer agents and gradually declines as agent count increases, illustrating a trade-off between speed to significance and overall accuracy.
Figure 8. Consistency analysis across four independent runs of the same experiment. While exact vote counts fluctuate slightly due to LLM stochasticity, the system consistently identifies the same winner (Challenger) with a significant margin. A stacked bar chart compares vote counts for Control, Challenger, and Neither across four runs. Challenger receives the highest number of votes in every run, showing consistent winner identification despite minor variation in counts.
Figure 9. Persona Consistency Analysis. We ran each persona 20 times on identical inputs. The x-axis shows the number of deviation votes (minority votes). The steep drop-off indicates that individual personas hold stable, deterministic preferences, with the majority showing 0 or 1 deviation out of 20 runs. A line chart shows persona frequency by minority vote count. Most personas show zero or one minority vote, with a sharp drop as deviation increases, indicating stable and consistent preferences.
Counter- balancing
Neutral Names
Control
Challenger
None
p-value
✗
✗
126
749
125
0.0000
✗
✓
52
887
61
0.0000
✓
✗
442
434
124
0.8131
✓
✓
460
469
71
0.7930
Table 2. Validation for our bias mitigation techniques counterbalancing and neutral names. The table displays the number of votes both variants received during a test conducted with identical images to isolate bias.
Figure 10. Persona Diversity Ablation study. We show the effect of not using personas at all (that is, a single fixed persona) versus multiple adaptive personas per experiment, subsequently increasing accuracy. A bar chart compares accuracy across random choice, no diversity, high diversity, and baseline conditions. Accuracy increases when multiple diverse personas are used, outperforming random and single-persona setups.
Figure 11. Voting distributions by different persona splits: By age, profession archetype, income level, and technical savviness. Vote share for control is given in blue vs. vote share for challenger in orange by group. Four stacked bar charts show vote share for Control and Challenger by age, archetype, income level, and technical savviness. Challenger receives a higher vote share across most groups, although different segments of generated personas vote significantly different.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Attribute
Persona 1
Persona 2
Persona 3
Name
“Creative Director Carlos”
“Freelance Designer Priya”
“Student Creator Alex”
Age
38–45
28–32
19–22
Occupation
Creative Director at ad agency
Freelance graphic designer
University student, design major
Income
High ($120k+)
Medium ($50–70k)
Low (part-time work)
Goals
Team license, reliable support
Cost-effective solo plan
Student discount, essential features
Pain Points
Complex procurement process
Unpredictable income, need flexibility
Limited budget, overwhelmed by options
Appendix
Table 3. Example personas generated for a creative software pricing page.
A/B testing is the gold standard for evaluating product changes, but each experiment requires real user traffic, engineering effort, and weeks of measurement. We propose a simulation framework that predicts A/B test outcomes using LLM-powered agents conditioned on data-driven personas grounded in real user behavioral signals. Unlike prior work that relies on synthetic or rule-based personas, our agents are constructed from anonymized behavioral data - activity patterns, engagement signals, and inferred demographics - enabling more faithful population modeling. We frame A/B test simulation as a structured question task and systematically study (i) question design formats, (ii) the impact of persona data source and domain alignment, (iii) the trade-off between per-persona behavioral depth and population diversity, and (iv) efficient population subsampling. On a benchmark of 40 A/B tests spanning two metric types, our best configuration achieves 0.75-0.80 directional accuracy depending on the test metric, demonstrating that data-driven personas are a viable path toward fast, low-cost experiment pre-screening.
A/B testing remains the standard for rolling out new features in the technology industry. Each experiment, however, consumes real traffic, engineering effort, and weeks of wall-clock time. Can AI agents---conditioned on behavioral profiles and contextual descriptions of the intervention---simulate outcomes accurately enough to vet candidate treatments before committing live traffic? We formalize this question as a Simulated Randomized Controlled Trial (S-RCT) and derive a two-layer error decomposition that separates agent approximation error from subsampling error, enabling targeted improvements to each. The framework is agent-agnostic: any behavioral model---from a fine-tuned specialist to a general-purpose foundation model---can serve as the simulation engine. Validated on 67 historical marketing A/B tests, a baseline S-RCT using an off-the-shelf foundation model captures directional signal (sign overlap 0.70) but systematically overshoots effect magnitudes. A two-phase pre-period calibration protocol reduces the squared prediction error (after removing irreducible measurement noise) by ∼77×; a within-subject design---where each agent is exposed to both arms---reduces standard errors by ∼2.4×. We discuss limitations of the current approach and identify applications where experimenters stand to benefit from agentic signals.
A/B testing remains the gold standard for evaluating modifications to e-commerce storefronts, yet it diverts traffic, requires weeks to reach statistical significance, and risks degrading user experience. We present SimGym, a framework for simulating A/B tests on e-commerce storefronts using vision-language model (VLM) agents operating in a live browser. The framework comprises three key components: (a) a traffic-grounded persona generation pipeline that derives per-shop buyer archetypes and intents from production clickstream data; (b) a live-browser agent architecture that combines multimodal perception over visual and browser-structured observations with episodic memory and guardrails to conduct coherent shopping sessions across control and treatment storefronts; and (c) an evaluation protocol that compares simulated outcome shifts with observed shifts in real buyer behavior. We validate SimGym on A/B tests of visually driven UI theme changes from a major e-commerce platform across diverse storefronts and product categories. Empirical results show that SimGym agents achieve strong agreement with observed outcome shifts, attaining 77% directional alignment with add-to-cart shifts observed across interface variants in real-buyer traffic. It reduces experimental cycles from weeks to under an hour, enabling rapid experimentation without exposing real buyers to candidate variants.