cs.HCMar 1, 2026
SaveSimAB: Simulating A/B Tests with Persona-Conditioned AI Agents for Rapid Design Evaluation
Organizations: ETH Zurich, Zurich, Switzerland · Adobe, Munich, Germany · Adobe, Basel, Switzerland · Adobe Research, Basel, Switzerland
Abstract
A/B testing is a standard method for validating design decisions, yet its reliance on real user traffic limits iteration speed and makes certain experiments impractical. We present SimAB, a system that reframes A/B testing as a fast, privacy-preserving simulation using persona-conditioned AI agents. Given design screenshots and a conversion goal, SimAB generates user personas, deploys them as agents that state their preference, aggregates results, and synthesizes rationales. Through a formative study with experimentation practitioners, we identified scenarios where traffic constraints hinder testing, including low-traffic pages, multi-variant comparisons, micro-optimizations, and privacy-sensitive contexts. Our design emphasizes speed, early feedback, actionable rationales, and audience specification. We evaluate SimAB against 47 historical A/B tests with known outcomes, achieving 67% overall accuracy, increasing to 83% for high-confidence cases. Additional experiments show robustness to naming and positional bias and demonstrate accuracy gains from personas. Practitioner feedback suggests that SimAB supports faster evaluation cycles and rapid screening of designs difficult to assess with traditional A/B tests.
Figures & tables
Figure 1. The SimAB pipeline reframes A/B testing as a fast simulation. (1) Input Processing : The system ingests a test specification, containing design variants, conversion goals, target audience definitions, and documents to provide context. (2) Persona Generation : It creates a diverse population of synthetic users (e.g., “Marketer Mia”) conditioned on the specific context. (3) Persona Simulation : Persona-driven agents interact with the designs. (4) Summary Generation : Individual verdicts and rationales are aggregated into actionable insights. This workflow reduces feedback latency from months to minutes. A pipeline diagram showing four stages. 1. Input Processing shows Control and Challenger screenshots with goals. 2. Persona Generation shows a neural network generating persona cards like 'Teacher Theo' and 'Marketer Mia'. 3. Persona Simulation shows the agents interacting with the designs via a counterbalancer and action predictor. 4. Summary Generation shows a funnel aggregating verdicts into pros/cons and actionable insights. The figure illustrates the SimAB pipeline for simulated A/B testing, organized into four stages from left to right. Input Processing: Two design variants, labeled Control and Challenger, are shown as webpage screenshots. Additional inputs include a conversion goal, such as increasing sign-ups, a target audience definition, such as young professionals, and optional context documents, such as analysis results. Persona Generation: A persona generator model creates multiple synthetic user personas. Example persona cards include different roles and backgrounds, such as teacher, marketer, developer, researcher, and others, representing diverse user perspectives. Persona Simulation: Each persona evaluates both design variants. The variants are labeled neutrally as “Image 1” and “Image 2” and passed through a counterbalancer to mitigate ordering bias. Personas interact with the designs through an action predictor that simulates navigation and next-screen behavior. Each persona produces a verdict and a rationale. Summary Generation: Individual verdicts and rationales are aggregated into a summary. The output includes a final verdict, grouped pros for the control and challenger designs, and a list of actionable design insights.
Figure 2. Example Input. Control can be found on the left; Challenger on the right. Together with a primary conversion goal (‘Will users donate to the Wikimedia Foundation?’), this constitutes the mandatory input to start an A/B test using SimAB . The test itself is created for this work, using an old Wikipedia A/B test from 2010 fitted to the current Wikipedia page for A/B testing . They tested 2 different group photo positions to determine the more effective one in terms of Wikimedia Foundation donations. The figure shows two side-by-side screenshots of the Wikipedia article “A/B testing,” illustrating two design variants used in an experiment. Both versions display the same article content, navigation sidebar, and page structure. The primary difference appears at the top of the article body. In the left variant, a banner reads “Please read: An appeal from our staff,” with a compact image and a call-to-action button. In the right variant, the same appeal appears as a full-width banner with a wider group photo.
Figure 3. Comparing the target distribution as provided in audience restrictions from a user to the distribution generated by the SimAB persona generation module. Blue: baseline without providing segment information to the LLM call, orange: using the segments as audience restriction to align generated personas on it. Left: absolute mean squared error (MSE), right: root mean squared error (RMSE). Term Frequency-Inverse Document Frequency (TF-IDF): evaluates a word’s importance in a document relative to the whole corpus; using cosine-similarity to measure the similarity of words in two collections. Both TF-IDF and direct requests to LLMs were used to classify generated personas to the most similar persona segment (the alignment objective provided in audience restrictions ). Note that outliers (RMSE) in particular are reduced significantly. The figure presents two bar charts comparing persona generation accuracy with and without audience alignment. Blue bars show a baseline without audience restrictions, and orange bars show results with audience alignment. The left chart reports mean absolute error, and the right chart reports root mean squared error. Results are shown for TF-IDF and OpenAI embeddings. In both metrics and embedding types, audience alignment reduces error, with a larger reduction visible in root mean squared error, indicating fewer large deviations.
Figure 4. Example Personas. The generated personas include a context, ranging from age, education, and background, to the present context (the main goals in the current browser session). Four example user personas shown as side-by-side cards. The personas are Growth focused Product Manager, Academic Researcher or PhD Student, E commerce Small Business Owner, and UX Designer or Freelance Consultant. Each card lists age range, education level, income level, tech savviness, key interests, and main goals related to A/B testing and experimentation.
Figure 5. Example Winner. After just 31 agents (1-2 minutes, all started in parallel) conclude, statistical significance is reached. The result coincides with the ground truth data gathered from real users. A results screen comparing two variants. Control has 6 votes, and Challenger has 25 votes. The interface highlights Challenger as the preferred option, notes a 95% confidence difference between variants, and states that the AI-based simulation suggests validation before confirmatory testing with real users.
Figure 6. Example Insights. The rationales of all agents are aggregated to show the top reasons why Control / Challenger / None is preferred (for those that preferred Control / Challenger / None ). Additionally, SimAB proposes immediate action items to improve and iterate on the Challenger. An insights dashboard summarizing reasons for preference. Three columns list the top reasons for Control, Challenger, and None, with bullets comparing banner clarity, visual load, human imagery, CTA visibility, and engagement. A section below presents actionable design recommendations to refine the Challenger through adjustments to imagery, headline clarity, CTA contrast, and layout.
| Ground Truth | ||||
| Challenger | Control | Precision | ||
| Predicted | Challenger | 22 (TP) | 9 (FP) | 71% |
| Control | 6 (FN) | 8 (TN) | ||
| Recall | 79% | Acc: 67% | ||
Table 1. Confusion matrix SimAB achieved on our corpus of 47 diverse historical A/B tests. Note that two tests did not reach statistical significance and subsequently are not taken into account for the confusion matrix.
Figure 7. Trade-off between agent count and accuracy. We plot the cumulative accuracy for the subset of experiments that concluded with agents. Experiments that reach statistical significance quickly exhibit substantially higher accuracy compared to those requiring more agents. A line chart shows agent accuracy versus the number of agents. Accuracy is higher with fewer agents and gradually declines as agent count increases, illustrating a trade-off between speed to significance and overall accuracy.
Figure 8. Consistency analysis across four independent runs of the same experiment. While exact vote counts fluctuate slightly due to LLM stochasticity, the system consistently identifies the same winner (Challenger) with a significant margin. A stacked bar chart compares vote counts for Control, Challenger, and Neither across four runs. Challenger receives the highest number of votes in every run, showing consistent winner identification despite minor variation in counts.
Figure 9. Persona Consistency Analysis. We ran each persona 20 times on identical inputs. The x-axis shows the number of deviation votes (minority votes). The steep drop-off indicates that individual personas hold stable, deterministic preferences, with the majority showing 0 or 1 deviation out of 20 runs. A line chart shows persona frequency by minority vote count. Most personas show zero or one minority vote, with a sharp drop as deviation increases, indicating stable and consistent preferences.
| Counter- balancing | Neutral Names | Control | Challenger | None | p-value |
| ✗ | ✗ | 126 | 749 | 125 | 0.0000 |
| ✗ | ✓ | 52 | 887 | 61 | 0.0000 |
| ✓ | ✗ | 442 | 434 | 124 | 0.8131 |
| ✓ | ✓ | 460 | 469 | 71 | 0.7930 |
Table 2. Validation for our bias mitigation techniques counterbalancing and neutral names. The table displays the number of votes both variants received during a test conducted with identical images to isolate bias.
Figure 10. Persona Diversity Ablation study. We show the effect of not using personas at all (that is, a single fixed persona) versus multiple adaptive personas per experiment, subsequently increasing accuracy. A bar chart compares accuracy across random choice, no diversity, high diversity, and baseline conditions. Accuracy increases when multiple diverse personas are used, outperforming random and single-persona setups.
Figure 11. Voting distributions by different persona splits: By age, profession archetype, income level, and technical savviness. Vote share for control is given in blue vs. vote share for challenger in orange by group. Four stacked bar charts show vote share for Control and Challenger by age, archetype, income level, and technical savviness. Challenger receives a higher vote share across most groups, although different segments of generated personas vote significantly different.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Attribute | Persona 1 | Persona 2 | Persona 3 |
| Name | “Creative Director Carlos” | “Freelance Designer Priya” | “Student Creator Alex” |
| Age | 38–45 | 28–32 | 19–22 |
| Occupation | Creative Director at ad agency | Freelance graphic designer | University student, design major |
| Income | High ($120k+) | Medium ($50–70k) | Low (part-time work) |
| Goals | Team license, reliable support | Cost-effective solo plan | Student discount, essential features |
| Pain Points | Complex procurement process | Unpredictable income, need flexibility | Limited budget, overwhelmed by options |
Table 3. Example personas generated for a creative software pricing page.
More in this topic
A/b Testing
A/B testing compares product variants through controlled experiments to estimate causal effects and guide decisions.
See all 29 papers