On Evaluating User Simulators for Interactive Agents
Organizations: UCLA · Arena
Abstract
User simulators offer a scalable way to evaluate interactive agents, but how to assess their quality remains underexplored. In particular, it is unclear whether existing user simulator evaluation frameworks reliably detect meaningful failures in simulated user behavior. This uncertainty makes it difficult to interpret simulator evaluation. To understand what a high benchmark score tells us about simulator quality, we (meta-)evaluate the metrics themselves. To instantiate this meta-evaluation, we deliberately perturb simulated replays to induce failures such as altered goals, omitted constraints, premature stopping, and acceptance of broken work, then test whether the metrics penalize these failures. Evaluation methods from three prior studies detect only 5-6 of the 10 perturbations; their combined scores even favor populations instructed to accept broken work over unmodified controls. Motivated by these gaps, we propose a new evaluation suite that combines utility of user simulators for evaluating agents and two measures of fidelity to real users: population-level fidelity assesses collective interaction patterns, while trace-level fidelity assesses the preservation of individual users' goals, constraints, and persona. Under the same detection criterion, our evaluation suite detects all 10 perturbations. Evaluating six user simulators on 351 real multi-turn sessions in which 348 distinct users across 23 languages and agents work together to create and refine artifacts such as full-stack web apps, games, design assets, and automated workflows, we find that population-level measures can miss when a simulated user pursues the wrong goal, ignores a constraint, or behaves out of character, highlighting the need to assess fidelity at comprehensive granularities.
Figures & tables
| Evaluation suite | Agent favoritism | Negative profile artifact | ||||||
| 10% | 25% | 50% | 100% | 10% | 25% | 50% | 100% | |
| Ours | 0.55 R | 0.69 R | 0.84 R | 0.99 R | 0.62 A | 0.75 A | 0.91 A | 1.00 A |
| SimulatorArena | 0.50 W | 0.54 W | 0.56 W | 0.64 W | 0.55 I | 0.60 I | 0.67 I | 0.84 I |
| VISTA | 0.49 K | 0.50 F | 0.50 F | 0.50 F | 0.60 F | 0.71 F | 0.86 F | 0.98 F |
| SWE-Together | 0.51 Q | 0.49 S | 0.49 S | 0.44 S | 0.67 U | 0.85 U | 0.98 U | 1.00 U |
| Sim2Real | 0.51 | 0.53 | 0.55 | 0.58 | 0.50 | 0.51 | 0.56 | 0.61 |
| Evaluation suite | Approve broken work | Premature stop | Front-loading | |||||||||
| 10% | 25% | 50% | 100% | 10% | 25% | 50% | 100% | 10% | 25% | 50% | 100% | |
| Ours, L2 | 0.49 D | 0.46 D | 0.45 P | 1.00 A | 0.49 P | 0.49 P | 0.49 P | 0.46 P | 0.56 A | 0.65 A | 0.78 A | 0.96 A |
| SimulatorArena | 0.53 W | 0.58 W | 0.66 W | 0.78 W | 0.53 T | 0.58 T | 0.62 T | 0.69 T | 0.56 W | 0.64 W | 0.74 W | 0.90 W |
| VISTA | 0.43 K | 0.31 K | 0.16 K | 0.01 K | 0.45 G | 0.38 G | 0.27 G | 0.12 G | 0.56 F | 0.64 F | 0.77 F | 0.92 F |
| SWE-Together | 0.50 S | 0.50 S | 0.52 S | 0.50 S | 0.69 Q | 0.88 Q | 0.99 Q | 1.00 C | 0.53 C | 0.59 C | 0.65 C | 0.80 S |
| Sim2Real | 0.49 | 0.48 | 0.45 | 0.55 | 0.48 | 0.45 | 0.44 | 0.60 | 0.50 | 0.52 | 0.56 | 0.61 |
| Control vs perturbed (control positive) | Real vs perturbed (real positive) | |||||
| Benchmark (best metric) | AUROC | control@5%FPR | control@10%FPR | AUROC | real@5%FPR | real@10%FPR |
| Ours | 0.62 persona | 0.10 | 0.17 | 0.90 persona | 0.68 | 0.77 |
| SimulatorArena | 0.55 Likert writing | 0.05 | 0.11 | 0.88 Likert writing | 0.28 | 0.55 |
| VISTA | 0.55 composite | 0.06 | 0.11 | 0.68 composite | 0.09 | 0.19 |
| SWE-Together | 0.52 coverage rate | 0.05 | 0.11 | 0.75 user correction | 0.12 | 0.24 |
| Simulator | L1 | L2 | L3 | Headline |
| Ours, Sonnet 5 backbone | 47 | 87 | 83 | 70 |
| SimulatorArena (gpt-5-mini) | 57 | 79 | 74 | 69 |
| Ours, Terra backbone | 42 | 73 | 84 | 64 |
| Ours, Luna backbone | 40 | 72 | 78 | 61 |
| -bench (gpt-4.1) | 33 | 82 | 74 | 59 |
| MIMESIS-9B | 27 | 82 | 61 | 51 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Perturbation | Scope | Change to the replay setup |
| Inspired by previous work | ||
| Adversarial goal | Trace-level fidelity | Replace the goal and deliverable with a plausible variant containing one decisive change. ( Zheng et al., 2026 ; Naous et al., 2026 ) |
| All constraints removed | Trace-level fidelity | Empty the profile’s constraint list (forgotten requirements; the real opener still states them). ( Zheng et al., 2026 ; Jain and Sandhu, 2026 ) |
| Approve broken work | Population-level fidelity | Instruct the user to accept delivered work and continue until at least its sixth message. ( Zhou et al., 2026 ; Jain and Sandhu, 2026 ) |
| Premature stop | Population-level fidelity | Cap the turn budget at half the real user’s turn count (minimum 2, maximum 15). ( Zheng et al., 2026 ; Jain and Sandhu, 2026 ) |
| Persona swap | Trace-level fidelity | Replace the register with terse-lowercase, bureaucratic, or formal-verbose. ( Zhou et al., 2026 ) |
| Metric | Agent favoritism | Negative profile artifact |
| (a) Change in Kendall’s : perturbed control [95% CI] | ||
| Ours: L1 ranking agreement | ||
| SimulatorArena: assistant rating | ||
| (b) Population-level win rate: control perturbed | ||
| L2: act distribution | 0.13 | 0.99 |
| L2: dynamics | 0.46 | 0.88 |
| Metric | Goal | AllC | Appr | Stop | Pers | Lang | CoD | Front |
|---|---|---|---|---|---|---|---|---|
| Ours, L2 population families (population metrics) | ||||||||
| act distribution | 0.30 | 0.49 | 1.00 | 0.00 | 0.49 | 0.77 | 0.72 | 1.00 |
| dynamics | 0.46 | 0.43 | 0.34 | 0.20 | 0.55 | 0.41 | 0.19 | 0.33 |
| outcome sensitivity | 0.45 | 0.49 | 0.12 | 0.09 | 0.80 | 0.82 | 0.84 | 0.80 |
| provenance | 0.54 | 0.40 | 0.60 | 0.44 | 0.90 | 0.62 | 0.94 | 0.97 |
| best metric | 0.54 provenance | 0.49 act | 1.00 act | 0.44 provenance | 0.90 provenance | 0.82 outcome | 0.94 provenance | 1.00 act |
| Metric | Goal | AllC | Appr | Stop |
|---|---|---|---|---|
| Ours, L2 population families (population metrics) | ||||
| act distribution | ||||
| dynamics | ||||
| outcome sensitivity | ||||
| provenance | ||||
| best metric | ||||
| Metric | Pers | Lang | CoD | Front |
|---|---|---|---|---|
| Ours, L2 population families (population metrics) | ||||
| act distribution | ||||
| dynamics | ||||
| outcome sensitivity | ||||
| provenance | ||||
| best metric | ||||
| approve-broken-work | front-loading | |||||||
| Sessions per population | 10 | 25 | 50 | 100 | 10 | 25 | 50 | 100 |
| Ours: L2 act distribution | 0.85 | 0.94 | 0.99 | 1.00 | 0.80 | 0.90 | 0.99 | 1.00 |
| Ours: L2 (mean of families) | 0.63 | 0.65 | 0.71 | 0.62 | 0.73 | 0.84 | 0.91 | 1.00 |
| Ours: best L3 family, population mean | 0.84 | 0.94 | 0.99 | 1.00 | 0.69 | 0.75 | 0.85 | 0.92 |
| SimulatorArena: best metric, population mean | 0.72 | 0.78 | 0.83 | 0.90 | 0.72 | 0.91 | 0.96 | 0.99 |
| VISTA: best metric, population mean | 0.19 | 0.12 | 0.04 | 0.01 | 0.70 | 0.80 | 0.92 | 0.98 |
| Evaluation suite / metric | Approve broken work | Premature stop | Front-loading |
| Ours: L2 (mean of families) | 1.00 | 0.53 | 1.00 |
| Ours: L2 family | 1.00 act distribution | 0.01 provenance | 1.00 act distribution |
| SimulatorArena | 1.00 mean of Likert writing | 0.00 mean of Turing detector | 1.00 mean of Likert writing |
| VISTA | 0.00 mean of goal consistency | 0.11 mean of goal consistency | 1.00 mean of profile consistency |
| SWE-Together | 1.00 mean of scope precision | 1.00 mean of weighted coverage | 1.00 mean of scope precision |
| Metric | Dir. | Goal | AllC | Appr | Stop | Pers | Lang | CoD | Front | Favor | ArtNeg |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Ours | |||||||||||
| persona | 0.67 | 0.54 | 0.63 | 0.55 | 0.70 | 0.75 | 0.68 | 0.46 | 0.50 | 0.55 | |
| intent | 0.63 | 0.52 | 0.53 | 0.55 | 0.51 | 0.56 | 0.53 | 0.53 | 0.51 | 0.51 | |
| constraints | 0.63 | 0.60 | 0.47 | 0.56 | 0.57 | 0.49 | 0.51 | 0.42 | 0.54 | 0.48 | |
| best metric | 0.67 persona | 0.60 constraint | 0.63 persona | 0.56 constraint | 0.70 persona | 0.75 persona | 0.68 persona | 0.53 intent | 0.54 constraint | 0.55 persona | |
| composite | 0.70 | 0.57 | 0.61 | 0.58 | 0.66 | 0.71 | 0.63 | 0.48 | 0.56 | 0.54 | |
| Metric | Dir. | Goal | AllC | Appr | Stop | Pers |
|---|---|---|---|---|---|---|
| Ours | ||||||
| persona | [-3pt] | [-3pt] | [-3pt] | [-3pt] | [-3pt] | |
| intent | [-3pt] | [-3pt] | [-3pt] | [-3pt] | [-3pt] | |
| constraints | [-3pt] | [-3pt] | [-3pt] | [-3pt] | [-3pt] | |
| best metric | [-3pt] | [-3pt] | [-3pt] | [-3pt] | [-3pt] | |
| composite | [-3pt] | [-3pt] | [-3pt] | [-3pt] | [-3pt] | |
| Metric | Dir. | Lang | CoD | Front | Favor | ArtNeg |
|---|---|---|---|---|---|---|
| Ours | ||||||
| persona | [-3pt] | [-3pt] | [-3pt] | [-3pt] | [-3pt] | |
| intent | [-3pt] | [-3pt] | [-3pt] | [-3pt] | [-3pt] | |
| constraints | [-3pt] | [-3pt] | [-3pt] | [-3pt] | [-3pt] | |
| best metric | [-3pt] | [-3pt] | [-3pt] | [-3pt] | [-3pt] | |
| composite | [-3pt] | [-3pt] | [-3pt] | [-3pt] | [-3pt] | |
| Perturbation | Acc. (unsure ) | Agreement |
| Agent favoritism | 0.53 | 0.65 |
| Negative profile artifact | 0.51 | 0.65 |
| Adversarial goal | 0.57 | 0.45 |
| All constraints removed | 0.53 | 0.75 |
| Approve broken work | 0.66 | 0.45 |
| Premature stop | 0.51 | 0.55 |
| Agent favoritism | ||||
| Metric (AUROC / ctrl@5%FPR) | =10% | 25% | 50% | 100% |
| L1: Kendall vs reference | 0.55 / 0.04 | 0.69 / 0.12 | 0.84 / 0.32 | 0.99 / 0.98 |
| SimulatorArena: composite | 0.45 / 0.05 | 0.39 / 0.01 | 0.29 / 0.00 | 0.08 / 0.00 |
| SimulatorArena: best metric | 0.50 / 0.06 Likert writing | 0.54 / 0.01 Likert writing | 0.56 / 0.01 Likert writing | 0.64 / 0.09 Likert writing |
| VISTA: composite | 0.48 / 0.04 | 0.49 / 0.05 | 0.47 / 0.04 | 0.42 / 0.04 |
| VISTA: best metric | 0.49 / 0.05 knowledge consistency | 0.50 / 0.05 profile consistency | 0.50 / 0.07 profile consistency | 0.50 / 0.08 profile consistency |
| Fraction of replays replaced by perturbed replays ( ) | ||||||||
| 10% | 25% | 50% | 100% | |||||
| Metric | AUROC | ctrl@ 5%FPR | AUROC | ctrl@ 5%FPR | AUROC | ctrl@ 5%FPR | AUROC | ctrl@ 5%FPR |
| Approve broken work | ||||||||
| L2: mean of families | 0.37 | 0.03 | 0.29 | 0.01 | 0.26 | 0.01 | 0.71 | 0.21 |
| L2: act distribution | 0.23 | 0.01 | 0.14 | 0.00 | 0.17 | 0.01 | 1.00 | 1.00 |
| L2: dynamics | 0.49 | 0.05 | 0.46 | 0.05 | 0.42 | 0.06 | 0.39 | 0.05 |
| Control vs perturbed (control positive) | Real vs perturbed (real positive) | |||||
| Metric | AUROC | control @5% FPR | control @10% FPR | AUROC | real @5% FPR | real @10% FPR |
| Ours: L3 (mean of families) ‡ | 0.61 | 0.07 | 0.17 | n/a | n/a | n/a |
| Ours: persona | 0.62 | 0.10 | 0.17 | 0.90 | 0.68 | 0.77 |
| Ours: intent ‡ | 0.55 | 0.06 † | 0.11 † | n/a | n/a | n/a |
| Ours: constraints ‡ | 0.54 | 0.06 † | 0.11 † | n/a | n/a | n/a |
| SimulatorArena: Likert writing | 0.55 | 0.05 † | 0.11 † | 0.88 | 0.28 † | 0.55 † |
User Simulation
User simulation models human behavior and interaction to test, train, and evaluate AI systems with synthetic users.
See all 143 papers