As NLP evaluation shifts from static benchmarks to multi-turn interactive settings, LLM-based simulators have become widely used as user proxies, serving two roles: generating user turns and providing evaluation signals. Yet, these simulations are frequently assumed to be faithful to real human behaviors, often without rigorous verification. We formalize the Sim2Real gap in user simulation and present the first study running the full τ-bench protocol with real humans (451 participants, 165 tasks), benchmarking 31 LLM simulators across proprietary, open-source, and specialized families using the User-Sim Index (USI), a metric we introduce to quantify how well LLM simulators resemble real user interactive behaviors and feedback. Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity, creating an "easy mode" that inflates agent success rates above the human baseline. In evaluations, real humans provide nuanced judgments across eight quality dimensions while simulated users produce uniformly more positive feedback; rule-based rewards are failing to capture rich feedback signals generated by human users. Overall, higher general model capability does not necessarily yield more faithful user simulation. These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle and motivate improved models for user simulation.
Figures & tables
Figure 1: User-Sim Index (USI) vs. Chatbot Arena Elo Score for LLM simulators. Solid lines and shaded regions show per-family linear regression with 80% confidence bands; error bars denote standard deviation across three annotator batches. Besides GPT-series, other LLMs’ general capability does not reliably translate to faithful user simulation.
Figure 2: Taxonomy of Sim2Real gaps in user simulation. We highlight the dimensions where the gaps between humans and all LLMs are significant. See Appendix § A.4 for exact operational definitions of the behavioral metrics.
Model
D1 Comm.
D2 Info.
D3 Clarif.
D4 React.
Eval
ECE ↓
USI
Human (inter-ann.)
87.4 ±6.8
97.9 ±0.9
88.0 ±1.3
93.5 ±2.5
97.4 ±5.0
0.081 ±0.010
92.7 ±1.1
Gemini2.0-Flash
51.6 ±1.6
88.9 ±1.1
68.2 ±2.1
76.9 ±3.7
73.7 ±0.8
0.111 ±0.022
74.7 ±0.8
GPT-5.1
47.3 ±6.9
77.4 ±0.6
73.3 ±2.0
88.1 ±2.6
72.1 ±1.5
0.172 ±0.011
73.5 ±1.0
GPT-5
49.7 ±5.6
73.7 ±0.7
73.2 ±2.3
73.4 ±3.3
74.5 ±1.1
0.102 ±0.022
72.4 ±1.8
GPT-4o-mini
40.6 ±1.8
84.7 ±0.9
70.2 ±3.7
73.7 ±1.4
75.7 ±0.3
0.123 ±0.024
72.1 ±0.8
GPT-5-mini
39.4 ±5.9
74.4 ±0.7
83.1 ±2.3
68.7 ±1.6
73.5 ±0.5
0.102 ±0.018
71.5 ±1.4
Table 1: Behavioral divergence between human and LLM-simulated users on τ -bench (mean ± std across three human annotation batches). D1–D4 : Sørensen–Dice coefficients (higher = closer to human). Eval : (1−MAE)×100 . USI : composite 0–100 alignment score aggregating D1–D4, Eval, and (1−ECE)×100 . Models are grouped into proprietary , open-source , and specialized categories, ranked by USI within each group. Bold indicates the best model per column. Full results for all 31 models are in Appendix Table 4 .
Figure 3: Concrete τ -bench annotation example. The right panel gives a human annotator a one-way New York–Seattle booking task and role-playing rules; the left panel shows the resulting conversation, including the agent’s tool calls. After the interaction, the same interface presents the survey used for task-success and quality ratings.
Figure 4: Per-metric behavioral comparison for selected models and humans. Metrics are grouped into four dimensions; human values appear as dark bars and red outlines indicate large divergence.
Figure 5: Score distributions for human annotators and GPT-5.1 across quality dimensions ( n=165×3 batches), with mean differences ( Δ = LLM − Human). The LLM evaluator is lenient on interaction quality but conservative on task success.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Human (Ground Truth)
Fail
Pass
Total
LLM Judge
Fail
9 (TN)
6 (FN)
15
Pass
2 (FP)
34 (TP)
36
Total
11
40
51
Appendix
Table 2: Confusion Matrix: LLM Judge vs. Human Ground Truth ( N=51 )
Metric
Human
Default
Persona
Polite%
11.0
44.0
28.0
Front-load%
32.0
43.0
38.0
Clarification%
0.8
0.0
1.0
Uncertainty%
6.0
7.0
13.0
Dash%
1.7
59.0
73.0
Emotion%
1.3
1.3
0.7
Appendix
Table 3: Persona prompting moves some metrics toward the same-task human reference but overshoots others, lowering overall fidelity.
Figure 6: Per-dimension human quality ratings by reward group ( n=495 ). Stacked distributions for reward=0 and reward=1 are nearly indistinguishable across all eight dimensions, confirming that the binary reward captures none of these quality aspects.
Figure 7: (a) Human task success judgment (Yes / No / Policy-constrained) vs. τ -bench binary reward ( n=495 , Cramér’s V=0.168 ). 70.6% of reward=0 cases are judged successful by humans; 81% of policy-constrained cases receive reward=1. (b) Composite human quality scores by reward group. The binary reward predicts neither task success nor interaction quality as perceived by humans.
Figure 8: Agent success rates (%) on τ -bench when paired with each user simulator. Models are grouped by category (proprietary, open-source, specialized). The dashed line indicates the human baseline. Most general-purpose LLM simulators inflate agent success rates above the human baseline, while specialized user-simulation models fall below it.
Model
D1 Comm.
D2 Info.
D3 Clarif.
D4 React.
Eval
ECE ↓
USI
Human (inter-ann.)
87.4 ±6.8
97.9 ±0.9
88.0 ±1.3
93.5 ±2.5
97.4 ±5.0
0.081 ±0.010
92.7 ±1.1
Gemini2.0-Flash
51.6 ±1.6
88.9 ±1.1
68.2 ±2.1
76.9 ±3.7
73.7 ±0.8
0.111 ±0.022
74.7 ±0.8
GPT-5.1
47.3 ±6.9
77.4 ±0.6
73.3 ±2.0
88.1 ±2.6
72.1 ±1.5
0.172 ±0.011
73.5 ±1.0
GPT-5
49.7 ±5.6
73.7 ±0.7
73.2 ±2.3
73.4 ±3.3
74.5 ±1.1
0.102 ±0.022
72.4 ±1.8
GPT-4o-mini
40.6 ±1.8
84.7 ±0.9
70.2 ±3.7
73.7 ±1.4
75.7 ±0.3
0.123 ±0.024
72.1 ±0.8
GPT-5-mini
39.4 ±5.9
74.4 ±0.7
83.1 ±2.3
68.7 ±1.6
73.5 ±0.5
0.102 ±0.018
71.5 ±1.4
Appendix
Table 4: Full behavioral divergence results across all 31 LLM user simulators on τ -bench tasks (mean ± std across three independent human annotation batches). D1–D4 are per-dimension Sørensen–Dice coefficients (higher = closer to human), see Appendix A.4 for definitions. Eval measures agreement between LLM and human evaluators on multi-dimensional surveys, computed as (1−MAE)×100 . The User-Sim Index (USI) aggregates D1–D4, Eval, and a calibration score (1−ECE)×100 into a single 0–100 measure of human alignment. Models are grouped into proprietary , open-source , and specialized categories, ranked by USI within each group. Bold indicates the best model per column.
User Simulator
D1 Conv.
D2 Info.
D3 Clarif.
D4 React.
Eval
ECE
USI
Human (GPT-5.2 agent)
87.4 ±6.8
97.9 ±0.9
88.0 ±1.3
93.5 ±2.5
97.4 ±5.0
0.069 ±0.022
92.9 ±0.9
Gemini-2.0-Flash
71.6 ±6.3
88.4 ±1.1
76.9 ±2.4
75.6 ±3.8
73.7 ±0.8
0.113 ±0.025
79.1 ±0.3
GPT-4o
42.6 ±4.0
87.3 ±0.8
78.4 ±1.1
78.0 ±3.0
73.7 ±1.1
0.121 ±0.025
74.6 ±1.4
GPT-5-mini
54.5 ±6.6
71.4 ±0.7
69.1 ±3.8
67.8 ±2.0
73.5 ±0.5
0.115 ±0.035
70.8 ±0.3
Gemini-3-Flash
55.8 ±6.9
76.1 ±0.7
67.4 ±5.5
44.0 ±2.6
71.7 ±1.3
0.156 ±0.025
66.6 ±1.1
GPT-3.5-turbo
49.6 ±1.2
76.6 ±0.7
59.5 ±3.3
43.9 ±0.6
73.9 ±0.7
0.105 ±0.025
65.5 ±0.5
Appendix
Table 5: Behavioral divergence when using Gemini-3.1-Pro as the agent instead of GPT-5.2. Human baseline scores are from the GPT-5.2 agent setting and thus not directly comparable (see text). D1–D4 and USI definitions follow Table 1 .
Tasks (dataset)
Task-type
Scale
Behavior Metric
Eval Metric
Sat.?
Seshadri et al. (2026)
τ -Bench (retail)
H–A
∼ 360 p / 1440 sess
ECE; miscalibration
Task success-gap
No
Bougie and Watanabe (2025)
RS
SimUser–RS
1000 agents
View ratio; click; rating acc.
RS metric rank/acc.
Yes
Ferraro et al. (2024)
Twitter (2020 US election)
Offline
2020 election dataset
Ling. patterns; homophily
Macro dynamics
No
Zhang et al. (2025)
Web search
Search
31 p / 296 sess
BLEU/BERTScore
Click/stop F1
No
Zhang et al. (2024)
Web search
Search
Public behavior dataset
Query gen; click/stop pred.
IR sim quality
No
Ren et al. (2024)
Web search (WARRIORS)
Search
WARRIORS dataset
Query/click consis. ( ∼ 90%)
MRR/NDCG
No
Appendix
Table 6: Comparison to prior work on measuring the gap between LLM-based user simulators and real users across domains, interaction types, evaluation scale, alignment metrics, downstream evaluation, and whether user satisfaction is modeled.
LLM-based user simulation is the primary mechanism for end-to-end agent evaluation, yet simulated users are poor proxies for real humans: unconstrained LLM defaults produce a Formalism Ceiling (style match rates of 6-8% against real users), while hand-crafted behavioral directives trigger Directive Amplification, where models hyper-interpret instructions into unnatural behavioral extremes that vary dramatically across simulator models. We present RealUserSim, the first user simulation framework grounded in real behavioral data. From 14,000+ authentic human-LLM conversations (WildChat), we extract 7,275 executable behavioral profiles and use them to ground LLM simulators. A fidelity benchmark (PT3) on 600 conversations across 71+ domains with anti-leakage controls shows that grounded simulation raises match rate from 24.2% to 45.3% across five behavioral dimensions. Agent evaluation on TauBench with 6 simulator models and extensive analysis shows that grounded simulation acts as a realistic stress test, surfacing three failure mechanisms invisible to cooperative simulators (mean -3.2% to -3.5% task success degradation), while Directive Amplification in existing benchmarks produces unrealistic behavior that compromises the validity of agent evaluation.
LLM-based user simulators are increasingly used to evaluate autonomous agents at scale, in place of costly human evaluations. Despite this promise, these simulators exhibit "assistant bias," a tendency to cooperate and pursue task goals. They rarely reproduce the frustration or disengagement that real users exhibit, compromising evaluation validity. Prior work outlines that this bias is baked in during model training, which role-playing prompts fail to override. We analyze this bias from model activations, extracting a user role vector by contrasting how the model represents user versus assistant perspectives on the same dialogue. We observe two findings: (i) the user direction is identifiable in activations, elicits user-like behaviors, and captures characteristics distinct from assistant traits; and (ii) although user-role activation associates with simulation realism and steering strengthens it, it can exaggerate user behaviors and override individual user profiles. Together, our findings provide a representation-level analysis of LLM user simulators, confirming that assistant bias is structurally identifiable and that user behavior can be directionally analyzed.
Daeheon Jeong, Yoonjoo Lee, Eugene Choi +2
1KAIST · University of Michigan · 3Seoul National University +2
Large Language Model (LLM) agents are increasingly deployed in settings where they interact with diverse users, including those who are unclear, impatient, or reluctant to share information. However, collecting real interaction data at scale remains expensive. The field has turned to LLM-based user simulators as stand-ins, but these simulators inherit the behavior of their underlying models: cooperative and homogeneous. As a result, agents that appear strong in simulation often fail in real human interactions. To narrow this gap, we introduce Persona Policies (PPol), a plug-and-play control layer that induces realistic behavioral variation in user simulators while preserving original task goals. Rather than hand-crafting personas, we employ an evolutionary coding agent to discover persona generation programs optimized for human-likeness and behavioral coverage over real user conversations. The evolved program generates diverse, human-like personas for any task in the domain. Across 4 benchmarks--including τ2-bench Retail and Airline, ColBench, and WildChat--evolved PPol yield 28-72% absolute gains in fitness score over the baseline simulator. In blinded evaluations, annotators judged PPol users as 'human' 80.4% of the time, nearly 2x more than the baseline simulators. Training agents with PPol also improves real-world performance: our user study with live human-agent interactions showed that fine-tuning with our method boosted task success by +23% over default baselines. PPol thus offers a novel approach to strengthen simulator-based evaluation and training without changing underlying tasks.