RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation

Ming Zhu; Juntao Tan; Rithesh Murthy; Jielin Qiu; Liangwei Yang; Wenting Zhao; Silvio Savarese; Shelby Heinecke

cs.HCApr 7, 2026

RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation

Authors: Ming Zhu, Juntao Tan, Rithesh Murthy, Jielin Qiu, Liangwei Yang, Wenting Zhao, Silvio Savarese, Shelby Heinecke, +1 more

Organizations: Salesforce AI Research

Abstract

LLM-based user simulation is the primary mechanism for end-to-end agent evaluation, yet simulated users are poor proxies for real humans: unconstrained LLM defaults produce a Formalism Ceiling (style match rates of 6-8% against real users), while hand-crafted behavioral directives trigger Directive Amplification, where models hyper-interpret instructions into unnatural behavioral extremes that vary dramatically across simulator models. We present RealUserSim, the first user simulation framework grounded in real behavioral data. From 14,000+ authentic human-LLM conversations (WildChat), we extract 7,275 executable behavioral profiles and use them to ground LLM simulators. A fidelity benchmark (PT3) on 600 conversations across 71+ domains with anti-leakage controls shows that grounded simulation raises match rate from 24.2% to 45.3% across five behavioral dimensions. Agent evaluation on TauBench with 6 simulator models and extensive analysis shows that grounded simulation acts as a realistic stress test, surfacing three failure mechanisms invisible to cooperative simulators (mean -3.2% to -3.5% task success degradation), while Directive Amplification in existing benchmarks produces unrealistic behavior that compromises the validity of agent evaluation.

Figures & tables

Figure 1: Linguistic trait clusters across 7,273 users. (a) t-SNE projection of 48-dimensional binary trait vectors with $k$ -means clustering ( $k{=}8$ ). Each cluster represents a behavioral archetype (e.g., C7: Everyday Casual, C4: Formal Expert, C6: Informal Texter, C0: Organized Writer). (b, c) Age and education composition per cluster, showing systematic demographic shifts: casual/informal clusters skew younger and lower-education, while formal/structured clusters skew older and higher-education.

Dimension	Description
Persona & Affective Traits	Demeanor, emotional state, patience level, personality cues
Linguistic Style & Mechanics	Vocabulary, phrasing, formality, typos, message length patterns
Tech Competency & Knowledge	Domain expertise, terminology usage, depth of questions
Interaction & Data Flow	Information-sharing habits (scattered vs. dense), questioning style
Pacing & Action Sequencing	Turn length, topic transitions, conversation conclusion patterns

Table 1: Five dimensions of conversational identity used in the PT3 fidelity evaluation.

By dimension
	Baseline	With Profile	$\boldsymbol{\Delta}$
Persona & Affective Traits	7.2	39.0	+31.8
Linguistic Style & Mechanics	6.2	26.2	+20.0
Tech Competency & Knowledge	93.3	93.2	$-$ 0.2
Interaction & Data Flow	7.7	36.0	+28.3
Pacing & Action Sequencing	6.5	32.0	+25.5

Table 2: PT3 fidelity evaluation across 600 WildChat conversations. Match rate (%) indicates the proportion of cases where the LLM judge finds the simulated user indistinguishable from the real user. Top: aggregate by dimension. Bottom: by evaluation subset (

n{=}100

each).

User Type	Original	Baseline	With Profile
Informal student	“Class 6th ncert civics questions can u list three things that the government does which have not been mentioned”	“Hi there! I’m looking to understand more about the roles of government…Could you help me identify three such functions?”	“hey can u tell me 3 things the government does that arent in the class 6 civics book? easy words pls”
ESL speaker	“well i believe there are many differences between between two types of study for example for example hmm through face to face teaching, hmm students can students have…”	“Hi there! I have a piece of text that I’d like to revise for clarity and accuracy…”	“well hmm i need a lil help with something can you help me revise a context i got hmm it needs to be more clear and accurate you know…”
Medical student	“.which of the following therapies is most likely to prolong survival in COPD? a) atenolol b) enalapril c) Oxygen d) prednisone e) theophylline”	“Hi there! I recently asked a question about ‘Pulsus bisferiens’…I’m looking for a similar type of question…”	“.the pulse characteristic known as ‘water-hammer pulse’ is associated with which of the following conditions: a) mitral stenosis b) aortic regurgitation…”

Table 3: Contrastive examples of user simulation fidelity. The Baseline normalizes diverse communication styles into polished prose; profile injection preserves distinctive patterns.

	Airline ( $n=50$ )				Retail ( $n=114$ )
Model	Orig	NP	Persona	$\Delta$	Orig	NP	Persona	$\Delta$
GPT-4o	52.0	50.0	41.3	$-$ 8.7	60.5	77.2	74.0	$-$ 3.2
GPT-5-mini	42.0	48.0	46.0	$-$ 2.0	49.1	50.9	56.6	+5.7
GPT-5	46.0	48.0	44.7	$-$ 3.3	75.4	77.2	71.6	$-$ 5.6
Llama 3-70b	56.0	54.0	44.0	$-$ 10.0	64.0	69.3	63.2	$-$ 6.1
gpt-oss-20b	44.0	38.0	45.3	+7.3	53.5	63.2	47.7	$-$ 15.5

Table 4: Agent task success rates (%) across Airline (

n=50

) and Retail (

n=114

) domains. Persona averages 3 independent runs with different random persona–task assignments.

\Delta

shows change from NP baseline.

Figure 2: Two key effects on agent task success rate across 6 simulator models. (a) Persona injection (NP

\to

Persona) degrades most models in both domains (mean

-3.2\%

Airline,

-3.5\%

Retail), confirming that real user communication styles add meaningful friction. (b) Stripping behavioral directives (Original

\to

NP) consistently improves Retail performance (all 6 models, up to

+16.7\%

), indicating that hand-crafted directives introduce artificial difficulty.

Mechanism	Description	Example
Information loss	Terse persona omits critical task details	Airline Task 42: NP says “cancel FDZ0T5 and HSR97W …don’t modify other passengers.” Terse persona says “proceed with cancelling the reservations as listed.” Agent over-cancels 4 instead of 2.
Agent misinterpretation	Aggressive style triggers wrong agent action	Airline Task 27: NP requests compensation, receives $150 certificate. ALL-CAPS persona writes “GIMME A REFUND PLS” —agent interprets as cancellation request.
Compliance failure	Agreeable persona fails adversarial scenario	Airline Task 47: Must insist on refund 5 $\times$ then give up without cancelling . Hesitant persona says “ahh yes please go ahead and cancel it” on first attempt.

Table 5: Three failure mechanisms from persona injection, identified by comparing NP (pass) vs. Persona (fail) trajectories.

Model	Metric	With Directives	Without (NP)
Llama 3-70b	Roleplay markers	8.6%	0.9%
Llama 3-70b	Stage directions	1.5%	0.0%
GPT-5-mini	Frustration words	16.7%	2.1%
GPT-5-mini	Avg message length	1218 chars	514 chars
gpt-oss-20b	Roleplay markers	19.4%	10.9%
gpt-oss-20b	Frustration words	8.2%	2.0%

Table 6: Behavioral metrics with and without directives. Roleplay markers are asterisk-bracketed actions (e.g., sobbing ). Theatrical behavior drops sharply when directives are stripped.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Test Cases	Inferred	Correct	Accuracy
Gender	100	41	35	85.4%
Marital Status	200	24	19	79.2%
Occupation	100	22	17	77.3%
Education	100	95	55	57.9%
Age	100	54	30	55.6%
Income	200	28	14	50.0%

Appendix

Table 7: Demographic inference validation (V9 Hybrid, strict categorical matching).

Figure 3: Demographic distributions across 7,273 WildChat users. The population skews young (72% under 35) and educated (57% bachelor’s degree). Gender and occupation have low coverage (12.6% and 8.2%) as they are rarely self-disclosed; education has the highest coverage (88.2%) due to inference.

Figure 4: Profile statistics across 7,273 users. Left: most profiles contain 8–10 commands (mean 9.4). Center: 66% of users have only a single conversation; the distribution follows a power law. Right: demographic completeness is low (mean 0.25), as most users have only 1–2 fields populated.

Figure 5: Conversation domain distribution (top 25 of 1,012 domains). Entertainment & Media dominates at 20.8%, followed by a cluster of technical domains (Technology & IT, Software Development, Game Development) each around 5–6%. The long tail spans 1,000+ niche domains.

Metric	GPT-4o	GPT-5-mini	GPT-5	Llama 3	oss-20b	Claude 3
Avg msg len (chars)	122	494	382	373	223	218
Avg msgs/conv	6.8	5.1	5.9	6.5	5.9	6.7
Contains “please”	24%	69%	54%	55%	32%	19%
Has lists	0.2%	32%	43%	2%	8%	2%
Multi-line ( $\geq$ 3)	1%	72%	69%	34%	18%	23%
Has emotes	0%	1%	1%	1%	10%	0%

Appendix

Table 8: Quantitative metrics for model default personas under the NP condition.

Model	NP	High-Edu	Low-Edu	Young	Oldest	Perfect
GPT-4o	77.2	69.3	67.5	66.7	73.7	80.7
GPT-5-mini	50.9	53.5	62.3	48.2	57.0	71.1
GPT-5	77.2	80.7	71.9	69.3	78.1	75.4
Llama 3-70b	69.3	71.9	61.4	65.8	61.4	66.7
gpt-oss-20b	63.2	58.8	43.0	49.1	47.4	58.8
Claude 3 Sonnet	72.8	76.3	77.2	74.6	72.8	71.1

Appendix

Table 9: Demographic persona results, Retail domain (

n=114

). All values are task success rates (%).

Model	NP	High-Edu	Low-Edu	Young	Oldest	Perfect
GPT-4o	50.0	46.0	50.0	50.0	52.0	46.0
GPT-5-mini	48.0	36.0	48.0	39.2	38.0	42.0
GPT-5	48.0	40.0	40.0	46.0	48.0	50.0
Llama 3-70b	54.0	50.0	40.0	58.0	46.0	46.0
gpt-oss-20b	38.0	44.0	44.0	46.0	34.0	36.0
Claude 3 Sonnet	52.0	48.0	58.0	48.0	46.0	54.0

Appendix

Table 10: Demographic persona results, Airline domain (

n=50

). All values are task success rates (%).

Figure 6: Persona sampling variance across three independent random runs (v7, v8, v9) with different persona–task assignments. Dashed line shows the NP baseline. Shaded region indicates the per-model range. Mean range is $8.0\%$ for Airline and $5.9\%$ for Retail.

Figure 7: Demographic-based persona sampling. High-Edu and Oldest personas generally outperform Low-Edu and Young in Retail, while Airline shows more varied patterns.

Figure 8: Perfect User vs. other conditions. The Perfect User consistently outperforms random personas in Retail (5/6 models), with GPT-5-mini showing the largest gain ( $+20.2\%$ vs. NP).

Explore similar work

Mind the Sim2Real Gap in User Simulation for Agentic Tasks
Mar 11, 2026Xuhui Zhou, Weiwei Sun, Qianou Ma +8User Simulation Sim-to-Real Transfer
Carnegie Mellon University, Language Technologies Institute
Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents
May 13, 2026Harshita Chopra, Kshitish Ghate, Aylin Caliskan +3Human Behavior Simulation LLM Agent Evaluation
University of Washington, Seattle, WA · Georgetown University, Washington, DC
AgentPersonaBench: Benchmarking Persona-Driven User Simulation
Oct 3, 2026Jintao Huang, Yifan Wang, Hongyu Shen +43AI Agent Benchmarks Persona-Conditioned Generation

Mar 11, 2026J/K move · Enter open · S save

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

Xuhui Zhou, Weiwei Sun, Qianou Ma, Yiqing Xie+7

Carnegie Mellon University, Language Technologies Institute

User Simulation Sim-to-Real Transfer LLM Agent Evaluation Human-in-the-Loop Evaluation

Open paper AskPDF arXiv

As NLP evaluation shifts from static benchmarks to multi-turn interactive settings, LLM-based simulators have become widely used as user proxies, serving two roles: generating user turns and providing evaluation signals. Yet, these simulations are frequently assumed to be faithful to real human behaviors, often without rigorous verification. We formalize the Sim2Real gap in user simulation and present the first study running the full

τ

-bench protocol with real humans (451 participants, 165 tasks), benchmarking 31 LLM simulators across proprietary, open-source, and specialized families using the User-Sim Index (USI), a metric we introduce to quantify how well LLM simulators resemble real user interactive behaviors and feedback. Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity, creating an "easy mode" that inflates agent success rates above the human baseline. In evaluations, real humans provide nuanced judgments across eight quality dimensions while simulated users produce uniformly more positive feedback; rule-based rewards are failing to capture rich feedback signals generated by human users. Overall, higher general model capability does not necessarily yield more faithful user simulation. These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle and motivate improved models for user simulation.

Abstract

Explore similar work

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents

AgentPersonaBench: Benchmarking Persona-Driven User Simulation