Users increasingly rely on AI systems for extended, multi-turn tasks, yet evidence indicates that LLM-based system performance degrades over prolonged interaction. To evaluate and improve AI assistants in such settings, we need user simulators that faithfully follow complex user intents while exhibiting realistic conversational behavior. Existing approaches manifest a trade-off: prompted assistants can follow complex user intents but lack behavioral fidelity, while specialized user simulators better approximate conversational behavior but struggle to remain faithful to complex intents. In this paper, we propose a reinforcement learning (RL) recipe that jointly rewards task intent adherence and behavioral naturalness. A crucial contribution of the method is operationalizing naturalness with a lightweight discriminator trained in an alternating, adversarial loop with the user simulator. We release the resulting model, UserLM-2.0, which produces valid simulated conversations (i.e., meeting both task-faithfulness and naturalness criteria) 46.2% of the time, over three times the rate of the strongest prior user simulator (14.1%).
Figures & tables
Figure 1: We train and release UserLM-2.0, a user simulator for User-AI interaction optimized via RL to precisely enact user intents while exhibiting natural conversational behavior. In comparison, assistants such as GPT-5.6-Sol follow intents, but sound unnatural, while SFT-trained simulators like UserLM-8B ( Naous et al., 2026 ) generate natural, but meandering conversations.
width
Task Fidelity &
Naturalness =
Validity
Model width
Qual.
Cov.
Term. width
Spec.
Gen.
Pan.
% Pass width
Valid %
Cost
DeepSeek-V4-Pro width
98.9
85.1
21.8 width
0.6
2.3
0.4
2.1 width
0.3
$1331.35
Llama-3.1-8B width
59.2
76.9
48.5 width
0.2
1.9
3.1
2.6 width
0.4
$11.98
Qwen3.5-9B width
80.4
79.2
91.1 width
0.5
1.3
2.1
2.8 width
1.6
$14.79
GPT-5.6-Luna width
98.6
85.8
45.5 width
0.4
2.2
3.6
4.9 width
3.4
$20.89
Kimi-K2.6 width
99.6
86.1
63.5 width
0.5
1.5
4.2
4.9 width
4.0
$63.48
Table 1: Evaluation results of user simulators. We compare prompted assistants (top), finetuned models (middle), and human reference conversations on task fidelity (Qual., Cov., Term.) and naturalness (component means Spec., Gen., Pan., and Naturalness Pass Rate, % Pass). We report overall conversation validity (Valid %) and estimated simulator cost in USD per 100 valid conversations.
width
Coverage
Naturalness Pass (%)
Overall Valid %
Model width
1
2
3
4
5
6
7 width
1
2
3
4
5
6
7 width
1
2
3
4
5
6
7
DeepSeek-V4-Pro width
95
91
88
85
81
78
75 width
5
4
2
1
1
1
1 width
2
0
0
0
0
0
0
Llama-3.1-8B width
96
86
82
76
70
65
59 width
7
4
4
1
0
2
1 width
1
1
1
1
0
0
0
Qwen3.5-9B width
90
88
83
78
74
71
66 width
11
4
2
2
1
0
0 width
9
2
1
1
0
0
0
GPT-5.6-Luna width
97
90
88
86
83
79
78 width
22
9
3
1
1
1
1 width
21
5
2
1
0
0
0
Kimi-K2.6 width
96
92
89
86
82
79
76 width
22
10
3
1
0
1
0 width
22
9
1
1
0
0
0
Table 2: Stress testing by number of task subgoals (1–7). We report mean Coverage, Naturalness Pass Rate (Naturalness Pass %), and overall Validity. Validity additionally requires quality and termination criteria. Scores are rounded to whole percentages.
Repetition
Verbosity
Model
Re- visit
Rep. Only
Emp.
W/goal
DeepSeek-V4-Pro
64.9
57.4
12.6
127
Llama-3.1-8B
53.1
43.2
24.2
190
Qwen3.5-9B
41.8
35.8
14.5
99
GPT-5.6-Luna
51.0
47.8
8.9
96
Kimi-K2.6
52.4
47.2
4.6
77
Table 3: Repetition and verbosity of user simulators. We report the rate of goal revisitation (Revisit), repetition-only turns (Rep. Only), goalless turns that do not advance the task intent (Emp.), and median user words per covered goal (W/goal). Rates are percentages; darker shading indicates closer agreement with humans for each metric.
Figure 2: We depict goal disclosure over time for the Human Reference, UserLM-2.0,and prompted and finetuned baselines. Each bar represents the average cumulative coverage at a given conversation turn across sampled conversations. For each turn, the bolded component represents the proportion of newly covered sub-goals, while the gray component represents the proportion of revisited sub-goals. UserLM-2.0 achieves the best alignment with the Human Reference.
width
Task Fidelity
Naturalness
Valid
Model width
Qual.
Cov.
Term. width
Spec.
Gen.
Pan.
Pass. width
%
Naturalness Design
Fixed Disc. width
95.8
88.7
73.4 width
0.8
3.2
10.1
11.3 width
7.8
Judge (Likert) width
98.0
91.6
80.7 width
1.0
1.8
13.1
14.0 width
10.5
Judge (Winrate) width
96.6
92.8
32.2 width
1.0
4.9
23.1
24.5 width
7.0
Reward Ablations
Table 4: Evaluation results of alternative training strategies. The top three models are trained with alternative reward designs for naturalness. The middle models are either SFT only, or ablate a reward component (Coverage or Naturalness). We report task-fidelity, naturalness and validity metrics.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Prompt template used to extract generic user intents c from conversations with Qwen3-30B-A3B-Instruct-2507. {conversation} is replaced by the user turns of the conversation, following the formatting in the few-shot examples.
Figure 4: Prompt template for the first step of subgoal extraction, which extracts a detailed intent with numbered, conversation-specific goals. {conversation} is replaced by the user turns of the input conversation, formatted in the same way as the few-shot examples. The few-shot examples have been shortened for space.
Figure 5: Prompt template for the second step of subgoal extraction, which genericizes subgoals to be attainable given only the generic intent. {generic} is the generic intent (Figure 3 ) and {detailed} is the detailed intent from the first step (Figure 4 ), both inserted verbatim.
Figure 6: Prompt template for the quality judge, with examples removed due to space. Each few-shot example and {conversation} are replaced by the user turns, preceded by the system prompts.
Figure 7: Prompt template for the coverage judge, with full few-shot examples removed due to space. Each {example_i} block and the {conversation} block are replaced by the associated conversation featuring only the user turns, with conditioning on the full task intent.
Figure 8: Prompt template for the LLM Judge - Winrate experiment. Each few-shot example pairs a WildChat conversation with a simulated conversation responding to the same intent. The {example_i_system} placeholders are replaced by the associated system prompts. Each individual option is then replaced by the real and simulated conversation, with orders swapping with each example.
Figure 9: Prompt template for the LLM Judge - Likert experiment. The few-shot examples contain one conversation from an SFT-initialized model, one conversation from a prompted assistant model and a real human conversation. Each conversation contains the system prompt, followed by the user turns.
Figure 10: System prompt for USP.
Figure 11: System prompt for HumanLM-Opinion, adapted from HumanLM’s own system prompt. As none of our models condition on persona, user or dataset statistic information, we leave those fields as missing and include the intent in the interests field.
Figure 12: Prompt for the first turn of the prompted assistant user simulator. The whole prompt is given to the model as a single user message and its response is taken as the first user turn.
Figure 13: Prompt for every turn after the first of the prompted assistant user simulator. Each turn, the whole prompt is re-rendered and given to the model as a single user message, and its response is taken as the next user turn.
Figure 14: Prompt template used to annotate which subgoals each simulated user turn covers, used with GPT-5.6-Luna in the main evaluation experiments to compute coverage.
width
Naturalness-Specific
Naturalness-General
Pangram
Model width
1
2
3
4
5
6
7 width
1
2
3
4
5
6
7 width
1
2
3
4
5
6
7
DeepSeek-V4-Pro width
1
1
1
0
0
0
0 width
7
3
2
1
1
2
1 width
1
1
0
0
0
0
0
Llama-3.1-8B width
1
0
0
0
0
0
0 width
6
2
2
2
1
1
1 width
7
5
4
2
1
2
1
Qwen3.5-9B width
1
1
1
0
0
0
0 width
4
2
1
1
1
0
0 width
9
4
2
1
1
0
0
GPT-5.6-Luna width
2
1
0
0
0
0
0 width
6
3
2
1
1
2
1 width
20
6
2
1
1
0
0
Kimi-K2.6 width
2
1
0
1
0
0
0 width
5
4
1
1
0
0
0 width
20
9
2
1
0
1
0
Appendix
Table 5: Naturalness-Specific, Naturalness-General and Pangram component means by number of task subgoals (1–7). Values are rounded to whole percentages; darker shading indicates higher unrounded scores, with one scale per metric across all seven buckets and simulated models. Human Reference remains unshaded.
width
Quality
Terminated
Model width
1
2
3
4
5
6
7 width
1
2
3
4
5
6
7
DeepSeek-V4-Pro width
99
99
99
99
99
99
97 width
31
28
25
23
18
14
8
Llama-3.1-8B width
71
61
59
57
56
55
60 width
33
37
47
51
56
58
55
Qwen3.5-9B width
85
84
81
81
78
78
75 width
91
90
92
91
91
92
91
GPT-5.6-Luna width
99
99
99
99
98
99
98 width
71
63
52
45
33
28
24
Kimi-K2.6 width
100
100
100
100
99
100
99 width
80
78
71
63
53
50
42
Appendix
Table 6: Quality and Terminated by number of task subgoals (1–7). Both task-fidelity metrics use blue shading. Values are rounded to whole percentages; darker shading indicates higher unrounded scores, with one scale per metric across all seven buckets and simulated models. Human Reference remains unshaded and is treated as terminated by construction.
Figure 15: (a) Naturalness reward throughout training for the Fixed Discriminator and LLM Judge - Likert and Winrate ablations; (b) Naturalness reward throughout training for the dynamic discriminator experiments; (c) Average of the Jensen-Shannon Divergence between utterance-length histograms and turn-count histograms for model-generated and real validation conversations. Lower is better.
Figure 16: For UserLM-2.0, we perform inference with each discriminator trained during RL on validation rollouts from each validation step and report Pdisc(y=real∣u1^...uk^,c) for generated conversations. Blue and red indicate high and low probabilities respectively. UserLM-2.0 learns not only to fool the most recent discriminator, but also keeps fooling older ones as well.
User simulators are increasingly leveraged to build interactive AI assistants, yet how to measure the quality of these simulators remains an open question. In this work, we show how simulator quality can be quantified in terms of its downstream utility: how an LLM assistant trained with this user simulator performs in the wild when interacting with real humans. In a controlled experiment where only the user simulator varies, we train LLM assistants via reinforcement learning against a spectrum of simulators, from an LLM prompted to role-play a user to one fine-tuned on human utterances from WildChat. As evaluation, we measure pairwise win rates in a user study with 283 participants and on WildBench, a benchmark derived from real human--AI conversations. Training against the role-playing LLM yields an assistant statistically indistinguishable from the initial assistant in our user study (51% win rate), whereas training against the fine-tuned simulator yields significant gains (58% over the initial and 57% over the one trained against role-playing). Closer inspection reveals three further patterns: methods for making role-playing LLMs more realistic (e.g., persona conditioning) improve trained assistants but do not close the gap to the fine-tuned simulator; scaling the simulator's model size benefits the fine-tuned simulator but yields no gain for role-playing ones; and assistants trained against role-playing simulators fail to generalize when paired with other simulators at test time, while the one trained against fine-tuned simulator does. Together, these results argue for grounding user simulators in real human behavior and measuring their quality by their downstream effect on real users.
User simulators are gaining attention as environments to train LLM assistants that can better cope with the nuances of multi-turn conversations with real users. Prior work on user simulation has emphasized generating realistic user utterances, leaving unexplored their potential as a source of direct feedback on the assistant's responses. Such evaluation signals may not be explicitly communicated in language, but can instead reside in the latent reasoning of users. In this work, we introduce user simulators with intrinsic feedback, namely, models equipped with latent evaluation states: internal thoughts, reactions, and scalar ratings that evaluate the assistant from a first-person user perspective at each conversation turn. We train LLM-based simulators using interaction logs between real users and assistants and demonstrate that intrinsic feedback improves simulation realism and critiquing behaviors versus prompted simulator baselines, while also providing turn-level evaluations that existing fine-tuned simulators lack. Further, evaluations generated by our trained simulators correlate better with human judgments than those of prompted simulators of the same size. Lastly, our experiments reveal that simulators with intrinsic feedback provide effective reward signals for multi-turn RL training, matching setups that rely on external judge LLMs for rewards while cutting RL compute by more than half.
Faithful user simulation is fundamental to building, evaluating, and improving interactive AI at scale. Yet current simulators often produce plausible individual responses without reproducing the intent evolution and outcomes observed in real interactions. We propose TRACER, a multi-turn user simulator that models evolving user intent and aligns simulated trajectories with real ones. TRACER is trained in two stages: supervised fine-tuning on real user dialogues, followed by multi-turn reinforcement learning. The RL stage combines hierarchical outcome- and trajectory-level rewards with deviation-aware advantage modulation, jointly addressing reward sparsity and credit assignment challenges in long dialogues. On real customer-service sessions organized into reference cohorts, TRACER-7B surpasses the strongest baseline by 11.4 conversion F1 points, while outperforming all baselines on group-level conversion-rate error and semantic trajectory distance and generalizing to out-of-distribution scenarios. In human Turing tests, annotators identified TRACER conversations at near-chance accuracy. Building on this simulator, we further introduce the Dynamic Marketing Benchmark, which jointly evaluates persuasion and response quality via simulated interactions, revealing that higher response quality does not necessarily correspond to higher conversion rates.
Geng Chen, Ruotong Pan, Zhirui Yang +9
Peking University · Kuaishou Technology · Xi’an Jiaotong University +3