cs.CLOct 22, 2026
SaveTowards User Simulators That Stay On Task And Still Sound Natural
Organizations: Cornell University · Microsoft Research
Abstract
Users increasingly rely on AI systems for extended, multi-turn tasks, yet evidence indicates that LLM-based system performance degrades over prolonged interaction. To evaluate and improve AI assistants in such settings, we need user simulators that faithfully follow complex user intents while exhibiting realistic conversational behavior. Existing approaches manifest a trade-off: prompted assistants can follow complex user intents but lack behavioral fidelity, while specialized user simulators better approximate conversational behavior but struggle to remain faithful to complex intents. In this paper, we propose a reinforcement learning (RL) recipe that jointly rewards task intent adherence and behavioral naturalness. A crucial contribution of the method is operationalizing naturalness with a lightweight discriminator trained in an alternating, adversarial loop with the user simulator. We release the resulting model, UserLM-2.0, which produces valid simulated conversations (i.e., meeting both task-faithfulness and naturalness criteria) 46.2% of the time, over three times the rate of the strongest prior user simulator (14.1%).
Figures & tables
Figure 1: We train and release UserLM-2.0, a user simulator for User-AI interaction optimized via RL to precisely enact user intents while exhibiting natural conversational behavior. In comparison, assistants such as GPT-5.6-Sol follow intents, but sound unnatural, while SFT-trained simulators like UserLM-8B ( Naous et al., 2026 ) generate natural, but meandering conversations.
| width | Task Fidelity & | Naturalness = | Validity | ||||||
| Model width | Qual. | Cov. | Term. width | Spec. | Gen. | Pan. | % Pass width | Valid % | Cost |
| DeepSeek-V4-Pro width | 98.9 | 85.1 | 21.8 width | 0.6 | 2.3 | 0.4 | 2.1 width | 0.3 | $1331.35 |
| Llama-3.1-8B width | 59.2 | 76.9 | 48.5 width | 0.2 | 1.9 | 3.1 | 2.6 width | 0.4 | $11.98 |
| Qwen3.5-9B width | 80.4 | 79.2 | 91.1 width | 0.5 | 1.3 | 2.1 | 2.8 width | 1.6 | $14.79 |
| GPT-5.6-Luna width | 98.6 | 85.8 | 45.5 width | 0.4 | 2.2 | 3.6 | 4.9 width | 3.4 | $20.89 |
| Kimi-K2.6 width | 99.6 | 86.1 | 63.5 width | 0.5 | 1.5 | 4.2 | 4.9 width | 4.0 | $63.48 |
Table 1: Evaluation results of user simulators. We compare prompted assistants (top), finetuned models (middle), and human reference conversations on task fidelity (Qual., Cov., Term.) and naturalness (component means Spec., Gen., Pan., and Naturalness Pass Rate, % Pass). We report overall conversation validity (Valid %) and estimated simulator cost in USD per 100 valid conversations.
| width | Coverage | Naturalness Pass (%) | Overall Valid % | ||||||||||||||||||
| Model width | 1 | 2 | 3 | 4 | 5 | 6 | 7 width | 1 | 2 | 3 | 4 | 5 | 6 | 7 width | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| DeepSeek-V4-Pro width | 95 | 91 | 88 | 85 | 81 | 78 | 75 width | 5 | 4 | 2 | 1 | 1 | 1 | 1 width | 2 | 0 | 0 | 0 | 0 | 0 | 0 |
| Llama-3.1-8B width | 96 | 86 | 82 | 76 | 70 | 65 | 59 width | 7 | 4 | 4 | 1 | 0 | 2 | 1 width | 1 | 1 | 1 | 1 | 0 | 0 | 0 |
| Qwen3.5-9B width | 90 | 88 | 83 | 78 | 74 | 71 | 66 width | 11 | 4 | 2 | 2 | 1 | 0 | 0 width | 9 | 2 | 1 | 1 | 0 | 0 | 0 |
| GPT-5.6-Luna width | 97 | 90 | 88 | 86 | 83 | 79 | 78 width | 22 | 9 | 3 | 1 | 1 | 1 | 1 width | 21 | 5 | 2 | 1 | 0 | 0 | 0 |
| Kimi-K2.6 width | 96 | 92 | 89 | 86 | 82 | 79 | 76 width | 22 | 10 | 3 | 1 | 0 | 1 | 0 width | 22 | 9 | 1 | 1 | 0 | 0 | 0 |
Table 2: Stress testing by number of task subgoals (1–7). We report mean Coverage, Naturalness Pass Rate (Naturalness Pass %), and overall Validity. Validity additionally requires quality and termination criteria. Scores are rounded to whole percentages.
| Repetition | Verbosity | |||
| Model | Re- visit | Rep. Only | Emp. | W/goal |
| DeepSeek-V4-Pro | 64.9 | 57.4 | 12.6 | 127 |
| Llama-3.1-8B | 53.1 | 43.2 | 24.2 | 190 |
| Qwen3.5-9B | 41.8 | 35.8 | 14.5 | 99 |
| GPT-5.6-Luna | 51.0 | 47.8 | 8.9 | 96 |
| Kimi-K2.6 | 52.4 | 47.2 | 4.6 | 77 |
Table 3: Repetition and verbosity of user simulators. We report the rate of goal revisitation (Revisit), repetition-only turns (Rep. Only), goalless turns that do not advance the task intent (Emp.), and median user words per covered goal (W/goal). Rates are percentages; darker shading indicates closer agreement with humans for each metric.
Figure 2: We depict goal disclosure over time for the Human Reference, UserLM-2.0,and prompted and finetuned baselines. Each bar represents the average cumulative coverage at a given conversation turn across sampled conversations. For each turn, the bolded component represents the proportion of newly covered sub-goals, while the gray component represents the proportion of revisited sub-goals. UserLM-2.0 achieves the best alignment with the Human Reference.
| width | Task Fidelity | Naturalness | Valid | |||||
| Model width | Qual. | Cov. | Term. width | Spec. | Gen. | Pan. | Pass. width | % |
| Naturalness Design | ||||||||
| Fixed Disc. width | 95.8 | 88.7 | 73.4 width | 0.8 | 3.2 | 10.1 | 11.3 width | 7.8 |
| Judge (Likert) width | 98.0 | 91.6 | 80.7 width | 1.0 | 1.8 | 13.1 | 14.0 width | 10.5 |
| Judge (Winrate) width | 96.6 | 92.8 | 32.2 width | 1.0 | 4.9 | 23.1 | 24.5 width | 7.0 |
| Reward Ablations | ||||||||
Table 4: Evaluation results of alternative training strategies. The top three models are trained with alternative reward designs for naturalness. The middle models are either SFT only, or ablate a reward component (Coverage or Naturalness). We report task-fidelity, naturalness and validity metrics.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Prompt template used to extract generic user intents from conversations with Qwen3-30B-A3B-Instruct-2507. {conversation} is replaced by the user turns of the conversation, following the formatting in the few-shot examples.
Figure 4: Prompt template for the first step of subgoal extraction, which extracts a detailed intent with numbered, conversation-specific goals. {conversation} is replaced by the user turns of the input conversation, formatted in the same way as the few-shot examples. The few-shot examples have been shortened for space.
Figure 5: Prompt template for the second step of subgoal extraction, which genericizes subgoals to be attainable given only the generic intent. {generic} is the generic intent (Figure 3 ) and {detailed} is the detailed intent from the first step (Figure 4 ), both inserted verbatim.
Figure 6: Prompt template for the quality judge, with examples removed due to space. Each few-shot example and {conversation} are replaced by the user turns, preceded by the system prompts.
Figure 7: Prompt template for the coverage judge, with full few-shot examples removed due to space. Each {example_i} block and the {conversation} block are replaced by the associated conversation featuring only the user turns, with conditioning on the full task intent.
Figure 8: Prompt template for the LLM Judge - Winrate experiment. Each few-shot example pairs a WildChat conversation with a simulated conversation responding to the same intent. The {example_i_system} placeholders are replaced by the associated system prompts. Each individual option is then replaced by the real and simulated conversation, with orders swapping with each example.
Figure 9: Prompt template for the LLM Judge - Likert experiment. The few-shot examples contain one conversation from an SFT-initialized model, one conversation from a prompted assistant model and a real human conversation. Each conversation contains the system prompt, followed by the user turns.
Figure 10: System prompt for USP.
Figure 11: System prompt for HumanLM-Opinion, adapted from HumanLM’s own system prompt. As none of our models condition on persona, user or dataset statistic information, we leave those fields as missing and include the intent in the interests field.
Figure 12: Prompt for the first turn of the prompted assistant user simulator. The whole prompt is given to the model as a single user message and its response is taken as the first user turn.
Figure 13: Prompt for every turn after the first of the prompted assistant user simulator. Each turn, the whole prompt is re-rendered and given to the model as a single user message, and its response is taken as the next user turn.
Figure 14: Prompt template used to annotate which subgoals each simulated user turn covers, used with GPT-5.6-Luna in the main evaluation experiments to compute coverage.
| width | Naturalness-Specific | Naturalness-General | Pangram | ||||||||||||||||||
| Model width | 1 | 2 | 3 | 4 | 5 | 6 | 7 width | 1 | 2 | 3 | 4 | 5 | 6 | 7 width | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| DeepSeek-V4-Pro width | 1 | 1 | 1 | 0 | 0 | 0 | 0 width | 7 | 3 | 2 | 1 | 1 | 2 | 1 width | 1 | 1 | 0 | 0 | 0 | 0 | 0 |
| Llama-3.1-8B width | 1 | 0 | 0 | 0 | 0 | 0 | 0 width | 6 | 2 | 2 | 2 | 1 | 1 | 1 width | 7 | 5 | 4 | 2 | 1 | 2 | 1 |
| Qwen3.5-9B width | 1 | 1 | 1 | 0 | 0 | 0 | 0 width | 4 | 2 | 1 | 1 | 1 | 0 | 0 width | 9 | 4 | 2 | 1 | 1 | 0 | 0 |
| GPT-5.6-Luna width | 2 | 1 | 0 | 0 | 0 | 0 | 0 width | 6 | 3 | 2 | 1 | 1 | 2 | 1 width | 20 | 6 | 2 | 1 | 1 | 0 | 0 |
| Kimi-K2.6 width | 2 | 1 | 0 | 1 | 0 | 0 | 0 width | 5 | 4 | 1 | 1 | 0 | 0 | 0 width | 20 | 9 | 2 | 1 | 0 | 1 | 0 |
Table 5: Naturalness-Specific, Naturalness-General and Pangram component means by number of task subgoals (1–7). Values are rounded to whole percentages; darker shading indicates higher unrounded scores, with one scale per metric across all seven buckets and simulated models. Human Reference remains unshaded.
| width | Quality | Terminated | ||||||||||||
| Model width | 1 | 2 | 3 | 4 | 5 | 6 | 7 width | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| DeepSeek-V4-Pro width | 99 | 99 | 99 | 99 | 99 | 99 | 97 width | 31 | 28 | 25 | 23 | 18 | 14 | 8 |
| Llama-3.1-8B width | 71 | 61 | 59 | 57 | 56 | 55 | 60 width | 33 | 37 | 47 | 51 | 56 | 58 | 55 |
| Qwen3.5-9B width | 85 | 84 | 81 | 81 | 78 | 78 | 75 width | 91 | 90 | 92 | 91 | 91 | 92 | 91 |
| GPT-5.6-Luna width | 99 | 99 | 99 | 99 | 98 | 99 | 98 width | 71 | 63 | 52 | 45 | 33 | 28 | 24 |
| Kimi-K2.6 width | 100 | 100 | 100 | 100 | 99 | 100 | 99 width | 80 | 78 | 71 | 63 | 53 | 50 | 42 |
Table 6: Quality and Terminated by number of task subgoals (1–7). Both task-fidelity metrics use blue shading. Values are rounded to whole percentages; darker shading indicates higher unrounded scores, with one scale per metric across all seven buckets and simulated models. Human Reference remains unshaded and is treated as terminated by construction.
Figure 15: (a) Naturalness reward throughout training for the Fixed Discriminator and LLM Judge - Likert and Winrate ablations; (b) Naturalness reward throughout training for the dynamic discriminator experiments; (c) Average of the Jensen-Shannon Divergence between utterance-length histograms and turn-count histograms for model-generated and real validation conversations. Lower is better.
Figure 16: For UserLM-2.0, we perform inference with each discriminator trained during RL on validation rollouts from each validation step and report for generated conversations. Blue and red indicate high and low probabilities respectively. UserLM-2.0 learns not only to fool the most recent discriminator, but also keeps fooling older ones as well.
More in this topic
User Simulation
User simulation models human behavior and interaction to test, train, and evaluate AI systems with synthetic users.
See all 127 papers