User simulators enable interactive training and evaluation of AI assistants at scale. Yet current simulators struggle to reproduce the diverse behaviors of real users, leading to assistants that generalize poorly to underrepresented users. We identify user representations as an underexplored bottleneck in user simulation. User representations are natural-language descriptions that condition simulators to reproduce a particular user's behavior. A representation's quality becomes apparent in hindsight, through the behavior it induces in a simulator. To evaluate and optimize user representations, we formulate a conversation reconstruction task: given an original conversation, we generate a representation, condition a simulator on it, simulate a conversation, and then measure fidelity by comparing user behavior in the original and simulated conversations. This comparison also yields hindsight feedback, which identifies where the representation falls short and how it can be improved. We introduce hindsight distillation, an online training procedure that distills feedback-conditioned improvements into the representation generator, enabling it to generate improved representations directly from the original conversation. We present UserRepGen-4B, a representation generator trained with hindsight distillation. Across 700 conversations spanning seven datasets, hindsight feedback helps improve user representations at test time, and UserRepGen-4B generates representations that enable more faithful simulation than those generated by GPT-5.5. Beyond conversation reconstruction, learned representations improve simulation fidelity for unseen tasks, better preserve the population-level diversity of real users, and better distinguish users across conversations.
Figures & tables
Figure 1: User representations are a bottleneck in user simulation. They capture the information a simulator needs to faithfully reproduce a user’s behavior. We train UserRepGen-4B to generate high-quality user representations and demonstrate their importance for reconstructing user conversations, simulating users in new scenarios, simulating diverse user populations, and distinguishing users across conversations.
Figure 2: Hindsight distillation for Gθ . A representation is sampled, and is improved with I rounds of hindsight feedback to get r(I) , which serves as a training target.
Figure 3: Simulation fidelity across iterations of hindsight feedback. Shaded regions show 95% bootstrap confidence intervals over conversations.
Model
Best-of-1
Best-of-3
Qwen3.5-4B
3.41
4.42
+ GRPO
3.57 ↑0.16
4.59 ↑0.17
+ SFT
4.64 ↑1.23
5.68 ↑1.26
+ OPSD
4.46 ↑1.05
5.60 ↑1.18
+ OSFT
4.95 ↑1.54
6.02 ↑1.60
Qwen3.5-9B
3.72 ↑0.31
4.68 ↑0.26
Table 1: Simulation fidelity across user representation generators. We report Best-of-1 and Best-of-3 scores. Subscripts indicate absolute improvements over the Qwen3.5-4B, and darker blue indicates stronger scores.
Model
Task
Behavior
Total
R-L (%)
Qwen3.5-4B
61
101
163
27.2
+ GRPO
61
101
162
27.8
+ SFT
58
126
185
25.8
+ OPSD
72
111
183
25.3
+ OSFT
62
148
210
23.9
Table 2: Quantitative analysis of user representations. We report the average word counts of the task, behavior and complete user representations, as well as ROUGE-L precision against the source conversation.
Figure 4: Simulation fidelity when conditioned under configurations \scriptsizeA⃝, \scriptsizeB⃝, \scriptsizeC⃝, and \scriptsizeD⃝. Each cell reports Best-of-3 scores. Subscripts indicate absolute improvement relative to the condition immediately to the left.
Task Only
Task + Behavior
Model
JS↓
SB ↓
JS↓
SB ↓
Qwen3.5-4B
0.389
0.257
0.315
0.227
+ GRPO
0.356
0.256
0.338
0.217
+ SFT
0.353
0.264
0.270
0.196
+ OPSD
0.326
0.239
0.308
0.202
+ OSFT
0.354
0.246
0.273
0.192
Table 3: Distributional gap between original and simulated user behaviors for our different representation generators when the simulator is conditioned on the task description alone or both the task and behavior descriptions. We report ( JS ) divergence and Self-BLEU (SB). Darker shades of blue and lower values indicate better performance.
Representation Generator
Accuracy (%)
Full conversation (baseline)
44.22
Qwen3.5-4B
38.78 ↓5.44
+ GRPO
41.50 ↓2.72
+ SFT
43.54 ↓0.68
+ OPSD
50.34 ↑6.12
+ OSFT
51.02 ↑6.80
Table 4: GPT-5.4 mini accuracy at distinguishing users across conversations. Subscripts indicate performance compared to the full conversation baseline, and darker blue indicates higher accuracy.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Score gap
Agreement
Rate
1
33 / 41
80.5%
2
17 / 20
85.0%
≥3
38 / 39
97.4%
Overall
88 / 100
88.0%
Appendix
Table 5: Agreement between our manually labeled preferences and the LLM judge. The score gap is the absolute difference between the two simulated conversations’ overall fidelity scores.
Hyperparameter
GRPO
OPSD
SFT
OSFT
Learning rate
5×10−6
5×10−6
5×10−6
5×10−6
Effective batch size
32
32
32
32
Training steps
100
100
100
100
LoRA rank ( r )
32
64
64
64
LoRA alpha ( α )
64
128
128
128
Maximum prompt length
8,192
8,192
8,192
8,192
Appendix
Table 6: Training hyperparameters for GRPO, OPSD, SFT, and OSFT.
Question
UserRepGen-4B Selected (%)
Task description
Captures constraints introduced in later turns
63.6
Captures shifts in direction or goal
61.0
Captures multiple objectives or subtasks
58.3
Has less repetition
33.1
Behavior description
Appendix
Table 7: Comparing user representations generated by our trained and untrained models along several dimensions. We report the percentage of times that GPT-5.4 mini selects the representation generated by UserRepGen-4B over the one generated by Qwen3.5-4B.
Users increasingly rely on AI systems for extended, multi-turn tasks, yet evidence indicates that LLM-based system performance degrades over prolonged interaction. To evaluate and improve AI assistants in such settings, we need user simulators that faithfully follow complex user intents while exhibiting realistic conversational behavior. Existing approaches manifest a trade-off: prompted assistants can follow complex user intents but lack behavioral fidelity, while specialized user simulators better approximate conversational behavior but struggle to remain faithful to complex intents. In this paper, we propose a reinforcement learning (RL) recipe that jointly rewards task intent adherence and behavioral naturalness. A crucial contribution of the method is operationalizing naturalness with a lightweight discriminator trained in an alternating, adversarial loop with the user simulator. We release the resulting model, UserLM-2.0, which produces valid simulated conversations (i.e., meeting both task-faithfulness and naturalness criteria) 46.2% of the time, over three times the rate of the strongest prior user simulator (14.1%).
User simulators are gaining attention as environments to train LLM assistants that can better cope with the nuances of multi-turn conversations with real users. Prior work on user simulation has emphasized generating realistic user utterances, leaving unexplored their potential as a source of direct feedback on the assistant's responses. Such evaluation signals may not be explicitly communicated in language, but can instead reside in the latent reasoning of users. In this work, we introduce user simulators with intrinsic feedback, namely, models equipped with latent evaluation states: internal thoughts, reactions, and scalar ratings that evaluate the assistant from a first-person user perspective at each conversation turn. We train LLM-based simulators using interaction logs between real users and assistants and demonstrate that intrinsic feedback improves simulation realism and critiquing behaviors versus prompted simulator baselines, while also providing turn-level evaluations that existing fine-tuned simulators lack. Further, evaluations generated by our trained simulators correlate better with human judgments than those of prompted simulators of the same size. Lastly, our experiments reveal that simulators with intrinsic feedback provide effective reward signals for multi-turn RL training, matching setups that rely on external judge LLMs for rewards while cutting RL compute by more than half.
User simulators are increasingly leveraged to build interactive AI assistants, yet how to measure the quality of these simulators remains an open question. In this work, we show how simulator quality can be quantified in terms of its downstream utility: how an LLM assistant trained with this user simulator performs in the wild when interacting with real humans. In a controlled experiment where only the user simulator varies, we train LLM assistants via reinforcement learning against a spectrum of simulators, from an LLM prompted to role-play a user to one fine-tuned on human utterances from WildChat. As evaluation, we measure pairwise win rates in a user study with 283 participants and on WildBench, a benchmark derived from real human--AI conversations. Training against the role-playing LLM yields an assistant statistically indistinguishable from the initial assistant in our user study (51% win rate), whereas training against the fine-tuned simulator yields significant gains (58% over the initial and 57% over the one trained against role-playing). Closer inspection reveals three further patterns: methods for making role-playing LLMs more realistic (e.g., persona conditioning) improve trained assistants but do not close the gap to the fine-tuned simulator; scaling the simulator's model size benefits the fine-tuned simulator but yields no gain for role-playing ones; and assistants trained against role-playing simulators fail to generalize when paired with other simulators at test time, while the one trained against fine-tuned simulator does. Together, these results argue for grounding user simulators in real human behavior and measuring their quality by their downstream effect on real users.