Training User Simulators with Latent Intrinsic Feedback

Tarek Naous; Craig Boutilier; Wei Xu; Ofer Meshi

cs.CLOct 9, 2026

Training User Simulators with Latent Intrinsic Feedback

Authors: Tarek Naous, Craig Boutilier, Wei Xu, Ofer Meshi

Organizations: Google Research · Georgia Institute of Technology

Abstract

User simulators are gaining attention as environments to train LLM assistants that can better cope with the nuances of multi-turn conversations with real users. Prior work on user simulation has emphasized generating realistic user utterances, leaving unexplored their potential as a source of direct feedback on the assistant's responses. Such evaluation signals may not be explicitly communicated in language, but can instead reside in the latent reasoning of users. In this work, we introduce user simulators with intrinsic feedback, namely, models equipped with latent evaluation states: internal thoughts, reactions, and scalar ratings that evaluate the assistant from a first-person user perspective at each conversation turn. We train LLM-based simulators using interaction logs between real users and assistants and demonstrate that intrinsic feedback improves simulation realism and critiquing behaviors versus prompted simulator baselines, while also providing turn-level evaluations that existing fine-tuned simulators lack. Further, evaluations generated by our trained simulators correlate better with human judgments than those of prompted simulators of the same size. Lastly, our experiments reveal that simulators with intrinsic feedback provide effective reward signals for multi-turn RL training, matching setups that rely on external judge LLMs for rewards while cutting RL compute by more than half.

Figures & tables

Figure 1: Example conversation between a user simulator ( ) with latent evaluation states and an LLM assistant ( ) where the simulator is trying to get help to create a browser extension. At each turn, before generating the next user utterance, the simulator generates latent states with multiple components (thoughts, ratings, reactions, etc.) focused on evaluating the current assistant response.

		Feedback Behavior		Simulation Realism
Simulator		F1 Score ( $\uparrow$ )	Correlation ( $\uparrow$ )	Statistical ( $\uparrow$ )	Perplexity ( $\downarrow$ )
Gemini3.1-Pro	IT	78.95	0.4904	0.6396	—
Gemma3-4B	IT	57.33	0.3692	0.5869	17.48
	SFT	72.14	—	0.7502	4.05
	LES	69.29	0.3944	0.7628	3.65
Gemma3-12B	IT	67.13	0.3747	0.5135	13.41

Table 1: Evaluation of the feedback behavior and simulation realism of different simulators on the held-out set of 4,000 conversations from WildChat. We compare prompting instruction-tuned (IT) models to roleplay users with fine-tuning pre-trained base models to generate user turns (SFT) and fine-tuning with LESs. All simulators are conditioned on a general user intent for each conversation before the first user turn. We report additional results by another model family (Qwen at the 3B, 7B, and 14B scales) in Appendix C.2 where we observe similar trends.

	SimulatorArena
	Doc	Math	All
Gemma3-4b-IT	0.073	0.378	0.244
Gemma3-4b-LES	0.273 ∗∗	0.440	0.384 ∗∗
Gemma3-12b-IT	0.158	0.402	0.295
Gemma3-12b-LES	0.330 ∗	0.412	0.401 ∗∗
Gemma3-27b-IT	0.225	0.421	0.343

Table 4: Spearman correlations between ratings generated by user simulators and ground-truth human ratings for conversations from the SimulatorArena out-of-domain dataset ( Dou et al., 2025 ) . LES simulators correlate more strongly with human judgments than prompted baselines at every scale. We report the full correlation differences

\Delta\rho

with confidence intervals in Appendix C.1 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Statistics from the latent evaluation states generated for user turns in the WildChat dataset. (Left): Distribution of Rating and Instruction_Followed latent sub-states for user turns stratified by Reaction_Type (Neutral, Praise, Critique). (Right) : Categorization of the extracted Feedback in critiquing turns based on clustering with Gemini3.1-Pro.

		Feedback Behavior
Simulator		Precision ( $\uparrow$ )	Recall ( $\uparrow$ )	F1 ( $\uparrow$ )
Gemini3.1-Pro	IT	100.00	65.22	78.95
Gemma3-4B	IT	91.00	41.84	57.33
	SFT	92.11	59.29	72.14
	LES	92.38	55.43	69.29
Gemma3-12B	IT	100.00	50.52	67.13

Appendix

Table 6: Precision and Recall results on the WildChat heldout set for Gemma3-based simulators.

Simulator	Subset	$n$	$\rho_{\text{LES}}$	$\rho_{\text{IT}}$	$\Delta\rho$	95% CI	$p$
Gemma3-4b	Doc	459	0.273	0.073	+0.200	[+0.079, +0.319]	0.008 ∗∗
	Math	450	0.440	0.378	+0.062	[-0.031, +0.154]	0.725
	All	909	0.384	0.244	+0.140	[+0.067, +0.214]	0.002 ∗∗
Gemma3-12b	Doc	459	0.330	0.158	+0.172	[+0.068, +0.276]	0.016 ∗
	Math	450	0.412	0.402	+0.010	[-0.077, +0.097]	0.945
	All	909	0.401	0.295	+0.106	[+0.039, +0.174]	0.008 ∗∗

Appendix

Table 7: Pairwise comparison of LES simulators and prompted (IT) simulators correlations on SimulatorArena human ratings at each scale.

\Delta\rho=\rho_{\text{LES}}-\rho_{\text{IT}}

, with a 95% percentile bootstrap interval over conversations (10,000 resamples) and a two-sided bootstrap

p

-value, Holm-corrected across the nine tests.

{}^{*}p<0.05

{}^{**}p<0.01

. LES-based simulators lead at every scale, significantly so at 4b and 12b.

	SimulatorArena
	Doc	Math	All
Qwen2.5-3b-Instruct	0.227	0.224	0.253
Qwen2.5-3b-LES	0.394 ∗∗	0.363 ∗	0.392 ∗∗
Qwen2.5-7b-Instruct	0.282	0.129	0.240
Qwen2.5-7b-LES	0.345	0.384 ∗∗	0.398 ∗∗
Qwen2.5-14b-Instruct	0.331	0.323	0.339

Appendix

Table 9: Spearman correlation results between ratings generated by Qwen2.5 simulators and ground-truth human judgments for SimulatorArena conversations.

Simulator	Subset	$n$	$\rho_{\text{LES}}$	$\rho_{\text{Instruct}}$	$\Delta\rho$	95% CI	$p$
Qwen2.5-3b	Doc	459	0.394	0.227	+0.167	[+0.066, +0.267]	0.010 ∗∗
	Math	450	0.363	0.224	+0.139	[+0.040, +0.240]	0.025 ∗
	All	909	0.392	0.253	+0.139	[+0.080, +0.197]	0.002 ∗∗
Qwen2.5-7b	Doc	459	0.345	0.282	+0.063	[-0.020, +0.149]	0.400
	Math	450	0.384	0.129	+0.256	[+0.155, +0.357]	0.002 ∗∗
	All	909	0.398	0.240	+0.158	[+0.102, +0.216]	0.002 ∗∗

Appendix

Table 10: Pairwise comparison of LES-based and prompted (Instruct) Qwen2.5 simulators.

\Delta\rho=\rho_{\text{LES}}-\rho_{\text{IT}}

with a 95% percentile bootstrap interval over conversations (10,000 resamples) and a two-sided bootstrap

p

-value, Holm-corrected across the nine tests.

{}^{*}p<0.05

{}^{**}p<0.01

LLM Assistant	Reward	Satisfy (%)
Qwen2.5-1.5B-Instruct	—	28.78 $\pm$ 0.17
$\hookrightarrow$ GRPO w/ Gemma3-4B-IT	Judge (Qwen2.5-3B-Instruct)	32.51 $\pm$ 0.94
$\hookrightarrow$ GRPO w/ Gemma3-4B-LES	Judge (Qwen2.5-3B-Instruct)	34.82 $\pm$ 0.93
$\hookrightarrow$ GRPO w/ Gemma3-4B-IT	Judge (Qwen2.5-14B-Instruct)	39.79 $\pm$ 0.70
$\hookrightarrow$ GRPO w/ Gemma3-4B-LES	Judge (Qwen2.5-14B-Instruct)	41.08 $\pm$ 0.97

Appendix

Table 11: Checklist satisfaction rates on WildBenchv2 by LLM assistants after multi-turn GRPO training with a small Qwen2.5-3B-Instruct judge vs a larger Qwen2.5-14B-Instruct.

Figure 3: Reward curves for RL training of Qwen2.5-1.5B using multi-turn GRPO with Gemma3-4B-IT and Gemma3-4B-LES user simulators. We show the training curves that compare using intrinsic rewards directly from the simulators vs. using an external judge model (Qwen2.5-14B-Instruct) to compute rewards. The curves show the mean reward per training step, smoothed using a time-weighted exponential moving average.

Panel of Figure 5 — Figure 5: Prompt template used to generate the generic user intents for WildChat conversations with Gemini3.1-Pro. We use the exact same prompt used by Naous et al. (2026) .

Panel of Figure 6 — Figure 6: Prompt template used to simulate users by prompting an assistant LLM (Gemma3-12B-IT, Gemma3-27B-IT, and Gemini3.1-Pro) to roleplay a user. We instruct the model to generate a realistic user response and also a rating that reflects the user statisfaction by the assistant’s last output.

User Simulation

User simulation models human behavior and interaction to test, train, and evaluate AI systems with synthetic users.

See all 127 papers

Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants
May 10, 2026Joseph Suh, Ayush Raj, Minwoo Kang +1LLM Agent Evaluation Human-AI Interaction
UC Berkeley
Learning User Simulators with Turing Rewards
Jun 17, 2026Yingshan Susan Wang, Cedegao E. Zhang, Linlu Qiu +5Reward Modeling RL for Language Models
Massachusetts Institute of Technology · Stanford University · MIT-IBM Watson AI Lab
Mind the Sim2Real Gap in User Simulation for Agentic Tasks
Mar 11, 2026Xuhui Zhou, Weiwei Sun, Qianou Ma +8User Simulation Sim-to-Real Transfer
Carnegie Mellon University, Language Technologies Institute

May 10, 2026J/K move · Enter open · S save

Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants

Joseph Suh, Ayush Raj, Minwoo Kang, Serina Chang

UC Berkeley

LLM Agent Evaluation Human-AI Interaction User Simulation

Open paper AskPDF arXiv

User simulators are increasingly leveraged to build interactive AI assistants, yet how to measure the quality of these simulators remains an open question. In this work, we show how simulator quality can be quantified in terms of its downstream utility: how an LLM assistant trained with this user simulator performs in the wild when interacting with real humans. In a controlled experiment where only the user simulator varies, we train LLM assistants via reinforcement learning against a spectrum of simulators, from an LLM prompted to role-play a user to one fine-tuned on human utterances from WildChat. As evaluation, we measure pairwise win rates in a user study with 283 participants and on WildBench, a benchmark derived from real human--AI conversations. Training against the role-playing LLM yields an assistant statistically indistinguishable from the initial assistant in our user study (51% win rate), whereas training against the fine-tuned simulator yields significant gains (58% over the initial and 57% over the one trained against role-playing). Closer inspection reveals three further patterns: methods for making role-playing LLMs more realistic (e.g., persona conditioning) improve trained assistants but do not close the gap to the fine-tuned simulator; scaling the simulator's model size benefits the fine-tuned simulator but yields no gain for role-playing ones; and assistants trained against role-playing simulators fail to generalize when paired with other simulators at test time, while the one trained against fine-tuned simulator does. Together, these results argue for grounding user simulators in real human behavior and measuring their quality by their downstream effect on real users.

Abstract

User Simulation

Explore similar work

Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants

Learning User Simulators with Turing Rewards

Mind the Sim2Real Gap in User Simulation for Agentic Tasks