CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking

Anjali Kantharuban; Jonas Mueller

cs.CLOct 1, 2026

CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking

Authors: Anjali Kantharuban, Jonas Mueller

Organizations: Handshake AI · Carnegie Mellon University

Abstract

Recent benchmarks rely on user simulators to evaluate AI agents in multi-turn interaction. While existing simulation techniques demonstrate surface fidelity to human style and behavior, ecologically valid interactive benchmarking also requires alignment in when and how agents fail across simulated and real user populations. We find that existing simulators lack outcome calibration: agreement with observed success rates and failure patterns when real users interact with the same agent. We introduce Calibrated User Embeddings (CUE), a framework that both encodes observed sessions and samples continuous representations, then decodes them into persona commands to steer LLMs to act as user simulators without training. Through this, we evaluate user-conditioned replay of past sessions and aggregate metric agreement when sampling novel personas for the same tasks. On $τ^2$ -Bench, CUEd simulators commit fewer simulator-attributed errors and more faithfully reproduce real-user agent failure modes, aggregate success rates, and outcomes for specific task-user pairs than other persona-based simulation methods. These gains coexist with competitive user fidelity as measured using metrics established in prior work. After being fit to mostly customer support interactions, the same CUEd simulators generalize to document creation, math tutoring, and casual conversation, and remain effective across different simulator LLMs without CUE retraining.

Figures & tables

Figure 1: Method ranks across user fidelity and outcome calibration. Each colored point represents a method’s normalized rank for one setting (base model

\times

evaluation mode), larger points indicate multiple observations at the same rank, and black points indicate mean rank. Ranks are normalized, allowing comparisons with different numbers of applicable methods. *No user-conditioned replay. **No sampled evaluation.

Figure 2: The three components of the CUE framework, with two ways to generate CUEs. : Encode an observed session for retrospective replay, : Sample a new, distributionally plausible CUE from the learned training distribution. Under either option, decoded commands and retrieved training examples augment the task-specific simulator prompt so the base model may act as the user.

Axis	Metric	Type	Source
Nat.	S2R-N $\uparrow$	Features	Chopra et al. (2026)
	TT $\downarrow$	LLM Judge	Wang et al. (2026)
Mim.	AVA $\uparrow$	Embedding	Wang et al. (2025)
	PT3 $\uparrow$	LLM Judge	Zhu et al. (2026)
Cov.	S2R-C $\uparrow$	Features	Chopra et al. (2026)
	SD-C $\uparrow$	Embedding	Patel et al. (2025)

Table 1: User-fidelity metrics adopted from prior work , by axis. Definitions in Appendix D .

Method	Naturalness				Mimicry		Coverage
Method	S2R-N $\uparrow$		TT $\downarrow$		AVA $\uparrow$	PT3 $\uparrow$	S2R-C $\uparrow$	SD-C $\uparrow$
Llama 3.1 8B
None	– /	–	0.44 /	0.44	0.04	0.19	0.01	0.45
UserLM	– /	0.56	– /	0.41	–	–	0.62	0.55
USP	0.26 /	0.27	0.46 /	0.47	0.06	0.02	0.00	0.44
PPol	0.47 /	–	0.18 /	–	0.26	0.13	–	–

Table 2: User fidelity on

\tau^{2}

-Bench. Naturalness entries are user-conditioned/sampled, mimicry uses user-conditioned and coverage uses sampled. Bold marks the best reported value for each variation.

Method	Role Adherence			Failure Reproduction			Success Alignment
Method	User Gap $\downarrow$		Attr. F1 $\uparrow$	Agent TVD $\downarrow$		Mode F1 $\uparrow$	Rate $\|\Delta\|$ $\downarrow$		Outcome F1 $\uparrow$
Llama 3.1 8B
None	0.14 /	0.14	0.35	0.36 /	0.36	0.22	0.21 /	0.21	0.50
UserLM	– /	0.54	–	– /	0.31	–	– /	0.39	–
USP	0.73 /	0.72	0.20	0.31 /	0.93	0.05	0.54 /	0.47	0.32
PPol	0.23 /	–	0.34	0.35 /	–	0.13	0.45 /	–	0.38

Table 3: Outcome calibration on

\tau^{2}

-Bench. Paired entries are user-conditioned/sampled; F1 metrics are only user-conditioned. Attribution F1 conditions on both episodes failing, mode F1 conditions on both having agent-attributed failures. Bold marks best value per variation.

Figure 3: Conditional failure composition over agent-attributed failures. Bars show the proportional distribution of failure modes within each method’s agent-attributed failures. TC = Tool Call .

Base	Method	MAE $\downarrow$
Base	Method	Document Quality	Interaction Quality
Llama 3.1 8B	USP	5.34	1.86
	RealUserSim	6.06	2.21
	CUE	3.55	1.48
GPT 5.4 Mini	RealUserSim	5.80	1.39
GPT 5.4 Mini	CUE	2.21	1.33

Table 4: Outcome calibration over SimulatorArena writing benchmark. Pairwise mean absolute error (MAE) over user-conditioned models using the benchmark’s LLM-judge grader.

Figure 4: Bidirectional stylistic proximity on PRISM. Left: Percent of generated trajectories with a human neighbor. Right: Percent of human trajectories with a generated neighbor. Methods use the same set for all measurements. Displayed intervals aggregate seeds and backbones into a 95% CI.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Train	Dev
LMSYS Chat 1M ( Zheng et al., 2024 )	10,000
WildChat ( Zhao et al., 2024 )	10,000
ABCD ( Chen et al., 2021 )	8,034	1,004
AirDialogue ( Wei et al., 2018 )	10,000	10,000
BiTOD ( Lin et al., 2021 )	2,952	295
CaSiNo ( Chawla et al., 2021 )	900	30

Appendix

Table A1 : CUE source inventory. Reported trajectories per source split; related dataset versions are not necessarily unique conversations, and inventory counts are not annotation-retention counts.

Figure A1 : Encoder architecture. Hierarchical encoder first creates turn embeddings then aggregates them into session-level CUE.

Figure A2 : Decoder architecture. One command (or no-op) is generated for each slot.

Group	Hyperparameter	Value
Encoder	User / system backbone	nomic-ai/modernbert-embed-base
	Bottleneck dimension	1024
	Session dimension	1024
	Session transformer layers	4
	Session attention heads	8
	Max turns per session	64

Appendix

Table A2: Joint encoder-decoder training hyperparameters. During the 200-step LM-format warm-up, the encoder is frozen, the decoder LM is trainable, and representation-level conditioning losses are disabled. After warm-up, the decoder LM is frozen, the remaining objectives are enabled, and the optimizer is rebuilt. ∗ Weight decay is the framework default and is not set explicitly in the configuration.

Figure A3 : Sampler Architecture. Demonstrates two possible paths for generating a novel CUE.

Group	Hyperparameter	Value
Architecture	Latent dimension	1024
	MLP hidden width	1536
	Depth (residual blocks)	6
	Timestep embedding dim	256
	Conditioner heads	4
	Parameters	$\sim$ 162M

Appendix

Table A3: Diffusion sampler hyperparameters. The sampler is trained on a single GPU. ∗ Rollout-time guidance is configured independently from the value used during training-time evaluation.

Metric	Axis	Modes
S2R-N, TT	Naturalness	Both
AVA, PT3	Mimicry	User-conditioned
S2R-C, SD-C	Coverage	Sampled

Appendix

Table A4 : Fidelity metrics and the simulator modes to which they are applied.

Method	S2R-N	TT	AVA	PT3	Success	Outcome F1
Llama 3.1 8B
None	–	0.44 [0.44, 0.44]	0.04 [0.04, 0.05]	0.19 [0.18, 0.19]	0.40 [0.39, 0.41]	0.50 [0.48, 0.52]
USP	0.26 [0.25, 0.26]	0.46 [0.44, 0.48]	0.06 [0.06, 0.06]	0.02 [0.01, 0.02]	0.07 [0.06, 0.08]	0.32 [0.32, 0.33]
PPol	0.47 [0.45, 0.49]	0.18 [0.18, 0.18]	0.26 [0.26, 0.27]	0.13 [0.13, 0.14]	0.16 [0.14, 0.18]	0.38 [0.37, 0.39]
RealUserSim	0.09 [0.09, 0.09]	0.30 [0.29, 0.30]	0.24 [0.24, 0.24]	0.17 [0.14, 0.19]	0.18 [0.18, 0.19]	0.44 [0.44, 0.44]
CUE	0.41 [0.40, 0.42]	0.22 [0.20, 0.30]	0.20 [0.20, 0.20]	0.21 [0.21, 0.21]	0.26 [0.26, 0.27]	0.46 [0.46, 0.47]

Appendix

Table A5 : Replay fidelity and success estimates with reported 95% bootstrap intervals. None has no persona injection; RealUserSim denotes RealUserSim.

Method	S2R-N	TT	S2R-C	SD-C	Success
Llama 3.1 8B
None	–	0.44 [0.44, 0.44]	0.01 [0.00, 0.02]	0.45 [0.44, 0.45]	0.40 [0.39, 0.41]
UserLM	0.56 [0.56, 0.56]	0.41 [0.40, 0.42]	0.62 [0.55, 0.66]	0.55 [0.55, 0.56]	0.22 [0.21, 0.24]
USP	0.27 [0.26, 0.27]	0.47 [0.46, 0.49]	0.00 [0.00, 0.00]	0.44 [0.44, 0.44]	0.14 [0.12, 0.15]
RealUserSim	0.09 [0.09, 0.10]	0.33 [0.30, 0.34]	0.00 [0.00, 0.00]	0.46 [0.45, 0.46]	0.26 [0.25, 0.29]
CUE	0.48 [0.47, 0.49]	0.26 [0.26, 0.26]	0.63 [0.62, 0.63]	0.55 [0.55, 0.55]	0.29 [0.28, 0.30]

Appendix

Table A6 : Sampled fidelity and success estimates with reported 95% bootstrap intervals.

Method	User Share	Success	Adjusted Success	Adjusted Gap $\downarrow$	Discordance
Llama 3.1 8B
None	0.35/0.35	0.40/0.40	0.53/0.53	0.22 /0.22	0.31
UserLM	–/0.75	–/0.22	–/0.70	–/0.05	–
USP	0.94/0.93	0.07/0.14	0.53/0.37	0.22/0.38	0.08
PPol	0.44/–	0.16/–	0.30/–	0.45/–	0.18
RealUserSim	0.50/0.48	0.18/0.26	0.36/0.45	0.39/0.30	0.22

Appendix

Table A7 : Supplementary outcome summaries. Paired values are replay/sampled. Human references are 0.21 for user share, 0.61 for success, and 0.75 for adjusted success. Adjusted success removes user- and environment-attributed failures from each population separately. Discordance is the replay outcome-discordance rate on same-task pairs with different human outcome.

Method	Attribution F1	$n_{FF}$	Failure Mode F1	$n_{AA}$
Llama 3.1 8B
None	0.35 [0.30, 0.40]	347	0.22 [0.15, 0.27]	117
USP	0.20 [0.17, 0.24]	481	0.05 [0.01, 0.08]	19
PPol	0.34 [0.30, 0.39]	446	0.13 [0.08, 0.18]	109
RealUserSim	0.34 [0.29, 0.38]	447	0.22 [0.15, 0.28]	109
CUE	0.38 [0.33, 0.43]	411	0.20 [0.14, 0.26]	118

Appendix

Table A8 : Conditional replay agreement and support. Reported 95% intervals are in brackets.

n_{FF}

counts overlapping human–simulator failures;

n_{AA}

counts pairs where both failures are agent-attributed. Each method selects a different subset.

Method	User Share	Agent TVD	Adjusted Success	Discordance
Llama 3.1 8B
None	0.35 [0.32, 0.38]	0.36 [0.31, 0.46]	0.53 [0.50, 0.56]	0.31 [0.25, 0.37]
USP	0.94 [0.93, 0.95]	0.31 [0.25, 0.48]	0.53 [0.45, 0.61]	0.08 [0.05, 0.11]
PPol	0.44 [0.41, 0.47]	0.35 [0.29, 0.44]	0.30 [0.27, 0.34]	0.18 [0.13, 0.23]
RealUserSim	0.50 [0.48, 0.53]	0.32 [0.27, 0.42]	0.36 [0.32, 0.39]	0.22 [0.18, 0.25]
CUE	0.34 [0.31, 0.37]	0.34 [0.29, 0.45]	0.43 [0.39, 0.46]	0.28 [0.22, 0.33]

Appendix

Table A9 : Replay conditional summaries with reported 95% intervals.

Method	User Share	Agent TVD	Adjusted Success
Llama 3.1 8B
None	0.35 [0.32, 0.38]	0.36 [0.31, 0.46]	0.53 [0.50, 0.56]
UserLM	0.75 [0.72, 0.77]	0.31 [0.25, 0.48]	0.70 [0.65, 0.75]
USP	0.93 [0.91, 0.94]	0.93 [0.87, 0.98]	0.37 [0.32, 0.42]
RealUserSim	0.48 [0.45, 0.50]	0.32 [0.27, 0.42]	0.45 [0.42, 0.49]
CUE	0.34 [0.31, 0.37]	0.33 [0.29, 0.44]	0.46 [0.42, 0.49]

Appendix

Table A10 : Sampled conditional summaries with reported 95% intervals.

Pairing	AVA	PT3
Paired	0.27	0.38
Shuffled	0.20	0.21

Appendix

Table A11 : Gemini mimicry: paired versus shuffled sessions.

Sampling Mode	S2R-C	SD-C
Unconditional	0.83	0.57
Neighborhood	0.84	0.49
Bank	0.77	0.56

Appendix

Table A12 : Gemini coverage for the three sampling modes.

Method	User Share
Llama 3.1 8B
None	0.35 /	0.35
UserLM	– /	0.75
USP	0.94 /	0.93
PPol	0.44 /	–
RealUserSim	0.50 /	0.48

Appendix

Table A13 : Conditional user-side shares among labeled failures from the expanded results. Human reference: 0.21. Results are reported as user-conditioned/sampled.

Base	Model	Naturalness				Mimicry		Coverage
Base	Model	S2R-N ↑		TT ↓		AVA ↑	PT3 ↑	S2R-C ↑	SD-C ↑
SimulatorArena
Llama 3.1 8B	UserLM	– /	0.64	– /	0.37	–		0.84	0.62
	USP	0.41 /	0.38	0.40 /	0.42	0.50	0.01	0.11	0.52
	RealUserSim	0.29 /	0.21	0.19 /	0.39	0.50	0.05	0.00	0.50
	CUE	0.23 /	0.27	0.17 /	0.21	0.44	0.06	0.83	0.56

Appendix

Table A14 : Out-of-domain user fidelity results for SimulatorArena and PRISM. Naturalness results are reported for user-conditioned/sampled methods while mimicry is only reported for user-conditioned methods and coverage is only reported for sampled methods.

Interaction	Executed replacement	Refund	Reward
Human	8555936349 (outside order)	$32.48	0
CUE, seed 0	8555936349 (outside order)	$32.48	0
None, seed 0	1646531091 (within order)	$26.48	1

Appendix

Table A15: Executed outcomes for the selected episode. All three interactions call exchange_delivered_order_items on order #W3470184 , replacing item 2757705742 and using gift_card_7245904 . The native task requires replacement 1646531091 .

Interaction	Reward	Executed outcome	Match?
Human	0	Incorrect target 8555936349	Ref.
CUE paired	0	Same incorrect target 8555936349	Yes
None	1	Correct target 1646531091	No
PPol	1	Corrects its choice before execution	No
RealUserSim paired, no examples	0	No exchange; unsupported order ID	No
UserLM	0	No exchange; unsupported item IDs	No

Appendix

Table A16: Observed outcomes on the selected episode and seed. “Match” requires the same incorrect exchange as the human, not merely reward 0. USP sampled is a population-sampling reference, not paired replay. Prompt provenance limits are described below.

User Simulation

User simulation models human behavior and interaction to test, train, and evaluate AI systems with synthetic users.

See all 128 papers

Mind the Sim2Real Gap in User Simulation for Agentic Tasks
Mar 11, 2026Xuhui Zhou, Weiwei Sun, Qianou Ma +8User Simulation Sim-to-Real Transfer
Carnegie Mellon University, Language Technologies Institute
UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training
Sep 29, 2026Ashish Jain, Armaan SandhuAI Agent Benchmarks User Simulation
Sarvam AI
RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation
Apr 7, 2026Ming Zhu, Juntao Tan, Rithesh Murthy +6User Simulation Human Behavior Simulation
Salesforce AI Research

Mar 11, 2026J/K move · Enter open · S save

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

Xuhui Zhou, Weiwei Sun, Qianou Ma, Yiqing Xie+7

Carnegie Mellon University, Language Technologies Institute

User Simulation Sim-to-Real Transfer LLM Agent Evaluation Human-in-the-Loop Evaluation

Open paper AskPDF arXiv

As NLP evaluation shifts from static benchmarks to multi-turn interactive settings, LLM-based simulators have become widely used as user proxies, serving two roles: generating user turns and providing evaluation signals. Yet, these simulations are frequently assumed to be faithful to real human behaviors, often without rigorous verification. We formalize the Sim2Real gap in user simulation and present the first study running the full

τ

-bench protocol with real humans (451 participants, 165 tasks), benchmarking 31 LLM simulators across proprietary, open-source, and specialized families using the User-Sim Index (USI), a metric we introduce to quantify how well LLM simulators resemble real user interactive behaviors and feedback. Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity, creating an "easy mode" that inflates agent success rates above the human baseline. In evaluations, real humans provide nuanced judgments across eight quality dimensions while simulated users produce uniformly more positive feedback; rule-based rewards are failing to capture rich feedback signals generated by human users. Overall, higher general model capability does not necessarily yield more faithful user simulation. These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle and motivate improved models for user simulation.

Abstract

User Simulation

Explore similar work

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training

RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation