ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders

Ofer Meshi; Krisztian Balog; Sally Goldman; Avi Caciularu; Guy Tennenholtz; Jihwan Jeong; Amir Globerson; Craig Boutilier

cs.CLFeb 18, 2026

ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders

Authors: Ofer Meshi, Krisztian Balog, Sally Goldman, Avi Caciularu, Guy Tennenholtz, Jihwan Jeong, Amir Globerson, Craig Boutilier

Organizations: Google

Abstract

The promise of LLM-based user simulators to improve conversational AI is hindered by a critical "realism gap," leading to systems that are optimized for simulated interactions, but may fail to perform well in the real world. We introduce ConvApparel, a new dataset of human-AI conversations designed to address this gap. Its unique dual-agent data collection protocol -- using both "good" and "bad" recommenders -- enables counterfactual validation by capturing a wide spectrum of user experiences, enriched with first-person annotations of user satisfaction. We propose a comprehensive validation framework that combines statistical alignment, a human-likeness score, and counterfactual validation to test for generalization. Our experiments reveal a significant realism gap across all simulators. However, the framework also shows that data-driven simulators outperform a prompted baseline, particularly in counterfactual validation where they adapt more realistically to unseen behaviors, suggesting they embody more robust, if imperfect, user models.

Figures & tables

Panel of Figure 1 — Figure 1: A conversation transcript from ConvApparel between a user and the “good” conversational recommender. (The carousels with recommended items are not shown for brevity; see Fig. 8 .)

Figure 3: Discriminator performance and HLS as a function of the number of training examples.

Panel of Figure 4 — Figure 4: Population-Level Statistical Alignment (PLSA) with the good (top) and bad (bottom) recommenders. Data-driven simulators (ICL, SFT) consistently align more closely with human behavioral distributions than the prompted baseline across general statistics, dialog acts, and inferred user experience.

Panel of Figure 5 — Figure 5: Counterfactual validation results. Simulators are trained on one agent type and tested on an unseen one (top: good $\rightarrow$ bad, bottom: bad $\rightarrow$ good). The ICL and SFT models show stronger generalization than the prompted baseline, adapting their behavior to the new agent and more closely matching human patterns.

	Training - Test
Metric	Prompted Prompted	Prompted SFT	SFT SFT	SFT Prompted
Accuracy	1.000	0.476	1.000	0.978
Precision	1.000	0.467	1.000	0.966
Recall	1.000	0.988	1.000	0.991
F1 Score	1.000	0.634	1.000	0.978
Specificity	1.000	0.041	1.000	0.964

Table 1: Discriminator Generalization Performance

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Confusion matrices comparing the agreement between different raters (Rows) across three evaluation metrics (Columns).

	Human-Prompted		Human-ICL		Human-SFT
Metric	MWU $p$ $\uparrow$	KS stat $\downarrow$	MWU $p$ $\uparrow$	KS stat $\downarrow$	MWU $p$ $\uparrow$	KS stat $\downarrow$
Num turns	0.000	0.791	0.000	0.343	0.000	0.622
Num user words	0.000	0.461	0.000	0.419	0.000	0.230
Turns with question	0.000	0.442	0.000	0.242	0.000	0.098
Critique	0.000	0.065	0.004	0.043	0.000	0.053
Greet/Thank	0.005	0.074	0.000	0.227	0.000	0.260

Appendix

Table 2: Mann-Whitney U (MWU) p-values and Kolmogorov-Smirnov (KS) statistics for different PLSA metrics across simulator types for the “good” recommender.

	Human-Prompted		Human-ICL		Human-SFT
Metric	MWU $p$ $\uparrow$	KS stat $\downarrow$	MWU $p$ $\uparrow$	KS stat $\downarrow$	MWU $p$ $\uparrow$	KS stat $\downarrow$
Num turns	0.000	0.872	0.000	0.542	0.000	0.673
Num user words	0.000	0.755	0.000	0.575	0.004	0.217
Turns with question	0.000	0.333	0.000	0.210	0.019	0.082
Critique	0.000	0.141	0.572	0.010	0.000	0.053
Greet/Thank	0.010	0.068	0.000	0.165	0.000	0.157

Appendix

Table 3: Mann-Whitney U (MWU) p-values and Kolmogorov-Smirnov (KS) statistics for different PLSA metrics across simulator types for the “bad” recommender.

	Human-Prompted		Human-ICL good		Human-SFT good
Metric	MWU $p$ $\uparrow$	KS stat $\downarrow$	MWU $p$ $\uparrow$	KS stat $\downarrow$	MWU $p$ $\uparrow$	KS stat $\downarrow$
Num turns	0.000	0.872	0.000	0.649	0.000	0.436
Num user words	0.000	0.755	0.000	0.557	0.241	0.154
Turns with question	0.000	0.333	0.008	0.083	0.110	0.055
Critique	0.000	0.141	0.006	0.050	0.155	0.023
Greet/Thank	0.010	0.068	0.000	0.147	0.000	0.177

Appendix

Table 4: Mann-Whitney U (MWU) p-values and Kolmogorov-Smirnov (KS) statistics for different PLSA metrics across simulator types for the “bad” recommender, where ICL/SFT are trained with data from the “good” recommender.

	Human-Prompted		Human-ICL bad
Metric	MWU $p$ $\uparrow$	KS stat $\downarrow$	MWU $p$ $\uparrow$	KS stat $\downarrow$
Num turns	0.000	0.791	0.001	0.283
Num user words	0.000	0.461	0.064	0.237
Turns with question	0.000	0.442	0.000	0.208
Critique	0.000	0.065	0.000	0.060
Greet/Thank	0.005	0.074	0.000	0.203

Appendix

Table 5: Mann-Whitney U (MWU) p-values and Kolmogorov-Smirnov (KS) statistics for different PLSA metrics across simulator types for the “good” recommender, where ICL uses data from the “bad” recommender.

Panel of Figure 7 — Figure 7: Human-vs-human baseline: evaluation metrics comparing two samples from the ConvApparel dataset.

	Human-Human
Metric	MWU $p$ $\uparrow$	KS stat $\downarrow$
Num turns	0.682	0.033
Num user words	0.613	0.045
Turns with question	0.707	0.021
Critique	0.810	0.025
Greet/Thank	0.065	0.079

Appendix

Table 6: Mann-Whitney U (MWU) p-values and Kolmogorov-Smirnov (KS) statistics for different PLSA metrics between two samples of human conversations.

Figure 8: The ConvApparel study interface.

Panel of Figure 9 — Figure 9: Task-level survey responses.

Panel of Figure 10 — Figure 10: Turn-level results.

Panel of Figure 11 — Figure 11: Turn-level ratings.

User Simulation

User simulation models human behavior and interaction to test, train, and evaluate AI systems with synthetic users.

See all 121 papers

Mind the Sim2Real Gap in User Simulation for Agentic Tasks
Mar 11, 2026Xuhui Zhou, Weiwei Sun, Qianou Ma +8User Simulation Sim-to-Real Transfer
Carnegie Mellon University, Language Technologies Institute
RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation
Apr 7, 2026Ming Zhu, Juntao Tan, Rithesh Murthy +6User Simulation Human Behavior Simulation
Salesforce AI Research
Prompt Optimization for User Simulation in Conversational Recommender Systems: A Multi-Objective Framework
May 8, 2026Nipun B Nair, Tongtong Wu, Weiqing WangAutomatic Prompt Optimization Recommender Systems
Monash University, Australia

Mar 11, 2026J/K move · Enter open · S save

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

Xuhui Zhou, Weiwei Sun, Qianou Ma, Yiqing Xie+7

Carnegie Mellon University, Language Technologies Institute

User Simulation Sim-to-Real Transfer LLM Agent Evaluation Human-in-the-Loop Evaluation

Open paper AskPDF arXiv

As NLP evaluation shifts from static benchmarks to multi-turn interactive settings, LLM-based simulators have become widely used as user proxies, serving two roles: generating user turns and providing evaluation signals. Yet, these simulations are frequently assumed to be faithful to real human behaviors, often without rigorous verification. We formalize the Sim2Real gap in user simulation and present the first study running the full

τ

-bench protocol with real humans (451 participants, 165 tasks), benchmarking 31 LLM simulators across proprietary, open-source, and specialized families using the User-Sim Index (USI), a metric we introduce to quantify how well LLM simulators resemble real user interactive behaviors and feedback. Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity, creating an "easy mode" that inflates agent success rates above the human baseline. In evaluations, real humans provide nuanced judgments across eight quality dimensions while simulated users produce uniformly more positive feedback; rule-based rewards are failing to capture rich feedback signals generated by human users. Overall, higher general model capability does not necessarily yield more faithful user simulation. These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle and motivate improved models for user simulation.

Abstract

User Simulation

Explore similar work

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation

Prompt Optimization for User Simulation in Conversational Recommender Systems: A Multi-Objective Framework