SimInter: Evidence-Grounded User Simulation with Interview-Based Profile Enrichment for Conversational Recommender Systems
Authors: Erfan Shahabi, Amirhossein Roshandel, Mohammad Sadegh Sadeghi, Azadeh Shakery
Organizations: School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran · Institute for Research in Fundamental Sciences (IPM), Tehran, Iran
Conversational Recommender Systems (CRSs) require reliable user simulators for cost-effective evaluation. LLM-based simulators have made realistic simulation possible but typically condition on profiles derived from explicit ratings, demographics, and static sources capturing surface preferences while omitting implicit behavioral characteristics such as decision-making style, aspect-level preferences, and domain expertise. We propose SimInter, a pipeline that elicits these attributes via synthetic interviews between two LLM agents, grounded in the user's statistical profile and historical conversational evidence. Before profile enrichment, an audit agent explicitly flags potential fabrication and stereotyping at the turn level. We instantiate and evaluate SimInter on movie recommendation. We simulate 300 ReDial users and, on a matched subset of 100 users, compare SimInter with a statistics-only baseline: SimInter significantly increases the diversity of expressed preference aspects (filtered aspect entropy: 1.619 vs. 0.859; paired d = 0.96, Holm-adjusted p < 0.001). This gain persists when the simulator converses with two trained recommenders, KBRD and UniCRS, under two independent evaluation judges (+0.28 to +0.56 nats; all p < 0.001). Human evaluation finds 94.3% of enriched-attribute claims supported or partially supported, while human judgments agree with 95.2% of the auditor's binary filtering decisions on a stratified sample.
Figures & tables
Figure 1: Overview of the SimInter pipeline.
Dimension
CSHI Zhu et al. (2024a)
PerCRS Zhao et al. (2025)
PUB Ma et al. (2025)
SimInter (ours)
Dataset
ReDial, OpenDialKG, MovieLens
DuRecDial 2.0
Amazon Reviews
ReDial
Conversational
Yes
Yes
No
Yes
Profile source
History; generated or editable profiles
History; injected Big Five
Behavioral logs; inferred Big Five
Statistics; audited interviews
Profile ground truth
Target items; held-out interactions
Injected personality traits
Held-out real interactions
Historical preferences; no trait labels
Target-item conditioning
Yes
Yes
N/A
No
Fabrication control
Target-leakage prevention
KG-based factual grounding
Not reported
Evidence-based turn-level auditing
Table 1: Conceptual comparison of user simulation frameworks. Ground truth denotes available evaluation references, not necessarily verified personality traits. Fabrication-control mechanisms differ in scope; N/A indicates not applicable.
Figure 2: Example ReDial user profile after Stage 2, combining historical preferences with derived behavioral statistics.
Figure 3: Example simulated interview between interviewer and interviewee agents.
Track
Metric
Full
StatOnly
CSHI
ΔF−S
ΔF−C
T1
Item entropy
2.034
2.409
1.454
−0.375⋆
+0.588⋆
T1
JSD vs. ReDial
0.681
0.676
0.682
+0.005⋆
−0.001
T2
Population Pearson r
0.171
0.133
0.074
—
—
T3
Aspect entropy (filtered)
1.619
0.859
0.194
+0.760⋆
+1.425⋆
T3
Aspect entropy (unfiltered)
1.857
1.490
0.694
+0.366⋆
+1.163⋆
T4
SBERT diversity (pairwise)
0.532
0.500
0.345
+0.032
+0.186⋆
Table 2: Evaluation results on the matched 100-user subset. StatOnly denotes StatisticalOnly . ΔF−S and ΔF−C denote paired mean differences between Full and StatisticalOnly and CSHI, respectively. ⋆ indicates significance after Holm correction ( padj<0.05 ). CSHI T1 is defined for 94 users (six mention a single movie). T2 correlations are descriptive and computed over condition-specific movie sets.
Figure 4: Effect of interview-based profile enrichment on filtered T3 aspect entropy across 100 matched users. (a) Per-user paired differences between Full and StatisticalOnly ; the dashed line indicates the mean difference and the solid line indicates zero. (b) Percentage of users with zero aspect entropy under both conditions.
Attribute
n
Support
Grounded
Viewing motivation
51
100.0%
47.1%
Aspect prefs. (liked)
60
100.0%
81.7%
Aspect prefs. (descr.)
24
100.0%
91.7%
Pickiness
66
97.0%
65.2%
Domain expertise
36
88.9%
41.7%
Novelty preference
15
86.7%
46.7%
Table 3: Human-evaluation support rates by attribute. Support = Supported + Partially supported; Grounded = Evidence-grounded. n is number of annotations (each claim has three).
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Full
StatOnly
Persona r (pooled)
0.603
0.261
n pairs
52
92
Var. redial
0.078
0.074
% liked (GT)
94.0%
96.3%
n high-conf. pairs
50
81
n users, both classes
2
2
Appendix
Table 4: T2 persona-anchored diagnostics on the matched 100-user subset. Ground-truth labels are near-constant in every condition, making Pearson r unreliable as a discriminative metric.
Issue type
High
Med.
Low
Tot.
fabrication
12
47
72
131
caricature
12
255
105
372
history_contradiction
55
68
50
173
structural_error
0
0
0
0
Total
79
370
227
676
Appendix
Table 5: Distribution of auditor flags by type and severity across 300 interviews (final attempts). Dialogue-level decision distribution: accept 209 , partial 91 , regenerate 0 (one interview regenerated once before receiving partial ).
Figure 5: T1 per-user item entropy. Full is shifted left of StatisticalOnly ( 2.034 vs. 2.409 ; Δ=−0.375 , dz=−0.95 , padj<0.001 ), reflecting a more concentrated movie-mention distribution.
Figure 6: T1 per-user JSD against the ReDial reference distribution. Full is marginally higher ( 0.681 vs. 0.676 ; dz=0.47 , padj<0.001 ); the absolute difference is small.
Figure 7: T3 per-user sentiment entropy. Full is slightly lower ( 0.153 vs. 0.201 ; dz=−0.26 , uncorrected p=0.011 ). Both conditions cluster at zero ( 72 and 64 users), reflecting a strong positive-sentiment imbalance.
Figure 8: T4 per-user SBERT semantic diversity. Full is slightly higher ( 0.532 vs. 0.500 ; dz=0.25 , padj=0.11 ); 10 Full and 15 StatisticalOnly users fall below 0.4 .
Figure 9: T4 per-user corpus TTR. Full TTR is systematically lower ( 0.674 vs. 0.728 ; dz=−0.68 , uncorrected p<0.001 ); enriched users repeat tokens more.
Figure 10: T4 per-user n-gram monotony (Jaccard overlap of bigrams/trigrams between turn pairs; higher = more repetitive). Full is skewed right of StatisticalOnly ( 0.028 vs. 0.019 ; dz=0.36 , uncorrected p<0.001 ), reflecting profile-vocabulary reuse.
Rank
Full aspect
Count
StatOnly aspect
Count
1
mood
164
movie overall
407
2
pacing
103
plot
58
3
movie overall
81
mood
57
4
atmosphere
78
premise
23
5
overall vibe
50
action
22
6
plot
43
overall concept
21
Appendix
Table 6: Top-20 most frequent aspects on the matched 100-user subset (after filtering generic placeholders). Full ’s vocabulary skews toward experiential dimensions ( mood , pacing , atmosphere , energy ); StatisticalOnly is dominated by the near-generic movie overall ( 43% of its aspect mentions vs. 7% for Full ) and broad genre labels. Total distinct non-generic aspects: 267 vs. 174 .
CRS
Evaluation judge
nu
Full
StatOnly
Δ T3
95% CI
pW
dz
KBRD
GPT-4o-mini
111
0.392
0.126
+0.265
[0.144, 0.389]
0.0002
0.40
KBRD
DeepSeek-chat
111
0.350
0.069
+0.280
[0.170, 0.394]
<0.0001
0.46
UniCRS
GPT-4o-mini
80
0.488
0.157
+0.330
[0.179, 0.487]
0.0001
0.47
UniCRS
DeepSeek-chat
80
0.498
0.129
+0.369
[0.214, 0.527]
<0.0001
0.52
Appendix
Table 7: Primary audit-filtered Cross-CRS results for per-user filtered T3 aspect entropy. Here, nu is the number of users with at least one Accept or Partial conversation in both conditions. For each user and condition, aspect–sentiment pairs were pooled across all retained conversations before computing entropy. Differences are Full − StatisticalOnly ; pW denotes the two-sided paired Wilcoxon signed-rank test. All bootstrap confidence intervals exclude zero. Because means and differences are computed from unrounded user-level values, a displayed difference need not equal the subtraction of the two rounded means. Absolute entropy values should not be compared across evaluation judges.
Figure 11: Paired effect of semantic enrichment on filtered T3 aspect entropy. Points represent mean user-level differences and horizontal bars show 95% bootstrap confidence intervals. The vertical line marks the null effect. Circles denote GPT-4o-mini and squares denote DeepSeek-chat; dark and light blue denote KBRD and UniCRS.
Attribute
Sup.
Part.
Uns.
Contr.
Viewing motivation
27.5%
72.5%
0.0%
0.0%
Aspect prefs. (liked)
75.0%
25.0%
0.0%
0.0%
Aspect prefs. (descr.)
33.3%
66.7%
0.0%
0.0%
Pickiness
72.7%
24.2%
0.0%
3.0%
Domain expertise
27.8%
61.1%
11.1%
0.0%
Novelty preference
26.7%
60.0%
13.3%
0.0%
Appendix
Table 8: Full per-attribute breakdown of support ratings (annotation level). The 1.3% Contradicted rate concentrates in disliked aspects ( 9.5% ) and pickiness ( 3.0% ); decision-making style has the highest Unsupported rate ( 21.4% ), consistent with cognitive-style claims being the most amenable to invented rationalization.
Pair
Unwgt. κ
Wgt. κ
Exact
±1 -rank
A vs. B
0.30
0.43
60.0%
98.1%
A vs. C
0.19
0.40
45.7%
97.1%
B vs. C
0.43
0.52
69.5%
98.1%
Appendix
Table 9: Pairwise inter-annotator agreement on the Support label. Weighted κ is quadratic-weighted; “ ±1 -rank” is the rate at which two annotators fell in the same or adjacent ordinal category. Annotator C applies the strictest interpretation of Supported.
Auditor outcome
n
Human
Human
Human
Excluded
Flagged–ret.
Unflagged
Excluded
35
31
4
0
Flagged–retained
35
0
31
4
Unflagged
35
1
5
29
Total
105
32
40
33
Appendix
Table 10: Auditor outcomes versus the majority judgments of three annotators for 105 synthetic interviewee turns. The diagonal corresponds to 91/105 ( 86.7% ) three-class human-referenced accuracy on the balanced sample.
Component
Model
Temp.
Max tok.
Interviewer (S3)
Gemini 2.5 Flash
0.7
512
Interviewee (S3)
Gemini 2.5 Flash
0.6
512
Audit: movie check (S4)
Claude Haiku 4.5
0.2
512
Audit: contradiction (S4)
Claude Haiku 4.5
0.2
512
Audit: fabr./caric. (S4)
Claude Haiku 4.5
0.2
1,024
Enrichment (S5)
Tencent Hy3
0.1
8,192
Appendix
Table 11: LLM components of the primary experiment. S3–S6 denote pipeline stages. The CSHI baseline uses the same CRS configuration.
Run
Calls
Tokens
Gen.
Eval.
Total
in/out (M)
($)
($)
($)
Full (300)
32,907
28.7 / 4.4
8.30
23.85
32.15
StatOnly (100)
11,149
8.5 / 0.9
1.41
8.93
10.34
CSHI (100)
16,589
14.8 / 1.4
1.37
16.90
18.27
Total
60,645
52.1 / 6.8
11.08
49.68
60.76
Appendix
Table 12: API usage and cost (USD) per run of the primary experiment. Hy3 output tokens include reasoning tokens. Numbers in parentheses give the number of users.
Conversational recommender systems (CRSs) are a core component of next-generation intelligent recommender systems because they enable users to actively elicit preferences, clarify intentions, and adapt recommendations in real time. However, there are two key obstacles in the CRS domain: evaluation and access to training data. Evaluating CRSs through real human studies is more critical than for traditional recommender systems, yet such studies are both costly and time-consuming. Moreover, CRS interaction data are often difficult to obtain for model training due to privacy concerns. Large language model (LLM)-based user simulators have shown promise in addressing both challenges by generating synthetic user interactions for evaluation and training. However, existing approaches suffer from systematic positive bias, data leakage, and limited behavioral diversity, and they rely on brittle manual prompt engineering that requires extensive domain expertise. In this paper, we propose a framework to automatically optimize prompts for LLM-based user simulators in CRSs, simultaneously mitigating these issues. Experimental results demonstrate that the proposed framework achieves improved behavioral alignment with human interaction patterns compared to baseline methods across diverse prompt settings.
Training conversational recommender systems (CRS) requires extensive dialogue data, which is challenging to collect at scale. To address this, researchers have used simulated user-recommender conversations. Traditional simulation approaches often utilize a single large language model (LLM) that generates entire conversations with prior knowledge of the target items, leading to scripted and artificial dialogues. We propose a reference-free simulation framework that trains two independent LLMs, one as the user and one as the conversational recommender. These models interact in real-time without access to predetermined target items, but preference summaries and target attributes, enabling the recommender to genuinely infer user preferences through dialogue. This approach produces more realistic and diverse conversations that closely mirror authentic human-AI interactions. Our reference-free simulators match or exceed existing methods in quality, while offering a scalable solution for generating high-quality conversational recommendation data without constraining conversations to pre-defined target items. We conduct both quantitative and human evaluations to confirm the effectiveness of our reference-free approach.
Jerome Ramos, Feng Xia, Xi Wang +4
University College London, London, UK · University of Sheffield, Sheffield, UK · Missouri University of Science and Technology, Rolla, USA
Conversational recommender systems (CRS) increasingly rely on user simulators for automated evaluation of sales agents. A key requirement for such simulators is the ability to model human decision-making. However, most existing simulation frameworks do not explicitly model the internal decision process, and LLM-based simulators often exhibit unrealistically strong information-processing capabilities, rarely exhibit the hesitation or decision deferral commonly observed in real consumer behavior, resulting in overly high acceptance probabilities. To address this limitation, we propose Hesitator, a theory-grounded user simulation framework that explicitly models human decision-making under choice overload. The framework introduces a modular Decision Module that separates utility-based item selection from overload-aware commitment decisions. Experiments across multiple user simulation frameworks, domains, sales modes, and LLM backbones show that integrating our module consistently mitigates unrealistic behaviors under increasing overload conditions. Furthermore, Hesitator reproduces established behavioral patterns from psychological economics, demonstrating its ability to model human decision behavior.
Yuan-Chi Li, Li-Chi Chen, Sung-Yi Wu +2
Department of Computer Science and Information Engineering National Taiwan University