SimInter: Evidence-Grounded User Simulation with Interview-Based Profile Enrichment for Conversational Recommender Systems

Erfan Shahabi; Amirhossein Roshandel; Mohammad Sadegh Sadeghi; Azadeh Shakery

cs.CLOct 13, 2026

SimInter: Evidence-Grounded User Simulation with Interview-Based Profile Enrichment for Conversational Recommender Systems

Authors: Erfan Shahabi, Amirhossein Roshandel, Mohammad Sadegh Sadeghi, Azadeh Shakery

Organizations: School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran · Institute for Research in Fundamental Sciences (IPM), Tehran, Iran

Abstract

Conversational Recommender Systems (CRSs) require reliable user simulators for cost-effective evaluation. LLM-based simulators have made realistic simulation possible but typically condition on profiles derived from explicit ratings, demographics, and static sources capturing surface preferences while omitting implicit behavioral characteristics such as decision-making style, aspect-level preferences, and domain expertise. We propose SimInter, a pipeline that elicits these attributes via synthetic interviews between two LLM agents, grounded in the user's statistical profile and historical conversational evidence. Before profile enrichment, an audit agent explicitly flags potential fabrication and stereotyping at the turn level. We instantiate and evaluate SimInter on movie recommendation. We simulate 300 ReDial users and, on a matched subset of 100 users, compare SimInter with a statistics-only baseline: SimInter significantly increases the diversity of expressed preference aspects (filtered aspect entropy: 1.619 vs. 0.859; paired d = 0.96, Holm-adjusted p < 0.001). This gain persists when the simulator converses with two trained recommenders, KBRD and UniCRS, under two independent evaluation judges (+0.28 to +0.56 nats; all p < 0.001). Human evaluation finds 94.3% of enriched-attribute claims supported or partially supported, while human judgments agree with 95.2% of the auditor's binary filtering decisions on a stratified sample.

Figures & tables

Figure 1: Overview of the SimInter pipeline.

Dimension	CSHI Zhu et al. (2024a)	PerCRS Zhao et al. (2025)	PUB Ma et al. (2025)	SimInter (ours)
Dataset	ReDial, OpenDialKG, MovieLens	DuRecDial 2.0	Amazon Reviews	ReDial
Conversational	Yes	Yes	No	Yes
Profile source	History; generated or editable profiles	History; injected Big Five	Behavioral logs; inferred Big Five	Statistics; audited interviews
Profile ground truth	Target items; held-out interactions	Injected personality traits	Held-out real interactions	Historical preferences; no trait labels
Target-item conditioning	Yes	Yes	N/A	No
Fabrication control	Target-leakage prevention	KG-based factual grounding	Not reported	Evidence-based turn-level auditing

Table 1: Conceptual comparison of user simulation frameworks. Ground truth denotes available evaluation references, not necessarily verified personality traits. Fabrication-control mechanisms differ in scope; N/A indicates not applicable.

Figure 2: Example ReDial user profile after Stage 2, combining historical preferences with derived behavioral statistics.

Figure 3: Example simulated interview between interviewer and interviewee agents.

Track	Metric	Full	StatOnly	CSHI	$\Delta_{F-S}$	$\Delta_{F-C}$
T1	Item entropy	2.034	2.409	1.454	$-0.375^{\star}$	$+0.588^{\star}$
T1	JSD vs. ReDial	0.681	0.676	0.682	$+0.005^{\star}$	$-0.001$
T2	Population Pearson $r$	0.171	0.133	0.074	—	—
T3	Aspect entropy (filtered)	1.619	0.859	0.194	$+0.760^{\star}$	$+1.425^{\star}$
T3	Aspect entropy (unfiltered)	1.857	1.490	0.694	$+0.366^{\star}$	$+1.163^{\star}$
T4	SBERT diversity (pairwise)	0.532	0.500	0.345	$+0.032$	$+0.186^{\star}$

Table 2: Evaluation results on the matched 100-user subset. StatOnly denotes StatisticalOnly .

\Delta_{F-S}

and

\Delta_{F-C}

denote paired mean differences between Full and StatisticalOnly and CSHI, respectively.

\star

indicates significance after Holm correction (

p_{\mathrm{adj}}<0.05

). CSHI T1 is defined for 94 users (six mention a single movie). T2 correlations are descriptive and computed over condition-specific movie sets.

Figure 4: Effect of interview-based profile enrichment on filtered T3 aspect entropy across 100 matched users. (a) Per-user paired differences between Full and StatisticalOnly ; the dashed line indicates the mean difference and the solid line indicates zero. (b) Percentage of users with zero aspect entropy under both conditions.

Attribute	$n$	Support	Grounded
Viewing motivation	51	100.0%	47.1%
Aspect prefs. (liked)	60	100.0%	81.7%
Aspect prefs. (descr.)	24	100.0%	91.7%
Pickiness	66	97.0%	65.2%
Domain expertise	36	88.9%	41.7%
Novelty preference	15	86.7%	46.7%

Table 3: Human-evaluation support rates by attribute. Support = Supported + Partially supported; Grounded = Evidence-grounded.

n

is number of annotations (each claim has three).

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

	Full	StatOnly
Persona $r$ (pooled)	0.603	0.261
$n$ pairs	52	92
Var. ${}_{\text{redial}}$	0.078	0.074
% liked (GT)	94.0%	96.3%
$n$ high-conf. pairs	50	81
$n$ users, both classes	2	2

Appendix

Table 4: T2 persona-anchored diagnostics on the matched 100-user subset. Ground-truth labels are near-constant in every condition, making Pearson

r

unreliable as a discriminative metric.

Issue type	High	Med.	Low	Tot.
fabrication	12	47	72	131
caricature	12	255	105	372
history_contradiction	55	68	50	173
structural_error	0	0	0	0
Total	79	370	227	676

Appendix

Table 5: Distribution of auditor flags by type and severity across

300

interviews (final attempts). Dialogue-level decision distribution: accept

209

, partial

91

, regenerate

0

(one interview regenerated once before receiving partial ).

Figure 5: T1 per-user item entropy. Full is shifted left of StatisticalOnly ( $2.034$ vs. $2.409$ ; $\Delta=-0.375$ , $d_{z}=-0.95$ , $p_{\mathrm{adj}}<0.001$ ), reflecting a more concentrated movie-mention distribution.

Figure 6: T1 per-user JSD against the ReDial reference distribution. Full is marginally higher ( $0.681$ vs. $0.676$ ; $d_{z}=0.47$ , $p_{\mathrm{adj}}<0.001$ ); the absolute difference is small.

Figure 7: T3 per-user sentiment entropy. Full is slightly lower ( $0.153$ vs. $0.201$ ; $d_{z}=-0.26$ , uncorrected $p=0.011$ ). Both conditions cluster at zero ( $72$ and $64$ users), reflecting a strong positive-sentiment imbalance.

Figure 8: T4 per-user SBERT semantic diversity. Full is slightly higher ( $0.532$ vs. $0.500$ ; $d_{z}=0.25$ , $p_{\mathrm{adj}}=0.11$ ); $10$ Full and $15$ StatisticalOnly users fall below $0.4$ .

Figure 9: T4 per-user corpus TTR. Full TTR is systematically lower ( $0.674$ vs. $0.728$ ; $d_{z}=-0.68$ , uncorrected $p<0.001$ ); enriched users repeat tokens more.

Figure 10: T4 per-user n-gram monotony (Jaccard overlap of bigrams/trigrams between turn pairs; higher = more repetitive). Full is skewed right of StatisticalOnly (

0.028

vs.

0.019

;

d_{z}=0.36

, uncorrected

p<0.001

), reflecting profile-vocabulary reuse.

Rank	Full aspect	Count	StatOnly aspect	Count
1	mood	164	movie overall	407
2	pacing	103	plot	58
3	movie overall	81	mood	57
4	atmosphere	78	premise	23
5	overall vibe	50	action	22
6	plot	43	overall concept	21

Appendix

Table 6: Top-20 most frequent aspects on the matched 100-user subset (after filtering generic placeholders). Full ’s vocabulary skews toward experiential dimensions ( mood , pacing , atmosphere , energy ); StatisticalOnly is dominated by the near-generic movie overall (

43\%

of its aspect mentions vs.

7\%

for Full ) and broad genre labels. Total distinct non-generic aspects:

267

vs.

174

CRS	Evaluation judge	$n_{u}$	Full	StatOnly	$\Delta$ T3	95% CI	$p_{\mathrm{W}}$	$d_{z}$
KBRD	GPT-4o-mini	111	0.392	0.126	+0.265	[0.144, 0.389]	0.0002	0.40
KBRD	DeepSeek-chat	111	0.350	0.069	+0.280	[0.170, 0.394]	$<0.0001$	0.46
UniCRS	GPT-4o-mini	80	0.488	0.157	+0.330	[0.179, 0.487]	0.0001	0.47
UniCRS	DeepSeek-chat	80	0.498	0.129	+0.369	[0.214, 0.527]	$<0.0001$	0.52

Appendix

Table 7: Primary audit-filtered Cross-CRS results for per-user filtered T3 aspect entropy. Here,

n_{u}

is the number of users with at least one Accept or Partial conversation in both conditions. For each user and condition, aspect–sentiment pairs were pooled across all retained conversations before computing entropy. Differences are Full

-

StatisticalOnly ;

p_{\mathrm{W}}

denotes the two-sided paired Wilcoxon signed-rank test. All bootstrap confidence intervals exclude zero. Because means and differences are computed from unrounded user-level values, a displayed difference need not equal the subtraction of the two rounded means. Absolute entropy values should not be compared across evaluation judges.

Figure 11: Paired effect of semantic enrichment on filtered T3 aspect entropy. Points represent mean user-level differences and horizontal bars show 95% bootstrap confidence intervals. The vertical line marks the null effect. Circles denote GPT-4o-mini and squares denote DeepSeek-chat; dark and light blue denote KBRD and UniCRS.

Attribute	Sup.	Part.	Uns.	Contr.
Viewing motivation	27.5%	72.5%	0.0%	0.0%
Aspect prefs. (liked)	75.0%	25.0%	0.0%	0.0%
Aspect prefs. (descr.)	33.3%	66.7%	0.0%	0.0%
Pickiness	72.7%	24.2%	0.0%	3.0%
Domain expertise	27.8%	61.1%	11.1%	0.0%
Novelty preference	26.7%	60.0%	13.3%	0.0%

Appendix

Table 8: Full per-attribute breakdown of support ratings (annotation level). The

1.3\%

Contradicted rate concentrates in disliked aspects (

9.5\%

) and pickiness (

3.0\%

); decision-making style has the highest Unsupported rate (

21.4\%

), consistent with cognitive-style claims being the most amenable to invented rationalization.

Pair	Unwgt. $\kappa$	Wgt. $\kappa$	Exact	$\pm 1$ -rank
A vs. B	0.30	0.43	60.0%	98.1%
A vs. C	0.19	0.40	45.7%	97.1%
B vs. C	0.43	0.52	69.5%	98.1%

Appendix

Table 9: Pairwise inter-annotator agreement on the Support label. Weighted

\kappa

is quadratic-weighted; “

\pm 1

-rank” is the rate at which two annotators fell in the same or adjacent ordinal category. Annotator C applies the strictest interpretation of Supported.

Auditor outcome	$n$	Human	Human	Human
		Excluded	Flagged–ret.	Unflagged
Excluded	35	31	4	0
Flagged–retained	35	0	31	4
Unflagged	35	1	5	29
Total	105	32	40	33

Appendix

Table 10: Auditor outcomes versus the majority judgments of three annotators for

105

synthetic interviewee turns. The diagonal corresponds to

91/105

(

86.7\%

) three-class human-referenced accuracy on the balanced sample.

Component	Model	Temp.	Max tok.
Interviewer (S3)	Gemini 2.5 Flash	0.7	512
Interviewee (S3)	Gemini 2.5 Flash	0.6	512
Audit: movie check (S4)	Claude Haiku 4.5	0.2	512
Audit: contradiction (S4)	Claude Haiku 4.5	0.2	512
Audit: fabr./caric. (S4)	Claude Haiku 4.5	0.2	1,024
Enrichment (S5)	Tencent Hy3	0.1	8,192

Appendix

Table 11: LLM components of the primary experiment. S3–S6 denote pipeline stages. The CSHI baseline uses the same CRS configuration.

Run	Calls	Tokens	Gen.	Eval.	Total
		in/out (M)	($)	($)	($)
Full (300)	32,907	28.7 / 4.4	8.30	23.85	32.15
StatOnly (100)	11,149	8.5 / 0.9	1.41	8.93	10.34
CSHI (100)	16,589	14.8 / 1.4	1.37	16.90	18.27
Total	60,645	52.1 / 6.8	11.08	49.68	60.76

Appendix

Table 12: API usage and cost (USD) per run of the primary experiment. Hy3 output tokens include reasoning tokens. Numbers in parentheses give the number of users.

User Simulation

User simulation models human behavior and interaction to test, train, and evaluate AI systems with synthetic users.

See all 121 papers

Prompt Optimization for User Simulation in Conversational Recommender Systems: A Multi-Objective Framework
May 8, 2026Nipun B Nair, Tongtong Wu, Weiqing WangAutomatic Prompt Optimization Recommender Systems
Monash University, Australia
Interplay: Training Independent Simulators for Reference-Free Conversational Recommendation
Mar 19, 2026Jerome Ramos, Feng Xia, Xi Wang +4Recommender Systems User Simulation
University College London, London, UK · University of Sheffield, Sheffield, UK · Missouri University of Science and Technology, Rolla, USA
Decision-aware User Simulation Agent for Evaluating Conversational Recommender Systems
May 5, 2026Yuan-Chi Li, Li-Chi Chen, Sung-Yi Wu +2Human Behavior Simulation Recommender Systems
Department of Computer Science and Information Engineering National Taiwan University

May 8, 2026J/K move · Enter open · S save

Prompt Optimization for User Simulation in Conversational Recommender Systems: A Multi-Objective Framework

Nipun B Nair, Tongtong Wu, Weiqing Wang

Monash University, Australia

Automatic Prompt Optimization Recommender Systems Large Language Model-Based Recommendation User Simulation

Open paper AskPDF arXiv

Conversational recommender systems (CRSs) are a core component of next-generation intelligent recommender systems because they enable users to actively elicit preferences, clarify intentions, and adapt recommendations in real time. However, there are two key obstacles in the CRS domain: evaluation and access to training data. Evaluating CRSs through real human studies is more critical than for traditional recommender systems, yet such studies are both costly and time-consuming. Moreover, CRS interaction data are often difficult to obtain for model training due to privacy concerns. Large language model (LLM)-based user simulators have shown promise in addressing both challenges by generating synthetic user interactions for evaluation and training. However, existing approaches suffer from systematic positive bias, data leakage, and limited behavioral diversity, and they rely on brittle manual prompt engineering that requires extensive domain expertise. In this paper, we propose a framework to automatically optimize prompts for LLM-based user simulators in CRSs, simultaneously mitigating these issues. Experimental results demonstrate that the proposed framework achieves improved behavioral alignment with human interaction patterns compared to baseline methods across diverse prompt settings.

Abstract

User Simulation

Explore similar work

Prompt Optimization for User Simulation in Conversational Recommender Systems: A Multi-Objective Framework

Interplay: Training Independent Simulators for Reference-Free Conversational Recommendation

Decision-aware User Simulation Agent for Evaluating Conversational Recommender Systems