Ask, Answer, and Detect: Role-Playing LLMs for Personality Detection with Question-Conditioned Mixture-of-Experts

Yifan Lyu; Liang Zhang

cs.CLDec 9, 2025

Ask, Answer, and Detect: Role-Playing LLMs for Personality Detection with Question-Conditioned Mixture-of-Experts

Authors: Yifan Lyu, Liang Zhang

Organizations: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · University of International Business and Economics, Beijing, China

Abstract

Understanding human personality is important for personalized recommendation and mental health assessment. Existing personality detection methods typically encode social media posts into user representations and predict personality labels. Despite advances in large language models (LLMs), these methods face limited supervision from scarce labels and an unclear mapping between user language and psychological constructs. We propose ROME, a framework that incorporates psychological knowledge into personality detection. Inspired by standardized self-assessment tests, ROME uses LLM role-playing to simulate users' responses to validated psychometric questionnaires. These question-level answers provide interpretable, questionnaire-grounded evidence and intermediate supervision. A question-conditioned Mixture-of-Experts module jointly routes post and question representations to predict questionnaire answers under explicit supervision. The predicted answers are aggregated into an interpretable answer vector and fused with user representations for final personality prediction in a multi-task learning framework. Experiments on two real-world datasets show that ROME outperforms state-of-the-art baselines, with relative improvements of 13.60% on Kaggle and 19.72% on Pandora. Code is available at https://github.com/JasmineLyf/ROME.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of modeling paradigms for personality detection.

Figure 3: Prompt design for role-play-based questionnaire response generation.

Methods	Kaggle					Pandora
Methods	I/E	S/N	T/F	P/J	Avg	I/E	S/N	T/F	P/J	Avg
XGBoost	56.67	52.85	75.42	65.94	62.72	45.99	48.93	63.51	55.55	53.50
BiLSTM	57.82	57.87	69.97	57.01	60.67	48.01	52.01	63.48	56.21	54.93
BERTconcat	58.33	53.88	69.36	60.88	60.61	54.22	49.15	58.31	53.14	53.71
BERTmean	64.05	57.82	77.06	65.25	66.04	56.60	48.71	64.70	56.07	56.52
AttRCNN	59.74	64.08	78.77	66.44	67.25	48.55	56.19	64.39	57.26	56.60

Table 1: Performance comparison on the Kaggle and Pandora datasets.

Methods	I/E	S/N	T/F	P/J	Avg
ROME ${}_{\text{w/o q-weighting}}$	90.13	94.82	96.36	72.55	88.47
ROME ${}_{\text{w/o gated fusion}}$	89.03	94.45	96.36	74.16	88.50
ROME ${}_{\text{w/o posts}}$	86.56	71.21	94.18	48.51	75.12
ROME ${}_{\text{w/o evidence}}$	42.33	92.35	75.11	53.27	65.77
ROME ${}_{\text{label-only Ask}}$	47.02	91.53	79.02	56.09	68.42
ROME ${}_{\text{posts-only Ask}}$	83.69	94.02	92.42	71.51	85.41

Table 2: Results of the ablation study on the Kaggle dataset.

Methods	I/E	S/N	T/F	P/J	Avg
ROME ${}_{\text{user-only MLP}}$	45.15	91.96	80.27	57.90	68.82
ROME ${}_{\text{q-cond. MLP}}$	64.14	92.83	88.82	66.76	78.14
ROME ${}_{\text{uniform MoE}}$	82.58	93.16	93.30	71.16	85.05
ROME ${}_{\text{user-cond. MoE}}$	91.46	93.46	95.29	76.98	89.30
ROME	92.24	95.90	97.45	81.32	91.73

Table 3: Ablation study of the Answer module on the Kaggle dataset.

Methods	I/E	S/N	T/F	P/J	Avg
ChatGPT-3.5	65.86	51.69	78.60	63.93	65.02
ChatGPT-3.5 ${}_{\text{3-shot}}$	70.61	58.35	76.58	65.43	67.74
ChatGPT-4o	81.70	78.15	85.13	74.67	79.91
ChatGPT-4o ${}_{\text{3-shot}}$	80.55	76.53	84.87	78.13	80.02
PsyCoT-3.5	66.56	61.70	74.80	57.83	65.22
HIPPD-4o	73.61	77.25	84.98	79.33	78.79

Table 4: Inference-Time LLM performance on the Kaggle dataset.

Figure 4: Effect of training data size on the Kaggle dataset.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Types	Train	Validation	Test
Kaggle	I/E	4032/1173	1330/405	1314/421
	S/N	724/4481	230/1505	243/1492
	T/F	2388/2817	802/933	791/944
	P/J	3160/2045	1007/728	1074/661
Pandora	I/E	4314/1126	1425/388	1403/411
	S/N	621/4819	202/1611	205/1609

Appendix

Table 5: Statistics of the two datasets.

Panel of Figure 5 — Figure 5: Prompt used for question-to-construct assignment.

Question IDs	Assigned dimensions
Q1–Q10	I/E, S/N, T/F, P/J, T/F, I/E, P/J, T/F, P/J, T/F
Q11–Q20	I/E, S/N, T/F, P/J, T/F, I/E, S/N, T/F, S/N, T/F
Q21–Q30	I/E, S/N, T/F, P/J, T/F, I/E, T/F, T/F, P/J, T/F
Q31–Q40	I/E, S/N, T/F, P/J, P/J, I/E, S/N, T/F, P/J, T/F
Q41–Q50	I/E, S/N, I/E, P/J, T/F, S/N, T/F, T/F, P/J, T/F
Q51–Q60	I/E, S/N, I/E, T/F, T/F, P/J, S/N, T/F, P/J, T/F

Appendix

Table 6: Fixed mapping from questionnaire items to MBTI dimensions.

Panel of Figure 6 — Figure 6: Effect of backbone LLMs and text encoders on Kaggle dataset.

Panel of Figure 7 — Figure 7: Effect of questionnaire length and the number of experts on the Kaggle dataset.

Dim.	Q#	Question	Direction	$\boldsymbol{\Delta}$ logit	Text evidence (snippet)
I/E	Q21	You enjoy solitary hobbies or activities more than group ones.	I	$-3.19$	“…Feeling cozy in over sized sweaters, reading a good book in a warm bed …”
I/E	Q41	You avoid making phone calls.	I	$-0.62$	“The very thought of having my personal space violated in that manner by a complete stranger is terrifying to me.”
S/N	Q2	Complex and novel ideas excite you more than simple and straightforward ones.	S	$-1.71$	“Usually, I don’t really like discussing topics like these but the posts here are so relatable.”
S/N	Q42	You enjoy exploring unfamiliar ideas and viewpoints.	N	$+0.83$	“That’s an interesting perspective – that we shouldn’t hold others’ imperfections against them.”
T/F	Q3	You usually feel more persuaded by what resonates emotionally with you than by factual arguments.	T	$-1.54$	“That map isn’t very reliable . I know INFPs who live outside of Europe/Americas and there isn’t a single dot…”
T/F	Q15	You rarely worry about whether you make a good impression on people you meet.	F	$+1.05$	“As a child and adolescent, I would lay awake for nights in a row with my stomach knotting and turning out of guilt , even if the person said it didn’t bother them.”

Appendix

Table 7: Illustrative question–post evidence for a representative ISTJ user.

Methods	I/E	S/N	T/F	P/J	Avg
ROME ${}_{\text{drop-max-question}}$	92.06	88.16	96.33	81.02	89.39
ROME ${}_{\text{drop-rand-question}}$	92.01	95.08	97.21	81.09	91.35
ROME ${}_{\text{drop-min-question}}$	92.16	95.81	97.45	81.43	91.71
ROME	92.24	95.90	97.45	81.32	91.73

Appendix

Table 8: Sensitivity of item removal on Kaggle dataset.

Figure 8: Expert activation analysis on Kaggle dataset.

Comparison Setting	Mean MAE	Median MAE
Same user, repeated generations	0.282	0.267
Same MBTI type, different users	0.529	0.477
Different MBTI types, different users	1.912	1.843

Appendix

Table 9: Pairwise MAE between role-played questionnaire-response vectors.

User ID	Role-Played Responses to Q37	Representative Post Evidence
3782	$-3,-3,-2,-3,-3$	“I have zero desire to write anything.”
6170	$+3,+3,+3,+3,+3$	“I’m enveloped in making and watching films. I’m always writing…”

Appendix

Table 10: Illustrative role-played responses to Q37 for two users sharing the same ISTP label.

Methods	I/E	S/N	T/F	P/J	Avg
EERPD	62.14	56.98	70.68	60.57	62.59
ROME	74.93	95.45	82.21	78.22	82.70

Appendix

Table 11: Comparison with EERPD on the Pandora dataset.

Methods	AGR	CON	EXT	NEU	OPN	Avg
EERPD	65.01	68.64	63.02	56.00	60.93	62.72
ROME	65.19	63.31	65.06	61.69	65.47	64.14

Appendix

Table 12: Performance comparison on the Essays dataset under the Big Five personality framework.

Personality Modeling in Language Models

Personality modeling in language models studies how traits are measured, represented, simulated, and controlled in model behavior.

See all 169 papers

Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality?
May 21, 2026Caixin Kang, Tianyu Yan, Sitong Gong +8LLM Grounding Personality Modeling in Language Models
1The University of Tokyo · 2Shanda AI Research Tokyo · 3Dalian University of Technology
Large-Language-Models-as-a-Judge in Theory-Agnostic Adaptive Metric-Alignment for Prototypical Networks in Personality Recognition
Jul 9, 2026Jing Jie Tan, Ban-Hoe Kwan, Danny Wee-Kiat Ng +6Personality Modeling in Language Models Metric Learning
Department of Mechatronics and Biomedical Engineering, Lee Kong Chian Faculty of Engineering and Science, Universiti Tunku Abdul Rahman, Malaysia · Institute of Communication Studies, National Yang Ming Chiao Tung University, Taiwan · Institute of Information Management, National Yang Ming Chiao Tung University, Taiwan +2
PACIFIC: Can LLMs Discern the Psychometric Traits Influencing Your Preferences? Personality-Driven Preference Alignment in LLMs
Feb 6, 2026Tianyu Zhao, Siqi Li, Yasser Shoukry +1LLM Personalization Preference Alignment
Department of Electrical Engineering and Computer Science University of California, Irvine Irvine, CA 92697, USA

May 21, 2026J/K move · Enter open · S save

Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality?

Caixin Kang, Tianyu Yan, Sitong Gong, Mingfang Zhang+7

1The University of Tokyo · 2Shanda AI Research Tokyo · 3Dalian University of Technology

LLM Grounding Personality Modeling in Language Models Multimodal Model Evaluation

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) are increasingly deployed in human-facing roles where personality perception is critical, yet existing benchmarks evaluate this capability solely on numerical Big Five score prediction, leaving open whether models truly perceive personality through behavioral understanding or merely prejudge through superficial pattern matching. We address this gap with three contributions. (i) A new task: we formalize Grounded Personality Reasoning (GPR), which requires MLLMs to anchor each Big Five rating in observable evidence through a chain of rating, reasoning, and grounding. (ii) A new dataset: we release MM-OCEAN (1,104 videos, 5,320 MCQs), produced by a multi-agent pipeline with human verification, with timestamped behavioral observations, evidence-grounded trait analyses, and seven categories of cue-grounding MCQs. (iii) Benchmark and analysis: we design a three-tier evaluation (rating, reasoning, grounding) plus four sample-level failure-mode metrics: Prejudice Rate (PR), Confabulation Rate (CR), Integration-failure Rate (IR), and Holistic-grounding Rate (HR), and benchmark 27 MLLMs (13 closed, 14 open). The analysis uncovers a striking Prejudice Gap: across the field, 51% of correct ratings are not grounded in retrieved cues, and the Holistic-Grounding Rate spans only 0-33.5%. These findings expose a disconnect between getting the right score and reasoning for the right reason, charting a roadmap for grounded social cognition in MLLMs.

Abstract

Personality Modeling in Language Models

Explore similar work

Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality?

Large-Language-Models-as-a-Judge in Theory-Agnostic Adaptive Metric-Alignment for Prototypical Networks in Personality Recognition

PACIFIC: Can LLMs Discern the Psychometric Traits Influencing Your Preferences? Personality-Driven Preference Alignment in LLMs