Deep Persona: A Psychologically Grounded Architecture and Evaluation Framework for Role-Playing Agents and Simulations

Rotem Dror; Zohar Elyoseph; Yuval Haber; Elad Refoua; Oshrat Ayalon; Adir Solomon

cs.CLSep 6, 2026

Deep Persona: A Psychologically Grounded Architecture and Evaluation Framework for Role-Playing Agents and Simulations

Authors: Rotem Dror, Zohar Elyoseph, Yuval Haber, Elad Refoua, Oshrat Ayalon, Adir Solomon

Organizations: Faculty of Computer and Information Science, University of Haifa, Israel · School of Therapy, Counseling and Human Development, University of Haifa, Israel · The PhD Program of Hermeneutics and Culture, Interdisciplinary Studies Unit, Bar-Ilan University, Israel · Department of Psychology, Bar-Ilan University, Israel

Abstract

Existing approaches to persona simulation with Large Language Models (LLMs) mostly rely on shallow character descriptions that fail to sustain coherent character behavior across extended interactions. We introduce Deep Persona, a psychologically grounded, three-layered architecture that organizes personas into hierarchical levels of observable expression, latent beliefs, and core motivational drives, for constructing highly convincing role-playing agents. Governed by the principles of scripted determinism and bounded agency, the architecture restricts the model to a reactive engine guided by a structured internal script. We further propose a reference-free evaluation framework that benchmarks dialogue naturalness against empirical human distributions using established psychological clinical instruments and adversarial stress-tests. Empirical evaluation reveals that while LLMs achieve high pragmatic fluency, they exhibit systematic limitations in emotional expression and joint attention. In addition, we present a case study of two Deep Personas and evaluate them using the proposed framework, demonstrating that structured personas can produce interactions that more closely align with human conversational behavior.

Figures & tables

Figure 1: An illustration of the differences between a flat persona and the Deep Persona framework.

Interview Dimension	Elicited Content	Prompt Component
Narrative Background	Personal history, social context	Narrative Background and Motivations
Initial Context	Entry setting (time/place)	Interaction Structure: Initialization
Emotional Tone	Affect, stance, typical mood	External Layer
Communication Patterns	Vocabulary, sentence length, questioning style	Interaction Structure: Parameters
Conscious Goals	Stated desires, explicit complaints, surface motivations	External Layer
Semi-conscious Material	Information revealed only after trust or progression	Middle Layer

Table 1: Mapping interview dimensions to prompt specification components in the Deep Persona methodology.

Dataset	$S_{\text{pragmatics}}$	$S_{\text{joint-attention}}$	$S_{\text{emotion}}$
DailyDialog	0.9400	0.3521	0.1152
CC (H-H)	0.9680	0.8854	1.4264

Table 2: Human baseline scores for the proposed dialogue evaluation metrics for the DailyDialog dataset and the human to human interactions from the CounselChat (human-human) (CC).

Dataset	$S_{\text{pragmatics}}$	$S_{\text{joint-attention}}$	$S_{\text{emotion}}$
RP Boss	0.8881	0.8390	0.2068
RP Classmate	0.8923	0.7808	0.2934
RP Vanilla	0.8918	0.8221	0.2646
ABC-Eval	0.8931	0.4594	0.1108
CC (H-LLM)	0.9676	0.9344	2.2054
Sarah	0.9402	0.5202	0.4918

Table 3: Scoring profiles showing the average values for the datasets of Role-Play (RP), ABC-Eval, and CounselChat (human-LLM) (CC). At the bottom are the scores of the deep personas Sarah and Evelyn.

	DailyDialog		CounselChat (H-H)		Combined Baseline
Evaluated Dataset	DNS	# Dialogues	DNS	# Turns	DNS	# Dialogues
Role-Play Boss	0.6618	26/28	0.0600	0/28	0.6899	26/28
Role-Play Classmate	0.6665	26/28	0.0529	0/28	0.7467	27/28
Role-Play Vanilla	0.6705	28/29	0.0645	0/29	0.7266	28/29
ABC-Eval	0.8181	505/528	0.0633	0/528	0.7929	504/528
CounselChat (H-LLM)	0.0479	124/3507	0.7940	3118/3507	0.5254	2107/3507

Table 4: DNS Scores for the datasets of Role-Play, ABC-Eval, CounselChat (human-LLM), and for the deep personas Sarah and Evelyn. The # Dialogues indicates the number of dialogues where the agent was statistically indistinguishable from a human (

p>0.05

). Scores are calculated with respect to human baselines from the DailyDialog dataset, the CounselChat (human-human) dataset, and a combined baseline pooling both human-human datasets.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Pragmatic Fluidity & Echolalia
Dataset	Metric	$\boldsymbol{\mu}$	$\boldsymbol{\sigma}$
RP Boss	$\bar{S}_{\text{overlap}}$	0.1772	0.0404
	$\bar{S}_{\text{self}}$	0.1522	0.0335
	$E_{\text{rate}}$	0.0045	0.0236
	$S_{\text{pragmatics}}$	0.8881	0.0222
RP Classmate	$\bar{S}_{\text{overlap}}$	0.1582	0.0371

Appendix

Table 5: Full table of scores for the datasets of Role-Play Tao et al. (2024) (RR), ABC-Eval Finch et al. (2023) , and CounselChat (human-LLM) Bird et al. (2024) (CC). For

S_{\text{pragmatics}}

\alpha=\beta=\gamma=0.33

, and for

S_{\text{emotion}}

\alpha=0.7

and

\beta=0.3

Explore similar work

PersonaArena: Dynamic Simulation for Evaluating and Enhancing Persona-Level Role-Playing in Large Language Models
May 16, 2026Wenlong Shi, Jianxun Lian, Mingqi Wu +5Large Language Model-Based Role-Play Simulation Personality Modeling in Language Models
College of Computer Science and Software Engineering, Shenzhen University, China · Microsoft Research Asia · Microsoft Gaming +2
Memory-Driven Role-Playing: Evaluation and Enhancement of Persona Knowledge Utilization in LLMs
Mar 14, 2026Kai Wang, Haoyang You, Yang Zhang +1Large Language Model-Based Role-Play Simulation LLM Prompting
Harbin Institute of Technology, China · Macquarie University, Australia
Eval4Sim: An Evaluation Framework for Persona Simulation
Mar 3, 2026Eliseo Bao, Anxo Perez, Javier Parapar +1Large Language Model-Based Role-Play Simulation Large Language Model-Based Social Simulation
IRLab, CITIC, Universidade da Coruña A Coruña, Spain · University of Sheffield Sheffield, United Kingdom

May 16, 2026J/K move · Enter open · S save

PersonaArena: Dynamic Simulation for Evaluating and Enhancing Persona-Level Role-Playing in Large Language Models

Wenlong Shi, Jianxun Lian, Mingqi Wu, Haiming Qin+4

College of Computer Science and Software Engineering, Shenzhen University, China · Microsoft Research Asia · Microsoft Gaming+2

Large Language Model-Based Role-Play Simulation Personality Modeling in Language Models Human Behavior Simulation LLM Agent Evaluation AI Agent Benchmarks Large Language Model-Based Social Simulation

Open paper AskPDF arXiv

Large language models (LLMs) increasingly serve as interactive social agents, yet their ability to maintain coherent and authentic persona-level role-playing remains limited, particularly in realistic social scenarios. Existing research predominantly focuses on character-level settings and relies on static evaluation formats, failing to capture the complexity of everyday social interactions. In this work, we present PersonaArena, a dynamic simulation framework for evaluating and improving persona-level role-playing in LLMs. PersonaArena leverages a large, filtered corpus of user-generated social content to construct a nuanced persona bank, and elicits multi-turn, context-rich interactions within simulated social environments. Our framework features a multi-agent debating judge for holistic and unbiased assessment. Through extensive experiments, we demonstrate that PersonaArena enables rigorous evaluation and enhancement of LLMs' role-playing capabilities, advancing the development of more authentic and socially adept AI agents.

Abstract

Explore similar work

PersonaArena: Dynamic Simulation for Evaluating and Enhancing Persona-Level Role-Playing in Large Language Models

Memory-Driven Role-Playing: Evaluation and Enhancement of Persona Knowledge Utilization in LLMs

Eval4Sim: An Evaluation Framework for Persona Simulation