From Memory to Behavior: A Behavior-Aware Role-Playing Framework for Social Media Influencers

Ji-Lun Peng; Yi-Zhen Zhang; Chun-Nan Chou; Yun-Nung Chen

cs.CLSep 18, 2026

From Memory to Behavior: A Behavior-Aware Role-Playing Framework for Social Media Influencers

Authors: Ji-Lun Peng, Yi-Zhen Zhang, Chun-Nan Chou, Yun-Nung Chen

Organizations: National Taiwan University, Taipei, Taiwan · CMoney Technology Corporation, Taiwan

Abstract

Large language models have shown strong potential as role-playing agents for real individuals, yet faithful impersonating remains challenging. Existing in-context learning-based methods fail to capture how individuals react under different situations. In addition, LLM-based evaluation is difficult for obscure individuals. To address these challenges, we propose Situation--Internal state--Behavior Persona method to incorporate situation-dependent behavioral strategies. We further design an evaluation protocol that provides LLM evaluators with references about the impersonated individual. We evaluate our approach on a newly constructed dataset for the task of generating replies on social media. Experimental results show that our proposed method outperforms state-of-the-art ICL-based baselines, while our evaluation protocol achieves moderate correlation with human judgment. Besides, experiments on fictional-character benchmarks demonstrate that our proposed method is applicable beyond the social media setting. These findings suggest that incorporating behavioral information broadly improves the fidelity of role-playing for real individuals on social media or fictional characters.

Figures & tables

Figure 1: In the social media comment-replying scenarios between influencers and followers, replies involve short texts where behavioral strategy is more important for determining the fidelity of impersonation.

Dataset	Public	Identifiable Individual History	#Individual	Comment- Reply Structure	Avg. Posts per Individual	Avg. Comment-Reply Pairs per Individual	Max Posts per Individual	Max Comment-Reply Pairs per Individual
BuzzFace Santia and Williams (2018)	✗	-	-	-	-	-	-	-
TWON Münker et al. (2026)	✗	-	-	-	-	-	-	-
BluePrint Bück-Kaeffer et al. (2025)	✓	✗	-	-	-	-	-	-
SYNTHIA Rahimzadeh et al. (2026)	✓	✗	-	-	-	-	-	-
PersonalDialog Zheng et al. (2019)	✓	✗	-	-	-	-	-	-
Persona-Chat Zhang et al. (2018)	✓	✗	-	-	-	-	-	-

Table 1: Feasibility of existing datasets for studying role-playing of real individuals on social media reply generation. ”-” indicates that the corresponding information is not applicable under our comparison criteria. Since our research goal is to impersonate influencers across diverse interaction situations, we construct a dataset centered on a small number of influencers with rich historical interactions.

Figure 2: Overview of the SIBPersona method. The upper part illustrates the process of extracting data-variant persona from historical data. The lower part presents reply generation process: retrieving the most relevant dimensional documents based on the comment enables the RPA to generate a high-fidelity reply.

Methods

GPT-5.2

Gemini-3-flash

DeepSeek-V3.2

Claude-Opus-4.6

Average

M

T

RP

M

T

RP

M

T

RP

M

T

RP

M

T

RP

Naive

97.50

46.40 ‡

67.53 ‡

94.70

42.47 ‡

69.27 ‡

95.13

48.93 ‡

70.83 ‡

96.77

46.93 ‡

76.87 ‡

96.03

46.18 ‡

70.78 ‡

CoT

97.47

44.93 ‡

70.73 ‡

95.83

41.17 ‡

69.53 ‡

96.27

49.23 ‡

71.47 ‡

97.33

47.00

77.33 ‡

96.73

45.58 ‡

72.27 ‡

IMPersona

96.07

47.50 ‡

71.93 ‡

94.90

44.60 ‡

70.10 ‡

96.00

50.80 ‡

72.50 ‡

97.43

48.67 ‡

78.13 ‡

96.10

47.89 ‡

73.17 ‡

Vanilla RAG

97.33

49.43 ‡

73.20 ‡

93.77

53.20

71.50 ‡

94.23

58.37 ‡

73.43

96.70

56.93

77.70 ‡

95.51

54.48 ‡

73.96 ‡

SIBPersona

96.90

54.33

77.63

93.77

56.47

75.50

94.83

63.67

76.43

96.93

62.77

81.93

95.61

59.31

77.88

Table 2: Performance on Influencer A . Bold : the best results;

\ddagger

: significant degradation compared to SIBPersona (

p<0.05

Methods	Deepseek-V3.2			Gemini-3-flash
	$M$	$T$	$RP$	$M$	$T$	$RP$
Naive	92.10 $\pm$ 3.04	42.03 ‡ $\pm$ 8.43	68.48 ‡ $\pm$ 5.86	92.50 $\pm$ 2.32	43.33 ‡ $\pm$ 7.35	70.64 ‡ $\pm$ 7.37
CoT	92.17 $\pm$ 3.03	43.92 ‡ $\pm$ 8.13	69.83 ‡ $\pm$ 6.14	92.15 $\pm$ 2.54	44.05 ‡ $\pm$ 7.08	70.48 ‡ $\pm$ 6.40
IMPersona	91.60 $\pm$ 2.21	45.54 ‡ $\pm$ 5.86	71.14 ‡ $\pm$ 6.02	91.42 $\pm$ 1.94	45.21 ‡ $\pm$ 6.40	71.56 ‡ $\pm$ 7.19
Vanilla RAG	91.39 $\pm$ 3.23	57.05 ‡ $\pm$ 2.43	72.43 ‡ $\pm$ 5.06	92.19 $\pm$ 2.46	53.43 ‡ $\pm$ 4.92	72.59 ‡ $\pm$ 4.79
SIBPersona	92.14 $\pm$ 3.26	61.09 $\pm$ 1.66	75.84 $\pm$ 4.90	92.21 $\pm$ 2.20	57.34 $\pm$ 3.20	76.48 $\pm$ 5.08

Table 3: Results on other five influencers. Bold indicates the better scores.

\ddagger

: significant degradation compared to SIBPersona (

p<0.05

Methods	$M$	$T$	$RP$
SIBPersona	92.67 $\pm$ 3.18	61.60 $\pm$ 1.82	75.96 $\pm$ 4.49
w/o $I$	92.61 $\pm$ 2.49	60.30 $\pm$ 2.37	75.44 $\pm$ 3.61
w/o $RP$	92.83 $\pm$ 2.44	57.82 ‡ $\pm$ 2.51	73.36 ‡ $\pm$ 4.64

Table 4: Dimension ablation study using DeepSeek-V3.2 on the six influencer. Bold : the best results;

\ddagger

: significant degradation compared to SIBPersona (

p<0.05

Methods	$M$	$T$	$RP$	Avg.
IMPersona	2.92 ‡	2.40 ‡	2.67 ‡	2.67 ‡
Vanilla RAG	3.68	3.72	3.62	3.67
SIBPersona	3.78	3.64	3.62	3.68

Table 5: Human evaluation results comparing our framework with the baseline. Bold : better results; ‡ : significant degradation compared to SIBPersona (

p<0.05

Conditions	$M$	$T$	$RP$
Uninterpretable	3.03	3.00	2.87
Interpretable	4.13 †	3.97 †	4.00 †

Table 6: Human evaluation scores for Interpretable (n=20) and Uninterpretable (n=10) groups. Bold : better results; † : significant improvement with

p<0.05

Methods	CA	RA	CK	CP
IMPersona	2.94	2.43	2.25	2.38
SIBPersona	3.28 †	2.66 †	2.32 †	2.69 †
Fine-tuned CoSER	3.35	2.61	2.28	2.68

Table 7: Results on CharacterEval. CA : Conversational Ability, RA : Role-Playing Attractiveness, CK : Character Consistency–Knowledge, and CP : Character Consistency–Persona. Bold : the best results;

{\dagger}

: significant improvement with

p<0.05

Methods	CA	RA	CK	CP
SIBPersona	3.28	2.66	2.32	2.69
w/o $T$	3.33	2.69	2.31	2.74
w/o $I$	3.28	2.66	2.31	2.68
w/o $RP$	3.16 ‡	2.61	2.30	2.62 ‡
w/o $RP$ & $T$	3.05 ‡	2.54 ‡	2.28	2.57 ‡

Table 8: Component ablation study on CharacterEval. Bold : the best results;

\ddagger

: significant degradation compared to SIBPersona (

p<0.05

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

	Influencer
Persona Score	A	B	C	D	E	F
High	50	50	50	35	50	9
Medium	50	50	50	50	22	39
Low	50	50	50	50	12	50

Appendix

Table 9: Number of data in the test samples.

Influencer	Post Length	Comment Length	Reply Length	High (%)	Med (%)	Low (%)
A	467.92 $\pm$ 471.09	17.58 $\pm$ 25.37	8.46 $\pm$ 9.69	13%	29%	58%
B	367.39 $\pm$ 231.62	30.70 $\pm$ 57.55	10.01 $\pm$ 10.54	12%	35%	53%
C	952.74 $\pm$ 740.92	66.83 $\pm$ 105.54	33.65 $\pm$ 28.14	34%	36%	53%
D	355.21 $\pm$ 508.89	14.20 $\pm$ 15.24	9.56 $\pm$ 10.17	16%	25%	59%
E	271.65 $\pm$ 221.00	27.54 $\pm$ 28.06	9.52 $\pm$ 10.21	5%	8%	87%
F	860.21 $\pm$ 574.41	26.53 $\pm$ 29.41	11.82 $\pm$ 10.39	8%	33%	60%

Appendix

Table 10: Dataset diversity statistics across the six influencers, including text lengths and the proportion of comment–reply pairs across high, medium, and low persona-score levels.

Models	$M$	$T$	$RP$
GPT-5.2	6,064	1,410	970
Gemini-3-flash	5,755	1,050	861
Deepseek-V3.2	5,143	1,022	749
Claude-Opus-4.6	2,307	407	703

Appendix

Table 11: Number of dimensional documents of Influencer A .

Influencer	$M$	$T$	$RP$
B	12,358	8,162	6,701
C	6,179	2,982	4,049
D	8,914	4,935	3,811
E	10,087	2,393	2,378
F	2,587	446	398

Appendix

Table 12: Number of dimensional documents of Influencer B-F extracted from Deepseek V3.2 .

Influencer	$M$	$T$	$RP$
B	13360	8253	6911
C	5296	3321	3643
D	10102	4839	4053
E	10309	3536	3287
F	2363	515	449

Appendix

Table 13: Number of dimensional documents of Influencer B-F extracted from Gemini-3-flash .

Figure 3: t-SNE projection of the embeddings of $mp_{jk}$ , $sp_{jk}$ , and $b_{jk}$ generated by Claude-Opus-4.6 .

Figure 4: t-SNE projection of representative memory documents generated by DeepSeek-v3.2 .

Figure 5: t-SNE projection of representative speaking style documents generated by DeepSeek-v3.2 .

Figure 6: t-SNE projection of representative reaction process documents generated by DeepSeek-v3.2 .

Methods	Reciprocal Ranking score
IMPersona	0.42 ‡
SIBPersona	0.71
Ground Truth	0.71

Appendix

Table 14: Human comparison evaluation reciprocal scores. Bold indicates the best score.

\ddagger

: significant degradation compared to SIBPersona (

p<0.05

)

	$M$	$T$	$RP$
Rater 1 & 2	0.364	0.429	0.406
Rater 1 & 3	0.417	0.506	0.508
Rater 2 & 3	0.426	0.495	0.487

Appendix

Table 15: Inter-rater agreement among the three annotators.

	Pass Rate	95% CI
Rater 1	0.80	[0.67, 0.89]
Rater 2	0.84	[0.72, 0.92]
Rater 3	0.92	[0.81, 0.97]
Total	0.94	[0.84, 0.98]

Appendix

Table 16: Extraction faithfulness evaluated by the three annotators. The Total column reports the pass rate of by applying majority-vote across all annotators.

Method	# Tokens
Vanilla RAG	593.51 $\pm$ 456.37
IMPersona	1972.64 $\pm$ 816.14
SIBPersona	1630.83 $\pm$ 812.02

Appendix

Table 17: Average number of tokens inserted into the prompt per test query across six influencers using DeepSeek-V3.2 .

Method	$M$	$T$	$RP$
Naive	95.89	46.43 ‡	70.85 ‡
CoT	96.58	45.73 ‡	72.33 ‡
IMPersona	95.94	48.10 ‡	73.20 ‡
Vanilla RAG	95.32	54.12 ‡	73.68 ‡
SIBPersona	95.46	59.22	77.81

Appendix

Table 18: Evaluation results on Influencer A averaged over four models after excluding overlapping test instances. ‡: significant degradation compared to SIBPersona (p < 0.05).

Methods	M	T	RP
Naive	97.28	46.48 ‡	77.33 ‡
CoT	97.61	47.93 ‡	78.08 ‡
IMPersona	97.14	48.63 ‡	78.80 ‡
Vanilla RAG	97.02	58.93 ‡	79.18 ‡
SIBPersona	96.69	62.37	81.52

Appendix

Table 19: Results on Influencer A averaged over four models using evaluator references extracted from the demonstration source. ‡ : significant degradation compared to SIBPersona (

p<0.05

Reference Format	Source of Reference	Pearson	Spearman
Raw comment–reply pairs	Test candidate pool	0.355	0.409
SIBPersona document	Demonstration source	0.448	0.462
SIBPersona document	Test candidate pool (Ours)	0.561	0.530

Appendix

Table 20: Correlation between different evaluator configurations and human judgments.

Methods	Deepseek-V3.2			Gemini-3-flash
	$M$	$T$	$RP$	$M$	$T$	$RP$
Naive	91.34 $\pm$ 3.24	42.63 ‡ $\pm$ 9.19	70.83 ‡ $\pm$ 5.49	92.18 $\pm$ 2.83	44.21 ‡ $\pm$ 10.84	68.91 ‡ $\pm$ 7.71
CoT	91.06 $\pm$ 3.35	46.65 ‡ $\pm$ 7.36	71.56 ‡ $\pm$ 4.79	92.14 $\pm$ 1.71	45.57 ‡ $\pm$ 8.63	68.53 ‡ $\pm$ 7.39
IMPersona	91.43 $\pm$ 2.61	48.44 ‡ $\pm$ 5.61	73.04 ‡ $\pm$ 5.20	91.34 $\pm$ 2.36	46.32 ‡ $\pm$ 8.20	69.88 ‡ $\pm$ 7.94
Vanilla RAG	91.30 $\pm$ 3.24	59.18 ‡ $\pm$ 4.58	74.30 ‡ $\pm$ 4.26	92.37 $\pm$ 2.60	56.47 ‡ $\pm$ 3.07	72.49 ‡ $\pm$ 3.97
SIBPersona	92.01 $\pm$ 3.53	62.96 $\pm$ 4.27	77.18 $\pm$ 4.44	92.29 $\pm$ 2.48	59.48 $\pm$ 3.00	75.60 $\pm$ 5.10

Appendix

Table 21: Evaluation results across the other five influencers using references extracted from the demonstration source.

\ddagger

: significant degradation compared to SIBPersona (

p<0.05

		Influencer
Split	Data type	A	B	C	D	E	F
Demonstration	Post	1955	2716	713	2903	2130	453
Demonstration	Comment-Reply pair	1361	15240	5553	6087	4383	705
Test Candidates	Post	652	285	54	192	287	84
Test Candidates	Comment-Reply pair	547	1719	361	227	261	120

Appendix

Table 22: Number of data in the training set and the test candidate pool.

Influencers	Models	$M$		$T$		$RP$
		All	Rep.	All	Rep.	All	Rep.
A	DeepSeek-V3.2	2424	35	446	15	274	12
	Claude-Opus-4.6	1156	25	189	10	244	12
	Gemini-3-flash	2477	36	419	15	318	13
	GPT-5.2	2926	39	588	18	375	14
B	DeepSeek-V3.2	1786	30	851	21	758	20

Appendix

Table 23: Statistics of extracted documents and representative documents across influencers and models. All denotes the total number of dimensional documents extracted from the candidates for testing, and Rep. denotes the number of representative dimensional documents selected from the corresponding dimensional corpus.

Figure 7: Examples of three level of data for Influencer A .

Figure 8: Correlation matrix between LLM and human evaluations. *** $p<0.001$ , ** $p<0.01$ , * $p<0.05$ .

Methods

M

T

RP

M

T

RP

M

T

RP

M

T

RP

M

T

RP

Deepseek v3.2

Naive

97.67

45.60 ‡

78.07 ‡

88.97

52.00 ‡

69.57 ‡

91.59

37.22 ‡

60.93 ‡

90.24

26.85 ‡

64.17 ‡

90.66

40.97 ‡

66.22 ‡

CoT

97.87

45.63 ‡

79.80 ‡

89.57

53.00 ‡

71.43

90.89

43.96 ‡

62.22 ‡

90.30

28.15 ‡

65.30 ‡

90.82

40.87 ‡

66.48 ‡

IMPersona

95.77

46.47 ‡

81.43 ‡

89.70

54.43 ‡

71.67

90.70

42.44 ‡

64.63 ‡

90.65

36.73 ‡

66.19 ‡

90.20

42.35 ‡

67.81

Vanilla RAG

97.33

60.50

81.33 ‡

88.20

55.43 ‡

70.83

90.81

58.41 ‡

70.44 ‡

89.82

53.87

69.70

89.34

55.10

66.33 ‡

Appendix

Table 24: Results of Influencer B–F . Bold : best results; ‡ : significant degradation compared to SIBPersona

p<0.05

		LLM penalty-based evaluation			Pair-wise Evaluation
Tasks	Methods	$M$	$T$	$RP$	Win	Tie	Lose
General	IMPersona	75.32	65.10	71.15	-	-	-
General	SIBPersona	76.64	66.75 †	73.97 †	0.43	0.32	0.25
Summary	IMPersona	75.28	62.37	70.44	-	-	-
Summary	SIBPersona	78.94 †	63.83	73.12 †	0.34	0.39	0.27

Appendix

Table 25: Experimental results on RoleAgentBench, comparing our method with the baseline. Bold indicates the better score between the two methods. † denotes a result that is significantly better than its counterpart (

p<0.05

Figure 9: An example where both human and LLM evaluators assign high scores, while traditional metrics assign low scores.

Figure 10: Example of top-1 dimensional document retrieval. The influencer names in the text are anonymized and replaced with "Influencer".

Figure 11: English translation of the example for top-1 dimensional document retrieval. The influencer names in the text are anonymized and replaced with "Influencer".

Figure 12: Qualitative examples where our method performs well. The influencer names in the text are anonymized and replaced with "Influencer".

Figure 13: Qualitative examples where our method performs poorly. The influencer names in the text are anonymized and replaced with "Influencer".

Panel of Figure 14 — Figure 14: English translation of the persona scoring prompt used in our dataset.

Panel of Figure 15 — Figure 15: Recognition prompt used in RoleAgentBench. All datasets share the same prompt except that the few-shot examples are replaced with the examples from the corresponding dataset.

Panel of Figure 16 — Figure 16: Memory extraction prompt used in RoleAgentBench. All datasets share the same prompt except that the few-shot examples are replaced with the examples from the corresponding dataset.

Panel of Figure 17 — Figure 17: Speaking style extraction prompt used in RoleAgentBench. All datasets share the same prompt except that the few-shot examples are replaced with the examples from the corresponding dataset.

Panel of Figure 18 — Figure 18: Reaction Process extraction prompt used in RoleAgentBench. All datasets share the same prompt except that the few-shot examples are replaced with the examples from the corresponding dataset.

Panel of Figure 19 — Figure 19: Evaluation prompt used in RoleAgentBench. All datasets share the same prefix prompt except for the data input format (question, model reply, and correct reply triplet).

Panel of Figure 20 — Figure 20: Evaluation rubrics used for all datasets

Panel of Figure 21 — Figure 21: Reply generation prompt used in RoleAgentBench. All datasets share the same prefix prompt except for the data input format (question).

Large Language Model-Based Role-Play Simulation

Large language model-based role-play simulation creates interactive agents that portray people, characters, or users for training, testing, and research.

See all 107 papers

PersonaArena: Dynamic Simulation for Evaluating and Enhancing Persona-Level Role-Playing in Large Language Models
May 16, 2026Wenlong Shi, Jianxun Lian, Mingqi Wu +5Large Language Model-Based Role-Play Simulation Personality Modeling in Language Models
College of Computer Science and Software Engineering, Shenzhen University, China · Microsoft Research Asia · Microsoft Gaming +2
Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents
May 13, 2026Harshita Chopra, Kshitish Ghate, Aylin Caliskan +3Human Behavior Simulation LLM Agent Evaluation
University of Washington, Seattle · Georgetown University, Washington DC
Deep Persona: A Psychologically Grounded Architecture and Evaluation Framework for Role-Playing Agents and Simulations
Sep 6, 2026Rotem Dror, Zohar Elyoseph, Yuval Haber +3Large Language Model-Based Role-Play Simulation Personality Modeling in Language Models
Faculty of Computer and Information Science, University of Haifa, Israel · School of Therapy, Counseling and Human Development, University of Haifa, Israel · The PhD Program of Hermeneutics and Culture, Interdisciplinary Studies Unit, Bar-Ilan University, Israel +1

May 16, 2026J/K move · Enter open · S save

PersonaArena: Dynamic Simulation for Evaluating and Enhancing Persona-Level Role-Playing in Large Language Models

Wenlong Shi, Jianxun Lian, Mingqi Wu, Haiming Qin+4

College of Computer Science and Software Engineering, Shenzhen University, China · Microsoft Research Asia · Microsoft Gaming+2

Large Language Model-Based Role-Play Simulation Personality Modeling in Language Models Human Behavior Simulation LLM Agent Evaluation AI Agent Benchmarks Large Language Model-Based Social Simulation

Open paper AskPDF arXiv

Large language models (LLMs) increasingly serve as interactive social agents, yet their ability to maintain coherent and authentic persona-level role-playing remains limited, particularly in realistic social scenarios. Existing research predominantly focuses on character-level settings and relies on static evaluation formats, failing to capture the complexity of everyday social interactions. In this work, we present PersonaArena, a dynamic simulation framework for evaluating and improving persona-level role-playing in LLMs. PersonaArena leverages a large, filtered corpus of user-generated social content to construct a nuanced persona bank, and elicits multi-turn, context-rich interactions within simulated social environments. Our framework features a multi-agent debating judge for holistic and unbiased assessment. Through extensive experiments, we demonstrate that PersonaArena enables rigorous evaluation and enhancement of LLMs' role-playing capabilities, advancing the development of more authentic and socially adept AI agents.

Abstract

Large Language Model-Based Role-Play Simulation

Explore similar work

PersonaArena: Dynamic Simulation for Evaluating and Enhancing Persona-Level Role-Playing in Large Language Models

Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents

Deep Persona: A Psychologically Grounded Architecture and Evaluation Framework for Role-Playing Agents and Simulations