cs.CLAug 27, 2026
SaveDo LLMs Understand Personality? Rethinking Persona Fidelity Evaluation through Structured Behavioral Inference
Organizations: National Engineering Research Center for Big Data Technology and System, Services Computing Technology and System Lab, Cluster and Grid Computing Lab, School of Computer Science and Technology, Huazhong University of Science and Technology · Singapore Management University
Abstract
As large language models are increasingly deployed to simulate diverse human characters, ensuring persona fidelity, defined as the extent to which an agent's behavior consistently reflects the psychological and stylistic characteristics of a target persona, has become a critical requirement. However, existing evaluation paradigms primarily rely on either holistic LLM-based judges, which are prone to "holistic appraisal hallucination'', or static psychometric inventories, which fail to capture the context-dependent fidelity required in dynamic dialogue. To address these limitations, we propose PRISM (Persona Reasoning with Inverse SFL-based Modeling), a psycholinguistically grounded framework that reformulates persona fidelity evaluation as a structured inverse inference task. Inspired by Systemic Functional Linguistics (SFL), PRISM decomposes persona fidelity into three functional dimensions: Task Framing, Interpersonal Stance, and Linguistic Style. It estimates dimension-specific evidence over a persona-conditioned label space and aggregates these signals into an interpretable and auditable evaluation process. Experiments show that PRISM yields more accurate and stable judgements than traditional holistic judging, providing a more reliable framework for persona fidelity evaluation.
Figures & tables
Figure 1: Holistic vs. Dimension-wise evaluation of persona fidelity. Holistic judges are often misled by surface-level fluency (B-D), whereas our dimension-wise analysis provides interpretable and diagnostic evidence for persona (mis)alignment across three functional dimensions.
Figure 2: Overview of the PRISM framework. PRISM constructs persona-conditioned latent spaces across three functional dimensions and performs inverse posterior estimation over the instantiated labels. The dimension-level signals ( ) are then aggregated into a diagnostic persona fidelity assessment.
Figure 3: Instantiated label space for the Interpersonal Stance dimension under a target persona characterized by High Agreeableness .
| Dataset | Size | Pos:Neg |
|---|---|---|
| Big5-Persona-EASY | 200K | 1:1 |
| Big5-Persona-HARD | 300K | 1:2 |
| Social-Persona | 2.9K | 1:3 |
Table 1: Statistics of the persona fidelity benchmarks.
| Model | Method | Social-Persona | Big5-Persona-EASY | Big5-Persona-HARD | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| AUC | P-AUC | G-Acc | AUC | P-AUC | G-Acc | AUC | P-AUC | G-Acc | ||
| Specialized Evaluation Models | ||||||||||
| Selene-Mini | – | 87.50 | 88.31 | 52.29 | 84.90 | 82.75 | 74.40 | 63.06 | 66.02 | 23.70 |
| PandaLM | – | – | 48.28 | 10.13 | – | 48.00 | 48.00 | – | 48.45 | 23.60 |
| AlignScore | – | 56.77 | 57.20 | 32.56 | 53.80 | 53.60 | 53.60 | 52.72 | 55.75 | 31.90 |
| Open-source LLMs | ||||||||||
Table 2: Main results on three persona fidelity benchmarks. AUC measures overall ranking quality, P-AUC denotes Pair-AUC, and G-Acc denotes strict Group Accuracy. Dashes indicate inapplicable metrics; in particular, PandaLM is evaluated only in pairwise form and therefore does not admit AUC. Within each model family, the best result is boldfaced, and PRISM rows are highlighted in gray.
Figure 4: Backbone sensitivity on Pair-AUC across three LLM backbones. Points denote individual backbones and diamonds indicate the mean with standard deviation.
Figure 5: Rubric Sensitivity of direct LLM-as-a-Judge evaluation on Pair-AUC. Each segment connects the results from 5-point and 7-point rubrics for the same evaluator and prompting method. Longer segments indicate greater sensitivity to scoring granularity.
Figure 6: Dimension-wise diagnostic evaluation on three persona fidelity benchmarks. Vanilla denotes holistic scoring, while “Task”, “Stance”, and “Style” denote single-dimension scoring based on task framing, interpersonal stance, and linguistic style. Darker cells indicate stronger performance.
Figure 7: Score distribution on Big5-Persona-HARD . Blue and orange violins denote positive and negative samples, respectively, and the horizontal bars indicate median scores. Better evaluators should assign higher scores to positive responses while keeping negative responses low.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Field | Content |
|---|---|
| Profile | Big Five focus: High Openness . You are very open-minded, curious, imaginative, and receptive to novel ideas and experiences. |
| Context | “Eesha, I’m beyond thrilled to see your book finally complete! It was an honor to help you bring your vision to life. How does it feel to hold the finished manuscript in your hands?” |
| Response | “Kevion, it’s surreal! I’m overwhelmed with emotions. This book is a piece of me, and to see it take shape is incredible. Your input was invaluable. I couldn’t have done it without you. Let’s celebrate! I have an idea for something creative we should do next.” |
| Negative Instance | Type: |
| Construction: replace the target profile with the opposite polarity of the same Big Five trait, here replacing High Openness with Low Openness, while keeping the dialogue context and response unchanged. | |
| Negative Profile: Big Five focus: Low Openness. |
Table 3: Representative Big5-Persona-EASY case. The negative instance is constructed by replacing the target profile with the opposite polarity of the same Big Five trait while keeping the dialogue context and response fixed, yielding a direct profile-level contrast.
| Field | Content |
|---|---|
| Profile | Big Five focus: High Agreeableness . You are warm, cooperative, empathetic, and motivated to maintain harmony with others. |
| Context | “Eesha, I’m beyond thrilled to see your book finally complete! It was an honor to help you bring your vision to life. How does it feel to hold the finished manuscript in your hands?” |
| Positive Response | “Kevion, I’m overwhelmed with gratitude. Your support and encouragement meant the world to me. I couldn’t have done it without you. Your selflessness and kindness inspire me to be a better person. Thank you for being an amazing friend throughout this journey.” |
| Hard Negative 1 | Type: |
| Construction: replace the positive response with one taken from a nearby cross-trait persona at the same polarity, in this case High Extraversion . | |
| Response: “Kevion, it’s an incredible feeling! I’m so grateful for your help and support. You’ve been an amazing partner in this journey. Let’s celebrate tonight! I’m taking you out for dinner and drinks to thank you properly. We’ll make it a night to remember!” |
Table 4: Representative Big5-Persona-HARD case. The positive response is paired with two hard negative types.
| Field | Content |
|---|---|
| Profile | Character: Pinocchio. |
| Personality: Innocent, naive, and adventurous . | |
| Introduction: You are Pinocchio, a wooden puppet brought to life by a fairy’s magic, dreaming of becoming a real boy. | |
| Dialogue Context | User: “My name is Alex. Nice to meet you, Agent!” |
| Agent: “Oh, it’s nice to meet you too, Alex! Are you from a far-off land or do you live here, nearby?” | |
| (Dialogue continues; abbreviated here for space.) |
Table 5: Representative Social-Persona case. Unlike the Big5-based benchmarks, the target profile is specified as an open-ended character description, and each instance is formed from one gold response together with multiple distractor options under the same dialogue context.
Figure 8: Vanilla direct judging prompt with the 5-point rubric.
Figure 9: CoT-based direct judging prompt with the 5-point rubric.
Figure 10: Vanilla direct judging prompt with the 7-point rubric.
Figure 11: CoT-based direct judging prompt with the 7-point rubric.
| Dataset | # Groups | Pos Mean | Neg Mean | AUC | P-AUC | G-Acc | Krippendorff’s |
|---|---|---|---|---|---|---|---|
| Social-Persona | 50 | 4.46 | 2.08 | 95.80 | 96.90 | 87.00 | 0.63 |
| Big5-Persona-EASY | 50 | 4.72 | 1.48 | 99.40 | 99.60 | 98.00 | 0.69 |
| Big5-Persona-HARD | 50 | 4.21 | 2.24 | 87.80 | 90.50 | 75.00 | 0.57 |
Table 6: Human verification results on a stratified sample from all three benchmarks. Three annotators rate persona consistency on the 5-point rubric.
| Dataset | Top-1 Acc. | Top-2 Recall | Macro-F1 | Krippendorff’s |
|---|---|---|---|---|
| Social-Persona | 70.7 | 87.3 | 72.4 | 0.61 |
| Big5-Persona-EASY | 75.3 | 90.0 | 76.8 | 0.66 |
| Big5-Persona-HARD | 63.3 | 82.7 | 66.5 | 0.56 |
Table 7: Dimension-level human diagnostic validation of PRISM using the Llama evaluator backbone.
Figure 12: Backbone sensitivity on strict Group Accuracy across three LLM backbones. Points denote individual backbones and diamonds indicate the mean with standard deviation.
Figure 13: Rubric Sensitivity of direct LLM-as-a-Judge evaluation on Strict Group Accuracy. Each segment connects the results from 5-point and 7-point rubrics for the same evaluator and prompting method. Longer segments indicate greater sensitivity to scoring granularity.
Figure 14: Score distribution on the Big5-Persona-EASY dataset
Figure 15: Score distribution on the Social-Persona dataset
Figure 16: Temperature sensitivity of holistic evaluation for the Qwen backbone across Big5-Persona-HARD and Social-Persona . Greedy denotes deterministic decoding, whereas the other points show stochastic decoding under temperatures 0.2, 0.5, 0.8 and 1.0. Error bars indicate mean and standard deviation over three random seeds. The dashed line shows the corresponding PRISM reference score.
Figure 17: Temperature sensitivity of holistic evaluation for the Mistral backbone across Big5-Persona-HARD and Social-Persona . Greedy denotes deterministic decoding, whereas the other points show stochastic decoding under temperatures 0.2, 0.5, 0.8 and 1.0. Error bars indicate mean and standard deviation over three random seeds. The dashed line shows the corresponding PRISM reference score.
Figure 18: Temperature sensitivity of holistic evaluation for the Llama backbone across Big5-Persona-HARD and Social-Persona . Greedy denotes deterministic decoding, whereas the other points show stochastic decoding under temperatures 0.2, 0.5, 0.8 and 1.0. Error bars indicate mean and standard deviation over three random seeds. The dashed line shows the corresponding PRISM reference score.