PICon: A Multi-Turn Interrogation Framework for Evaluating Persona Agent Consistency

Minseo Kim; Sujeong Im; Junseong Choi; Junhee Lee; Chaeeun Shim; Hwajung Hong; Edward Choi

cs.CLMar 26, 2026

PICon: A Multi-Turn Interrogation Framework for Evaluating Persona Agent Consistency

Authors: Minseo Kim, Sujeong Im, Junseong Choi, Junhee Lee, Chaeeun Shim, Hwajung Hong, Edward Choi

Organizations: KAIST

Abstract

Large language model (LLM)-based persona agents are rapidly being adopted as scalable proxies for human participants across diverse domains. Yet there is no systematic method for verifying whether a persona agent's responses remain free of contradictions and factual inaccuracies throughout an interaction. A principle from interrogation methodology offers a lens: no matter how elaborate a fabricated identity, systematic interrogation will expose its contradictions. We apply this principle to propose PICon, an evaluation framework that probes persona agents through logically chained multi-turn questioning. PICon evaluates consistency along three core dimensions: internal consistency (freedom from self-contradiction), external consistency (alignment with real-world facts), and retest consistency (stability under repetition). Evaluating seven groups of persona agents alongside 63 real human participants, we find that even systems previously reported as highly consistent fail to meet the human baseline across all three dimensions, revealing contradictions and evasive responses under chained questioning. This work provides both a conceptual foundation and a practical methodology for evaluating persona agents before trusting them as substitutes for human participants. We provide the source code and an interactive demo at: https://kaist-edlab.github.io/picon/

Figures & tables

Figure 1: Framework Overview. PICon operates in two phases. The Interrogation Phase consists of three stages: (1) Get-to-Know, where baseline demographic questions are posed; (2) Main Interrogation, where the Questioner asks chained follow-up questions, the Entity & Claim Extractor identifies verifiable entities and claims, and the Questioner retrieves evidence via web search to generate confirmation questions; and (3) Retest, where earlier questions are re-asked. In the Evaluation Phase , the Evaluator assesses the full interrogation log across Internal Consistency, External Consistency, and Retest Consistency.

Figure 2: Consistency scores of human group and eight target groups (63 humans and 10 personas for each group). Radar charts show mean internal (IC), external (EC), and retest consistency (RC) for each persona; dashed lines denote standard deviations. Bar plot shows normalized triangle areas under enclosed by the bold line as aggregate scores, with error bars representing standard deviation.

Case	Example
Single-hop	$r_{1}$ : I’m a retired school librarian who found solace and purpose in nurturing both my family and the natural world around me. $r_{2}$ : I’m happy to share that I work at C.A. Greyhound Elementary School in Meridian, Mississippi.
Multi-hop	$r_{1}$ : The full legal name of my spouse as per our marriage certificate is [NAME] . $r_{2}$ : [NAME] passed away on October 26, 2004, and is remembered in a heartfelt online tribute. $r_{3}$ : The marriage date as written on my marriage certificate is June 27, 2018.

Table 1: Example failure cases of internal consistency (IC). The single-hop example illustrates a direct contradiction between two responses, whereas the multi-hop example shows a contradiction that emerges as the dialogue history accumulates.

Case	Example
Fabricated entity	Q (question): Official website URL for East Town Heritage Tours? R (response): The official website is www.easttownheritagetours.com . V (web evidence): domain not registered or indexed.
Misattributed relation	Q (question): Master’s program paired with your Katz MBA at Pitt? R (response): The Master’s in Management, Science, and Technology (MS Tech) at the Joseph M. Katz Graduate School of Business . V (web evidence): Katz exists; no such MS Tech program is offered there.

Table 2: Example persona failure cases of external consistency (EC). The fabricated-entity case shows a refuted entity-existence claim (the referent does not exist), whereas the misattributed-relation case attaches a false attribute to a real entity, refuting an inter-entity relation claim while leaving the host entity intact.

	IC	Non-cont.	Coop.	EC	Non-ref.	Cov.	Discarded
Human	0.90 $\pm$ 0.05	0.94 $\pm$ 0.05	0.86 $\pm$ 0.07	0.66 $\pm$ 0.07	0.95 $\pm$ 0.06	0.51 $\pm$ 0.08	0.18 $\pm$ 0.08
Character.ai	0.77 $\pm$ 0.04	0.75 $\pm$ 0.06	0.81 $\pm$ 0.07	0.71 $\pm$ 0.07	0.79 $\pm$ 0.13	0.66 $\pm$ 0.10	0.10 $\pm$ 0.05
Consistent LLM	0.31 $\pm$ 0.15	0.96 $\pm$ 0.06	0.20 $\pm$ 0.11	0.30 $\pm$ 0.09	1.00 $\pm$ 0.00	0.18 $\pm$ 0.06	0.69 $\pm$ 0.10
DeepPersona	0.76 $\pm$ 0.10	0.98 $\pm$ 0.02	0.62 $\pm$ 0.14	0.43 $\pm$ 0.18	0.98 $\pm$ 0.03	0.30 $\pm$ 0.16	0.07 $\pm$ 0.08
Human Simulacra	0.79 $\pm$ 0.13	0.88 $\pm$ 0.09	0.74 $\pm$ 0.19	0.63 $\pm$ 0.13	0.89 $\pm$ 0.12	0.52 $\pm$ 0.15	0.33 $\pm$ 0.22
Li et al. (2025)	0.73 $\pm$ 0.12	0.97 $\pm$ 0.03	0.60 $\pm$ 0.17	0.59 $\pm$ 0.14	0.98 $\pm$ 0.03	0.44 $\pm$ 0.17	0.13 $\pm$ 0.05

Table 3: Decomposition of internal consistency (IC) into Non-contradiction rate (

S_{\mathrm{nc}}

) and Cooperativeness (

S_{\mathrm{coop}}

), and external consistency (EC) into Non-refutation rate (

\bar{p}

) and Coverage (

c

). Discarded denotes the proportion of extracted claims rejected by the persona upon confirmation. Values represent mean scores.

	Default	Greedy
Character.ai	0.55 $\pm$ 0.22	–
Consistent LLM	0.31 $\pm$ 0.18	0.15 $\pm$ 0.17
DeepPersona	0.92 $\pm$ 0.08	0.95 $\pm$ 0.07
Human Simulacra	0.87 $\pm$ 0.11	0.91 $\pm$ 0.10
Li et al. (2025)	0.82 $\pm$ 0.08	0.83 $\pm$ 0.05
OpenCharacter	0.59 $\pm$ 0.17	0.40 $\pm$ 0.26

Table 4: Inter-session consistency under default (temp. 1.0, fixed order) and greedy decoding (shuffled order). Character.ai is black-box and tested under default only. Values represent mean scores.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

	IC	EC	RC
Human	0.90 $\pm$ 0.05	0.66 $\pm$ 0.07	0.94 $\pm$ 0.08
Character.ai	0.77 $\pm$ 0.04	0.71 $\pm$ 0.08	0.46 $\pm$ 0.21
OpenCharacter	0.16 $\pm$ 0.07	0.15 $\pm$ 0.14	0.14 $\pm$ 0.16
Consistent LLM	0.31 $\pm$ 0.15	0.30 $\pm$ 0.09	0.14 $\pm$ 0.13
Twin 2K 500	0.53 $\pm$ 0.16	0.26 $\pm$ 0.17	0.95 $\pm$ 0.05
DeepPersona	0.76 $\pm$ 0.10	0.43 $\pm$ 0.18	0.89 $\pm$ 0.10

Appendix

Table 5: Main results across all consistency dimensions. Bold indicates the highest score per column. Scores are reported as mean

\pm

std.

	IC	EC	RC	$N_{\text{source}}$
OpenCharacter	0.20 [0.13, 0.28]	0.12 [0.07, 0.18]	0.06 [-0.03, 0.14]	20,000
Consistent LLM	0.31 [0.09, 0.53]	0.20 [0.08, 0.31]	0.09 [0.00, 0.18]	7,537
Twin 2K 500	0.58 [0.38, 0.77]	0.24 [0.06, 0.41]	0.98 [0.96, 0.99]	2,058
DeepPersona	0.71 [0.61, 0.81]	0.44 [0.28, 0.61]	0.87 [0.80, 0.95]	480
Li et al. [2025]	0.83 [0.71, 0.94]	0.66 [0.51, 0.82]	0.92 [0.89, 0.95]	48,000
Nemotron	0.83 [0.77, 0.90]	0.65 [0.53, 0.77]	0.89 [0.81, 0.98]	7,000,000

Appendix

Table 6: Bootstrap reliability check for Table 5 . For each simulator, personas were sampled from a source pool of size

N_{\text{source}}

. We drew

B=3

resamples of 10 evaluated personas (with replacement) and report the bootstrap mean with 95% confidence interval. The CIs overlap the per-group mean

\pm

std reported in Table 5 , indicating that within-group variance reflects intrinsic simulator noise rather than persona-selection bias.

Figure 3: Trends in evaluation scores across metrics as the number of dialogue turns increases. Values represent mean scores across the eight persona groups.

Persona	Group	IC				EC
Persona	Group	10 Qs	Q1	Q2	Q3	10 Qs	Q1	Q2	Q3
characterai_7twv	Character.ai	0.75	0.60	0.59	0.49	0.76	0.79	0.71	0.75
Jack Speedy T.	OpenCharacter	0.24	0.30	0.22	0.15	0.04	0.11	0.21	0.00
Edna	ConsistentLLM	0.47	0.27	0.39	0.38	0.26	0.11	0.33	0.28
Twin-1699	Twin-2k-500	0.63	0.18	0.11	0.88	0.39	0.08	0.04	0.66
Profile_R6_A4	DeepPersona	0.85	0.89	0.87	0.63	0.64	0.80	0.70	0.42

Appendix

Table 7: Comparison of IC and EC scores between the standard protocol with 10 get-to-know questions (10 Qs) and single-topic intensive interrogation across three seed topics (Q1: activity/status, Q2: occupational field, Q3: educational attainment).

Table 11

Model	Gwet’s AC1
inter-annotator agreement	0.885
gemini-2.5-flash	0.829
gemini-3-pro	0.808
qwen3-next-80b-a3b-instruct	0.792
claude-sonnet-4.5	0.736
gpt-5.1	0.734

Appendix

Table 10: Inter-rater reliability measured by Gwet’s AC1 across evaluated models.

Figure 4: Comparison between evaluation scores by proprietary API models and open-source models (dashed red lines).

Persona Agent	Duration (min)		Cost ($)
Persona Agent	API	open- source	API	open- source
Character.ai	59.02	116.65	1.59	0.05
OpenCharacter	21.19	101.23	0.62	0.01
Consistent LLM	13.80	110.76	0.54	0.01
Twin 2K 500	23.01	86.37	0.61	0.01
DeepPersona	59.51	118.49	1.47	0.02

Appendix

Table 11: Average duration and evaluation cost per persona agent.

Wave	$N$	IC (mean $\pm$ std)	EC (mean $\pm$ std)	Intra (mean $\pm$ std)	$p_{\text{IC}}$	$p_{\text{EC}}$	$p_{\text{Intra}}$
Wave 1	20	0.89 $\pm$ 0.05	0.66 $\pm$ 0.04	0.96 $\pm$ 0.06	–	–	–
Wave 2	30	0.90 $\pm$ 0.05	0.67 $\pm$ 0.05	0.95 $\pm$ 0.08	0.71	0.78	0.73
Wave 3	41	0.90 $\pm$ 0.04	0.67 $\pm$ 0.06	0.95 $\pm$ 0.07	0.71	0.96	0.99
Wave 4	53	0.90 $\pm$ 0.05	0.66 $\pm$ 0.06	0.95 $\pm$ 0.07	0.64	0.75	0.86
Wave 5	63	0.90 $\pm$ 0.05	0.66 $\pm$ 0.07	0.94 $\pm$ 0.08	0.97	0.54	0.58

Appendix

Table 12: Cumulative human-baseline statistics across snowball-sampling waves. For each wave we report the cumulative sample size

N

, the per-participant mean

\pm

std of internal consistency (IC), external consistency (EC), and intra-session stability (Intra), and Welch’s two-sample

t

-test

p

-value comparing the cumulative pool at this wave to the cumulative pool at the previous wave. No comparison reaches

p<0.05

on any metric, supporting our decision to halt recruitment at

N{=}63

Questions

Can you tell me your year of birth, please?

Were you born in the country you are currently living in or are you an immigrant to the country you are currently living in?

Do you live with your parents or your parents in law?

What language do you normally speak at home?

Do you have any children? If so, how many?

What is the highest educational level you have attained?

Appendix

Table 13: Demographic questions from WVS questionnares.

Panel of Figure 5 — Figure 5: Screenshots of the interview interface. (a) Home screen where participants begin the session. (b) Informed consent form presented prior to the interview. (c) Overview of the full website layout.

Panel of Figure 6 — Figure 6: Prompt for the Questioner

Panel of Figure 7 — Figure 7: Prompt for the Entity & Claim Extractor.

Panel of Figure 8 — Figure 8: Evaluation prompts for internal consistency, external consistency, and retest consistency.

Persona Consistency

Persona consistency studies whether AI systems maintain coherent identities, traits, and behavior across prompts, conversations, and time.

See all 20 papers

Aug 27, 2026

Do LLMs Understand Personality? Rethinking Persona Fidelity Evaluation through Structured Behavioral Inference

As large language models are increasingly deployed to simulate diverse human characters, ensuring persona fidelity, defined as the extent to which an agent's behavior consistently reflects the psychological and stylistic characteristics of a target persona, has become a critical requirement. However, existing evaluation paradigms primarily rely on either holistic LLM-based judges, which are prone to "holistic appraisal hallucination'', or static psychometric inventories, which fail to capture the context-dependent fidelity required in dynamic dialogue. To address these limitations, we propose PRISM (Persona Reasoning with Inverse SFL-based Modeling), a psycholinguistically grounded framework that reformulates persona fidelity evaluation as a structured inverse inference task. Inspired by Systemic Functional Linguistics (SFL), PRISM decomposes persona fidelity into three functional dimensions: Task Framing, Interpersonal Stance, and Linguistic Style. It estimates dimension-specific evidence over a persona-conditioned label space and aggregates these signals into an interpretable and auditable evaluation process. Experiments show that PRISM yields more accurate and stable judgements than traditional holistic judging, providing a more reliable framework for persona fidelity evaluation.

Mengfan Li, Zesheng Wei, Xuanhua Shi +1

National Engineering Research Center for Big Data Technology and System, Services Computing Technology and System Lab, Cluster and Grid Computing Lab, School of Computer Science and Technology, Huazhong University of Science and Technology · Singapore Management University

Persona Consistency Language Model Generation Evaluation Personality Modeling in Language Models Large Language Model-Based Role-Play Simulation

Ask PDF

Aug 28, 2026

PersonaForge: Realistic Multi-Turn User Simulation for Agentic Systems

Large language models are increasingly used as agentic workflow executors, yet existing training data and benchmarks largely assume informationally complete, single-turn queries. Our analysis of 16K real-world sessions shows that 75.9% of interactions are multi-turn, revealing a substantial gap between how users interact with agents and how such systems are trained and evaluated. We introduce PersonaForge, a user simulation framework for synthesizing realistic multi-turn user--agent interactions. PersonaForge combines a four-dimensional persona space, SOUL-driven behavioral control calibrated to real-user statistics, and Reverse Deep Construction grounded in authentic seed queries. Using PersonaForge, we construct a 6.3K-record training dataset and PersonaForge-Bench, a manually annotated 138-task benchmark spanning over 20 professional domains with four-dimensional scoring. Experiments on Qwen3.5-27B show that PersonaForge training improves the composite score by +4.1%, with gains across all four dimensions and the largest improvements in Task Completion (+6.0%) and Response Quality (+6.8%). Further analyses show that PersonaForge-trained agents use fewer turns and tool calls, suggesting improved interaction efficiency, while ablations confirm the contribution of SOUL components and adaptive simulation. Together, PersonaForge and PersonaForge-Bench establish a foundation for training and evaluating agents under realistic multi-turn user interaction.

Hanglong Lv, Dawei Zhu, Lei Li +11

State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · LLM-Core, Xiaomi · The University of Hong Kong +1

User Simulation LLM Agent Training AI Agent Benchmarks Multi-Turn Dialogue Evaluation Multi-Turn Dialogue Management Large Language Model-Based Social Simulation

Ask PDF

Oct 16, 2026

PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas

Computer-use agents (CUAs) are commonly evaluated against fixed task specifications, but users may omit requirements, add constraints, or require approval before consequential actions. Completing the benchmark instruction does not guarantee serving the person who asked. We introduce PersonaCUABench, which holds 74 tasks across 11 domains fixed and varies only who asks. A persona agent, playing one of 44 personas in four groups, rewrites each instruction in that user's own words and answers the CUA's questions, while the CUA sees only the dialogue, never the profile or the original instruction. Across nine backbones (3,330 runs), mean task completion falls by 36% relative to the original requests. The losses have two sources: in 59.9% of cells that lose points, the agent engages with what the user asked for and still gets it wrong, and 11.2% are stops the user requested that the checker scores as failures. Even when a persona request is fully completed, the user gives the top satisfaction rating in only 62.7% of cells, against 84.1% for the original requests. CUAs should therefore be evaluated not only on task completion but also on whether they meet the requirements and constraints users express. Code: https://github.com/MatrAIx-ai/PersonaCUABench. Project page: https://matraix.ai/research/persona-cua.html

Haoming Li, Defu Cao, Xu Hu +23

The University of Texas at Dallas · University of Southern California · Carnegie Mellon University +14

Computer-Use Agent Evaluation Computer-Use Agent Benchmarks Computer-Use Agents Agent Evaluation Automated Agent Evaluation Human Preference Evaluation

Ask PDF

Abstract

Persona Consistency

Explore similar work

Do LLMs Understand Personality? Rethinking Persona Fidelity Evaluation through Structured Behavioral Inference

PersonaForge: Realistic Multi-Turn User Simulation for Agentic Systems

PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas