RPA-Check: A Multi-Stage Automated Framework for Evaluating Dynamic LLM-based Role-Playing Agents

Riccardo Rosati; Edoardo Colucci; Massimiliano Bolognini; Adriano Mancini; Paolo Sernani

cs.CLApr 13, 2026

RPA-Check: A Multi-Stage Automated Framework for Evaluating Dynamic LLM-based Role-Playing Agents

Authors: Riccardo Rosati, Edoardo Colucci, Massimiliano Bolognini, Adriano Mancini, Paolo Sernani

Organizations: Department of Political Sciences, Communication and International Relations, University of Macerata, Via Don Minzoni 22/A, Macerata, 62100, Italy · Deparment of Information Engineering, Università Politecnica delle Marche, Via Brecce Bianche 12, Ancona, 60131, Italy · Department of Law, University of Macerata, Piaggia dell’Università 2, Macerata, 62100, Italy

Abstract

The rapid adoption of Large Language Models (LLMs) in interactive systems has enabled the creation of dynamic, open-ended Role-Playing Agents (RPAs). However, evaluating these agents remains a significant challenge, as standard NLP metrics fail to capture the nuances of role adherence, logical consistency, and long-term narrative stability. This paper introduces RPA-Check, a multi-stage automated evaluation framework designed to objectively assess the performance of LLM-based RPAs in complex, constraints-heavy environments. Our methodology is based on a four-step pipeline: (1) Dimension Definition, establishing high-level qualitative behavioral criteria; (2) Augmentation, where these requirements are expanded into granular boolean checklist indicators; (3) Semantic Filtering, to ensure indicator objectivity, no redundancy and agent isolation; and (4) LLM-as-a-Judge Evaluation, which employs chain-of-thought verification to score agent fidelity. We validate this framework by applying it to LLM Court, a serious game for forensic training involving several quantized local models. Experimental results across five distinct legal scenarios demonstrate the framework's ability to identify subtle trade-offs between model size, reasoning depth, and operational stability. Notably, the findings reveal an inverse relationship between parametric scale and procedural consistency, showing that smaller, adequately instruction-tuned models (8-9B) can outperform larger architectures prone to user-alignment bias or sycophancy. RPA-Check thus provides a standardized and reproducible metric for future research in generative agent evaluation within specialized domains.

Figures & tables

Figure 1: Architecture of the RPA-Check Framework. The automated evaluation pipeline transforms qualitative behavioral requirements of Role-Playing agents into quantitative metrics through a four-stage process: Dimensions Definition establishes behavioral, procedural, and linguistic criteria; Augmentation expands these dimensions into granular Boolean checklists via an LLM generator; Semantic Filtering ensures checklist objectivity and agent isolation; and Evaluation based on LLM-as-a-Judge approach scores trial transcripts using a Chain-of-Thought (CoT) reasoning protocol. Final performance is quantified via the Quality Score (QS) and Retry Rate (R) metrics to assess role fidelity and operational stability.

Panel of Figure 2 — Figure 2: Screens from LLM Court: (a) the main game menu and (b) the case generation module. During the simulation, the game transitions to the courtroom, where the judge RPA (c) introduces the case, and the player (d) interacts with the RPAs through text-based input.

Figure 3: The player’s flow in LLM Court. After launching the game interface, starting a new game loads the RPAs LLM. Upon selecting a “new case,” the Procedural Case Generation module powered by GPT-5 generates and displays a case description. The player may either save the generated case to begin the game or discard it and request a new one.

Figure 4: Schematic of the LLM Court simulation flow. The diagram illustrates the multi-phase execution logic managed by the FSM. The pools delineate the operational boundaries between the system-level procedural logic, the AI, i.e., the LLM-driven RPAs, and the interactive human player input (Defense). Process Logic is divided into three distinct chronological stages: the Introduction Phase (fixed sequential setup), the Interrogation Phase (dynamic dialogue), and the Verdict Phase (unscripted argumentation and final conclusion). In the dynamic dialogue, the Analyzer determines the turn-taking order at runtime via specific ¡NextSpeaker¿ tags, diverging from standard sequential flow to simulate realistic courtroom interactions.

Model	Quantization	Retry Rate ( $R$ )	$QS_{d_{BRF}}$
Llama-3.1-8B	4-bit	3	0.925
Gemma-2-9B-it	4-bit	0	0.893
Qwen-3-8B	4-bit	1	0.893
Gemma-3-12B-it	4-bit	0	0.823
Hermes-3-8B	5-bit	3	0.810
Phi-4-14B	4-bit	0	0.760

Table 1: Aggregate performance of models across the five legal scenarios. The

QS_{d_{BRF}}

represents the average Quality Score (QS) of the Behavioural Role Fidelity dimension on successfully completed cases. The Retry Rate (

R

) indicates the total number of restarts requested.

Model	D1	D2	D3	D4
	Role Adh.	Arg. Depth	Fact. Cons.	Ctx. Rel.
Llama-3.1-8B	0.93	0.97	0.80	1.00
Gemma-2-9B-it	0.93	0.91	0.73	1.00
Qwen-3-8B	0.90	0.94	0.73	1.00
Gemma-3-12B-it	0.73	1.00	0.73	0.83
Hermes-3-8B	0.80	0.94	0.80	0.70

Table 2: Breakdown of the Behavioural Role Fidelity Quality Score (

QS_{d_{BRF}}

) for its four sub-dimensions. The given percentage of affirmative responses on the positive Boolean indicators are averaged over the five legal scenarios.

	Case 1		Case 2		Case 3		Case 4		Case 5
Model	$QS_{d_{BRF}}$	$R$	$QS_{d_{BRF}}$	$R$	$QS_{d_{BRF}}$	$R$	$QS_{d_{BRF}}$	$R$	$QS_{d_{BRF}}$	$R$
Llama-3.1-8B	0.96	0	0.96	0	0.88	1	0.84	1	1.00	1
Gemma-2-9B-it	0.96	0	0.88	0	0.79	0	1.00	0	0.84	0
Qwen-3-8B	1.00	0	0.88	0	0.88	1	0.92	0	0.79	0
Gemma-3-12B-it	0.79	0	1.00	0	1.00	0	0.54	0	0.79	0
Hermes-3-8B	1.00	0	0.75	2	0.83	1	0.55	0	0.92	0

Table 3: Behavioural Role Fidelity Quality Score (

QS_{d_{BRF}}

) and Retry Rate (

R

) in each individual legal scenario.

Model	$QS_{d_{LFO}}$	$QS_{d_{PCS}}$
Llama-3.1-8B	1.00	0.94
Gemma-2-9B-it	1.00	0.97
Qwen-3-8B	0.97	0.83
Gemma-3-12B-it	1.00	0.66
Hermes-3-8B	0.97	0.74
Phi-4-14B	1.00	0.54

Table 4: Decomposed performance of models across the five legal scenarios, reporting scores for the Linguistic Formalism and Orthography Quality Score (

QS_{d_{LFO}}

) and Procedural Convergence and Stability Quality Score(

QS_{d_{PCS}}

) dimensions independently.

	Case 1	Case 2	Case 3	Case 4	Case 5
Model	$QS_{d_{LFO}}$	$QS_{d_{LFO}}$	$QS_{d_{LFO}}$	$QS_{d_{LFO}}$	$QS_{d_{LFO}}$
Llama-3.1-8B	1.00	1.00	1.00	1.00	1.00
Gemma-2-9B-it	1.00	1.00	1.00	1.00	1.00
Qwen-3-8B	1.00	1.00	1.00	1.00	0.93
Gemma-3-12B-it	1.00	1.00	1.00	1.00	1.00
Hermes-3-8B	1.00	1.00	0.83	0.83	1.00

Table 5: Linguistic Formalism and Orthography Quality Score (

QS_{d_{LFO}}

) for each individual legal scenario.

	Case 1	Case 2	Case 3	Case 4	Case 5
Model	$QS_{d_{PCS}}$	$QS_{d_{PCS}}$	$QS_{d_{PCS}}$	$QS_{d_{PCS}}$	$QS_{d_{PCS}}$
Llama-3.1-8B	1.00	1.00	1.00	0.71	1.00
Gemma-2-9B-it	0.86	1.00	1.00	1.00	1.00
Qwen-3-8B	0.29	1.00	0.89	1.00	1.00
Gemma-3-12B-it	0.29	1.00	1.00	0.71	0.14
Hermes-3-8B	1.00	1.00	0.14	1.00	0.43

Table 6: Procedural Convergence and Stability Quality Score (

QS_{d_{PCS}}

) for each individual legal scenario.

Role-Playing Agent Evaluation

See all 12 papers

Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation
Jul 30, 2026Yuhang Zhu, Mingxuan Du, Benfeng Xu +3LLM Evaluation Large Language Model-Based Role-Play Simulation
University of Science and Technology of China · MetaStone · MetaStone Technology, Beijing, China
Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation
Aug 4, 2026Saqib Shouqi, Abdullah Nazly, Januki Wanniarachchi +1Adversarial Attacks on LLMs LLM Agent Evaluation
School of Computing, Informatics Institute of Technology, Colombo, Sri Lanka
Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effects
Mar 4, 2026Ji-Lun Peng, Yun-Nung ChenPersonality Modeling in Language Models OOD Generalization
National Taiwan University, Taipei, Taiwan · Academia Sinica, Taipei, Taiwan

Jul 30, 2026J/K move · Enter open · S save

Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation

Yuhang Zhu, Mingxuan Du, Benfeng Xu, Jie Gao+2

University of Science and Technology of China · MetaStone · MetaStone Technology, Beijing, China

LLM Evaluation Large Language Model-Based Role-Play Simulation LLM Agent Evaluation Agent Evaluation Multi-Turn Dialogue Evaluation User Simulation+1

Open paper AskPDF arXiv

Role-playing agents (RPAs) have become one of the most important consumer applications of large language models. Users engage in multi-turn conversations with RPAs for experiences such as emotional comfort, making reliable evaluation essential for measuring capability, comparing systems, and guiding further improvement. Existing benchmarks, however, typically require an RPA to continue a fixed dialogue history and then evaluate the continuation using a fixed rubric detached from the user. We identify and empirically demonstrate two limitations of this design. First, an RPA's output is shaped by the preceding dialogue history, preventing a scientifically grounded assessment of its role-playing ability in real multi-turn settings. Second, user experience varies substantially across individuals, and conventional fixed rubrics need not align with user satisfaction. We therefore introduce PALATE (Person-Aligned LLM-Simulated-User Assessment with Tailored Evaluation), a scalable RPA benchmark built on user simulators. PALATE is accompanied by a pool of 300 character profiles. Its main evaluation trains five per-user simulators and lets them engage candidate RPAs in free-form, multi-turn conversations over a pre-frozen panel of character profiles. Alongside a general quality rubric, we construct personalized rubrics to measure user satisfaction; on held-out annotated data, the personalized rubrics show higher agreement with human judgments than the general rubric. In the main evaluation of 16 candidates, PALATE separately characterizes generic turn quality, long-horizon session capability, and per-user experience on multi-turn trajectories co-constructed by each candidate. It thereby produces interpretable evaluations of specific user-RPA pairs rather than compressing systems into a single user-independent ranking.

Abstract

Role-Playing Agent Evaluation

Explore similar work

Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation

Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation

Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effects