MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents

Zhenyu Wang; Xiaofen Xing; Yirong Chen; Xiangmin Xu

cs.CLFeb 24, 2026

MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents

Authors: Zhenyu Wang, Xiaofen Xing, Yirong Chen, Xiangmin Xu

Organizations: South China University of Technology, Guangzhou 510641, China · Foshan University, Foshan 528225, China

Abstract

Multimodal Role-Playing Agents (MRPAs) are attracting increasing attention due to their ability to deliver more immersive multimodal emotional interactions. However, existing studies still rely on pure textual benchmarks to evaluate the text responses of MRPAs, while delegating the assessment of their multimodal expressions solely to modality-synthesis metrics. This evaluation paradigm, on the one hand, entangles semantic assessment with modality generation, leading to ambiguous error attribution, and on the other hand remains constrained by the heavy reliance on human judgment. To this end, we propose MERRY, a semantically decoupled evaluation framework for assessing Multimodal Emotional and Role consistencies of Role-playing agents. This framework introduce five refined metrics for EC and three for RC. Notably, we transform the traditional subjective scoring approach into a novel bidirectional-evidence-finding task, significantly improving the human agreement of LLM-as-Judge evaluations. Based on MERRY, we conduct extensive evaluations. Our empirical results primarily reveal that: (1) Training on synthetic datasets tends to reduce emotional consistency, whereas training on real-world datasets improves it; (2) Existing models suffer from emotional templatization and simplification, exhibiting positive-bias and performance bottleneck in fine-grained negative emotions; (3) Simple prompting method strengthens the weak models but constrains the strong ones, while simple fine-tuning method suffers from poor role generalization. Codes and dataset are available.

Figures & tables

Fig. 1: Current end-to-end evaluations lead to ambiguous error attribution and strong dependence on human assessment. For instance, given a same video output and end-to-end human evaluation result, the issue can be ambiguously attributed to either MLLM (top) or Talker (bottom).

	Semantically Decoupled MM Expression Eval	EC	RC	Cross-modal Eval
CharacterEval [ 5 ]	✘	✘	✔	✘
CharacterBox [ 12 ]	✘	✘	✔	✘
InCharacter [ 10 ]	✘	✘	✔	✘
TimeChara [ 13 ]	✘	✘	✔	✘
MMRole [ 1 ]	✘	✘	✔	✘
Beyond Dialogue [ 14 ]	✘	✔	✔	✘

TABLE I: Existing Role-Playing Benchmarks.

Dataset	Modalities (per turn)	Data Type	Role Profiles	Emotion Labels	Semantic Descriptions of MM data	Language	Dialogue Source
Multimodal Emotional/Empathetic Dialogue Datasets
MELD [ 15 ]	(a, v, l)	MM dialogue	✘	✔	✘	EN	“Friends” TV
M3ED [ 16 ]	(a, v, l)	MM dialogue	✘	✔	✘	CN	56 TVs
CPED [ 11 ]	(a, v, l)	MM dialogue	✘	✔	✘	CN	40 TVs
MC-EIU [ 17 ]	(a, v, l)	MM dialogue	✘	✔	✘	EN&CN	3 English TVs 4 Chinese TVs
MESC [ 18 ]	(a, v, l)	MM dialogue	✘	✔	✘	EN	“In Treatment” TV

TABLE II: Comparison with Existing Dataset. a, v, and l refer audio, visual, and text respectively. MM Refers to MultiModal. EN and CN Refer to English and Chinese respectively.

Fig. 2: Framework of MERRY . Upper Left : dialogue format of evaluation tasks. Lower Left : evaluation pipeline. Right : eight metrics for emotional and role consistencies.

	$emo_{x}$ in set(PD)	$emo_{x}$ not in set(PD)
$emo_{x}$ in set(GT)	TP	FN
$emo_{x}$ not in set(GT)	FP	TN

TABLE III: Confusion matrix.

emo_{x}

denotes the x-th class emotion. GT and PD refers to

emos_{gt}

and

emos_{fusion}

, respectively. “set(list)” function is used to convert the list into unique elements.

AgreeFlag	0	1	1	1	1	0
DisagreeFlag	0	0	1	1	1	1
evidences (agree vs. disagree)	\	\	>	=	<	\
FinalScore	dropped	5	4	3	2	1

TABLE IV: Role consistency mapping table.

Fig. 3: MERRY-Data Construction Pipeline

Split	# Samples	# Roles	# Dialogues
Train	23817	239	3577
Test	1690	18	401

TABLE V: Statistics of train and test splits of MERRY-Data .

#samples	25507
#dialogues	3978
Avg. tokens of previous information	35.33
#roles	257
Avg. of #utterances per role	114.73
Avg. of #turns per dialogue	7.41
Avg. durations of video (seconds)	6.00

TABLE VI: Statistics of whole proposed dataset MERRY-Data .

Models	Type	MEC $\uparrow$		CEC $\uparrow$		EDD $\downarrow$		RCD $\leftrightarrow$		ED $\downarrow$				RC $\uparrow$
Models	Type	lower	upper	lower	upper	intra	inter	intra	inter	all	spe	fac	bod	Exp	Cha	Rel
Close-source MLLMs
Doubao-1.5v $\dagger$	All	0.243	0.546	0.658	0.745	0.410	0.455	-0.033	-0.006	0.648	0.634	0.632	0.615	4.891	4.720	4.877
	Prof	0.232	0.489	0.592	0.745	0.414	0.449	-0.017	0.011	0.610	0.577	0.596	0.587	4.634	4.805	4.800
	Prev	0.243	0.524	0.615	0.755	0.425	0.456	-0.030	0.003	0.598	0.563	0.586	0.573	4.909	4.804	4.923
	None	0.230	0.504	0.637	0.727	0.412	0.454	-0.026	-0.006	0.661	0.636	0.645	0.644	4.439	4.527	4.757

TABLE VII: Performance results of base models with training-free role-playing methods on the entire testset. Best results are in bold , and the second best result are underlined . Type Prev , Prof , All denotes that the model is equipped with previous information, profile, both of them, respectively. While None denotes that the model are asked to play a role without profile and previous information.

\dagger

means the audio files are replaced as speech prompts for these models.

Trainset	Method	MEC $\uparrow$		CEC $\uparrow$		EDD $\downarrow$		RCD $\leftrightarrow$		ED $\downarrow$				RC $\uparrow$
Trainset	Method	lower	upper	lower	upper	intra	inter	intra	inter	all	spe	fac	bod	Exp	Cha	Rel
None	None	0.264	0.536	0.586	0.652	0.381	0.398	-0.002	-0.017	0.612	0.537	0.482	0.471	4.026	4.117	4.170
OmniCharacter	SFT	0.256	0.534	0.491	0.514	0.365	0.416	-0.005	-0.001	0.683	0.451	0.553	0.471	4.205	4.276	4.472
MMRole	SFT	0.275	0.517	0.550	0.586	0.388	0.402	0.016	-0.061	0.539	0.204	0.167	0.167	4.267	4.274	4.417
MERRY (Ours)	SFT	0.291	0.541	0.934	0.962	0.379	0.412	-0.024	-0.021	0.257	0.117	0.106	0.152	4.357	4.167	4.305

TABLE VIII: Performance results of different training methods on the entire testset, with MiniCPM-o-2.6(8B) as the unified base model for fair comparison. Best results are in bold , and the second best result are underlined .

Fig. 4: Emotional transition matrices of Andy in Ode to Joy . Groundtruth matrices are list in the first column. The rest columns are role-played by GPT-5-chat with different types.

Fig. 5: The visualization of Recall and Precision scores of each emotions when calculating

MEC_{lower}

on all models with

type=All

Fig. 6: Average of RC scores of Gemini-2.5-pro based on different episodes of Ode to Joy . Notably, Ode to Joy is a 2-series TV. Episodes 1 to 18 are from Series 1, while episodes 19 to 38 are from Series 2.

	facial expression	body movement	speech prompt
$\bar{s}$	4.348	4.323	4.719
$\alpha$	0.585	0.539	0.571

TABLE IX: Results of human agreement experiment on description quality.

\alpha

refers to Krippendorff’s alpha.

\bar{s}

refers to the average score.

Evaluator	Type	Exp	Cha	Rel
Avg. (Ours)	finding bidirectional evidences	0.745	0.619	0.721
GPT-5.2		0.631	0.552	0.619
Doubao-1.6		0.503	0.443	0.492
Avg.	subjective scoring with explanations	0.558	0.386	0.482
GPT-5.2		0.490	0.354	0.407
Doubao-1.6		0.367	0.238	0.339

TABLE X: Krippendorff’s alpha between human and different evaluator with different evaluation type. Best results are in bold .

	facial expression	body movement	speech prompt	fusion
$\alpha$	0.696	0.656	0.709	0.721

TABLE XI: Krippendorff’s alpha between human and ERC experts across different modalities.

Explore similar work

MM-tau-p $^2$ : Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settings
Mar 10, 2026Anupam Purwar, Aditya ChoudharyAgent Evaluation AI Agent Benchmarks
Sprinklr AI, Gurugram, India
Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation
Jul 30, 2026Yuhang Zhu, Mingxuan Du, Benfeng Xu +3LLM Evaluation Large Language Model-Based Role-Play Simulation
University of Science and Technology of China · MetaStone · MetaStone Technology, Beijing, China
Through the Lens of Character: Resolving Modality-Role Interference in Multimodal Role-Playing Agent
May 10, 2026Yihong Tang, Kehai Chen, Xuefeng Bai +1Multimodal Large Language Models Multimodal Agents
Harbin Institute of Technology, Shenzhen, China · Shenzhen Loop Area Institute (SLAI), Shenzhen, China

Mar 10, 2026J/K move · Enter open · S save

MM-tau-p $^2$ : Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settings

Anupam Purwar, Aditya Choudhary

Sprinklr AI, Gurugram, India

Agent Evaluation AI Agent Benchmarks Multimodal Model Evaluation Multimodal Agents Multi-Turn Dialogue Evaluation

Open paper AskPDF arXiv

Current evaluation frameworks and benchmarks for LLM powered agents focus on text chat driven agents, these frameworks do not expose the persona of user to the agent, thus operating in a user agnostic environment. Importantly, in customer experience management domain, the agent's behaviour evolves as the agent learns about user personality. With proliferation of real time TTS and multi-modal language models, LLM based agents are gradually going to become multi-modal. Towards this, we propose the MM-tau-p

^2

benchmark with metrics for evaluating the robustness of multi-modal agents in dual control setting with and without persona adaption of user, while also taking user inputs in the planning process to resolve a user query. In particular, our work shows that even with state of-the-art frontier LLMs like GPT-5, GPT 4.1, there are additional considerations measured using metrics viz. multi-modal robustness, turn overhead while introducing multi-modality into LLM based agents. Overall, MM-tau-p

^2

builds on our prior work FOCAL and provides a holistic way of evaluating multi-modal agents in an automated way by introducing 12 novel metrics. We also provide estimates of these metrics on the telecom and retail domains by using the LLM-as-judge approach using carefully crafted prompts with well defined rubrics for evaluating each conversation.

Abstract

Explore similar work

MM-tau-p2^22: Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settings

Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation

Through the Lens of Character: Resolving Modality-Role Interference in Multimodal Role-Playing Agent

MM-tau-p $^2$ : Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settings