MPCEval: A Benchmark for Multi-Party Conversation Generation

Minxing Zhang; Yi Yang; Zhuofan Jia; Xuan Yang; Jian Pei; Yuchen Zang; Xingwang Deng; Xianglong Chen

cs.CLMar 5, 2026

MPCEval: A Benchmark for Multi-Party Conversation Generation

Authors: Minxing Zhang, Yi Yang, Zhuofan Jia, Xuan Yang, Jian Pei, Yuchen Zang, Xingwang Deng, Xianglong Chen

Organizations: Duke University USA · Tanka AI USA

Abstract

Multi-party conversation generation, such as smart reply and collaborative assistants, is an increasingly important capability of generative AI, yet its evaluation remains a critical bottleneck. Compared to two-party dialogue, multi-party settings introduce distinct challenges, including complex turn-taking, role-dependent speaker behavior, long-range conversational structure, and multiple equally valid continuations. Accordingly, we introduce MPCEval, a task-aware evaluation and benchmarking suite for multi-party conversation generation. MPCEval decomposes generation quality into speaker modeling, content quality, and speaker--content consistency, and explicitly distinguishes local next-turn prediction from global full-conversation generation. It provides novel, quantitative, reference-free, and reproducible metrics that scale across datasets and models. We apply MPCEval to diverse public and real-world datasets and evaluate modern generation methods alongside human-authored conversations. The results reveal systematic, dimension-specific model characteristics in participation balance, content progression and novelty, and speaker--content consistency, demonstrating that evaluation objectives critically shape model assessment and that single-score evaluation obscures fundamental differences in multi-party conversational behavior. The implementation of MPCEval and the associated evaluation code are publicly available at https://github.com/Owen-Yang-18/MPCEval.

Figures & tables

Figure 1. Overall design of the MPCEval framework.

Aspect	Metric and Description
Local Measures
Speaker	DNR (Direct Name Reference): binary indicator of whether the predicted next speaker is explicitly addressed in the recent context.
	IR (Implicit Reference): higher values indicate the predicted speaker participated more recently when no explicit addressing cue is present.
	PF (Participation Frequency): fraction of recent turns contributed by the speaker; a higher score means more frequent short-term engagement.
	LS-ES (Embedding Similarity): similarity between recent context and the predicted speaker’s prior utterances. $\uparrow$ score means $\uparrow$ similarity.
	LS-TA (Topic Alignment): similarity between topic distributions of recent context and the speaker’s prior utterances. $\uparrow$ score means $\uparrow$ similarity.

Table 1. Summary of MPCEval metrics, which are quantitative, reference-free, and reproducible.

\uparrow

indicates a higher value.

	Local Speaker Modeling					Local Content Quality					Local Consistency
Model	DNR	IR	PF	LS-ES-avg	LS-TA	LNR-E-w	M-SNS-avg	DAF	LL	TES	LSCC-ES-avg
LLaMa-3.3	.278 $\pm$ .448	.182 $\pm$ .205	.276 $\pm$ .096	.444 $\pm$ .088	.778 $\pm$ .193	.280 $\pm$ .159	.735 $\pm$ .091	.281 $\pm$ .243	.893 $\pm$ .091	.357 $\pm$ .399	.358 $\pm$ .198
GPT-4-Turbo	.278 $\pm$ .448	.185 $\pm$ .203	.303 $\pm$ .100	.456 $\pm$ .097	.783 $\pm$ .193	.403 $\pm$ .177	.736 $\pm$ .077	.363 $\pm$ .237	.793 $\pm$ .178	.142 $\pm$ .168	.421 $\pm$ .206
DeepSeek	.389 $\pm$ .488	.213 $\pm$ .223	.278 $\pm$ .099	.450 $\pm$ .100	.774 $\pm$ .196	.314 $\pm$ .133	.688 $\pm$ .061	.307 $\pm$ .240	.918 $\pm$ .074	.112 $\pm$ .185	.381 $\pm$ .161
Claude-3.5	.333 $\pm$ .471	.125 $\pm$ .143	.298 $\pm$ .101	.454 $\pm$ .096	.836 $\pm$ .202	.343 $\pm$ .176	.698 $\pm$ .081	.279 $\pm$ .236	.838 $\pm$ .203	.500 $\pm$ .497	.331 $\pm$ .178
MultiLIGHT	.333 $\pm$ .471	.227 $\pm$ .216	.309 $\pm$ .102	.448 $\pm$ .088	.781 $\pm$ .174	.156 $\pm$ .233	.749 $\pm$ .072	.329 $\pm$ .267	.757 $\pm$ .222	.303 $\pm$ .407	.297 $\pm$ .211

Table 2. Performance comparison on the DeliData dataset using MPCEval ’s next-message prediction (local) measures (mean

\pm

standard deviation). Bold and underlined values indicate the highest and lowest scores in each column, respectively. For LNR-E-w , M-SNS-avg, and TES , values closest to 0.5 are bolded , while those farthest from 0.5 are underlined . ChatGPT-solver does not report LSCC-ES-avg, as local speaker–content consistency is not applicable (see Appendix D.1 ).

Conversation History & Next Message

Metric Comparison

Tiger: So what do you guys think Ox: I would turn A and 6, but it can’t be that easy Falcon: I was thinking the same too, but we need to discuss Falcon: I think turning A would be a no-brainer Falcon: @Tiger, what is your opinion Tiger: I was thinking, maybe we should turn all odd and even numbers Generated Next Message: Ox: Actually, we only need to turn the A and 5. A to check if there’s an even number on the other side, and 5 to ensure there’s no vowel on its other side. Human-Authored Next Message: Ox: hm.. why the odd numbers

Existing Metric Gen. Human MPCEval Gen. Human BLEU-4 0.006 — LNR-E-w 0.591 0.469 BERTScore (F1) 0.840 — M-SNS-avg 0.774 0.673 BARTScore -3.073 -5.983 M-SNS-min 0.490 0.327 G-EVAL (Engage.) 2.700 1.000 DAF 0.700 0.000 Uni-Eval (Engage.) 1.995 0.985 LL 0.956 0.906 Uni-Eval (Understand.) 0.290 0.530 TES 0.516 0.000

Figure 2. Case study comparing MPCEval with existing evaluation metrics. “Gen.” denotes model-generated messages. BLEU-4 and BERTScore are excluded for human-authored next message as comparing the reference to itself is uninformative.

	Global Speaker		Global Content		Consistency
Model	NSE	SC-Gini	PD	HMP	GSCC
MPC + LLaMa-3.1	.978 $\pm$ .022	.445 $\pm$ .197	1.127 $\pm$ .163	.869 $\pm$ .107	.894 $\pm$ .040
MPC + Qwen	.983 $\pm$ .024	.392 $\pm$ .176	1.648 $\pm$ .215	1.021 $\pm$ .114	.951 $\pm$ .061
MPC + LLaMa-3.3	.960 $\pm$ .000	.430 $\pm$ .163	1.144 $\pm$ .148	.835 $\pm$ .074	.915 $\pm$ .026
MPC + GPT-4-Turbo	.984 $\pm$ .020	.420 $\pm$ .185	1.198 $\pm$ .232	.875 $\pm$ .095	.885 $\pm$ .039
MPC + DeepSeek	.979 $\pm$ .024	.416 $\pm$ .197	1.270 $\pm$ .145	.807 $\pm$ .095	.918 $\pm$ .041

Table 3. Performance on the DeliData dataset via MPCEval ’s global metrics (mean

\pm

std). GSCC denotes GSCC-DC-avg under single-centroid speaker–content consistency. Bold and underlined mark the highest and lowest values per column. MPC is shorthand for the MPC-constraints method ( Penzo et al., 2025 ) .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
Conversation primitives
$C$	Conversation history.
$\tilde{C}$	Generated conversation.
$\hat{C}=C\oplus\tilde{C}$	Full conversation (or $\hat{C}=\tilde{C}$ if generated from scratch).
$\oplus$	Concatenation of turn sequences.
$(s_{i},u_{i})$	The $i$ -th turn with speaker $s_{i}$ and utterance $u_{i}$ .

Appendix

Table 4. Core notations used in Appendix B and C .

	Local Speaker Modeling
Model	DNR	IR	PF	LS-ES-avg	LS-ES-max	LS-TA
LLaMa-3.3	0.278 $\pm$ 0.448	0.182 $\pm$ 0.205	0.276 $\pm$ 0.096	0.444 $\pm$ 0.088	0.845 $\pm$ 0.154	0.778 $\pm$ 0.193
GPT-4-Turbo	0.278 $\pm$ 0.448	0.185 $\pm$ 0.203	0.303 $\pm$ 0.100	0.456 $\pm$ 0.097	0.838 $\pm$ 0.149	0.783 $\pm$ 0.193
Deepseek	0.389 $\pm$ 0.488	0.213 $\pm$ 0.223	0.278 $\pm$ 0.099	0.450 $\pm$ 0.100	0.831 $\pm$ 0.158	0.774 $\pm$ 0.196
Claude-3.5	0.333 $\pm$ 0.471	0.125 $\pm$ 0.143	0.298 $\pm$ 0.101	0.454 $\pm$ 0.096	0.837 $\pm$ 0.149	0.836 $\pm$ 0.202
MultiLIGHT	0.333 $\pm$ 0.471	0.227 $\pm$ 0.216	0.309 $\pm$ 0.102	0.448 $\pm$ 0.088	0.819 $\pm$ 0.145	0.781 $\pm$ 0.174

Appendix

Table 5. Performance comparison on the Delidata dataset using MPCEval ’s local speaker modeling measures (mean

\pm

standard deviation). Bold and underlined values denote the highest and lowest scores in each column, respectively.

	Local Content Quality
Model	LNR-E-w	M-SNS-min	M-SNS-avg	DAF	LL	TES
LLaMa-3.3	$0.280\pm 0.159$	$0.336\pm 0.095$	$0.735\pm 0.091$	$0.281\pm 0.243$	$0.893\pm 0.091$	$0.357\pm 0.399$
GPT-4-Turbo	$\textbf{0.403}\pm 0.177$	$\textbf{0.376}\pm 0.131$	$0.736\pm 0.077$	$\textbf{0.363}\pm 0.237$	$0.793\pm 0.178$	$0.142\pm 0.168$
Deepseek	$0.314\pm 0.133$	$0.283\pm 0.095$	$\textbf{0.688}\pm 0.061$	$0.307\pm 0.240$	$\textbf{0.918}\pm 0.074$	$0.112\pm 0.185$
Claude-3.5	$0.343\pm 0.176$	$0.332\pm 0.125$	$0.698\pm 0.081$	$0.279\pm 0.236$	$0.838\pm 0.203$	$\textbf{0.500}\pm 0.497$
MultiLIGHT	0.156 $\pm$ 0.233	0.233 $\pm$ 0.154	0.749 $\pm$ 0.072	$0.329\pm 0.267$	$0.757\pm 0.222$	$0.303\pm 0.407$

Appendix

Table 6. Performance comparison on the DeliData dataset using MPCEval ’s local content quality measures (mean

\pm

standard deviation). Bold and underlined values denote the highest and lowest scores in each column, respectively. For LNR-E-w , M-SNS-min, M-SNS-avg, and TES , values closest to 0.5 are bolded , while those farthest from 0.5 are underlined .

Table 9

	Local Speaker Modeling
Model	DNR	IR	PF	LS-ES-avg	LS-ES-max	LS-TA
LLaMa-3.3	0.102 $\pm$ 0.302	0.440 $\pm$ 0.222	0.400 $\pm$ 0.134	0.688 $\pm$ 0.117	0.789 $\pm$ 0.142	0.660 $\pm$ 0.119
GPT-4-Turbo	0.100 $\pm$ 0.301	0.468 $\pm$ 0.210	0.408 $\pm$ 0.127	0.692 $\pm$ 0.109	0.793 $\pm$ 0.134	0.666 $\pm$ 0.112
Deepseek	0.105 $\pm$ 0.306	0.463 $\pm$ 0.211	0.400 $\pm$ 0.131	0.689 $\pm$ 0.122	0.792 $\pm$ 0.136	0.661 $\pm$ 0.115
Claude-3.5	$0.102\pm 0.303$	$0.474\pm 0.212$	$0.401\pm 0.133$	$0.689\pm 0.114$	$0.791\pm 0.138$	$0.660\pm 0.117$
MultiLIGHT	-	-	-	-	-	-

Appendix

Table 9. Performance comparison on the MPDD dataset using MPCEval ’s local speaker modeling measures (mean

\pm

standard deviation). Bold and underlined values denote the highest and lowest scores in each column, respectively.

	Local Content Quality
Model	LNR-E-w	M-SNS-min	M-SNS-avg	DAF	LL	TES
LLaMa-3.3	0.545 $\pm$ 0.200	$0.341\pm 0.123$	$0.577\pm 0.094$	$0.405\pm 0.290$	$0.641\pm 0.234$	$0.450\pm 0.404$
GPT-4-Turbo	$0.588\pm 0.179$	$\mathbf{0.376\pm 0.127}$	$0.594\pm 0.100$	$0.427\pm 0.285$	$0.678\pm 0.203$	$0.447\pm 0.407$
Deepseek	0.572 $\pm$ 0.177	0.341 $\pm$ 0.118	0.562 $\pm$ 0.096	0.428 $\pm$ 0.283	0.825 $\pm$ 0.151	0.453 $\pm$ 0.406
Claude-3.5	$0.589\pm 0.164$	$0.359\pm 0.114$	$0.573\pm 0.095$	$0.426\pm 0.286$	$0.788\pm 0.165$	$0.446\pm 0.407$
MultiLIGHT	-	-	-	-	-	-

Appendix

Table 10. Performance comparison on the MPDD dataset using MPCEval ’s local content quality measures (mean

\pm

standard deviation). Bold and underlined values denote the highest and lowest scores in each column, respectively; for LNR-E-w , M-SNS-min, M-SNS-avg, and TES , values closest to 0.5 are bolded , while those farthest from 0.5 are underlined .

	Global Speaker Modeling		Global Content Quality		Global Speaker–Content Consistency
					Single Centroid ( $k=1$ )		Multiple Centroids ( $k>1$ )
Model	NSE	SC-Gini	PD	HMP	GSCC-DC-avg	GSCC-DC-max	GSCC-DC-avg	GSCC-DC-max
MPC-constrains + LLaMa-3.1	$0.978\pm 0.022$	$\mathbf{0.445\pm 0.197}$	$1.127\pm 0.163$	$0.869\pm 0.107$	$0.894\pm 0.040$	$0.898\pm 0.040$	$0.894\pm 0.040$	$0.898\pm 0.040$
MPC-constrains + Qwen	$0.983\pm 0.024$	$0.392\pm 0.176$	$\mathbf{1.648\pm 0.215}$	$\mathbf{1.021\pm 0.114}$	$\mathbf{0.951\pm 0.061}$	$\mathbf{0.951\pm 0.061}$	$\mathbf{0.951\pm 0.061}$	$\mathbf{0.951\pm 0.061}$
MPC-constrains + LLaMa-3.3	$0.960\pm 0.000$	$0.430\pm 0.163$	$1.144\pm 0.148$	$0.835\pm 0.074$	$0.915\pm 0.026$	$0.915\pm 0.026$	$0.915\pm 0.026$	$0.915\pm 0.026$
MPC-constrains + GPT-4-Turbo	$0.984\pm 0.020$	$0.420\pm 0.185$	$1.198\pm 0.232$	$0.875\pm 0.095$	$0.885\pm 0.039$	$0.888\pm 0.035$	$0.885\pm 0.039$	$0.888\pm 0.035$

Appendix

Table 11. Performance comparison on the Delidata dataset using MPCEval ’s global measures, including global speaker modeling, global content quality, and global speaker–content consistency (mean

\pm

standard deviation). Bold and underlined values denote the highest and lowest scores in each column, respectively.

	Global Speaker Modeling		Global Content Quality		Global Speaker–Content Consistency
					Single Centroid ( $k=1$ )		Multiple Centroids ( $k>1$ )
Model	NSE	SC-Gini	PD	HMP	GSCC-DC-avg	GSCC-DC-max	GSCC-DC-avg	GSCC-DC-max
LLaMa-3.3	$0.796\pm 0.082$	$0.573\pm 0.118$	$0.559\pm 0.122$	$0.620\pm 0.319$	$\mathbf{0.841\pm 0.048}$	$0.899\pm 0.039$	$0.869\pm 0.057$	$0.933\pm 0.049$
GPT-4-Turbo	$0.801\pm 0.103$	$0.557\pm 0.140$	$0.516\pm 0.094$	$\underline{0.592\pm 0.252}$	$0.832\pm 0.025$	$0.893\pm 0.013$	$0.834\pm 0.025$	$0.895\pm 0.013$
Deepseek	$0.770\pm 0.067$	$0.548\pm 0.107$	$\underline{0.466\pm 0.088}$	$0.738\pm 0.249$	$0.837\pm 0.027$	$\mathbf{0.903\pm 0.022}$	$\mathbf{0.909\pm 0.045}$	$\mathbf{0.974\pm 0.025}$
Claude-3.5	$\mathbf{0.822\pm 0.078}$	$\underline{0.543\pm 0.173}$	$\mathbf{0.572\pm 0.084}$	$0.725\pm 0.289$	$0.797\pm 0.019$	$0.873\pm 0.021$	$0.809\pm 0.017$	$0.892\pm 0.019$

Appendix

Table 12. Performance comparison on the Tanka dataset using MPCEval ’s global measures, including global speaker modeling, global content quality, and global speaker–content consistency (mean

\pm

standard deviation). Bold and underlined values denote the highest and lowest scores in each column, respectively.

Conversation History (Last 10 Turns)

Next Message & Metrics

Leopard: but it would make sense to falsify the rule by flipping 3 Bat: but the rule is say with the vowel card there is even number on the other side Leopard: as if 3 had a vowel it would be incorrect Lion: Yes but I am not fussed if it’s incorrect if I can’t see it, I’m not looking to disprove the rule I’m looking to prove Lion: Thats my thinking Leopard: right…. Lion: FINAL ANSWER 2 Leopard: what about i Lion: That’s a vowel that can stay Leopard: are they gonna give us the correct answer Lion: I’m looking for vowels, I hope not Leopard: can you give us the correct answer when we are done Leopard: thanks!

LLM-Generated Next Message: Leopard: Sure, let’s finish up and then we’ll see if we got it right! Human-Authored Next Message: Leopard: @all where you gone Metric Comparison: LLM Human LNR-E-w 0.475 0.753 M-SNS-avg 0.714 0.825 LL 0.924 0.038

Appendix

Figure 3. Case study. Human-authored next message is not the only valid response.

Multi-Turn Dialogue Evaluation

Multi-turn dialogue evaluation measures whether AI systems remain accurate, coherent, safe, and useful as conversations evolve.

See all 201 papers

MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant
Sep 11, 2026Yi-Jen Shih, Shih-Yun Shan Kuan, Guan-Ting Lin +7Spoken Dialogue Systems AI Agent Benchmarks
The University of Texas at Austin · National Taiwan University · Massachusetts Institute of Technology +2
Duplex-MPE: Benchmarking Multi-Party Interaction in Full-Duplex Dialogue
Sep 25, 2026Chengqian Ma, Wenhao Feng, Weixuan Jin +7Voice Agent Evaluation Full-Duplex Spoken Dialogue Systems
Peking University · Renmin University of China · Tsinghua University +3
DRAMA-Eval: A Benchmark for Context-Aware, Multi-Turn, Multi-Speaker Speech Generation
Oct 22, 2026Jingbin Hu, Zihang Yang, Xiaoge Peng +21Speech Generation Evaluation Multi-Speaker TTS
Shenzhen Pimei Technology Co., Ltd. · Yutu Zhineng

Sep 11, 2026J/K move · Enter open · S save

MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant

Yi-Jen Shih, Shih-Yun Shan Kuan, Guan-Ting Lin, Kai-Wei Chang+6

The University of Texas at Austin · National Taiwan University · Massachusetts Institute of Technology+2

Spoken Dialogue Systems AI Agent Benchmarks

Open paper AskPDF arXiv

Conversational voice agents have advanced significantly, offering increasingly natural human-machine interactions through both cascaded and end-to-end architectures. However, while recent benchmarks extensively evaluate dyadic interactions and passive audio comprehension, they largely overlook a prevalent real-world scenario: multi-party conversations. Evaluating agents in these settings is fundamentally more challenging than in dyadic interactions due to the exponentially greater conversational complexity. For voice agents to integrate seamlessly into human group dynamics, they must not only generate contextually appropriate responses but also demonstrate a nuanced understanding of open turn-taking. To address this gap, we introduce Multiparty Bench (MP-Bench), the first benchmark specifically designed to objectively evaluate conversational speech systems as active participants within multi-party contexts. MP-Bench assesses agent behavior along two primary dimensions: turn-taking awareness and response appropriateness. Additionally, we incorporate comprehension-based question-answering tasks as a complementary evaluation. By benchmarking 12 voice agents, we find that real-time voice agents stay at or below 22% on multiparty comprehension and remain near chance on multiparty turn-taking, exposing an open challenge for real-time voice agents under multiparty scenario.

Abstract

Multi-Turn Dialogue Evaluation

Explore similar work

MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant

Duplex-MPE: Benchmarking Multi-Party Interaction in Full-Duplex Dialogue

DRAMA-Eval: A Benchmark for Context-Aware, Multi-Turn, Multi-Speaker Speech Generation