Discourse Coherence and Response-Guided Context Rewriting for Multi-Party Dialogue Generation

Zhiyu Cao; Peifeng Li; Qiaoming Zhu

cs.CLApr 8, 2026

Discourse Coherence and Response-Guided Context Rewriting for Multi-Party Dialogue Generation

Authors: Zhiyu Cao, Peifeng Li, Qiaoming Zhu

Organizations: School of Computer Science and Technology, Soochow University, China

Abstract

Previous research on multi-party dialogue generation has predominantly leveraged structural information inherent in dialogues to directly inform the generation process. However, the prevalence of colloquial expressions and incomplete utterances in dialogues often impedes comprehension and weakens the fidelity of dialogue structure representations, which is particularly pronounced in multi-party dialogues. In this work, we propose a novel framework DRCR (Discourse coherence and Response-guided Context Rewriting) to improve multi-party dialogue generation through dialogue context rewriting. Specifically, DRCR employs two complementary feedback signals, discourse coherence and response quality, to construct preference data for both context rewriting and response generation. Moreover, we propose a dynamic self-evolution learning method that allows the rewriter and responder to continuously enhance their capabilities through mutual interaction in an iterative training loop. Comprehensive experiments conducted on four multi-party dialogue datasets substantiate the effectiveness of DRCR.

Figures & tables

Figure 1: An example of multi-party dialogue generation. The start and end of the arrow represent the addressee and sender of an utterance, respectively.

Figure 2: An overview of DRCR, which includes three stages: 1) Train Addressee Recognition Classifier: Identifying the addressee corresponding to each utterance in preparation for scoring the discourse coherence of the dialogue context, 2) Preference Data Construction: Constructing preference data based on the discourse coherence of the dialogue and the quality of responses, and 3) Mutual Self-Evolution: Iteratively optimizing both rewriter and responder without relying on external data.

Method	Ubuntu IRC-16						Ubuntu IRC-19
Method	$\text{B}_{\text{1}}$	$\text{B}_{\text{2}}$	$\text{B}_{\text{3}}$	$\text{B}_{\text{4}}$	MR	$\text{R}_{\text{L}}$	$\text{B}_{\text{1}}$	$\text{B}_{\text{2}}$	$\text{B}_{\text{3}}$	$\text{B}_{\text{4}}$	MR	$\text{R}_{\text{L}}$
SLM-based MDG
GSN Hu et al. (2019)	6.32	2.28	1.10	0.61	3.27	7.39	10.23	3.57	1.70	0.97	4.10	9.91
HeterMPC Gu et al. (2022)	11.40	4.29	2.43	1.74	4.57	10.44	12.26	4.80	2.42	1.49	4.94	11.20
EMMDG Li and Zhao (2023)	11.67	4.73	2.64	1.81	5.12	10.43	12.31	5.39	3.34	2.45	5.52	11.71
MADNet Gu et al. (2023)	11.82	4.58	2.65	1.91	4.90	10.74	12.73	5.12	2.64	1.63	5.31	11.74

Table 1: Experimental results on IRC-16 and IRC-19 (Section 4.6 for the results on HLA-Chat++ and Friends).

Variant	$\text{B}_{\text{1}}$	$\text{B}_{\text{2}}$	$\text{B}_{\text{3}}$	$\text{B}_{\text{4}}$	MR	$\text{R}_{\text{L}}$
DRCR	17.81	8.02	5.79	4.26	8.49	14.82
w/o. DC	17.09	7.22	5.13	3.54	7.75	13.94
w/o. RQ	16.98	7.16	5.21	3.57	7.64	14.03
w/o. CoV	17.18	7.36	5.27	3.62	7.81	14.11
w/o. Warm-up	15.17	6.72	4.26	3.03	6.84	13.32
w/o. MSE	15.87	7.24	4.62	3.37	7.12	13.77

Table 2: Ablation experiments ( Qwen3-8B as backbone) and different LLMs as teacher rewriter

\pi_{t}

on IRC-19.

Datasets	Origin	w/ Warm-up	DRCR
Ubuntu IRC-16	75.32	81.59	84.16
Ubuntu IRC-19	84.16	87.30	91.37

Table 3: Using Qwen3-8B as the backbone to analyze the impact of rewriting on the discourse coherence.

Method	Ubuntu IRC-16	Ubuntu IRC-19
MADNet	72.32	81.79
RL-TRC	76.49	85.72
SS-MPC	76.93	86.51
Qwen3-8B+SFT	74.07	84.26
Qwen3-8B+DRCR	78.96	89.34
Ground Truth	85.67	96.42

Table 4: Accuracy of identifying the addressee based on the generated response.

Method	$\text{B}_{\text{1}}$	$\text{B}_{\text{2}}$	$\text{B}_{\text{3}}$	$\text{B}_{\text{4}}$	MR	$\text{R}_{\text{L}}$
DRCR-LLM	17.75	8.06	5.82	4.21	8.44	14.57
DRCR-AR	17.81	8.02	5.79	4.26	8.49	14.82

Table 5: Comparison of using LLM for coherence scoring and addressee recognition classifier for coherence measurement on Ubuntu IRC-19.

Method	$\text{B}_{\text{1}}$	$\text{B}_{\text{2}}$	D 1	D 2
HeterMPC	11.65	10.82	0.76	4.71
MADNet	12.39	11.84	0.94	5.49
SS-MPC	13.07	13.82	1.42	6.65
PersonaTKG+sent	13.18	14.17	1.59	6.90
Qwen3-8B+SFT	13.42	14.48	1.72	7.04
Qwen3-8B+DRCR (Ours)	15.83	17.06	3.53	9.23

Table 6: Performance evaluation of DRCR and baselines on the HLA-Chat++ dataset.

Method	$\text{B}_{\text{1}}$	$\text{B}_{\text{2}}$	$\text{B}_{\text{3}}$	$\text{R}_{\text{L}}$
HeterMPC	5.30	1.41	0.40	4.74
MADNet	6.00	1.54	0.63	5.95
SS-MPC	7.81	2.42	1.47	7.24
Qwen3-8B+SFT	7.13	2.06	1.39	6.82
Qwen3-8B+DRCR (Ours)	9.27	5.14	2.83	9.56

Table 7: Performance evaluation of DRCR and baselines on the Friends dataset.

Method	Coh.	Flu.	Info.	Help.	Overall
Ground Truth	0.82	0.89	0.93	0.87	3.51
MADNet	0.58	0.87	0.76	0.63	2.84
RL-TRC	0.69	0.90	0.81	0.72	3.12
SS-MPC	0.73	0.91	0.82	0.79	3.25
Qwen3-8B+SFT	0.71	0.90	0.82	0.76	3.19
Qwen3-8B+DRCR	0.79	0.94	0.86	0.82	3.41

Table 8: Human evaluation on DRCR and four strong baselines. Coh., Flu., Info., and Help. are abbreviations for Coherence, Fluency, Informativeness, and Helpfulness, respectively.

Figure 3: Case study on context rewriting and response generation. We provide the original context in Figure 1 .

	Ubuntu IRC-19
Variant	$\text{B}_{\text{1}}$	$\text{B}_{\text{2}}$	$\text{B}_{\text{3}}$	$\text{B}_{\text{4}}$	MR	$\text{R}_{\text{L}}$
DRCR	17.81	8.02	5.79	4.26	8.49	14.82
DRCR (MPC)	17.63	7.86	5.68	4.18	8.36	14.73
DRCR (Acc)	17.32	7.47	5.44	3.78	7.82	14.27

Table 9: Analysis of using MPC-BERT as the addressee recognition classifier for discourse coherence scoring and using accuracy as the coherence proxy.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Train	Valid	Test
Ubuntu IRC-16	461,120	28,570	32,668
Ubuntu IRC-19	311,725	5,000	5,000
HLA-Chat++	853,057	10,281	9,985
Friends	4,500	500	653

Appendix

Table 10: Statistics of the MDG datasets.

Method	$\text{B}_{\text{1}}$	$\text{B}_{\text{2}}$	$\text{B}_{\text{3}}$	$\text{B}_{\text{4}}$	MR	$\text{R}_{\text{L}}$
DRCR (GPT-3.5)	17.81	8.02	5.79	4.26	8.49	14.82
DRCR (GPT-4.1)	17.97	8.16	5.87	4.37	8.62	14.93
DRCR (Qwen3-4B)	16.53	6.74	4.92	3.26	7.31	13.74
DRCR (Qwen3-8B)	17.02	7.12	5.14	3.47	7.52	14.03
DRCR (Qwen3-32B)	18.13	8.32	5.98	4.52	8.79	15.12

Appendix

Table 11: Analysis of training data constructed using different LLMs on Ubuntu IRC-19.

Method	$\text{B}_{\text{1}}$	$\text{B}_{\text{2}}$	$\text{B}_{\text{3}}$	$\text{B}_{\text{4}}$	MR	$\text{R}_{\text{L}}$
GPT-4.1 (zero-shot)	15.37	7.16	4.35	2.91	7.13	13.49
GPT-4.1 (1-shot)	15.54	7.42	4.55	3.06	7.35	13.72
GPT-4.1 (4-shot)	15.46	7.28	4.31	2.97	7.22	13.56
Qwen3-32B (zero-shot)	15.68	7.37	4.56	3.24	7.46	13.61
Qwen3-32B (1-shot)	15.77	7.51	4.69	3.43	7.65	13.74
Qwen3-32B (4-shot)	15.73	7.46	4.63	3.39	7.54	13.68

Appendix

Table 12: Analysis of using powerful LLMs for multi-party dialogue generation on Ubuntu-19 without rewriting.

Figure 4: Analysis of scaling self-evolution rounds on Ubuntu IRC-19.

Figure 5: Analysis of scaling sampled candidate contexts on Ubuntu IRC-19.

Figure 6: The impact of rewriting threshold

\phi

on model performance in Ubuntu IRC-16 (top) and Ubuntu IRC-19 (bottom).

Stage	Ubuntu IRC-16	Ubuntu IRC-19
Data Construction	13.2	18.7
Warm-up Training	7.4	11.8
Mutual Self-evolution	12.1	17.2

Appendix

Table 13: Analysis of time consumption TC (in hours) in the data construction and training phase.

Variant	Ubuntu IRC-16	Ubuntu IRC-19
Non-rewriting	0.37	0.42
Adaptive Rewriting	0.58	0.71

Appendix

Table 14: Taking Qwen3-8B as an example to analyze the inference latency PL (in seconds) of non-rewriting and adaptive rewriting.

Method	Coh.	Flu.	Info.	Help.	Overall
Ground Truth	3.67	4.42	3.81	4.32	4.06
MADNet	2.85	3.64	3.19	3.37	3.26
RL-TRC	3.01	3.89	3.40	3.62	3.48
SS-MPC	3.18	4.12	3.53	3.87	3.68
Qwen3-8B+SFT	3.12	4.03	3.51	3.83	3.62
Qwen3-8B+DRCR	3.41	4.36	3.87	4.12	3.94

Appendix

Table 15: The generated responses are evaluated from four dimensions: coherence, fluency, informativeness, and helpfulness using GPT-4. Coh., Flu., Info., and Help. are abbreviations for coherence, fluency, informativeness, and helpfulness, respectively.

Panel of Figure 7 — Figure 7: The prompt used for utterance-level rewriting in Section 3.3 .

Panel of Figure 8 — Figure 8: The prompt used for response generation in Section 3.3 .

Panel of Figure 9 — Figure 9: Prompt used to evaluate the quality of responses using LLM.

Panel of Figure 10 — Figure 10: A case where incorrect rewriting affects response generation.

MPCEval: A Benchmark for Multi-Party Conversation Generation
Mar 5, 2026Minxing Zhang, Yi Yang, Zhuofan Jia +5Multi-Turn Dialogue Evaluation Conversational Agents
Duke University USA · Tanka AI USA
Improving Multi-turn Dialogue Consistency with Self-Recall Thinking
May 14, 2026Renning Pang, Tian Lan, Leyuan Liu +3Memory-Augmented Language Models Long-Term Conversational Memory
University of Electronic Science and Technology of China
When2Speak: A Dataset for Temporal Participation and Turn-Taking in Multi-Party Conversations for Large Language Models
May 7, 2026Vihaan Nama, Shreya Mendi, Zian Ye +1Synthetic Data Generation Synthetic Data Generation for Language Models
Pratt School of Engineering Duke University

Mar 5, 2026J/K move · Enter open · S save

MPCEval: A Benchmark for Multi-Party Conversation Generation

Minxing Zhang, Yi Yang, Zhuofan Jia, Xuan Yang+4

Duke University USA · Tanka AI USA

Multi-Turn Dialogue Evaluation Conversational Agents Reference-Free Evaluation

Open paper AskPDF arXiv

Multi-party conversation generation, such as smart reply and collaborative assistants, is an increasingly important capability of generative AI, yet its evaluation remains a critical bottleneck. Compared to two-party dialogue, multi-party settings introduce distinct challenges, including complex turn-taking, role-dependent speaker behavior, long-range conversational structure, and multiple equally valid continuations. Accordingly, we introduce MPCEval, a task-aware evaluation and benchmarking suite for multi-party conversation generation. MPCEval decomposes generation quality into speaker modeling, content quality, and speaker--content consistency, and explicitly distinguishes local next-turn prediction from global full-conversation generation. It provides novel, quantitative, reference-free, and reproducible metrics that scale across datasets and models. We apply MPCEval to diverse public and real-world datasets and evaluate modern generation methods alongside human-authored conversations. The results reveal systematic, dimension-specific model characteristics in participation balance, content progression and novelty, and speaker--content consistency, demonstrating that evaluation objectives critically shape model assessment and that single-score evaluation obscures fundamental differences in multi-party conversational behavior. The implementation of MPCEval and the associated evaluation code are publicly available at https://github.com/Owen-Yang-18/MPCEval.

Abstract

Explore similar work

MPCEval: A Benchmark for Multi-Party Conversation Generation

Improving Multi-turn Dialogue Consistency with Self-Recall Thinking

When2Speak: A Dataset for Temporal Participation and Turn-Taking in Multi-Party Conversations for Large Language Models