DraDDP: A Multimodal Multi-Party Dialogue Discourse Parsing Dataset

Shannan Liu; Peifeng Li; Yaxin Fan; Qiaoming Zhu

cs.CLApr 13, 2026

DraDDP: A Multimodal Multi-Party Dialogue Discourse Parsing Dataset

Authors: Shannan Liu, Peifeng Li, Yaxin Fan, Qiaoming Zhu

Organizations: School of Computer Science and Technology, Soochow University, Suzhou, China

Abstract

Multi-party dialogue discourse parsing aims to identify dependency structures and relation types between utterances in conversations. Previous studies are mostly limited to textual modality or two-party dialogue, failing to meet the multimodal and multi-party settings. In this paper, we construct the first publicly available English multimodal dataset DraDDP for multi-party dialogue discourse parsing, based on American TV dramas. DraDDP contains 495 dialogue segments with 6,374 utterances and 9.1 hours of parallel video content, covering rich multi-party interaction scenarios. Moreover, we establish comprehensive benchmarks by evaluating this task on DraDDP and conducting in-depth analysis on the impact of different modalities. Experimental results demonstrate the value of multimodal information in capturing dialogue structures and relation types. We will publicly release the dataset, annotation guidelines, and code to promote future research in multimodal dialogue understanding.

Figures & tables

Figure 1: An example of multimodal dialogue discourse parsing.

Modalities	Dataset	#Dialogues	#Utterances	Data Source	Language	Participants
T	STAC	1.1K	10.7K	Online game	English	Multi-party
T	Molweni	10K	88.3K	Online forums	English	Multi-party
T	DialogueDSA	705	24.5K	TV drama	Chinese	Multi-party
T	MSDC	541	22.6K	Online game	English	Two-party
T+I	JDDC 2.1	246K	3.46M	E-commerce platform	Chinese	Two-party
T+V+A	MODDP	864	18K	TV drama	Chinese	Two-party

Table 1: Comparison with existing dialogue discourse parsing datasets (T/V/A/I: Text/Video/Audio/Image).

Figure 2: Error statistics from pre-annotation where { X

\to

Y } indicates misclassifying relation X as Y and the relations are defined in Appendix A .

Figure 3: Analysis of pre-annotation results, including total number and correct number of each relation type.

Figure 4: Data analysis of the DraDDP dataset.

Category	Model	DraDDP		MODDP
Category	Model	Link	Link&Rel	Link	Link&Rel
Unimodal	RLTST	76.94	40.10	90.76	41.41
	BERTLine	78.65	45.82	90.73	40.96
	MODDP ${}_{\text{T}}$	83.31	45.06	92.46	43.66
	LLaMIPa	84.71	53.39	91.32	53.05
	LLaMIPa †	85.03	54.58	91.55	53.68

Table 2: Experimental results (F1 scores) on DraDDP and MODDP test sets. † : Without concatenating dialogue structure history previously predicted by the model. ‡ : Built upon the LLaMIPa † framework with the backbone model replaced by Qwen series (7B). Qwen2.5-VL: Text+Video; Qwen2-Audio: Text+Audio; Qwen2.5-Omni: Text+Video+Audio.

Category	Train	Dev	Test	Total
Dialogues	345	75	75	495
Utterances	4,447	962	965	6,374
Avg. Utt/Dialog	12.89	12.83	12.87	12.88
Avg. Utt Length	9.54	10.19	9.59	9.64

Table 3: Data statistics of the DraDDP dataset.

Model	LLaMIPa ‡ (Qwen2.5)		LLaMIPa ‡ (Qwen2.5-VL)		LLaMIPa ‡ (Qwen2-Audio)		LLaMIPa ‡ (Qwen2.5-Omni)
Model	Link	Link&Rel	Link	Link&Rel	Link	Link&Rel	Link	Link&Rel
$s\leq 2$	88.87	53.20	88.28	55.28	88.8	54.09	89.17	55.05
$2<s\leq 6$	80.80	53.30	81.60	51.30	83.20	54.90	80.60	48.90
$s>6$	79.85	55.42	81.77	51.58	87.54	61.19	87.54	59.27

Table 4: Performance on DraDDP with different models and number of speakers.

Figure 5: Error pattern statistics of the DraDDP test set under different modalities, where { X

\to

Y } indicates misclassifying relation X as Y.

Figure 6: A case study on unimodal (T) and multimodal (T+V+A) dialogue parsing using the DraDDP dataset.

Discourse Relation	Definition	DraDDP (%)	STAC (%)	MODDP (%)
Question-Answer Pair (QAP)	Arg2 is the answer to the question raised by Arg1.	17.8	24.2	9.4
Clarification_Question (Clafi)	Arg2 provides clarification for Arg1.	16.8	2.5	6.4
Comment (Comt)	Arg2 expresses a viewpoint or evaluation of Arg1.	14	17.6	16.4
Acknowledgement (Ack)	Arg2 shows approval or acknowledgment for Arg1.	9.6	9.6	2.7
Contrast (Contrast)	Arg1 and Arg2 differ on a shared theme.	7.6	4.7	6.6
Elaboration (Elab)	Arg2 elaborates on Arg1 in detail.	6.8	8.3	15.7

Table 5: Definition and proportion of discourse relations in DraDDP, MODDP, and STAC datasets.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Speaker Count	D=1	D=2	D=3	D=4	D $\geq$ 5
$s\leq 2$	88.6%	9.0%	1.2%	0.8%	0.3%
$2<s\leq 6$	80.2%	13.5%	3.0%	1.5%	1.8%
$s>6$	78.1%	12.6%	3.3%	2.1%	3.9%

Appendix

Table 6: Distribution of dependency relation distances across different speaker counts.

Modalities	Link	Link&Rel
T	84.67	53.69
V	43.38	22.21
A	47.39	38.83
T+V	83.61	52.97
T+A	84.83	54.76
V+A	50.12	40.39

Appendix

Table 7: Ablation results (F1-score) of different modalities.

Figure 7: Controversial annotation cases in the DraDDP dataset.

Category	Model	Link	Link&Rel
Unimodal	Claude-Sonnet-4	71.46	23.18
	GPT-4o	78.11	35.87
	Qwen3-235B	71.48	42.18
Multimodal	Gemini 2.5 Pro	68.17	21.69
Multimodal	Qwen3-VL-235B	71.64	37.71

Appendix

Table 8: Baseline performance of LLM on the DraDDP dataset (F1 score).

Type	Distinction	Example
Clafi	Clarification_Question is to ask about the “current situation, details, or ambiguities of an existing statement” to confirm facts or eliminate information bias.	$u_{0}$ : There’s nothing to tell. It’s just some guy I work with. $u_{1}$ : C’mon, you’re going out with the guy. There’s gotta be something wrong with him!
Q-Elab	Q-Elab is to further inquire or expand on an “already raised question” to obtain more comprehensive information, focusing on the “question itself”.	$u_{0}$ : Who’s Paul? $u_{1}$ : Paul, the wine guy? Paul?
Continu	Continuation is generally a “parallel extension” to the current topic, with no strict order, which is just horizontal expansion of the topic.	$u_{0}$ : I say push her down the stairs. $u_{1}$ : Push her down the stairs! Push her down the stairs! Push her down the stairs!
Narr	Narration is a “vertical expansion” of the current topic, with a clear implicit sequence, process, or logical connection, presenting complete events or information.	$u_{0}$ : Yeah, right! See, he gave up something, but then he got those magic beans. $u_{1}$ : And then he woke up, and there was a big plant outside his window, full of possibilities and stuff…
Expl	Explanation provides a clear reason, basis, or reasoning process for a “certain viewpoint, phenomenon, or result”, focusing on “answering why”.	$u_{0}$ : Oh my god, oh, you guys are great. $u_{1}$ : We all chipped in.
Backg	Background has a weaker causal relationship, providing relevant background information (such as time, scene, premise, etc.) for the “current topic”.	$u_{0}$ : I’m smoking. I’m smoking, I’m smoking. $u_{1}$ : Oh, I can’t believe you! You’ve been so good, for three years!

Appendix

Table 9: Distinction between easily confusable discourse relations.

Multimodal Dataset

See all 47 papers

TEIDAN: A Multilingual Multiparty Dialogue Corpus
Sep 1, 2026Taiga Mori, Koji Inoue, Mikey Elmers +2Spoken Dialogue Systems Turn-Taking in Dialogue
Graduate School of Informatics, Kyoto University · Japan
DuplexDrama: A Synthesized Dialogue Dataset with Scenarios, Full-Duplex Behaviors, Expressive Speech, and Sound Events
Sep 11, 2026Qingxiang Guo, Wenke Fan, Shuofeng Zhao +9Spoken Dialogue Systems Synthetic Data Generation for Language Models
Zuoyebang Education Technology, Beijing, China
Fine-grained Fragment Retrieval in Multi-modal Long-form Dialogues
Jun 3, 2026Hanbo Bi, Zhiqiang Yuan, Chongyang Li +7Multimodal IR Conversational Search
Pattern Recognition Center, WeChat AI, Tencent Inc, China · Aerospace Information Research Institute, Chinese Academy of Sciences

Sep 1, 2026J/K move · Enter open · S save

TEIDAN: A Multilingual Multiparty Dialogue Corpus

Taiga Mori, Koji Inoue, Mikey Elmers, Divesh Lala+1

Graduate School of Informatics, Kyoto University · Japan

Spoken Dialogue Systems Turn-Taking in Dialogue

Open paper AskPDF arXiv

Multi-party interaction is a central setting for human communication and a necessary target for human-agent interaction systems that must participate in group conversation. Yet available corpora often focus on meetings, task-oriented interaction, text-based interaction, or acted scenarios, and fewer resources support cross-linguistic comparison of spontaneous face-to-face triadic discussion. This paper presents TEIDAN, a multilingual multimodal corpus that currently consists of Japanese and English three-party conversations. TEIDAN records groups of three participants discussing open-ended topics with individual pin microphones, a microphone array, and participant-facing cameras, and provides IPU-based transcripts for both language portions. Earlier studies used subsets of the Japanese portion for task-specific benchmarks in multi-party dialogue modeling; in contrast, this paper presents TEIDAN as a corpus resource spanning both Japanese and English, with planned expansion to additional languages. We describe the collection design, participants, recording setup, transcription format, and corpus statistics, and provide preliminary analyses to illustrate how TEIDAN can support research on turn-taking, addressee recognition, and multimodal grounding in human-human and human-agent interaction.

Abstract

Multimodal Dataset

Explore similar work

TEIDAN: A Multilingual Multiparty Dialogue Corpus

DuplexDrama: A Synthesized Dialogue Dataset with Scenarios, Full-Duplex Behaviors, Expressive Speech, and Sound Events

Fine-grained Fragment Retrieval in Multi-modal Long-form Dialogues