Japanese Discourse Structure Analysis Based on Segmented Discourse Representation Theory Using LLMs

Satomu Tomisaki; Yosuke Mikami; Amane Watahiki; Hitomi Yanaka

cs.CLOct 27, 2026

Japanese Discourse Structure Analysis Based on Segmented Discourse Representation Theory Using LLMs

Authors: Satomu Tomisaki, Yosuke Mikami, Amane Watahiki, Hitomi Yanaka

Organizations: The University of Tokyo · RIKEN · Tohoku University

Abstract

Discourse parsing based on Segmented Discourse Representation Theory (SDRT) is crucial for capturing the structural dynamics of dialogues. However, most prior research focuses on English, and its effectiveness for Japanese remains unclear owing to the lack of Japanese SDRT corpora. To address this issue, we construct Japanese SDRT corpora by translating existing English SDRT corpora using an LLM. Through a comparative evaluation of prompt-based and fine-tuned LLMs, we show that the models achieve comparable performance between English and Japanese, suggesting the cross-lingual applicability of SDRT-based discourse parsing. In order to investigate why the English and Japanese parsers achieve similar performance, we conduct ablation analyses on discourse markers and contextual inputs. Our results reveal that most of the Japanese discourse markers we examined have little impact, and the models' predictions depend heavily on contextual discourse relations rather than on the content of the context clauses. This structural reliance might explain the comparable performance despite the substantial typological differences between English and Japanese.

Figures & tables

Figure 1: An extract from MSDC, represented as a discourse graph. EDUs and EEUs are shown in roman and italic fonts, respectively.

	Annotator1	Annotator2	Annotator3
Mean	97.2	88.3	88.7
Standard error	0.3	0.2	0.3

Table 1: Mean translation evaluation scores and standard errors for each annotator.

Model	Llama-3	Swallow-3	Llama-3.1			Swallow-3.1			GPT-5.5
Method	Llamipa		0shot	1shot	4shot	0shot	1shot	4shot	0shot	1shot	4shot	Supports
Ackn.	0.86	0.85	0.56	0.43	0.43	0.51	0.36	0.15	0.56	0.76	0.78	792
Alte.	0.81	0.90	0.00	0.00	0.42	0.00	0.00	0.13	0.76	0.65	0.88	24
Clari-Q.	0.67	0.68	0.08	0.24	0.27	0.01	0.19	0.08	0.40	0.58	0.60	134
Comm.	0.58	0.58	0.16	0.23	0.12	0.19	0.14	0.03	0.34	0.47	0.49	382
Cond.	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.22	0.22	0.00	13

Table 2: F1-scores for Llamipa and prompt-based parsers on the Japanese MSDC. "Supports" represents the number of instances per relation in the test data, and discourse relations are abbreviated. Bold values indicate the highest F1-scores. The models were evaluated on a subset of gold relations from the MSDC corpus (cf. Appendix E ).

Dataset	MSDC				STAC
Language	EN	JA	JA	Supports	EN	JA	JA	Supports
Base model	Llama-3	Llama-3	Swallow-3		Llama-3	Llama-3	Swallow-3
Ackn.	0.86	0.86	0.85	792	0.61	0.62	0.65	173
Alte.	0.90	0.81	0.90	24	0.67	0.59	0.62	10
Back.	n/a	n/a	n/a	n/a	0.00	0.20	0.00	9
Clari-Q.	0.69	0.67	0.68	134	0.50	0.46	0.50	48

Table 3: F1-scores of Llamipa on the MSDC and STAC datasets. "Supports" indicates the number of instances per relation in the test data. Discourse relations are abbreviated. "n/a" indicates that the relation is not included in the respective dataset. Bold values denote the best F1-scores.

Expression	Relation	Occurrence	Position
それから ( sorekara )	Narr.	36/54	$y$
そしたら ( sositara )	Narr.	36/40	$y$
そしたら ( sositara )	Resu.	35/40	$y$
じゃあ ( jaa )	Narr.	284/379	$y$
じゃあ ( jaa )	Resu.	316/379	$y$
だから ( dakara )	Elab.	41/47	$y$

Table 4: The statistics of the ablation study on target expressions. The “Occurrence” column is presented in the format

(a/b)

, where

a

is the number of times the linguistic expression appears at the position specified in the “Position” column, and

b

is the total number of clauses in which the expression occurs. In the “Position” column,

x

and

y

denote the first and second arguments of the discourse relation, respectively.

Figure 2: Depiction of the context clause masking.

Model		Llama-3-base		Swallow-base
Expression	Rel.	F1	Pred.	F1	Pred.
それから ( sorekara )	Narr.	0.89/0.90	46/31	0.89/0.90	43/33
そしたら ( sositara )	Narr.	0.89/0.89	35/33	0.89/0.89	36/34
そしたら ( sositara )	Resu.	0.91/0.91	35/34	0.91/0.91	36/35
じゃあ ( jaa )	Narr.	0.89/0.89	282/272	0.89/0.88	290/274
じゃあ ( jaa )	Resu.	0.91/0.91	308/300	0.91/0.91	313/298

Table 5: F1-scores and the number of predictions (Pred.) before and after ablation. In the “F1” column, the notation

b/a

indicates the scores before (

b

) and after (

a

) ablation, respectively. Similarly, in the “Pred.” column,

b/a

denotes that out of

b

original predictions,

a

remained unchanged after the target expressions were ablated.

Model	Llama-3-base		Swallow-base
Language	EN	JA	JA
	Before/After	Before/After	Before/After
Ackn.	0.86/0.71	0.86/0.70	0.86/0.74
Alte.	0.88/0.65	0.79/0.67	0.90/0.83
Clari-Q.	0.70/0.33	0.68/0.21	0.68/0.52
Comm.	0.59/0.40	0.58/0.46	0.59/0.48

Table 6: F1-scores before and after ablation of input discourse relations. The “Weighted” row indicates the overall performance weighted by relation frequency.

Model	Llama-3-base		Swallow-base
Language	EN	JA	JA
	Before/After	Before/After	Before/After
Ackn.	0.86/0.81	0.86/0.80	0.86/0.70
Alte.	0.88/0.88	0.79/0.82	0.90/0.91
Clari-Q.	0.70/0.62	0.68/0.65	0.68/0.66
Comm.	0.59/0.54	0.58/0.52	0.59/0.54

Table 7: F1-scores before and after masking of the content of the context clauses. The “Weighted” indicates the overall performance weighted by relation frequency.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Index	Speaker	Text
2	Builder	“lets do this”
		“やろうか”
3	Architect	“place a red block in front of the blue block”
		“青いブロックの前に赤いブロックを置いてください”
4	Architect	“along the line toward the purple block”
		“紫のブロックに向かう線に沿って”

Appendix

Table 8: Examples of translation of MSDC. The sentence above is the original English, while the sentence below is the Japanese translation.

Speaker	Text
Builder	“any color”
Builder	“or still orange?”
Builder	“どんな色でもいいよ”
Builder	“それともオレンジのままでいい?”
Architect	“on the top orange block”
Architect	“put a yellow block to the left of it”

Appendix

Table 9: Examples of translation errors. In each example, the upper segment provides the original English excerpt, and the lower segment presents its corresponding Japanese translation.

GPUs
NVIDIA Hopper H100
Hyperparameters
training epochs	3
batch size	1
optimizer	Adam
learning rate	2e-4

Appendix

Table 10: Hyperparameters for the training.

	EN (Llama-3-base)			JA (Llama-3-base)			JA (Swallow-base)
Relation	Prec.	Recall	F1	Prec.	Recall	F1	Prec.	Recall	F1	Supports
Ackn.	0.86	0.85	0.86	0.86	0.87	0.86	0.85	0.85	0.85	792
Alte.	0.85	0.96	0.90	0.70	0.96	0.81	0.85	0.96	0.90	24
Clari-Q.	0.67	0.72	0.69	0.66	0.69	0.67	0.66	0.70	0.68	134
Comm.	0.62	0.54	0.58	0.63	0.54	0.58	0.65	0.53	0.58	382
Cond.	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	13

Appendix

Table 11: Performance of the constructed SDRT parsers. The “Supports” column indicates the number of instances for each relation in the test data. The Relation labels are shown in abbreviated form. The best F1-scores among the models are highlighted in bold.

	EN (Llama-3-base)			JA (Llama-3-base)			JA (Swallow-base)
Relation	Prec.	Recall	F1	Prec.	Recall	F1	Prec.	Recall	F1	Supports
Ackn.	0.64	0.58	0.61	0.63	0.61	0.62	0.65	0.66	0.65	173
Alte.	0.75	0.60	0.67	0.71	0.50	0.59	0.83	0.50	0.62	10
Back.	0.00	0.00	0.00	1.00	0.11	0.20	0.00	0.00	0.00	9
Clari-Q.	0.44	0.58	0.50	0.39	0.54	0.46	0.46	0.54	0.50	48
Comm.	0.58	0.62	0.60	0.58	0.63	0.60	0.58	0.62	0.60	222

Appendix

Table 12: Performance of the constructed SDRT parsers on STAC. The “Supports” column indicates the number of instances for each relation in the test data. The Relation labels are shown in abbreviated form. The best F1-scores among the models are highlighted in bold.

	Llama-3		Swallow-3	Llama-3.1						Swallow-3.1			GPT-5.5
	Llamipa			0shot		1shot		4shot		0shot	1shot	4shot	0shot		1shot		4shot
Relation	EN	JA	JA	EN	JA	EN	JA	EN	JA	JA	JA	JA	EN	JA	EN	JA	EN	JA	Support
Ackn.	0.86	0.86	0.85	0.56	0.56	0.47	0.43	0.40	0.43	0.51	0.36	0.15	0.55	0.56	0.76	0.76	0.78	0.78	792
Alte.	0.90	0.81	0.90	0.00	0.00	0.00	0.00	0.32	0.42	0.00	0.00	0.13	0.68	0.76	0.88	0.65	0.88	0.88	24
Clari-Q.	0.69	0.67	0.68	0.06	0.08	0.22	0.24	0.29	0.27	0.01	0.19	0.08	0.44	0.40	0.60	0.58	0.56	0.60	134
Comm.	0.58	0.58	0.58	0.16	0.16	0.28	0.23	0.25	0.12	0.19	0.14	0.03	0.33	0.34	0.45	0.47	0.52	0.49	382

Appendix

Table 13: F1-scores for Llamipa and prompt-based methods, evaluated on a subset of gold relations from the MSDC corpus (cf. Appendix E ). "Support" represents the number of instances per relation in the test data, and discourse relations are abbreviated.

	Llama-3		Swallow-3	Llama-3.1						Swallow-3.1			GPT-5.5
	Llamipa			0shot		1shot		4shot		0shot	1shot	4shot	0shot		1shot		4shot
Relation	EN	JA	JA	EN	JA	EN	JA	EN	JA	JA	JA	JA	EN	JA	EN	JA	EN	JA	Support
Ackn.	0.86	0.86	0.85	0.56	0.56	0.47	0.43	0.40	0.43	0.51	0.36	0.15	0.55	0.56	0.76	0.76	0.78	0.78	792
Alte.	0.90	0.81	0.90	0.00	0.00	0.00	0.00	0.32	0.42	0.00	0.00	0.13	0.68	0.76	0.88	0.65	0.88	0.88	24
Clari-Q.	0.69	0.67	0.68	0.06	0.08	0.22	0.24	0.29	0.27	0.01	0.19	0.08	0.44	0.40	0.60	0.58	0.56	0.60	134
Comm.	0.54	0.54	0.54	0.15	0.14	0.26	0.21	0.23	0.11	0.18	0.13	0.03	0.31	0.32	0.42	0.43	0.48	0.46	434

Appendix

Table 14: F1-scores for Llamipa and prompt-based methods on the MSDC corpus, evaluated across all gold relations. "Support" represents the number of instances per relation in the test data, and discourse relations are abbreviated.

Multilingual Language Model Evaluation

Multilingual Language Model Evaluation measures how reliably language models perform across languages, scripts, dialects, and cultural contexts.

See all 485 papers

Worth Their Weights? Evaluating Generative Models for Discourse Parsing
Oct 13, 2026Luke Gessler, Amir ZeldesLanguage Modeling
Indiana University Bloomington Department of Linguistics · Georgetown University Department of Linguistics
Can Large Language Models Handle Discourse Particles? A Case Study of Colloquial Malay
May 27, 2026Mariah Al Giptiah Binte Yusoff, Jakin Tan, Bocheng Chen +2Multilingual Language Model Evaluation Pragmatic Reasoning in Language Models
Nanyang Technological University · University of Mississippi · Indiana University
Interpretable Cross-Lingual Alignment in Small Language Models: Probing Cultural and Pragmatic Reasoning in Japanese-English Bilingual LLMs
Aug 14, 2026Florian BraunPragmatic Reasoning in Language Models Cross-Lingual Reasoning in Language Models
Independent Researcher

Oct 13, 2026J/K move · Enter open · S save

Worth Their Weights? Evaluating Generative Models for Discourse Parsing

Luke Gessler, Amir Zeldes

Indiana University Bloomington Department of Linguistics · Georgetown University Department of Linguistics

Language Modeling

Open paper AskPDF arXiv

Rhetorical Structure Theory (RST) parsing has historically been dominated by discriminative models based on pretrained encoder-only language models. The recent availability of high-quality generative language models has made it unclear whether RST parsing is better approached as a generative task. In this work, we investigate whether generative language models are better at parsing RST than traditional discriminative ones. We find that at matched parameter counts, discriminative parsers perform much better than generative ones, though much larger generative models sometimes outperform them. Generative models hold a particular advantage in elementary discourse unit (EDU) segmentation, and we report new state-of-the-art results on the RST-DT benchmark in end-to-end parsing as well as segmentation.

Abstract

Multilingual Language Model Evaluation

Explore similar work

Worth Their Weights? Evaluating Generative Models for Discourse Parsing

Can Large Language Models Handle Discourse Particles? A Case Study of Colloquial Malay

Interpretable Cross-Lingual Alignment in Small Language Models: Probing Cultural and Pragmatic Reasoning in Japanese-English Bilingual LLMs