Ontology-Driven Structural Regularization for Document-Level Relation Extraction

Laura Menotti; Stefano Marchesin; Gianmaria Silvello

cs.CLAug 21, 2026

Ontology-Driven Structural Regularization for Document-Level Relation Extraction

Authors: Laura Menotti, Stefano Marchesin, Gianmaria Silvello

Organizations: Department of Information Engineering, University of Padua, Italy

Abstract

Document-Level Relation Extraction (DocRE) relies heavily on costly manually annotated datasets, while large distant supervision resources such as DocRED distant remain underexploited due to noise. We show that a critical yet overlooked source of noise lies in structural inconsistencies within relational triples, including violations of ontology constraints and logical contradictions. We introduce an ontology-driven framework to quantify and enforce structural consistency in DocRE datasets. Our analysis reveals substantial structural noise in DocRED distant and demonstrates that such inconsistencies propagate to model predictions. Enforcing structural well-formedness during training significantly reduces logical contradictions and consistently improves generalization performance. These findings establish structural consistency as a missing axis of supervision in DocRE and highlight structural regularization as an effective strategy for leveraging distant data at scale.

Figures & tables

Dataset	Split	Triples	Invalid Triples ( $\delta$ )	Missing Inverse ( $\sigma$ )	Asymmetric Inc. ( $\gamma$ )	Cardinality Viol. ( $\theta$ )
DocRED	Train	38,180	602 (1.58%)	2,696 (64.36%)	154 (0.41%)	10 (0.28%)
ReDocRED	Train	85,932	810 (0.94%)	947 (7.89%)	284 (0.34%)	20 (0.40%)
ReDocRED	Dev	17,284	122 (0.71%)	243 (8.92%)	56 (0.33%)	3 (0.32%)
ReDocRED	Test	17,448	141 (0.81%)	242 (9.27%)	82 (0.48%)	6 (0.65%)
DocRED	Train DS	1,505,638	46,211 (3.07%)	85,065 (40.26%)	12,408 (0.84%)	25 (0.01%)

Table 1: Structural consistency analysis of DocRED datasets. Splits “Train”, “Dev”, and “Test” consider manually annotated datasets, while split “Train DS” reports the quality of the DocRED distant dataset. For each rule, we report the absolute number of errors in the dataset and its percentage.

	PLM	Training Data	Invalid ( $\delta$ )	Inverse ( $\sigma$ )	Asymmetric ( $\gamma$ )
ATLOP	BERT	DocRED DS	110 (1.61%)	396 (43.04%)	38 (0.56%)
	BERT	Corrected	46 (0.66%)	65 (5.61%)	4 (0.06%)
	RoBERTa	DocRED DS	110 (1.56%)	407 (45.22%)	42 (0.60%)
	RoBERTa	Corrected	42 (0.59%)	66 (5.68%)	0 (0.00%)
DREEAM	BERT	DocRED DS	59 (1.05%)	266 (41.18%)	30 (0.53%)
	BERT	Corrected	21 (0.31%)	115 (11.97%)	4 (0.06%)

Table 2: Structural consistency of DocRE models predictions trained on DocRED Distant (“ DocRED DS ”) and its structurally-consistent version (“ Corrected ”) and evaluated on ReDocRED test dataset . For each rule, we report the absolute number of errors in the predictions and its percentage. Column “ Invalid ” refers to invalid triples, “ Inverse ” to missing inverse triples, and “ Asymmetric ” to asymmetric inconsistencies.

	PLM	Training Data	Precision	IgnPrec	Recall	F1	IgnF1
ATLOP	BERT	DocRED DS	0.7503	0.5877	0.2945	0.4230	0.3924
		Corrected	0.7665 (+2.16%)	0.6165 (+4.90%)	0.3070 (+4.24%)	0.4384 (+3.64%)	0.4099 (+4.46%)
		Random	0.7528 (+0.33%)	0.5973 (+1.63%)	0.2964 (+0.65%)	0.4253 (+0.43%)	0.3962 (+0.97%)
	RoBERTa	DocRED DS	0.7571	0.6006	0.3060	0.4358	0.4054
		Corrected	0.7657 (+1.14%)	0.6172 (+2.76%)	0.3120 (+1.96%)	0.4434 (+1.74%)	0.4245 (+4.71%)
		Random	0.7578 (+0.09%)	0.6082 (+1.27%)	0.3049 (-0.36%)	0.4348 (-0.23%)	0.4061 (+0.17%)

Table 3: Models trained on DocRED distant (“DocRED DS”) and its structurally-consistent version (“Corrected”) and evaluated on the ReDocRED test dataset . Random removal (“Random”) was performed ten times with different random seed and the mean performance is reported. The standard deviation is always below 1%. The percentage performance improvement compared to DocRED distant is reported between parentheses.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Instances	Invalid Triples ( $\delta$ )	Missing Inverse ( $\sigma$ )	Asymmetric Inc. ( $\gamma$ )	Cardinality Viol. ( $\theta$ )
UGDRE-DocRED	1,717,161	51,671 (3.01%)	112,912 (53.50%)	13,026 (0.77%)	74 (0.03%)
DOREMI-DocRED	1,704,161	29,799 (1.75%)	128,064 (75.13%)	6,030 (0.36%)	1,179 (0.73%)
UGDRE-ReDocRED	3,379,590	94,290 (2.79%)	122,913 (25.42%)	19,416 (0.59%)	1,426 (0.52%)
DOREMI-ReDocRED	3,957,238	69,035 (1.74%)	111,089 (22.00%)	15,244 (0.39%)	3,428 (1.43%)

Appendix

Table 4: Structural consistency analysis of the denoised distant datasets. For each rule, we report the absolute number of errors in the dataset and its percentage.

Model	Training Data	Invalid ( $\delta$ )	Inverse ( $\sigma$ )	Asymmetric ( $\gamma$ )
ATLOP BERT	UGDRE-Doc	106 (1.36%)	528 (59.26%)	38 (0.49%)
	Corrected	42 (0.54%)	65 (5.40%)	2 (0.03%)
	DOREMI-Doc	47 (0.72%)	335 (79.01%)	26 (0.40%)
	Corrected	33 (0.46%)	90 (10.79%)	10 (0.14%)
	UGDRE-ReD	188 (1.24%)	579 (26.82%)	66 (0.44%)
	Corrected	118 (0.79%)	149 (6.16%)	6 (0.04%)

Appendix

Table 5: Structural consistency of DocRE models predictions. The trained on different denoised distant datasets and evaluated on the ReDocRED test dataset . For space reasons, we use suffix “-Doc” as a diminutive for DocRED and “-ReD” as a diminutive for ReDocRED. For each rule, we report the absolute number of errors in the predictions and its percentage. Column “ Invalid ” refers to invalid triples, “ Inverse ” to missing inverse triples, and “ Asymmetric ” to asymmetric inconsistencies.

Model	Training Data	Precision	IgnPrec	Recall	F1	IgnF1
ATLOP BERT	UGDRE-Doc	0.7881	0.6592	0.3512	0.4858	0.4582
	Corrected	0.8002 (+1.54%)	0.6791 (+3.02%)	0.3593 (+2.31%)	0.4959 (+2.08%)	0.4700 (+2.58%)
	DOREMI-Doc	0.8686	0.7903	0.3234	0.4702	0.4589
	Corrected	0.8439 (-2.84%)	0.7716 (-2.37%)	0.3458 (+6.93%)	0.4905 (+4.32%)	0.4776 (+4.07%)
	UGDRE-ReD	0.8044	0.7445	0.7007	0.7490	0.7220
	Corrected	0.8069 (+0.31%)	0.7093 (-4.73%)	0.6937 (-1.00%)	0.7461 (-0.39%)	0.7014 (-2.85%)

Appendix

Table 6: ATLOP and DREEAM trained on different denoised distant datasets and evaluated on the ReDocRED test dataset . For space reasons, we use suffix “-Doc” as a diminutive for DocRED and “-ReD” as a diminutive for ReDocRED.

	PLM	Training Data	Precision	IgnPrec	Recall	F1	IgnF1
ATLOP	BERT	DocRED DS	0.7657	0.6069	0.2983	0.4293	0.4000
		w/o invalid	0.7684 (+0.35%)	0.6196 (+2.08%)	0.3009 (+0.87%)	0.4324 (+0.73%)	0.4050 (+1.27%)
		w/o inverse	0.7605 (-0.68%)	0.5980 (-1.46%)	0.3088 (+3.55%)	0.4393 (+2.33%)	0.4073 (+1.84%)
		w/o asymmetric	0.7660 (+0.03%)	0.6094 (+0.41%)	0.3009 (+0.89%)	0.4321 (+0.65%)	0.4029 (+0.73%)
		All rules	0.7704 (+0.60%)	0.6197 (+2.11%)	0.3103 (+4.05%)	0.4424 (+3.06%)	0.4136 (+3.41%)
	RoBERTa	DocRED DS	0.7603	0.6041	0.3106	0.4410	0.4102

Appendix

Table 7: Models trained on DocRED distant (“DocRED DS”) and its structurally-consistent versions and evaluated on the ReDocRED dev dataset . Row “All rules" considers the dataset cleaned from all structural constraints, row “w/o invalid" considers the dataset dataset cleaned from invalid triples only, row “w/o inverse" dataset cleaned from missing inverse triples only, and row “w/o asymmetric" dataset cleaned from asymmetric inconsistencies only. The percentage performance improvement compared to DocRED distant is reported between parentheses.

Wikidata ID	Name	Domain	Range
P6	head of government	LOC\|ORG\|MISC	PER\|MISC
P17	country	ORG\|LOC\|PER\|MISC\|TIME	LOC\|ORG\|MISC
P19	place of birth	PER\|MISC	LOC\|MISC
P20	place of death	PER\|MISC	LOC\|MISC
P22	father	PER\|MISC	PER\|MISC
P25	mother	PER\|MISC	PER\|MISC

Appendix

Table 8: Domain and Range constraints of DocRED relations (I). For each relation, we report the Wikidata ID and a textual description.

Wikidata ID	Name	Domain	Range
P206	located in or next to body of water	LOC\|ORG\|MISC	LOC\|MISC
P241	military branch	PER\|ORG\|MISC	ORG\|MISC
P264	record label	PER\|ORG\|MISC	ORG\|MISC
P272	production company	ORG\|MISC	ORG\|MISC
P276	location	PER\|ORG\|LOC\|TIME\|NUM\|MISC	ORG\|LOC\|MISC
P279	subclass of	PER\|ORG\|LOC\|TIME\|NUM\|MISC	PER\|ORG\|LOC\|TIME\|NUM\|MISC

Appendix

Table 9: Domain and Range constraints of DocRED relations (II). For each relation, we report the Wikidata ID and a textual description (column “Name”).

Wikidata ID	Name	Inverse ID	Inverse Name
P22	father	P40	child
P25	mother	P40	child
P26	spouse	P26	spouse
P36	capital	P1376	capital of
P50	author	P800	notable work
P57	director	P800	notable work

Appendix

Table 10: DocRED relations with an inverse relation. For each relation, we report its Wikidata ID, its textual description (“Name”) and the Wikidata ID (“Inverse ID”) and description (“Inverse Name”) of its inverse relation. Note that inverse relations also include symmetric relations.

Wikidata ID	Name	Max Cardinality
P19	place of birth	2
P20	place of death	2
P22	father	2
P25	mother	2
P569	date of birth	2
P570	date of death	2

Appendix

Table 11: DocRED relations with bounded maximum cardinality. For each relation, we report its Wikidata ID, its textual description (“Name”) and the Wikidata ID (“Inverse ID”) and description (“Inverse Name”) of its inverse relation. Note that inverse relations also include symmetric relations (Wikidata ID = Inverse ID).

Relation Extraction

Relation extraction identifies and classifies relationships between entities or events in text and multimodal data.

See all 82 papers

When Does Consistency Pay Off? Relational Constraints for LLM-based Document-Level Relation Extraction under Incomplete Annotations
Jul 27, 2026Mingxuan SunRelation Extraction Document Understanding
Department of Computer Science, Université de Sherbrooke, Sherbrooke, Canada
ReaORE: Reasoning-Guided Progressive Open Relation Extraction Empowered by Large Reasoning Models
Jun 25, 2026Xin Lin, Liang Zhang, Guoqi Ma +2Relation Extraction
National Institute for Data Science in Health and Medicine, Xiamen University, China · School of Informatics, Xiamen University
HCRE: LLM-based Hierarchical Classification for Cross-Document Relation Extraction with a Prediction-then-Verification Strategy
Apr 9, 2026Guoqi Ma, Liang Zhang, Hongyao Tu +6Relation Extraction Hierarchical Classification
School of Informatics, Xiamen University · Li Auto Inc. · Alibaba International Digital Commerce Group

Jul 27, 2026J/K move · Enter open · S save

When Does Consistency Pay Off? Relational Constraints for LLM-based Document-Level Relation Extraction under Incomplete Annotations

Mingxuan Sun

Department of Computer Science, Université de Sherbrooke, Sherbrooke, Canada

Relation Extraction Document Understanding LLM Reliability

Open paper AskPDF arXiv

Consistency-based repair is widely assumed to improve LLM-based document-level relation extraction (DocRE), but this assumption can fail when the benchmark itself is incompletely annotated. Whether such repair helps depends strongly on how closed the gold annotations are under the constraints it enforces. We introduce

Δ_{\rm closure}

, the rate at which gold annotations lack facts implied by the very rules used for repair, computable from the annotations and rules alone before any repair is run. The same typed constraints drive CONSISTRE, which applies them at inference time (constraint feedback for reflection, deterministic closure) and at training time (distillation and consistency-aware reinforcement learning). On 998 DocRED documents, whose Re-DocRED revision lowers

Δ_{\rm closure}

from 1.71% to 0.257%, rescoring identical predictions reverses the method ranking: Self-Refine leads under DocRED, while a constraint-feedback variant moves from fifth place to first. On the near-closed DWIE, constraint feedback with chain-of-thought beats three generic reflection baselines by 0.046-0.107 micro-F1 (

p=0.0002

), and deterministic closure of the base output scores higher still at

3.6\times

lower cost. A reinforcement-learning gain on DocRED (+0.025) does not survive Re-DocRED rescoring. Consistency repair pays off where annotations are closed, and

Δ_{\rm closure}

tells in advance whether they are.

Abstract

Relation Extraction

Explore similar work

When Does Consistency Pay Off? Relational Constraints for LLM-based Document-Level Relation Extraction under Incomplete Annotations

ReaORE: Reasoning-Guided Progressive Open Relation Extraction Empowered by Large Reasoning Models

HCRE: LLM-based Hierarchical Classification for Cross-Document Relation Extraction with a Prediction-then-Verification Strategy