HCRE: LLM-based Hierarchical Classification for Cross-Document Relation Extraction with a Prediction-then-Verification Strategy

Guoqi Ma; Liang Zhang; Hongyao Tu; Hao Fu; Hui Li; Yujie Lin; Longyue Wang; Weihua Luo

cs.CLApr 9, 2026

HCRE: LLM-based Hierarchical Classification for Cross-Document Relation Extraction with a Prediction-then-Verification Strategy

Authors: Guoqi Ma, Liang Zhang, Hongyao Tu, Hao Fu, Hui Li, Yujie Lin, Longyue Wang, Weihua Luo,

Organizations: School of Informatics, Xiamen University · Li Auto Inc. · Alibaba International Digital Commerce Group

Abstract

Cross-document relation extraction (RE) aims to identify relations between the head and tail entities located in different documents. Existing approaches typically adopt the paradigm of ``Small Language Model (SLM) + Classifier''. However, the limited language understanding ability of SLMs hinders further improvement of their performance. In this paper, we conduct a preliminary study to explore the performance of Large Language Models (LLMs) in cross-document RE. Despite their extensive parameters, our findings indicate that LLMs do not consistently surpass existing SLMs. Further analysis suggests that the underperformance is largely attributed to the challenges posed by the numerous predefined relations. To overcome this issue, we propose an LLM-based Hierarchical Classification model for cross-document RE (HCRE), which consists of two core components: 1) an LLM for relation prediction and 2) a hierarchical relation tree derived from the predefined relation set. This tree enables the LLM to perform hierarchical classification, where the target relation is inferred level by level. Since the number of child nodes is much smaller than the size of the entire predefined relation set, the hierarchical relation tree significantly reduces the number of relation options that LLM needs to consider during inference. However, hierarchical classification introduces the risk of error propagation across levels. To mitigate this, we propose a prediction-then-verification inference strategy that improves prediction reliability through multi-view verification at each level. Extensive experiments show that HCRE outperforms existing baselines, validating its effectiveness.

Figures & tables

Figure 1: Comparison between the paradigm of existing cross-document RE model (Direct Classification) and ours (Hierarchical Classification). Green arrows indicate hierarchical classification based on the relation tree.

Baselines	max F1	micro F1	binary F1
End-to-End	49.07	33.33	41.76
ECRIM	61.64	39.25	42.98
NEPD	63.53	25.77	32.00

Table 1: Comparison among maximum F1, micro F1 and binary F1.

Figure 2: Comparison among P@K, micro F1, and binary F1 under varying dataset sizes.

Figure 3: Our prompt template. Each prompt contains the preprocessed text path, head and tail entities, and relation options.

Figure 4: Effect of the number of relation options on LLMs. Note that CodRED involves 277 predefined relations. The horizontal dashed lines denote SLM performance serving as baselines.

Figure 5: Overview of HCRE. (a) Guided by the hierarchical relation tree, the LLM $\mathcal{M}_{1}$ performs hierarchical classification in a top-down manner (See green arrows). (b) At each level, we refine model prediction by prediction-then-verification inference strategy, which is composed of the prediction step and the verification step .

	Closed		Open
	Train	Dev	Dev
#Text Path (Pos.)	8,263	2,558	4,558
#Text Path (NA)	120,925	38,182	15,072
#Tokens / Doc	4,938.6	5,031.6	62,863

Table 2: Statistics of CodRED. (Pos.: Positive; NA: Not Available.)

Cross-Document RE Baselines
Backbone	Model	Closed		Open
Backbone	Model	micro F1	binary F1	micro F1	binary F1
BERT-base	End-to-End Yao et al. (2021)	33.33	41.76	22.23	28.85
	ECRIM Wang et al. (2022)	39.25	42.98	19.81	22.06
	KXDocRE Jain et al. (2024a)	37.53	38.25	18.80	19.39
	REIC Na et al. (2024)	38.75	46.25	19.50	23.77

Table 3: Experimental results on CodRED under both closed and open settings. The subscript denotes the corresponding standard deviation (e.g., 45.35 3 represents 45.35

\pm

0.3). ‡ indicates significance at

p<0.01

over the second-best baseline NEPD, based on 1,000 bootstrap tests Tibshirani and Efron (1993) .

Model	micro F1	binary F1
Ours	45.35	58.19
w/o multi-view	39.37	49.63
w/o PtV	37.66	47.28
w/o LTC	43.18	56.60
w/o LTC, PtV	32.55	45.44
w/o HRT	38.14	41.43

Table 4: Ablation study of our model on CodRED under the closed setting. Note that LTC, PtV, and HRT denote level-wise tree construction, prediction-then-verification, and hierarchical relation tree, respectively.

Figure 6: The accuracy and error propagation ratio of our model, with and without the PtV strategy.

Figure 7: Model performance on the CodRED development set using trees with varying tree depth

L

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Illustration of the level-wise pipeline for hierarchical relation tree construction.

Tree Construction Parameters	Value	micro F1	binary F1
Random Seed	42	45.35	58.19
	666	44.50	57.42
	1024	44.55	57.98
Criterion Set	(Domain, Entity Type)	45.35	58.19
	(Domain, Polarity)	45.10	57.08
	(Entity Type, Polarity)	44.14	57.36

Appendix

Table 5: Experiment results with varying tree construction parameters.

#Input Tokens	#Output Tokens	Total
334,793	12,713	347,506
Input Cost	Output Cost	Total
$0.8370	$0.1271	$0.9641

Appendix

Table 6: API cost of hierarchical relation tree construction.

Figure 9: Visualization of a partial hierarchical relation tree.

Seed	42	666	1024
42	100.0	–	–
666	46.67	100.0	–
1024	47.82	47.72	100.0

Appendix

Table 7: The tree edit similarity (%) matrix between constructed tree pairs.

Statistic	Value
Tree Depth	5
#Node at Level 0	1
#Node at Level 1	2
#Node at Level 2	12
#Node at Level 3	136
#Node at Level 4	675

Appendix

Table 8: The statistics of our hierarchical relation tree.

	Coherence (%)
Level 1 $\rightarrow$ 2	82.52
Level 2 $\rightarrow$ 3	80.27
Level 3 $\rightarrow$ 4	83.42

Appendix

Table 9: The semantic coherence scores for each pair of adjacent tree levels.

Figure 10: The tree construction pipeline for the w/o LTC variant.

Level	Model	%CP	%WP	%SC
1	HCRE w/o PtV	34.64	0.00	65.36
1	HCRE (Ours)	72.67	0.00	27.33
2	HCRE w/o PtV	32.13	65.36	2.51
2	HCRE (Ours)	68.92	27.33	3.75
3	HCRE w/o PtV	31.31	67.87	0.82
3	HCRE (Ours)	64.86	31.08	4.06

Appendix

Table 10: The error transfer details for HCRE w/o PtV and HCRE. Note that CP , WP , and SC refer to “Correct Prediction”, “Wrong Parent”, and “Sibling Confusion”, respectively.

Model	Closed		Open
Model	bag-level micro F1	bag-level binary F1	bag-level micro F1	bag-level binary F1
RoBERTa+NEPD	54.17	72.04	50.39	64.38
Vanilla	47.53	52.47	47.67	54.49
HCRE (Ours)	64.82	86.18	58.98	79.90

Appendix

Table 11: Bag-level evaluation results on CodRED under both closed and open settings.

Setting	Model	Dev				Test		Avg.
Setting	Model	F1	AUC	P@500	P@1000	F1	AUC	Avg.
Closed	End-to-End Yao et al. (2021)	51.26	47.94	62.80	51.00	51.02	47.46	49.42
	ECRIM Wang et al. (2022)	61.12	60.91	78.89	60.17	62.48	60.67	61.30
	KXDocRE Jain et al. (2024a)	64.97	64.30	–	–	66.30	65.55	65.28
	REIC Na et al. (2024)	63.47	66.41	80.20	63.50	65.02	65.88	65.20
	NEPD Yue et al. (2024)	63.63	65.01	77.84	64.03	64.41	66.23	64.82

Appendix

Table 12: Experiment results on conventional metrics under both closed and open settings. F1 denotes maximum F1, and Avg. scores are computed based on F1 and AUC only. Baseline results are taken from their respective original papers.

Backbone	Model	Closed		Open
Backbone	Model	micro F1	binary F1	micro F1	binary F1
Qwen2.5-0.5B-Instruct	Vanilla	18.63	26.41	7.00	10.15
Qwen2.5-0.5B-Instruct	HCRE (Ours)	25.83	44.10	21.00	37.63
Qwen2.5-7B-Instruct	Vanilla	36.10	41.16	14.20	16.59
Qwen2.5-7B-Instruct	HCRE (Ours)	37.29	52.35	34.40	53.43
Gemma2-9B-Instruct	Vanilla	37.34	41.77	14.53	17.04

Appendix

Table 13: Experiment results on CodRED with different backbones.

LLM-based Doc-level RE Baselines
Model	Dev		Test
Model	F1	Ign F1	F1	Ign F1
AutoRE	50.37	48.79	50.35	48.58
EP-RSR	56.17	53.74	56.47	53.22
Ours
Vanilla	61.30	60.21	60.77	59.63

Appendix

Table 14: Experiment results on DocRED.

Model	Input Tok.	#LLM Calls	Latency
Vanilla	1,499.76	40,740	0.21s
HCRE (Ours)	575.75	152,663	0.29s

Appendix

Table 15: Comparison of computation efficiency between Vanilla and HCRE. Input Len. refers to the average input tokens.

Figure 11: The micro F1 and per-instance inference latency using PtV with varying numbers of verification nodes.

Relation Extraction

Relation extraction identifies and classifies relationships between entities or events in text and multimodal data.

See all 82 papers

Jul 27, 2026

When Does Consistency Pay Off? Relational Constraints for LLM-based Document-Level Relation Extraction under Incomplete Annotations

Consistency-based repair is widely assumed to improve LLM-based document-level relation extraction (DocRE), but this assumption can fail when the benchmark itself is incompletely annotated. Whether such repair helps depends strongly on how closed the gold annotations are under the constraints it enforces. We introduce

Δ_{\rm closure}

, the rate at which gold annotations lack facts implied by the very rules used for repair, computable from the annotations and rules alone before any repair is run. The same typed constraints drive CONSISTRE, which applies them at inference time (constraint feedback for reflection, deterministic closure) and at training time (distillation and consistency-aware reinforcement learning). On 998 DocRED documents, whose Re-DocRED revision lowers

Δ_{\rm closure}

from 1.71% to 0.257%, rescoring identical predictions reverses the method ranking: Self-Refine leads under DocRED, while a constraint-feedback variant moves from fifth place to first. On the near-closed DWIE, constraint feedback with chain-of-thought beats three generic reflection baselines by 0.046-0.107 micro-F1 (

p=0.0002

), and deterministic closure of the base output scores higher still at

3.6\times

lower cost. A reinforcement-learning gain on DocRED (+0.025) does not survive Re-DocRED rescoring. Consistency repair pays off where annotations are closed, and

Δ_{\rm closure}

tells in advance whether they are.

Mingxuan Sun

Department of Computer Science, Université de Sherbrooke, Sherbrooke, Canada

Relation Extraction Document Understanding LLM Reliability

Ask PDF

Jun 30, 2026

Cross-lingual Relation Extraction with Large Language Models: Zero-Shot, Few-Shot, and Fine-Tuned Evaluation on Romanian

Relation extraction (RE) for low-resource languages is typically constrained by the lack of annotated corpora. We investigate the feasibility of cross-lingual RE for Romanian by combining automatic dataset translation with large language model (LLM) inference. We translate the SemEval-2010 Task 8 benchmark from English to Romanian using an LLM-based translation pipeline and evaluate Gemma 4 31B under zero-shot, few-shot, and QLoRA fine-tuned configurations, against four encoder baselines spanning 125M to 560M parameters: XLM- RoBERTa (base and large), Romanian BERT, and RoBERT- large. We assess two task formulations: relation classification with marked entities and end-to-end extraction. Our results show that Romanian incurs a 3 to 5 percentage point (pp) drop relative to English in prompt-only settings, that few-shot prompting provides marginal gains over zero-shot, and that QLoRA fine-tuning improves macro F1-Score by more than 22 percentage points in both languages while reducing the cross-lingual gap from 3.3 to 1.4pp. The encoder baselines come within 1-4pp of QLoRA Gemma on Romanian despite being 50-250 times smaller, with monolingual Romanian BERT at 125M parameters matching multilingual XLM-R at 278M. The case for using a 31B model for single-task RE on Romanian is therefore weak in deployment scenarios where compute matters. We release the translated dataset, evaluation code, and trained models.

Dragos-Mitrut Vasile, Elena-Simona Apostol, Stefan-Adrian Toma +2

National University of Science and Technology POLITEHNICA Bucharest, Splaiul Independent,ei 313, Bucures,ti 060042, Romania · Military Technical Academy ‘Ferdinand I’, Bulevardul George Cos,buc 39-49, 050141, Bucharest, Romania · Freie Universität Berlin, Arnimallee 14, Berlin, 14195, Germany +2

Relation Extraction LLM Fine-Tuning LLM Information Extraction Low-Resource Language Processing Cross-Lingual Transfer

Ask PDF

Aug 4, 2026

ANCHOR-RE: An Agentic Biomedical Relation Extraction Framework with Retrieval Augmentation and Error-Driven Verification

Biomedical relation extraction (BioRE) supports knowledge graph construction and hypothesis generation. In real-world settings, most candidate entity pairs are unrelated. Symbolic systems such as SemRep offer high precision and interpretability but limited recall, whereas large language models (LLMs) can produce false-positive relations. We developed ANCHOR-RE, an agentic framework that combines retrieved demonstrations, ontology-guided semantic constraints, external biomedical knowledge, and error-driven verification to improve BioRE reliability without task-specific fine-tuning. We evaluated ANCHOR-RE on SemRepGS, DDI, and ChemProt using proprietary and open-weight LLMs. To assess generalizability while reducing potential pretraining contamination, we also conducted a post-cutoff temporal precision evaluation using 100 biomedical articles published in 2026. With the proprietary backbone, ANCHOR-RE improved micro-F1 over direct prompting from 0.654 to 0.676 on SemRepGS, from 0.769 to 0.881 on DDI, and from 0.937 to 0.941 on ChemProt, outperforming previously reported inference-only methods on DDI and ChemProt. Gains were also observed with the open-weight backbone. Additional experiments demonstrated cross-family model compatibility and sensitivity to external knowledge quality. Manual assessment of 500 randomly sampled positive predictions from the post-cutoff set yielded 69% precision. These results show that contextual evidence and error-driven verification can improve BioRE reliability across benchmarks, model families, and newly published literature, supporting ANCHOR-RE as a practical training-free approach to biomedical literature mining.

Shufan Ming, Yikun Han, Gibong Hong +2

School of Information Sciences, University of Illinois Urbana-Champaign, 501 E Daniel St., Champaign, 61820, IL, USA · Division of Computational Health Sciences, Department of Surgery, University of Minnesota, 516 Delaware St SE, Minneapolis, 55455, MN, USA

Relation Extraction LLM Grounding LLM Reliability Neuro-Symbolic Reasoning

Ask PDF

Abstract

Relation Extraction

Explore similar work

When Does Consistency Pay Off? Relational Constraints for LLM-based Document-Level Relation Extraction under Incomplete Annotations

Cross-lingual Relation Extraction with Large Language Models: Zero-Shot, Few-Shot, and Fine-Tuned Evaluation on Romanian

ANCHOR-RE: An Agentic Biomedical Relation Extraction Framework with Retrieval Augmentation and Error-Driven Verification