LLMs Underperform Graph-Based Parsers on Supervised Relation Extraction for Complex Graphs

Paolo Gajo; Domenic Rosati; Hassan Sajjad; Alberto Barrón-Cedeño

cs.CLApr 9, 2026

LLMs Underperform Graph-Based Parsers on Supervised Relation Extraction for Complex Graphs

Authors: Paolo Gajo, Domenic Rosati, Hassan Sajjad, Alberto Barrón-Cedeño

Organizations: University of Bologna · Dalhousie University

Abstract

Relation extraction represents a fundamental component in the process of creating knowledge graphs, among other applications. Large language models (LLMs) have been adopted as a promising tool for relation extraction, both in supervised and in-context learning settings. However, in this work we show that their performance still lags behind much smaller architectures when the linguistic graph underlying a text has great complexity. To demonstrate this, we evaluate four LLMs against a graph-based parser on six relation extraction datasets with sentence graphs of varying sizes and complexities. Our results show that the graph-based parser increasingly outperforms the LLMs, as the number of relations in the input documents increases. This makes the much lighter graph-based parser a superior choice in the presence of complex linguistic graphs.

Figures & tables

Dataset	Min	Mean ( $\bm{\overline{k}}$ )	Max	$\bm{k\leq 5}$	Avg. chars.
CoNLL04	1	1.42	11	98.54 %	158.10
ADE	1	1.59	21	97.77 %	135.33
SciERC	1	2.38	17	93.49 %	159.44
enEWT	5	17.83	158	25.03 %	91.77
SciDTB	5	23.41	100	2.50 %	150.50
ERFGC	4	49.19	167	3.33 %	603.90

Table 1: Statistics of the number

k

of relations for the documents contained in the datasets, for all splits, along with mean document length in characters.

Prompt	#ICL	Steps	CoNLL04	ADE	SciERC	enEWT	SciDTB	ERFGC
Mistral-7B-Instruct-v0.3 (5 seeds)
NoDesc	0	$\times$	0.115 $\pm$ 0.000	0.047 $\pm$ 0.000	0.021 $\pm$ 0.000	0.006 $\pm$ 0.000	0.001 $\pm$ 0.000	0.002 $\pm$ 0.000
NoDesc	1	$\times$	0.123 $\pm$ 0.009	0.281 $\pm$ 0.031	0.039 $\pm$ 0.005	0.026 $\pm$ 0.000	0.032 $\pm$ 0.001	0.056 $\pm$ 0.013
NoDesc	0	1 ep.	0.597 $\pm$ 0.011	0.776 $\pm$ 0.016	0.320 $\pm$ 0.005	0.784 $\pm$ 0.000	0.793 $\pm$ 0.003	0.248 $\pm$ 0.017
NoDesc	1	1 ep.	0.613 $\pm$ 0.011	0.775 $\pm$ 0.007	0.346 $\pm$ 0.005	0.805 $\pm$ 0.001	0.830 $\pm$ 0.001	0.331 $\pm$ 0.013
Qwen3-14B-Base (4 seeds)

Table 2: Micro-F 1 for the LLMs and graph-based parser. Best in bold, second-best underlined.

	CoNLL04	ADE	SciERC	enEWT	SciDTB	ERFGC
Qwen3-14B-Base	-0.143 $\pm 0.067$	-0.152 $\pm 0.049$	0.171 $\pm 0.054$	-0.270 $\pm 0.025$	-0.185 $\pm 0.048$	-0.639 $\pm 0.072$
Graph-based parser	0.012 $\pm$ 0.024	-0.046 $\pm 0.045$	0.067 $\pm 0.029$	-0.043 $\pm 0.013$	-0.195 $\pm 0.012$	-0.206 $\pm 0.051$

Table 3: Pearson

r

between edge number

k

and micro-F 1 for Qwen3-14B-Base and the graph-based parser.

Model	Mean $\sigma$	Max $\sigma$
Mistral-7B-Instruct-v0.3	0.006	0.031
Qwen3-14B-Base	0.008	0.031
Qwen3-32B	0.012	0.039

Table 4: Mean and maximum variance for the F 1 scores obtained by the LLMs evaluated on all test samples.

Dataset (Train / Dev / Test)
ADE (2,563 / 854 / 300) Gurulingappa et al. (2012)
Entities: disease, drug
Relations: adverseEffect
CoNLL04 (922 / 231 / 288) Roth and Yih (2004)
Entities: organization, person, location
Relations: kill, locatedIn, workFor, orgBasedIn, liveIn

Table 5: Samples per partition and entity/relation classes for the datasets used in this paper.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Diagram of the graph-based parser.

$\bm{L_{\psi}}$	$\bm{L_{\phi}}$	CoNLL04	ADE	SciERC	enEWT	SciDTB	ERFGC
0	0	0.617 $\pm 0.012$	0.567 $\pm 0.016$	0.172 $\pm 0.032$	0.692 $\pm 0.003$	0.785 $\pm 0.002$	0.630 $\pm 0.005$
	1	0.618 $\pm 0.007$	0.589 $\pm 0.028$	0.183 $\pm 0.023$	0.715 $\pm 0.004$	0.820 $\pm 0.004$	0.640 $\pm 0.003$
	2	0.562 $\pm 0.038$	0.575 $\pm 0.009$	0.145 $\pm 0.043$	0.686 $\pm 0.007$	0.813 $\pm 0.001$	0.631 $\pm 0.010$
	3	0.536 $\pm 0.026$	0.521 $\pm 0.085$	0.056 $\pm 0.033$	0.641 $\pm 0.011$	0.789 $\pm 0.003$	0.633 $\pm 0.004$
1	0	0.649 $\pm 0.013$	0.678 $\pm 0.041$	0.317 $\pm 0.029$	0.850 $\pm 0.002$	0.903 $\pm 0.001$	0.695 $\pm 0.002$
	1	0.658 $\pm 0.002$	0.705 $\pm 0.011$	0.323 $\pm 0.005$	0.850 $\pm 0.001$	0.902 $\pm 0.005$	0.697 $\pm 0.004$

Appendix

Table 6: Exact evaluation micro-F1 for the graph-based BERT model. Best in bold.

Relation Extraction

Relation extraction identifies and classifies relationships between entities or events in text and multimodal data.

See all 82 papers

Are Large Language Models Suitable for Graph Computation? Progress and Prospects
Jun 5, 2026Yuting Zhang, Yi Han, Kai Wang +3LLM Reasoning with Graphs LLM Tool Use
University of New South Wales · Antai College of Economics and Management, Shanghai Jiao Tong University · Edith Cowan University +1
GRAIN: Bridging Name and Narrative Shifts in Real-World Graph Reasoning through Invariance-Rewarded Agentic RL
Aug 27, 2026Zike Yuan, Han Zhang, Jianzhi Yan +10LLM Reasoning with Graphs Language Model Robustness
Harbin Institute of Technology, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China · Xidian University, Xi’an, China
Position: How can Graphs Help Large Language Models?
May 4, 2026Xiyuan Wang, Yi Hu, Yanbo Wang +2LLM Grounding Knowledge Augmentation for Language Models
Institute for Artificial Intelligence, Peking University, Beijing 100871, China · School of Computer Sciences, Beijing University of Posts and Telecommunications, Beijing 100876, China

Jun 5, 2026J/K move · Enter open · S save

Are Large Language Models Suitable for Graph Computation? Progress and Prospects

Yuting Zhang, Yi Han, Kai Wang, Wei Ni+2

University of New South Wales · Antai College of Economics and Management, Shanghai Jiao Tong University · Edith Cowan University+1

LLM Reasoning with Graphs LLM Tool Use Graph Reasoning LLMs for Graph Learning

Open paper AskPDF arXiv

Large language models (LLMs) have been increasingly explored for graph computation, where tasks require reasoning over structured relationships and algorithmic operations. Yet, it remains unclear when LLMs can reliably support such computation and how they should be incorporated into graph-solving pipelines. Existing surveys at the intersection of LLMs and graphs primarily focus on graph learning, text-attributed graphs, or graph-language modeling. To bridge this gap, we provide a comprehensive review of LLMs for graph computation through a role-based taxonomy. Specifically, we identify two major paradigms: i) LLMs as executors, where models directly solve graph tasks from graph descriptions and instructions; and ii) LLMs as planners, where models formulate problems, decompose reasoning steps, and invoke external tools or agents for execution. Based on this taxonomy, we analyze the strengths and limitations of current methods. Our review indicates that LLMs are promising for simple, small-scale tasks, but remain unreliable for large-scale and exactness-demanding tasks. Finally, we summarize available datasets and suggest four future directions.

Abstract

Relation Extraction

Explore similar work

Are Large Language Models Suitable for Graph Computation? Progress and Prospects

GRAIN: Bridging Name and Narrative Shifts in Real-World Graph Reasoning through Invariance-Rewarded Agentic RL

Position: How can Graphs Help Large Language Models?