An Empirical Study of Recent Large Language Models for Knowledge Graph to Text Generation on Standard Benchmarks

Priyanka; Renaud Delbru; Souhail Bakkali; Mickaël Coustaty; Antoine Doucet

cs.CLOct 22, 2026

An Empirical Study of Recent Large Language Models for Knowledge Graph to Text Generation on Standard Benchmarks

Authors: Priyanka, Renaud Delbru, Souhail Bakkali, Mickaël Coustaty, Antoine Doucet

Organizations: La Rochelle Université, La Rochelle, France · Siren, Bordeaux, France · Université de Rennes, Rennes, France

Abstract

Knowledge graph-to-text (KG-to-text) generation verbalizes a set of knowledge graph triples into fluent natural language. While fine-tuned encoder-decoder models have long been the standard approach, recent LLMs are becoming increasingly competitive. We benchmark recent LLMs against fine-tuned encoder-decoder models, using reference-based metrics and graph-grounded reference-free LLM-based evaluation validated against human annotations. We observed that recent LLMs substantially narrow, and in several settings close, the gap with fine-tuned models on reference-based metrics, while performing better on graph-grounded metrics. LLM-based evaluation scores show limited correlation with reference-based metrics, suggesting that the two approaches capture different aspects of generation quality. Graph-grounded evaluation reveals that prompted models produce fewer contradictions and triple omissions than fine-tuned models, and these errors increase more rapidly with graph size for fine-tuned models, highlighting differences between the two approaches that reference-based metrics may not fully capture.

Figures & tables

Model	BLEU	METEOR	ChrF++	ROUGE-L	BLEURT	BERTScore	BARTScore	PARENT F1
T5-large	58.57 [57.54, 59.56]	0.776 $\pm 0.007$	72.54 [71.91, 73.16]	0.735 $\pm 0.008$	0.820 $\pm 0.003$	0.883 $\pm 0.004$	-0.962 $\pm 0.022$	0.693 $\pm 0.009$
GPT-5 mini	51.94 [51.01,52.92]	0.761 $\pm 0.007$	70.70 [70.13,71.27]	0.702 $\pm 0.008$	0.814 $\pm 0.004$	0.866 $\pm 0.004$	-1.117 $\pm 0.023$	0.649 $\pm 0.010$
Gemma-4-31B	53.74 [52.87, 54.62]	0.776 $\pm 0.007$	71.68 [71.11, 72.24]	0.717 $\pm 0.008$	0.820 $\pm 0.003$	0.873 $\pm 0.003$	-1.035 $\pm 0.021$	0.667 $\pm 0.009$
Granite-4.2-30B	52.86 [51.92, 53.74]	0.770 $\pm 0.007$	71.49 [70.95, 72.03]	0.702 $\pm 0.008$	0.812 $\pm 0.003$	0.869 $\pm 0.003$	-1.082 $\pm 0.023$	0.664 $\pm 0.009$
Nemotron-3-Ultra	55.51 [54.57,56.42]	0.771 $\pm 0.007$	71.69 [71.12,72.21]	0.723 $\pm 0.008$	0.821 $\pm 0.003$	0.875 $\pm 0.003$	-1.019 $\pm 0.021$	0.669 $\pm 0.009$
T5-Gemma2-4B-4B	55.43 [54.39, 56.45]	0.765 $\pm 0.007$	71.36 [70.74, 71.93]	0.705 $\pm 0.009$	0.811 $\pm 0.004$	0.872 $\pm 0.004$	-1.095 $\pm 0.024$	0.652 $\pm 0.010$

Table 1: Comparison of all models on the WebNLG17 test set. In case of BLEU and ChrF++, we present corpus score and bootstrap 95% CI. For the other metrics we present the mean scores and 95% CI through t-distribution

Model	BLEU	METEOR	ChrF++	ROUGE-L	BLEURT	BERTScore	BARTScore	PARENT F1
T5-large	54.23 [53.30,55.08]	0.745 $\pm 0.006$	69.48 [68.93,70.00]	0.692 $\pm 0.008$	0.802 $\pm 0.003$	0.863 $\pm 0.003$	-1.079 $\pm 0.024$	0.649 $\pm 0.009$
GPT-5-mini	52.08 [51.20,52.90]	0.745 $\pm 0.007$	69.19 [68.66,69.73]	0.683 $\pm 0.008$	0.807 $\pm 0.003$	0.859 $\pm 0.003$	-1.121 $\pm 0.021$	0.637 $\pm 0.009$
Gemma-4-31B	54.85 [53.95, 55.69]	0.764 $\pm 0.007$	70.65 [70.10, 71.21]	0.701 $\pm 0.008$	0.817 $\pm 0.003$	0.869 $\pm 0.003$	-1.045 $\pm 0.021$	0.658 $\pm 0.009$
Granite-4.2-30B	51.89 [51.06,52.77]	0.749 $\pm 0.006$	69.55 [69.04,70.06]	0.675 $\pm 0.008$	0.805 $\pm 0.003$	0.857 $\pm 0.003$	-1.136 $\pm 0.023$	0.640 $\pm 0.009$
Nemotron-3-Ultra	53.99 [53.10,54.87]	0.749 $\pm 0.007$	69.73 [69.18,70.28]	0.691 $\pm 0.008$	0.812 $\pm 0.003$	0.863 $\pm 0.003$	-1.035 $\pm 0.020$	0.638 $\pm 0.009$
T5-Gemma2-4B-4B	51.45 [50.52,52.37]	0.728 $\pm 0.007$	67.93 [67.37,68.48]	0.672 $\pm 0.008$	0.793 $\pm 0.004$	0.852 $\pm 0.004$	-1.167 $\pm 0.025$	0.633 $\pm 0.009$

Table 2: Comparison of all models on the WebNLG20 test set. In case of BLEU and ChrF++, we present corpus score and bootstrap 95% CI. For the other metrics we present the mean scores and 95% CI through t-distribution

Model	BLEU	METEOR	ChrF++	ROUGE-L	BLEURT	BERTScore	BARTScore	PARENT F1
T5-large	47.87 [47.21, 48.51]	0.706 $\pm 0.005$	65.11 [64.69,65.53]	0.647 $\pm 0.005$	0.779 $\pm 0.003$	0.844 $\pm 0.003$	-1.182 $\pm 0.017$	0.609 $\pm 0.006$
GPT-5-mini	41.80 [41.22,42.35]	0.686 $\pm 0.005$	63.40 [63.02,63.80]	0.615 $\pm 0.005$	0.773 $\pm 0.003$	0.824 $\pm 0.002$	-1.427 $\pm 0.017$	0.588 $\pm 0.006$
Gemma-4-31B	43.95 [43.35,44.50]	0.696 $\pm 0.005$	64.21 [63.83,64.60]	0.632 $\pm 0.005$	0.784 $\pm 0.002$	0.836 $\pm 0.002$	-1.279 $\pm 0.016$	0.609 $\pm 0.006$
Granite-4.2-30B	42.81 [42.20,43.43]	0.691 $\pm 0.005$	63.93 [63.54,64.34]	0.617 $\pm 0.005$	0.777 $\pm 0.003$	0.831 $\pm 0.003$	-1.381 $\pm 0.017$	0.597 $\pm 0.006$
Nemotron-3-Ultra	45.04 [44.40,45.69]	0.689 $\pm 0.005$	62.82 [62.40,63.24]	0.635 $\pm 0.005$	0.783 $\pm 0.003$	0.836 $\pm 0.003$	-1.262 $\pm 0.016$	0.609 $\pm 0.006$
T5-Gemma2-4B-4B	45.04 [44.40,45.69]	0.685 $\pm 0.005$	62.83 [62.40,63.24]	0.623 $\pm 0.006$	0.762 $\pm 0.003$	0.833 $\pm 0.003$	-1.306 $\pm 0.018$	0.598 $\pm 0.006$

Table 3: Comparison of all models on the DART test set. In case of BLEU and ChrF++, we present corpus score and bootstrap 95% CI. For the other metrics we present the mean scores and 95% CI through t-distribution

	Faithfulness						Coverage
System	WebNLG17		WebNLG20		DART		WebNLG17		WebNLG20		DART
	DeepSeek	GLM	DeepSeek	GLM	DeepSeek	GLM	DeepSeek	GLM	DeepSeek	GLM	DeepSeek	GLM
T5-Large	0.878	0.925	0.882	0.921	0.860	0.918	0.956	0.965	0.948	0.956	0.951	0.963
T5-Gemma 2	0.840	0.890	0.824	0.868	0.819	0.853	0.954	0.963	0.936	0.945	0.912	0.914
GPT-5-mini	0.956	0.983	0.959	0.989	0.971	0.986	0.992	0.997	0.993	0.996	0.993	0.997
Granite-4.2	0.943	0.969	0.946	0.968	0.949	0.977	0.989	0.993	0.989	0.992	0.987	0.994

Table 4: Faithfulness and coverage scores across six systems, three datasets, and two LLM judges. DeepSeek stands for DeepSeek-V-4.1-Flash and GLM stands for GLM-5.3-Flash.

	Q1: Unsupported facts (%)		Q2: Incorrect Names and Values (%)		Q3: Incorrect Relations (%)		Any question (%)		All three
System	DeepSeek	GLM	DeepSeek	GLM	DeepSeek	GLM	DeepSeek	GLM	DeepSeek	GLM
T5-Large	12.5	7.8	13.4	7.8	10.5	6.8	17.5	12.1	146	72
T5-Gemma2	15.4	11.2	19.4	12.9	13.0	8.8	23.2	16.4	198	119
Gemma-4-31B	3.0	1.5	2.7	1.0	2.3	1.2	4.5	2.4	23	7
Granite-4.2-30B	6.3	4.2	6.5	2.5	4.4	2.5	9.7	5.3	46	24
GPT-5 mini	3.7	2.2	6.6	1.8	2.9	1.1	9.2	3.4	20	7

Table 5: Share of outputs per model (system) where each judge answered "no" to the corresponding question.

Dimension	System	Human Negatives	DeepSeek Precision, Recall, $\kappa$	GLM Precision, Recall, $\kappa$
Faithfulness	T5-Large	14	11/21, 11/14, 0.56	11/14, 11/14, 0.75
Faithfulness	Nemotron-3-Ultra	9	1/6, 1/9, 0.07	0/2, 0/9, -0.03
Coverage	T5-Large	11	10/16, 10/11	9/13, 9/11
Coverage	Nemotron-3-Ultra	1	1/2, 1/1	1/1, 1/1

Table 6: LLM judge agreement with manual annotation on 110 WebNLG T5-Large and Nemotron-3-Ultra outputs (

\kappa

denotes Cohen’s kappa).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

WebNLG	Example
Graph	<H> Abel Hernandez <R> club <T> Uruguay national football team
Target	Abel Hernandez’s club is the Uruguay National football team.
GPT-5-mini	Abel Hernandez is a club for the Uruguay national football team. (Note: If you intended a different relationship phrasing, e.g., “Abel Hernandez plays for the Uruguay national football team,” that would also be correct.)
Graph	<H> 66063 1998 RO1 <R> epoch <T> 2013 11 04
Target	66063 1998 RO1 has an epoch date of 4 November 2013.
GPT-5-mini	From the graph, the subject H has the following relation: In 1998 RO1 66063, epoch R, with T 2013-11-04. However, the given data is a bit ambiguous as RDF. A clearer fluent rendering could be: - In 1998, the resource H has the identifier 66063 and is related to RO1 via the property named epoch, with the value 2013-11-04. - The triple: (H, epoch, 2013-11-04) includes the additional contextual numbers 66063, 1998, RO1, and T. If you can provide the intended predicate names and the exact meaning of 66063, RO1, and the T value, I can produce a more natural fluent sentence. For now a best guess: - The resource H, identified by 66063, originated in 1998 under RO1 and has an epoch timestamp of 2013-11-04. )

Appendix

Table 7: Sample text generated by GPT-5-mini with the prompt "Generate text from graph" .

Model	BLEU	ChrF++
T5-Large on WebNLG17 Ribeiro et al. (2021)	59.70	75.40
Our reproduced results on WebNLG17	58.57	72.54
T5-Large on DART Li et al. (2024)	50.66	n/r
Our reproduced results on DART	47.87	65.11

Appendix

Table 8: Comparison of the published results on WebNLG17 and DART on T5-Large and our reported results. It should be noted that we did not reproduce the evaluation protocol followed by the cited works; the BLEU and ChrF++ implementations used by us are mentioned in section 5.3 .

Figure 1: Mean metric scores per model on WebNLG17, with 95% confidence intervals. For visualization, metric scores are rescaled to

[0,1]

: BARTScore is transformed via

\exp(\cdot)

; BLEU and chrF++ are divided by 100 to match the

[0,1]

range of the remaining metrics, which require no rescaling.

Figure 2: Mean metric scores per model on WebNLG20, with 95% confidence intervals. For visualization, metric scores are rescaled to

[0,1]

: BARTScore is transformed via

\exp(\cdot)

; BLEU and chrF++ are divided by 100 to match the

[0,1]

range of the remaining metrics, which require no rescaling.

Figure 3: Mean metric scores per model on DART, with 95% confidence intervals. For visualization, metric scores are rescaled to

[0,1]

: BARTScore is transformed via

\exp(\cdot)

; BLEU and chrF++ are divided by 100 to match the

[0,1]

range of the remaining metrics, which require no rescaling.

Model	BLEU	ChrF++	ROUGE-L	BLEURT	BERTScore	BARTScore	PARENT F1
GPT-3 Yuan and Färber (2023)	20.36	57.95	45.64	13.39	n/r	n/r	n/r
LLaMA2-7b-chat fine-tuned on PlanGTG He et al. (2025)	46.35	63.92	n/r	n/r	n/r	-1.91	n/r
GPT-5 mini (3-shot prompted)	51.94	70.70*	70.16*	81.38*	0.86	-1.11	0.64
Gemma-4-31B (3-shot prompted)	53.74	71.67*	71.73*	82.01*	0.87	-1.03	0.66
Granite-4.2-30B (3-shot prompted)	52.86	71.48*	70.23*	81.26*	0.86	-1.08	0.66
Nemotron-3-Ultra (3-shot prompted)	55.51	71.69*	72.31*	82.09*	0.87	-1.01	0.66

Appendix

Table 9: Comparison of the recent LLMs with results reported in Yuan and Färber (2023) and He et al. (2025) on WebNLG17. *Scores are multiplied by 100 to match the scale of reported scores.

You are a system that converts RDF triples from a knowledge graph into natural English text.

Instructions:

- The triples describe facts about entities.

- Each RDF triple is in the format: <H> subject <R> predicate <T> object where the tokens <H>, <R>, and <T> are structural markers and must NOT appear in the output.

- Do not refer to entities as "entity H", "entity R", or "entity T". - Relations are undirected; express them in the most natural way. - Combine all triples into a coherent paragraph.

- Use one or more fluent sentences if needed.

- Do NOT use bullet points, numbering, or lists.

Appendix

Table 10: Prompt provided to all decoder-only models.

Figure 4: Few shot result comparison on GPT-5-mini on WebNLG17. For visualization, metric scores are rescaled to

[0,1]

: BARTScore is transformed via

\exp(\cdot)

; BLEU and chrF++ are divided by 100 to match the

[0,1]

range of the remaining metrics, which require no rescaling.

Figure 5: Few shot result comparison on Gemma-4-E4B-it on WebNLG17. For visualization, metric scores are rescaled to

[0,1]

: BARTScore is transformed via

\exp(\cdot)

; BLEU and chrF++ are divided by 100 to match the

[0,1]

range of the remaining metrics, which require no rescaling.

	Fluency
System	WebNLG17		WebNLG20
	DeepSeek	GLM	DeepSeek	GLM
T5-Large	0.943	0.899	0.951	0.925
T5-Gemma2	0.938	0.895	0.949	0.926
GPT-5-mini	0.961	0.934	0.966	0.958
Granite-4.2	0.949	0.915	0.945	0.942

Appendix

Table 11: Fluency scores by system, dataset, and LLM judge. DeepSeek stands for DeepSeek-V-4.1-Flash and GLM stands for GLM-5.3-Flash.

Question 1	Is every fact stated in the text supported by the input graph?
Violation Example	Input graph: <H> A Loyal Character Dancer <R> country <T> United States <H> United States <R> ethnic Group <T> Asian Americans
	Candidate text: A Loyal Character Dancer is published in English in the United States where the Asian Americans are one of the ethnic groups.
	Explanation: The text states that A Loyal Character Dancer is published in English, which is not supported by the input graph.
Question 2	Are all entity names, attribute values, and numerical values reproduced correctly from the graph?
Violation Example	Input graph: <H> A Loyal Character Dancer <R> country <T> United States <H> United States <R> ethnic Group <T> Asian Americans
	Candidate text: The Loyal Character Dancers are from the United States where the Asian Americans are one of the ethnic groups.

Appendix

Table 12: Questions and violation example provided to the judge LLMs for evaluating Faithfulness using BinEval framework

Question 1	Is the text easy to read and understand?
Violation Example	Candidate text: Mount Early is near the head of Scott Glacier on the west side. It is high and stands north of D’Angelo Bluff. It is a solitary volcanic cone.
	Explanation: The text presents the information in a complex manner introducing the location first, then the height and type of landform in the end, which makes it hard to follow.
Question 2	Does the text avoid awkward or repetitive wording?
Violation Example	Candidate text: Interior is an 1895 play. It is a play in rhymed dialogue written by Belgian playwright Maurice Maeterlinck.
	Explanation: The text contains unnecessary repetition of the fact that Interior is a play.
Question 3	Are the sentences in the text connected in a coherent manner and progress naturally?

Appendix

Table 13: Questions and violation example provided to the judge LLMs for evaluating Fluency using BinEval framework

WebNLG	Example
Graph	<H> Austria <R> language <T> Austrian German
Target 1	In Austria, the language is Austrian German.
Target 2	The language in Austria is Austrian German.
Target 3	The spoken language of Austria is Austrian German.
GPT-5-mini	Austrian German is a language spoken in Austria.
Graph	<H> Binignit <R> ingredient <T> Sweet potato <H> Sweet potato <R> division <T> Flowering plant <H> Binignit <R> main Ingredients <T> Sago <H> Sweet potato <R> order <T> Solanales

Appendix

Table 14: Some examples where the text generated by GPT-5-mini conveys the same meaning as the graph and references, but the surface overlap with all the references is low.

WebNLG	Example
Graph	<H> Above the Veil <R> media Type <T> Print
T5-large output	Above the Veil was produced in print
GPTScore	-5.500
Graph	<H> Bananaman <R> broadcasted By <T> BBC
T5-large output	Bananaman was broadcast by the BBC
GPTScore	-3.699

Appendix

Table 15: Some examples where the GPTScore value is low despite the output being correct and high despite the output misses one triple

Triples	$n$	T5-Large		T5-Gemma2 (LoRA)		Gemma-4-31B		Granite-4.2-30B		GPT-5 mini		Nemotron-3-Ultra
		DeepSeek	GLM	DeepSeek	GLM	DeepSeek	GLM	DeepSeek	GLM	DeepSeek	GLM	DeepSeek	GLM
1	454	4.6	3.3	4.6	3.5	2.0	0.9	3.7	1.1	3.7	1.5	2.6	1.3
2	349	12.6	7.7	14.6	8.3	4.9	3.2	8.0	5.4	4.3	1.5	8.0	3.2
3	386	21.5	15.0	27.2	17.6	6.0	2.1	12.4	8.3	10.1	2.8	10.6	4.1
4	363	23.7	15.4	34.2	24.5	5.5	2.8	12.1	4.4	11.8	4.7	11.3	4.7
5	265	30.9	24.2	40.0	32.1	5.3	4.2	13.2	7.2	11.3	3.4	12.5	5.7

Appendix

Table 16: Faithfulness by number of input triples on WebNLG17, evaluated by DeepSeek-V-4.1-Flash and GLM-5.3-Flash.

n

denotes number of graphs

Triples	$n$	T5-Large		T5-Gemma2 (LoRA)		Gemma-4-31B		Granite-4.2-30B		GPT-5 mini		Nemotron-3-Ultra
		DeepSeek	GLM	DeepSeek	GLM	DeepSeek	GLM	DeepSeek	GLM	DeepSeek	GLM	DeepSeek	GLM
1	454	0.9	0.4	0.7	0.9	0.0	0.0	0.4	0.4	0.2	0.0	0.0	0.0
2	349	5.4	3.4	4.9	4.0	2.0	1.4	2.3	1.1	2.0	0.6	1.4	1.1
3	386	13.2	12.4	14.0	11.1	2.8	1.8	5.2	3.9	2.6	1.0	4.1	1.3
4	363	22.3	17.1	20.1	14.3	3.6	1.4	4.4	2.8	3.3	2.2	4.7	3.6
5	265	34.3	29.8	35.5	29.8	6.4	2.3	6.0	2.6	5.3	1.5	6.8	1.1

Appendix

Table 17: Coverage scores by number of input triples on WebNLG17 dataset by LLM judges.

n

denotes number of graphs

Panel of Figure 6 — Figure 6: Spearman correlations between evaluation metrics for each model’s generated outputs on WebNLG17. BinEval1 and BinEval2 denote faithfulness scores assigned by DeepSeek-V4.1-Flash and GLM-5.3-Flash, respectively.

Knowledge Graphs

Knowledge graphs represent entities and relationships as structured, queryable networks for integrating data and supporting reasoning.

See all 232 papers

Knowledge-Graph Grounding Helps LLMs Only for Out-of-Training Knowledge: A Controlled Study on Clinical Question Answering
Jun 21, 2026Madhulatha Mandarapu, Sandeep KunkunuruFactual Knowledge in Language Models LLM Reasoning with Graphs
samyama
KoRe: Compact Knowledge Representations for Large Language Models
May 19, 2026Davide Cavicchini, Fausto Giunchiglia, Jacopo StaianoLLM Grounding Knowledge Augmentation for Language Models
University of Trento, Via Calepina, 14, 38122 Trento TN, Italy
Ground-Truth Subgraphs for Better Training and Evaluation of Knowledge Graph Augmented LLMs
Nov 6, 2025Alberto Cattaneo, Carlo Luschi, Daniel JustusGraph Retrieval-Augmented Generation Synthetic Benchmark Generation
Graphcore AI Research

Jun 21, 2026J/K move · Enter open · S save

Knowledge-Graph Grounding Helps LLMs Only for Out-of-Training Knowledge: A Controlled Study on Clinical Question Answering

Madhulatha Mandarapu, Sandeep Kunkunuru

samyama

Factual Knowledge in Language Models LLM Reasoning with Graphs Clinical QA LLM Grounding Clinical Language Model Evaluation Biomedical KGs+1

Open paper AskPDF arXiv

A recent Nature Medicine study reports that general-purpose frontier LLMs outperform specialized retrieval-augmented clinical tools on medical benchmarks, and that retrieval can hurt strong models. We ask the natural follow-up: does structured knowledge-graph (KG) grounding change this, and when does grounding help at all? We contribute two results. First, a reproduction: the study's headline HealthBench score (~88) is the Consensus variant, not full HealthBench, where frontier models and ideal completions both score ~46-47 under a physician-calibrated grader (agreement 82.5%); we reproduce GPT-5.2 Consensus =90.9 and flag a score-deflating grader bug. Second, a knowledge-boundary result. Using a graph+vector engine (samyama-graph) over the public biomedical KG PrimeKG, neither naive triple retrieval nor an agentic natural-language-to-Cypher loop (82% successful queries) improves MedQA across a weak-to-strong model ladder (all |Delta| <= 3.4). On a synthetic counterfactual KG, and on a hybrid benchmark mixing known and novel facts, the identical pipeline lifts out-of-training accuracy from chance to ~100% (+68 to +79) while adding nothing on known facts (a no-LLM arm answers both). Across three regimes (no-knowledge, graph-aided, hybrid), grounding helps only insofar as the decisive fact lies outside the model's training -- public-KG facts are redundant, private and novel data are where it pays -- matching the study's institutional-data caveat.

Abstract

Knowledge Graphs

Explore similar work

Knowledge-Graph Grounding Helps LLMs Only for Out-of-Training Knowledge: A Controlled Study on Clinical Question Answering

KoRe: Compact Knowledge Representations for Large Language Models

Ground-Truth Subgraphs for Better Training and Evaluation of Knowledge Graph Augmented LLMs