TailNLG: A Multilingual Benchmark Addressing Verbalization of Long-Tail Entities

Lia Draetta; Michael Oliverio; Virginia Ramón-Ferrer; Pier Felice Balestrucci; Flaviana Corallo; Carlos Badenes-Olmedo; Alessandro Mazzei; Marco Antonio Stranisci

cs.CLMar 29, 2026

TailNLG: A Multilingual Benchmark Addressing Verbalization of Long-Tail Entities

Authors: Lia Draetta, Michael Oliverio, Virginia Ramón-Ferrer, Pier Felice Balestrucci, Flaviana Corallo, Carlos Badenes-Olmedo, Alessandro Mazzei, Marco Antonio Stranisci,

Organizations: University of Turin, Italy · Universidad Politécnica de Madrid, Spain

Abstract

The automatic verbalization of structured knowledge is a key task for making knowledge graphs accessible to non-expert users and supporting retrieval-augmented generation systems. Although recent advances in Data-to-Text generation have improved multilingual coverage, little attention has been paid to potential biases in the verbalization of rare entities, frequently known as long-tail entities. In this work, we present the first systematic study of long-tail entities in Data-to-Text generation. We introduce TailNLG, a new multilingual benchmark in English, Italian, and Spanish, built from Wikidata and covering entities with varying levels of popularity. We evaluate three different families of large language models in zero-shot settings and compare their performance on rare versus common entities, as well as against the established WebNLG benchmark. Our results reveal a consistent bias against long-tail entities: embedding-based scores are lower, and model uncertainty is higher for rare entities. We further show that the impact of long-tail entities varies across models and languages, and that existing evaluation metrics do not consistently capture these differences, highlighting the need for more reliable evaluation frameworks.

Figures & tables

Figure 1: Distribution of entities by claims in the Wikidata category Artist . A small number of entities are associated with many claims, while the majority (long-tail) are associated with few relations. The red dashed line indicates the Pareto cut-off, while the green dashed line marks the head–tail threshold

Figure 2: PPL distributions for head and long-tail entities across models (log10 scale). Boxes represent the interquartile range (IQR, Q1–Q3) with the median shown as a line inside the box. Statistical significance was assessed using the Mann-Whitney U test : *** p < 0.001, ns = not significant.

$\textbf{Source}\rightarrow\textbf{Target}$	Accuracy	Fluency
$es\rightarrow en$	$4.45\pm 0.58$	$4.99\pm 0.07$
$en\rightarrow es$	$4.57\pm 0.53$	$4.95\pm 0.15$
$it\rightarrow en$	$4.71\pm 0.49$	$4.89\pm 0.32$
$en\rightarrow it$	$4.72\pm 0.48$	$4.97\pm 0.12$
$es\rightarrow it$	$4.51\pm 0.49$	$4.91\pm 0.22$
$it\rightarrow es$	$4.63\pm 0.49$	$4.93\pm 0.20$

Table 1: Average scores (

0

5

) and standard deviation of the human evaluation on Accuracy e Fluency

Type	Lang	BERT r $\uparrow$	BLEU $\uparrow$	chrF $\uparrow$	R-1 $\uparrow$	R-2 $\uparrow$	R-L $\uparrow$	PPL $\downarrow$
				Llama-8B
Long-tail	en	0.61	0.25	60.39	0.66	0.43	0.56	93.36
	es	0.58	0.17	53.30	0.58	0.35	0.49	222.16
	it	0.46	0.13	47.73 *	0.49 *	0.29 *	0.43 *	88.06
Head	en	0.65 *	0.26	61.20	0.67	0.44	0.57	59.03 *
	es	0.60	0.19	53.23	0.58	0.36	0.48	94.52 *

Table 2: Results on TailNLG long-tail vs head entitities verbalizations by data type and language. Metrics include BERTScore Rescaled (BERT r ), BLEU, chrF++ (chrF), ROUGE 1 (R-1) , ROUGE 2 (R2), ROUGE L (R-L) and Perplexity (PPL). Asterisks ( ∗ ) denote a statistically significant difference (

p<0.05

Llama-8B
Type	BERT r $\uparrow$	BLEU $\uparrow$	chrF $\uparrow$	R-1 $\uparrow$	R-2 $\uparrow$	R-L $\uparrow$	PPL $\downarrow$
Long-tail	0.55	0.18	53.81	0.58	0.36	0.50	134.53
WebNLG	0.53	0.21 *	53.77	0.65 *	0.43 *	0.53 *	68.23 *
Gemma-12B
Long-tail	0.62	0.24	60.38	0.64	0.42	0.55	455.57
WebNLG	0.65 *	0.30 *	63.63 *	0.73 *	0.50 *	0.59 *	290.10 *

Table 3: Performance comparison of models between long-tail triples from TailNLG and WebNLG test set. Metrics: BERTScore Rescaled (BERT r ), BLEU, chrF++ (chrF), ROUGE-1 (R-1), ROUGE-2 (R-2), ROUGE-L (R-L), and Perplexity (PPL). Asterisks ( ∗ ) denote a statistically significant difference (

p<0.05

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Examples of Chain, sibling and mixed triples configuration

Category	Count
Politician	47
Airport	46
Model	30
SportsTeam	37
Building	47
Company	44

Appendix

Table 4: Distribution of data entries across categories and entity types.

Language	Gold	Silver	Total
English	$168$	$447$	$615$
Spanish	$173$	$442$	$615$
Italian	$384$	$231$	$615$
Total	$725$	$1,120$	$1,845$

Appendix

Table 5: Distribution of gold and silver verbalizations by language in TailNLG.

Source	Target	$\rho$	$r$
es	en	0.23	0.25
en	es	0.32*	0.27
it	en	0.15	0.02
en	it	0.56*	0.64
es	it	0.26	0.22
it	es	0.53*	0.49*

Appendix

Table 6: Spearman (

\rho

) and Pearson (

r

) correlation coefficients across language pairs. Source and Target denote the source language and the target language into which the source is translated, respectively. Asterisks (*) denote a statistically significant difference (p < 0.05).

Model	Type	BERT r	BLEU $\uparrow$	chrF $\uparrow$	R-1 $\uparrow$	R-2 $\uparrow$	R-L $\uparrow$	PPL $\downarrow$
Qwen-3B	Long-tail	0.50	0.13	50.10	0.53	0.30	0.45	182.87
Qwen-3B	Top-head	0.51	0.13	49.12	0.52	0.29	0.44	70.45 *
Qwen-7B	Long-tail	0.56	0.19	54.27	0.58	0.35	0.49	138.70
Qwen-7B	Top-head	0.58	0.18	53.41	0.57	0.35	0.48	87.96 *
Gemma-4b	Long-tail	0.53	0.17	54.32	0.57	0.35	0.48	4724.94
Gemma-4b	Top-head	0.56	0.17	53.48	0.56	0.34	0.46	3488.00 *

Appendix

Table 7: Comparison of models by type. Asterisks ( ∗ ) denote a statistically significant difference with

p<0.05

Model	Lang	BERT r	BLEU $\uparrow$	chrF $\uparrow$	R-1 $\uparrow$	R-2 $\uparrow$	R-L $\uparrow$	PPL $\downarrow$
Llama-3B	en	0.58 *	0.21 *	56.67 *	0.63 *	0.40 *	0.53 *	73.51 *
	es	0.46	0.11	46.40	0.47	0.26	0.39	114.99
	it	0.44	0.08	44.26	0.42	0.22	0.35	81.98
Llama-8B	en	0.63 *	0.25 *	60.72 *	0.66 *	0.43 *	0.56 *	79.37 *
	es	0.59	0.18	53.27	0.58	0.35	0.49	170.14
	it	0.45	0.13	46.85	0.48	0.28	0.42	80.23

Appendix

Table 8: Average scores for different languages for each model (zero-shot). Asterisks ( ∗ ) denote a statistically significant difference with

p<0.05

Model	Quality	BERT r $\uparrow$	BLEU $\uparrow$	chrF $\uparrow$	R-1 $\uparrow$	R-2 $\uparrow$	R-L $\uparrow$	PPL $\downarrow$
Llama-3B	gold	0.46	0.12	48.04	0.48	0.27	0.40	95.30
Llama-3B	silver	0.52 *	0.14	49.80	0.52 *	0.30 *	0.44 *	86.86 *
Llama-8B	gold	0.52	0.18	52.68	0.50	0.55 *	0.48	90.94
Llama-8B	silver	0.58 *	0.19	54.22	0.59 *	0.36	0.50 *	122.10 *
Qwen-3B	gold	0.50	0.13	50.62	0.53	0.30	0.44	94.12
Qwen-3B	silver	0.51	0.13	49.08	0.52	0.29	0.45	165.11

Appendix

Table 9: Average scores for different quality levels (silver, gold) for each model (zero-shot). Asterisks ( ∗ ) denote a statistically significant difference with

p<0.05

Llama-3B
Type	Quality	BERT r	BLEU $\uparrow$	chrF $\uparrow$	R-1 $\uparrow$	R-2 $\uparrow$	R-L $\uparrow$	PPL $\downarrow$
Long-tail	Silver	0.50 *	0.14	49.63	0.52	0.30	0.44	89.70 *
Long-tail	Gold	0.47	0.13	49.95	0.50	0.29	0.42	122.69
Top-head	Silver	0.53 *	0.16 *	50.04 *	0.52 *	0.31 *	0.44 *	82.59
Top-head	Gold	0.46	0.11	45.40	0.45	0.25	0.38	57.57
Llama-8B

Appendix

Table 10: Performance comparison of models broken down by entity type (long-tail / top-head) and annotation quality (silver / gold). Asterisks ( ∗ ) denote a statistically significant difference between silver and gold within the same type (

p<0.05

, Mann-Whitney U test).

Llama-3B
Type	Lang	BERT r	BLEU $\uparrow$	chrF $\uparrow$	R-1 $\uparrow$	R-2 $\uparrow$	R-L $\uparrow$	PPL $\downarrow$
Long-tail	en	0.57	0.21	56.58	0.62	0.39	0.53	86.17
	es	0.45	0.11	47.07	0.48	0.26	0.40	144.99
	it	0.44	0.09	45.61	0.44	0.23	0.37	75.80
	total	0.49	0.13	49.75	0.51	0.30	0.43	102.32
WebNLG	en	0.56	0.23	57.17	0.71 *	0.47 *	0.56	53.67 *

Appendix

Table 11: Performance comparison of models on long-tail triples from TailNLG and WebNLG test set. Asterisks ( ∗ ) denote a statistically significant difference with

p<0.05

Text Generation

Text generation studies methods for producing coherent, useful language from prompts, data, and dialogue context.

See all 422 papers

All Entities are Not Created Equal: Examining the Long Tail for Ultra-Fine Entity Typing
Oct 22, 2024Advait Deshmukh, Ashwin Umadi, Dananjay Srinivas +1Language Model Pretraining Long-Tail Learning
University of Colorado Boulder
The Chronicles of RiDiC: Generating Datasets with Controlled Popularity Distribution for Long-form Factuality Evaluation
Mar 11, 2026Pavel Braslavski, Dmitrii Iarosh, Nikita Sushko +4Language Model Generation Evaluation Factual Knowledge in Language Models
HSE University · Ural Federal University · ITMO University +3
Think Before You Link: Rarity, Reasoning, and Retrieval in Multilingual Entity Linking
Sep 9, 2026Parinthapat Pengpun, Simran Khanuja, Graham NeubigMultimodal Grounding VLM Reasoning
Carnegie Mellon University

Oct 22, 2024J/K move · Enter open · S save

All Entities are Not Created Equal: Examining the Long Tail for Ultra-Fine Entity Typing

Advait Deshmukh, Ashwin Umadi, Dananjay Srinivas, Maria Leonor Pacheco

University of Colorado Boulder

Language Model Pretraining Long-Tail Learning Knowledge Augmentation for Language Models Factual Knowledge in Language Models

Open paper AskPDF arXiv

Due to their capacity to acquire world knowledge from large corpora, pre-trained language models (PLMs) are extensively used in ultra-fine entity typing tasks where the space of labels is extremely large. In this work, we explore the limitations of the knowledge acquired by PLMs by proposing a novel heuristic to approximate the pre-training distribution of entities when the pre-training data is unknown. Then, we systematically demonstrate that entity-typing approaches that rely solely on the parametric knowledge of PLMs struggle significantly with entities at the long tail of the pre-training distribution, and that knowledge-infused approaches can account for some of these shortcomings. Our findings suggest that we need to go beyond PLMs to produce solutions that perform well for infrequent entities.

Abstract

Text Generation

Explore similar work

All Entities are Not Created Equal: Examining the Long Tail for Ultra-Fine Entity Typing

The Chronicles of RiDiC: Generating Datasets with Controlled Popularity Distribution for Long-form Factuality Evaluation

Think Before You Link: Rarity, Reasoning, and Retrieval in Multilingual Entity Linking