When More Metadata Misleads: Probing Field-Level Grounding in LLM-Generated Museum Descriptions

Clemencia Siro; Laura Hollink

cs.CLOct 22, 2026

When More Metadata Misleads: Probing Field-Level Grounding in LLM-Generated Museum Descriptions

Authors: Clemencia Siro, Laura Hollink

Organizations: Centrum Wiskunde & Informatica (CWI), Amsterdam, The Netherlands

Abstract

Metadata-conditioned generation is often framed as a data-to-text task in which additional input fields should improve grounding. Museum description generation challenges this assumption because catalogue records are heterogeneous, partially visual, and unevenly documented across geographic regions. We study how metadata composition shapes the generation of artefact descriptions through a two-phase framework. Phase 1 analyses natural metadata variation in a large museum collection, where field count, field type, image availability, and Place co-vary. Phase 2 introduces controlled field probes that hold object identity fixed while varying metadata quantity, field type, and image access. Our results show that metadata quantity alone does not explain generation behaviour. Field type determines whether models ground, omit, or infer information, while images can both support visual grounding and increase unsupported interpretation. More broadly, our findings show that metadata-conditioned data-to-text evaluation should assess field- and claim-level support, rather than relying only on aggregate input coverage or reference similarity.

Figures & tables

Place	Assoc. name	Culture	Find spot	Inscription	Materials	Prod. date	Subjects	Technique
Overall	$44.0$	$100.0$	$68.2$	$41.1$	$99.8$	$86.7$	$54.3$	$61.7$
British Isles	$1.0$	$100.0$	$98.0$	$1.0$	$100.0$	$93.0$	$6.0$	$66.0$
Central America	$1.0$	$100.0$	$100.0$	$0.0$	$100.0$	$22.0$	$3.0$	$25.0$
China	$0.0$	$100.0$	$100.0$	$0.0$	$100.0$	$100.0$	$12.0$	$100.0$
Egypt	$100.0$	$100.0$	$100.0$	$100.0$	$100.0$	$16.0$	$8.0$	$100.0$
Greece	$61.0$	$100.0$	$86.0$	$16.0$	$100.0$	$100.0$	$100.0$	$89.0$

Table 1: Percentage of metadata available per place and overall; overall availability is computed across all

10{,}250

objects.

Cond.	Fields provided	$\|F\|$	Img.
R	All available	6–9	✓
T	All available	6–9	×
S	Materials, Culture, Prod. date	3	✓
S_noi	Materials, Culture, Prod. date	3	×
S_ctx4	S + Find spot	4	✓
S_ctx5	S + Find spot, Place	5	✓

Table 2: Phase 2 controlled prompt conditions. Matched-count arm conditions compare field type while holding object identity, prompt template, and image access fixed.

Figure 1: Mean BERTScore and generated word count as a function of the number of populated metadata fields

n

, across Phase 1 (left) and Phase 2 (right).

Figure 2: Mean primary fields’ (coverage, hallucination, silence; left) and bonus fields’ (correct, wrong, silent; right) as a function of observed metadata field count

n

	Primary fields			Bonus fields
Cond.	Cov	Hall	Sil	Cor	Wro	Sil b
S	0.933	0.058	0.009	0.092	0.223	0.685
S_noi	0.885	0.091	0.024	0.055	0.146	0.799
S_ctx4	0.949	0.040	0.011	0.116	0.308	0.577
S_ctx5	0.941	0.044	0.016	0.128	0.361	0.511
S_ctx6	0.937	0.042	0.021	0.218	0.421	0.360

Table 3: Primary fields rates (Cov, Hall, Sil) and bonus completion rates (Cor, Wro, Sil b ) by condition, averaged across all models; Bold indicates best value per column: highest for Cov, Cor, Sil b ; lowest for Hall, Sil, Wro. R and T contain no withheld fields by design.

	Primary			Bonus
Field	Cov	Hall	Sil	Cor	Wro	Sil b
Subjects	74.9	24.2	0.9	14.6	41.8	43.6
Technique	78.3	21.3	0.3	9.8	51.7	38.5
Assoc name	87.1	11.9	1.0	5.5	6.5	88.0
Materials	88.7	10.9	0.4	—	—	—
Inscription	87.5	9.1	3.4	16.7	10.6	72.6

Table 4: Primary fields rates (Cov, Hall, Sil) and bonus completion rates (Cor, Wro, Sil b ) by metadata field, averaged across all conditions and models.

	Primary fields			Bonus fields
region	Cov	Hall	Sil	Cor	Wro	Sil b
Central A.	0.877	0.112	0.010	0.050	0.304	0.646
China	0.859	0.094	0.047	0.062	0.268	0.669
British I.	0.936	0.054	0.010	0.059	0.232	0.710
Japan	0.918	0.072	0.010	0.114	0.185	0.701
India	0.901	0.082	0.017	0.137	0.133	0.730

Table 5: Primary and bonus fields results from the claim-level factuality.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Place	3	4	5	6	7	8	9
British Isles	17	45	276	616	54	5	–
Iran	15	18	155	352	332	125	2
Japan	5	239	225	200	180	20	–
Central America	–	304	132	48	3	1	–
India	–	7	77	50	621	112	1
Iraq	–	–	34	400	355	159	52

Appendix

Table 6: Distribution of objects by Place and number of available metadata fields. Bold cells indicate place-relative high-completeness tiers used for Phase 2 sampling.

Model	Version	Task	API
GPT-4o ( OpenAI, 2023 )	gpt-4o-2024-08-06	Generation	OpenAI
Llama-4-Maverick ( Meta, 2025 )	Llama-4-Maverick-17B-128E-Instruct-FP8	Generation	DeepInfra
Llama-3.2-Vision ( Llama Team, AI @ Meta, 2024 )	Llama-3.2-11B-Vision-Instruct	Generation	DeepInfra
Qwen2.5-VL ( Bai et al., 2025 )	Qwen2.5-VL-32B-Instruct	Generation	DeepInfra
Gemini 2.5 Flash ( Gemini Team, 2025 )	gemini-2.5-flash	Evaluation	Google

Appendix

Table 7: Model versions and APIs. LLaMA-3.2-Vision is restricted to the four base Phase 2 conditions. Gemini 2.5 Flash is used only for nugget extraction and verification.

Field	Template	Example $h_{f}$
Materials	This object is made of [ $v$ ].	…made of bronze.
Culture	This object belongs to the [ $v$ ] culture.	…Roman culture.
Production date	This object was produced [ $v$ ].	…produced 100 BC.
Find spot	This object was found at [ $v$ ].	…found at Pompeii.
Place	This object originates from [ $v$ ].	…from Roman Empire.
Associated name	This object is associated with [ $v$ ].	…associated with Augustus.

Appendix

Table 8: NLI hypothesis templates and instantiated examples. [

v

] is substituted with ground-truth value

v_{f}

Error type	Definition
wrong_material	Incorrect material stated (e.g., iron for ground-truth bronze )
wrong_date	Incorrect production date, century, or period
wrong_culture	Incorrect cultural or regional attribution
wrong_place	Incorrect geographic or archaeological provenance
wrong_subject	Incorrect iconographic or figurative content
wrong_technique	Incorrect production or manufacturing technique

Appendix

Table 9: Hallucination error taxonomy. Each hallucinated nugget receives one error type and a severity label.

Model	BLEU	ROUGE-L	METEOR	BERTScore
GPT-4o	0.240 †	0.159 †	0.353 †	0.856 †
LLaMA-Maverick	0.309	0.196	0.389	0.860
LLaMA-3.2-Vision	0.229 †	0.159 †	0.334 †	0.848 †
Qwen2.5-32B	0.152 †	0.105 †	0.277 †	0.841 †

Appendix

Table 10: Phase 1 surface metrics ecaluation of generated descriptions. Bold = best score.

\dagger

= significantly worse than best model under paired bootstrap and Wilcoxon tests (

p{<}0.05

Place	GPT-4o	LLaMA-3.2-Vision	LLaMA-Maverick	Qwen2.5-32B	Mean
British Isles	0.862	0.850	0.867	0.847	0.857
Central America	0.872	0.857	0.875	0.858	0.866
China	0.870	0.855	0.872	0.853	0.863
Egypt	0.850	0.845	0.855	0.837	0.847
Greece	0.846	0.842	0.851	0.831	0.843
India	0.852	0.849	0.859	0.839	0.850

Appendix

Table 11: BERTScore by place and model. Bold indicates the best model per place; gray shading indicates the lowest score per place. The mean column summarizes overall difficulty per region. Differences across places are significant for all models (

p<0.001

, Kruskal–Wallis and permutation tests).

Region	3	4	5	6	7	8	9
British Isles ∗	0.863	0.864	0.857	0.856	0.852	0.854	–
Central America †	–	0.865	0.865	0.865	0.858	–	–
China †	–	–	–	0.863	0.860	0.858	–
Egypt †	–	–	–	–	0.846	0.846	0.841
Greece ∗	–	–	0.853	0.851	0.845	0.840	0.835
India ∗	–	0.861	0.868	0.862	0.847	0.845	–

Appendix

Table 12: BERTScore by region and number of available metadata fields, averaged across all models. Bold indicates the highest score per region. ∗ scores differ significantly across field counts for all models (Kruskal–Wallis,

p<0.05

). † significant for at least one model. Roman Empire shows no significant variation across field counts.

Region	Spearman’s $\rho$	Slope
India	$-0.411$	21.0
Greece	$-0.350$	20.6
Iran	$-0.337$	14.9
Egypt	$-0.089$	14.1
Iraq	$-0.155$	13.8
British Isles	$-0.164$	8.8

Appendix

Table 13: Within-region relationships between metadata field count and BERTScore in Phase 1. The correlation column reports Spearman’s

\rho

between field count and BERTScore. The slope column reports the average increase in output length, measured as additional words per metadata field. † not significant at

p<0.05

; all other correlations are significant.

Model	BLEU-1	ROUGE-L	METEOR	BERTScore
GPT-4o	0.381	0.271	0.251	0.884
LLaMA-Maverick	0.327	0.244	0.213	0.877
Qwen2.5-32B	0.398	0.246	0.247	0.875

Appendix

Table 14: Phase 2 model-level surface-metric performance averaged across Phase 2 conditions. Values are condition-level means averaged over all probing conditions.

region	GPT-4o	LLaMA-Mav.	LLaMA-Vis.	Qwen2.5	Overall
Central America	0.9091	0.9033	0.8769	0.8823	0.8957
China	0.9084	0.8952	0.8762	0.8767	0.8914
British Isles	0.9001	0.8967	0.8749	0.8821	0.8908
Japan	0.8991	0.8947	0.8761	0.8802	0.8895
India	0.8964	0.8924	0.8760	0.8832	0.8890
Iraq	0.8978	0.8915	0.8739	0.8800	0.8879

Appendix

Table 15: Phase 2 mean BERTScore by region and model. Place differences are significant for all models (Kruskal–Wallis, all

p<0.001

; Bold indicates highest value per column. LLaMA-Vision ran on conditions R, S, S_noi, T only.

	Field count ( $n$ )		Word count
region	$r$ ***	$\rho$ ***	$r$ ***	$\rho$ ***
Central America	$+0.248$	$+0.257$	$-0.697$	$-0.704$
China	$+0.216$	$+0.223$	$-0.700$	$\mathbf{-0.711}$
British Isles	$+0.223$	$+0.263$	$-0.647$	$-0.618$
Japan	$+0.320$	$+0.363$	$-0.592$	$-0.578$
India	$+0.359$	$+0.417$	$-0.424$	$-0.398$

Appendix

Table 16: Phase 2 Pearson (

r

) and Spearman (

\rho

) correlations between BERTScore and number of metadata fields

n

(left panel) and generated word count (right panel), computed per region across all conditions and models. All correlations are significant at

p<0.001

(***). Bold marks the strongest correlation per column.

Figure 3: Mean BERTScore and generated word count as a function of observed metadata field count

n

for each of the ten regions in Phase 2, averaged across all models and conditions.

	S_ctx arm			S_cont arm
Field	Cor	Wro	Sil	Cor	Wro	Sil
Technique	10.6	51.1	38.3	9.8	56.8	33.4
Subjects	17.9	49.7	32.4	—	—	—
Inscription	27.2	13.3	59.5	10.6	9.4	80.1
Assoc name	2.9	4.3	92.9	8.1	7.7	84.2
Place	11.2	20.2	68.5	3.7	2.5	93.8

Appendix

Table 17: Phase 2 bonus field rates split by arm type: the S_ctx arm and S_cont arm . All values are percentages. Bold indicates best value per column within each arm: highest for Cor and Sil; lowest for Wro. Fields absent from a arm are shown as —.

Figure 4: Nugget-verified field outcomes by region and field, averaged across all models and conditions. Each row pairs a primary outcome (field provided, left) with its corresponding bonus outcome (field withheld, right): coverage with bonus correct (top), hallucination with bonus wrong (middle), and silence with bonus silent (bottom). Blue panels indicate outcomes where higher values are better; yellow–red panels indicate outcomes where higher values are worse. Hatched cells indicate fewer than 15 observations.

Error pattern	Main fields	Mechanism	Example pattern
Field substitution	Place , Find spot	The model transfers a value from one metadata field into another, especially using an excavation site, city, or acquisition location as the broader Place value.	Find spot : Babylon / Nineveh / Marcianopolis $\rightarrow$ Place .
Geographic over-specification	Place	The model narrows a broad catalogue place label into a more specific region, city, or modern location that is not supported by the target field value.	Central America $\rightarrow$ northern Peten / Guatemala; Roman Empire $\rightarrow$ Rome.
Material-to-object inference	Materials	The model uses a material label to infer an object type or function, rather than restricting the claim to material composition.	Earthenware $\rightarrow$ vessel; stone $\rightarrow$ stele; gold $\rightarrow$ coin / jewellery.
Field-boundary confusion	Materials , Technique	The model assigns a value to a neighbouring field, producing a plausible claim that violates the catalogue schema.	Glaze or velvet treated as material when recorded as technique.
Art-historical over-specification	Technique	The model refines a broad catalogue technique into a more specific art-historical technique using domain priors or visual cues.	Painted $\rightarrow$ red-figure / black-figure.
Visual elaboration beyond vocabulary	Subjects	The model adds visible motifs, attributes, posture, or iconographic details not present in the controlled subject vocabulary.	Bird; phoenix $\rightarrow$ foliage, vines, secondary motifs.

Appendix

Table 18: Qualitative schema-boundary error patterns identified from hallucinated and silent field-level outcomes. The examples illustrate recurring mechanisms rather than exhaustive error categories.

Figure 5: Generated descriptions for three objects across conditions S (sparse floor), S_CONT6 (sparse floor plus Subjects , Technique , Inscription ), and R (all available fields with image), alongside the museum reference description. Each row shows the same object; descriptions are truncated to 100 words.

Figure 6: Representative GPT-4o generated descriptions under condition R (all available fields with image) for one object per civilisation. Each cell shows the museum object and the generated description.

Grounded Text Generation

Grounded text generation produces language tied to verifiable evidence, structured data, or perceptual context to limit unsupported claims.

See all 31 papers

Faithful or Findable? Evaluating LLM-Generated Metadata for RDF Dataset Search
Jul 7, 2026Riccardo Terrenzi, Serkan AyvazIR Evaluation Synthetic Data Evaluation
University of Southern Denmark Sønderborg, Denmark
Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images
Apr 8, 2026Yuechen Jiang, Enze Zhang, Md Mohsinul Kabir +4Cultural Knowledge in Language Models VLM Evaluation
University of Manchester Manchester, UK · School of Artificial Intelligence, Wuhan University Wuhan, Hubei, China · Getty Conservation Institute Los Angeles, CA, USA
Natural Language Access to Domain-Specific Metadata: A Reusable Framework for LLM Query Generation
Jul 20, 2026Blake G. Fitch, Cato Elia KurtzKnowledge Representation Knowledge Graph Question Answering
Max Planck Institute for Biological Cybernetics Tübingen, Germany

Jul 7, 2026J/K move · Enter open · S save

Faithful or Findable? Evaluating LLM-Generated Metadata for RDF Dataset Search

Riccardo Terrenzi, Serkan Ayvaz

University of Southern Denmark Sønderborg, Denmark

IR Evaluation Synthetic Data Evaluation Text Generation

Open paper AskPDF arXiv

Dataset search depends heavily on metadata, making LLM-generated metadata a consequential form of synthetic content in retrieval systems. We study six metadata-generation settings for RDF datasets, ranging from simple rewriting to profile-grounded and agentic graph-based generation, and evaluate them jointly for retrieval effectiveness and faithfulness. Unconstrained metadata rewriting delivers the strongest retrieval gains over the original metadata, but it is also the least faithful, showing that search improvements can be driven by unsupported semantic expansion. More grounded settings substantially improve faithfulness, and profile-grounded rewriting provides the most balanced trade-off between retrieval effectiveness and grounding. These findings position synthetic metadata as a system-level IR problem in which effectiveness, provenance, and trust must be evaluated together.

Abstract

Grounded Text Generation

Explore similar work

Faithful or Findable? Evaluating LLM-Generated Metadata for RDF Dataset Search

Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images

Natural Language Access to Domain-Specific Metadata: A Reusable Framework for LLM Query Generation