Metadata-conditioned generation is often framed as a data-to-text task in which additional input fields should improve grounding. Museum description generation challenges this assumption because catalogue records are heterogeneous, partially visual, and unevenly documented across geographic regions. We study how metadata composition shapes the generation of artefact descriptions through a two-phase framework. Phase 1 analyses natural metadata variation in a large museum collection, where field count, field type, image availability, and Place co-vary. Phase 2 introduces controlled field probes that hold object identity fixed while varying metadata quantity, field type, and image access. Our results show that metadata quantity alone does not explain generation behaviour. Field type determines whether models ground, omit, or infer information, while images can both support visual grounding and increase unsupported interpretation. More broadly, our findings show that metadata-conditioned data-to-text evaluation should assess field- and claim-level support, rather than relying only on aggregate input coverage or reference similarity.
Figures & tables
Place
Assoc. name
Culture
Find spot
Inscription
Materials
Prod. date
Subjects
Technique
Overall
44.0
100.0
68.2
41.1
99.8
86.7
54.3
61.7
British Isles
1.0
100.0
98.0
1.0
100.0
93.0
6.0
66.0
Central America
1.0
100.0
100.0
0.0
100.0
22.0
3.0
25.0
China
0.0
100.0
100.0
0.0
100.0
100.0
12.0
100.0
Egypt
100.0
100.0
100.0
100.0
100.0
16.0
8.0
100.0
Greece
61.0
100.0
86.0
16.0
100.0
100.0
100.0
89.0
Table 1: Percentage of metadata available per place and overall; overall availability is computed across all 10,250 objects.
Cond.
Fields provided
∣F∣
Img.
R
All available
6–9
✓
T
All available
6–9
×
S
Materials, Culture, Prod. date
3
✓
S_noi
Materials, Culture, Prod. date
3
×
S_ctx4
S + Find spot
4
✓
S_ctx5
S + Find spot, Place
5
✓
Table 2: Phase 2 controlled prompt conditions. Matched-count arm conditions compare field type while holding object identity, prompt template, and image access fixed.
Figure 1: Mean BERTScore and generated word count as a function of the number of populated metadata fields n , across Phase 1 (left) and Phase 2 (right).
Figure 2: Mean primary fields’ (coverage, hallucination, silence; left) and bonus fields’ (correct, wrong, silent; right) as a function of observed metadata field count n .
Primary fields
Bonus fields
Cond.
Cov
Hall
Sil
Cor
Wro
Sil b
S
0.933
0.058
0.009
0.092
0.223
0.685
S_noi
0.885
0.091
0.024
0.055
0.146
0.799
S_ctx4
0.949
0.040
0.011
0.116
0.308
0.577
S_ctx5
0.941
0.044
0.016
0.128
0.361
0.511
S_ctx6
0.937
0.042
0.021
0.218
0.421
0.360
Table 3: Primary fields rates (Cov, Hall, Sil) and bonus completion rates (Cor, Wro, Sil b ) by condition, averaged across all models; Bold indicates best value per column: highest for Cov, Cor, Sil b ; lowest for Hall, Sil, Wro. R and T contain no withheld fields by design.
Primary
Bonus
Field
Cov
Hall
Sil
Cor
Wro
Sil b
Subjects
74.9
24.2
0.9
14.6
41.8
43.6
Technique
78.3
21.3
0.3
9.8
51.7
38.5
Assoc name
87.1
11.9
1.0
5.5
6.5
88.0
Materials
88.7
10.9
0.4
—
—
—
Inscription
87.5
9.1
3.4
16.7
10.6
72.6
Table 4: Primary fields rates (Cov, Hall, Sil) and bonus completion rates (Cor, Wro, Sil b ) by metadata field, averaged across all conditions and models.
Primary fields
Bonus fields
region
Cov
Hall
Sil
Cor
Wro
Sil b
Central A.
0.877
0.112
0.010
0.050
0.304
0.646
China
0.859
0.094
0.047
0.062
0.268
0.669
British I.
0.936
0.054
0.010
0.059
0.232
0.710
Japan
0.918
0.072
0.010
0.114
0.185
0.701
India
0.901
0.082
0.017
0.137
0.133
0.730
Table 5: Primary and bonus fields results from the claim-level factuality.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Place
3
4
5
6
7
8
9
British Isles
17
45
276
616
54
5
–
Iran
15
18
155
352
332
125
2
Japan
5
239
225
200
180
20
–
Central America
–
304
132
48
3
1
–
India
–
7
77
50
621
112
1
Iraq
–
–
34
400
355
159
52
Appendix
Table 6: Distribution of objects by Place and number of available metadata fields. Bold cells indicate place-relative high-completeness tiers used for Phase 2 sampling.
Model
Version
Task
API
GPT-4o ( OpenAI, 2023 )
gpt-4o-2024-08-06
Generation
OpenAI
Llama-4-Maverick ( Meta, 2025 )
Llama-4-Maverick-17B-128E-Instruct-FP8
Generation
DeepInfra
Llama-3.2-Vision ( Llama Team, AI @ Meta, 2024 )
Llama-3.2-11B-Vision-Instruct
Generation
DeepInfra
Qwen2.5-VL ( Bai et al., 2025 )
Qwen2.5-VL-32B-Instruct
Generation
DeepInfra
Gemini 2.5 Flash ( Gemini Team, 2025 )
gemini-2.5-flash
Evaluation
Google
Appendix
Table 7: Model versions and APIs. LLaMA-3.2-Vision is restricted to the four base Phase 2 conditions. Gemini 2.5 Flash is used only for nugget extraction and verification.
Field
Template
Example hf
Materials
This object is made of [ v ].
…made of bronze.
Culture
This object belongs to the [ v ] culture.
…Roman culture.
Production date
This object was produced [ v ].
…produced 100 BC.
Find spot
This object was found at [ v ].
…found at Pompeii.
Place
This object originates from [ v ].
…from Roman Empire.
Associated name
This object is associated with [ v ].
…associated with Augustus.
Appendix
Table 8: NLI hypothesis templates and instantiated examples. [ v ] is substituted with ground-truth value vf .
Error type
Definition
wrong_material
Incorrect material stated (e.g., iron for ground-truth bronze )
wrong_date
Incorrect production date, century, or period
wrong_culture
Incorrect cultural or regional attribution
wrong_place
Incorrect geographic or archaeological provenance
wrong_subject
Incorrect iconographic or figurative content
wrong_technique
Incorrect production or manufacturing technique
Appendix
Table 9: Hallucination error taxonomy. Each hallucinated nugget receives one error type and a severity label.
Model
BLEU
ROUGE-L
METEOR
BERTScore
GPT-4o
0.240 †
0.159 †
0.353 †
0.856 †
LLaMA-Maverick
0.309
0.196
0.389
0.860
LLaMA-3.2-Vision
0.229 †
0.159 †
0.334 †
0.848 †
Qwen2.5-32B
0.152 †
0.105 †
0.277 †
0.841 †
Appendix
Table 10: Phase 1 surface metrics ecaluation of generated descriptions. Bold = best score. † = significantly worse than best model under paired bootstrap and Wilcoxon tests ( p<0.05 ).
Place
GPT-4o
LLaMA-3.2-Vision
LLaMA-Maverick
Qwen2.5-32B
Mean
British Isles
0.862
0.850
0.867
0.847
0.857
Central America
0.872
0.857
0.875
0.858
0.866
China
0.870
0.855
0.872
0.853
0.863
Egypt
0.850
0.845
0.855
0.837
0.847
Greece
0.846
0.842
0.851
0.831
0.843
India
0.852
0.849
0.859
0.839
0.850
Appendix
Table 11: BERTScore by place and model. Bold indicates the best model per place; gray shading indicates the lowest score per place. The mean column summarizes overall difficulty per region. Differences across places are significant for all models ( p<0.001 , Kruskal–Wallis and permutation tests).
Region
3
4
5
6
7
8
9
British Isles ∗
0.863
0.864
0.857
0.856
0.852
0.854
–
Central America †
–
0.865
0.865
0.865
0.858
–
–
China †
–
–
–
0.863
0.860
0.858
–
Egypt †
–
–
–
–
0.846
0.846
0.841
Greece ∗
–
–
0.853
0.851
0.845
0.840
0.835
India ∗
–
0.861
0.868
0.862
0.847
0.845
–
Appendix
Table 12: BERTScore by region and number of available metadata fields, averaged across all models. Bold indicates the highest score per region. ∗ scores differ significantly across field counts for all models (Kruskal–Wallis, p<0.05 ). † significant for at least one model. Roman Empire shows no significant variation across field counts.
Region
Spearman’s ρ
Slope
India
−0.411
21.0
Greece
−0.350
20.6
Iran
−0.337
14.9
Egypt
−0.089
14.1
Iraq
−0.155
13.8
British Isles
−0.164
8.8
Appendix
Table 13: Within-region relationships between metadata field count and BERTScore in Phase 1. The correlation column reports Spearman’s ρ between field count and BERTScore. The slope column reports the average increase in output length, measured as additional words per metadata field. † not significant at p<0.05 ; all other correlations are significant.
Model
BLEU-1
ROUGE-L
METEOR
BERTScore
GPT-4o
0.381
0.271
0.251
0.884
LLaMA-Maverick
0.327
0.244
0.213
0.877
Qwen2.5-32B
0.398
0.246
0.247
0.875
Appendix
Table 14: Phase 2 model-level surface-metric performance averaged across Phase 2 conditions. Values are condition-level means averaged over all probing conditions.
region
GPT-4o
LLaMA-Mav.
LLaMA-Vis.
Qwen2.5
Overall
Central America
0.9091
0.9033
0.8769
0.8823
0.8957
China
0.9084
0.8952
0.8762
0.8767
0.8914
British Isles
0.9001
0.8967
0.8749
0.8821
0.8908
Japan
0.8991
0.8947
0.8761
0.8802
0.8895
India
0.8964
0.8924
0.8760
0.8832
0.8890
Iraq
0.8978
0.8915
0.8739
0.8800
0.8879
Appendix
Table 15: Phase 2 mean BERTScore by region and model. Place differences are significant for all models (Kruskal–Wallis, all p<0.001 ; Bold indicates highest value per column. LLaMA-Vision ran on conditions R, S, S_noi, T only.
Field count ( n )
Word count
region
r ***
ρ ***
r ***
ρ ***
Central America
+0.248
+0.257
−0.697
−0.704
China
+0.216
+0.223
−0.700
−0.711
British Isles
+0.223
+0.263
−0.647
−0.618
Japan
+0.320
+0.363
−0.592
−0.578
India
+0.359
+0.417
−0.424
−0.398
Appendix
Table 16: Phase 2 Pearson ( r ) and Spearman ( ρ ) correlations between BERTScore and number of metadata fields n (left panel) and generated word count (right panel), computed per region across all conditions and models. All correlations are significant at p<0.001 (***). Bold marks the strongest correlation per column.
Figure 3: Mean BERTScore and generated word count as a function of observed metadata field count n for each of the ten regions in Phase 2, averaged across all models and conditions.
S_ctx arm
S_cont arm
Field
Cor
Wro
Sil
Cor
Wro
Sil
Technique
10.6
51.1
38.3
9.8
56.8
33.4
Subjects
17.9
49.7
32.4
—
—
—
Inscription
27.2
13.3
59.5
10.6
9.4
80.1
Assoc name
2.9
4.3
92.9
8.1
7.7
84.2
Place
11.2
20.2
68.5
3.7
2.5
93.8
Appendix
Table 17: Phase 2 bonus field rates split by arm type: the S_ctx arm and S_cont arm . All values are percentages. Bold indicates best value per column within each arm: highest for Cor and Sil; lowest for Wro. Fields absent from a arm are shown as —.
Figure 4: Nugget-verified field outcomes by region and field, averaged across all models and conditions. Each row pairs a primary outcome (field provided, left) with its corresponding bonus outcome (field withheld, right): coverage with bonus correct (top), hallucination with bonus wrong (middle), and silence with bonus silent (bottom). Blue panels indicate outcomes where higher values are better; yellow–red panels indicate outcomes where higher values are worse. Hatched cells indicate fewer than 15 observations.
Error pattern
Main fields
Mechanism
Example pattern
Field substitution
Place , Find spot
The model transfers a value from one metadata field into another, especially using an excavation site, city, or acquisition location as the broader Place value.
The model assigns a value to a neighbouring field, producing a plausible claim that violates the catalogue schema.
Glaze or velvet treated as material when recorded as technique.
Art-historical over-specification
Technique
The model refines a broad catalogue technique into a more specific art-historical technique using domain priors or visual cues.
Painted → red-figure / black-figure.
Visual elaboration beyond vocabulary
Subjects
The model adds visible motifs, attributes, posture, or iconographic details not present in the controlled subject vocabulary.
Bird; phoenix → foliage, vines, secondary motifs.
Appendix
Table 18: Qualitative schema-boundary error patterns identified from hallucinated and silent field-level outcomes. The examples illustrate recurring mechanisms rather than exhaustive error categories.
Figure 5: Generated descriptions for three objects across conditions S (sparse floor), S_CONT6 (sparse floor plus Subjects , Technique , Inscription ), and R (all available fields with image), alongside the museum reference description. Each row shows the same object; descriptions are truncated to 100 words.
Figure 6: Representative GPT-4o generated descriptions under condition R (all available fields with image) for one object per civilisation. Each cell shows the museum object and the generated description.
Dataset search depends heavily on metadata, making LLM-generated metadata a consequential form of synthetic content in retrieval systems. We study six metadata-generation settings for RDF datasets, ranging from simple rewriting to profile-grounded and agentic graph-based generation, and evaluate them jointly for retrieval effectiveness and faithfulness. Unconstrained metadata rewriting delivers the strongest retrieval gains over the original metadata, but it is also the least faithful, showing that search improvements can be driven by unsupported semantic expansion. More grounded settings substantially improve faithfulness, and profile-grounded rewriting provides the most balanced trade-off between retrieval effectiveness and grounding. These findings position synthetic metadata as a system-level IR problem in which effectiveness, provenance, and trust must be evaluated together.
Riccardo Terrenzi, Serkan Ayvaz
University of Southern Denmark Sønderborg, Denmark
Recent advances in vision-language models (VLMs) have improved image captioning for cultural heritage. However, inferring structured cultural metadata (e.g., creator, origin, period) from visual input remains underexplored. We introduce a multi-category, cross-cultural benchmark for this task and evaluate VLMs using an LLM-as-Judge framework that measures semantic alignment with reference annotations. To assess cultural reasoning, we report exact-match, partial-match, and attribute-level accuracy across cultural regions. Results show that models capture fragmented signals and exhibit substantial performance variation across cultures and metadata types, leading to inconsistent and weakly grounded predictions. These findings highlight the limitations of current VLMs in structured cultural metadata inference beyond visual perception.
Yuechen Jiang, Enze Zhang, Md Mohsinul Kabir +4
University of Manchester Manchester, UK · School of Artificial Intelligence, Wuhan University Wuhan, Hubei, China · Getty Conservation Institute Los Angeles, CA, USA
Researchers need to answer ad-hoc questions about the contents of domain-specific archives but often lack the expertise to write structured queries on the metadata. We show that when domain vocabulary and semantics are captured in a well-designed Web Ontology Language (OWL) ontology, Large Language Models (LLMs) can generate accurate structured queries zero-shot, without task-specific fine-tuning, retrieval augmentation, or multi-agent orchestration. We present the Natural Language Knowledge Graph Query (NLKGQ) system, a framework and development process that enables natural language access to metadata in such archives. The framework includes a web interface that helps researchers pose natural language questions, which a domain-agnostic harness translates to SPARQL via an LLM and executes against a knowledge graph. The development process begins with capturing domain vocabulary and semantics in a formal OWL ontology. Domain-specific code then extracts metadata from archive sources and imports it into a knowledge graph defined by the ontology. Both are designed for reuse across domains. We demonstrate the system on metadata derived from a large-scale neuroimaging research archive, evaluating multiple LLMs and ontology representations. The best configurations achieve 100% accuracy on a 21-question competency and regression test set developed with domain experts. An ablation study across eight ontology representations reveals that readable entity names and semantic annotations are the dominant factors in accuracy, more significant than model choice or prompt engineering. We also compare SPARQL to an auto-generated SQL database as query backends, showing that OWL's structural features provide a substantial advantage over SQL DDL for LLM-driven query generation. Our demonstration domain requires local LLMs on modest institutional hardware to address privacy concerns for human subject data.
Blake G. Fitch, Cato Elia Kurtz
Max Planck Institute for Biological Cybernetics Tübingen, Germany