A Dataset for Named Entity Recognition and Relation Extraction from Art-historical Image Descriptions

Stefanie Schneider; Miriam Göldl; Julian Stalter; Ricarda Vollmer

cs.CLFeb 22, 2026

A Dataset for Named Entity Recognition and Relation Extraction from Art-historical Image Descriptions

Authors: Stefanie Schneider, Miriam Göldl, Julian Stalter, Ricarda Vollmer

Organizations: Marburg University, Marburg, Germany · Ruhr University Bochum, Bochum, Germany · University of Munich, Munich, Germany

Abstract

This paper introduces FRAME (Fine-grained Recognition of Art-historical Metadata and Entities), a manually annotated dataset of art-historical image descriptions for Named Entity Recognition (NER) and Relation Extraction (RE). Descriptions were collected from museum catalogs, auction listings, open-access platforms, and scholarly databases, then filtered to ensure that each text focuses on a single artwork and contains explicit statements about its material, composition, or iconography. FRAME provides stand-off annotations in three layers: a metadata layer for object-level properties, a content layer for depicted subjects and motifs, and a co-reference layer linking repeated mentions. Across layers, entity spans are labeled with 37 types and connected by typed RE links between mentions. Entity types are aligned with Wikidata to support Named Entity Linking (NEL) and downstream knowledge-graph construction. The dataset is released as UIMA XMI Common Analysis Structure (CAS) files with accompanying images and bibliographic metadata, and can be used to benchmark and fine-tune NER and RE systems, including zero- and few-shot setups with Large Language Models (LLMs).

Figures & tables

Panel of Figure 1 — Figure 1: Each record in the dataset includes (1) the referenced artwork image, (2) basic artwork metadata, and (3) an art-historical text excerpt labeled with NER and RE annotations.

Panel of Figure 2 — Figure 2: Our dataset creation process involves a modular, multi-stage pipeline, integrating both manual and automated components.

Name	URL	Language
Austrian Gallery Belvedere	https://www.belvedere.at/en	DE
Bavarian State Painting Collections	https://www.pinakothek.de/en	EN/DE
Berlin State Museums	https://www.smb.museum/en/	EN
Bonhams	https://www.bonhams.com/	EN
Hamburger Kunsthalle	https://hamburger-kunsthalle.de/en	DE
Hampel Fine Art Auctions	https://www.hampel-auctions.com/	DE

Table 1: We collected over

250{,}000

art-historical image descriptions from

13

German and English sources. Each description contains at at least one sentence that directly addresses the object or its creation, including references to materials, techniques, its composition, or iconography.

Property	Property Label	Frequency	Include
P31	instance of	7.82 %	yes
P195	collection	5.75 %	no
P18	image	5.57 %	no
P276	location	5.55 %	no
P217	inventory number	5.20 %	no
P170	creator	4.71 %	yes

Table 2: The Wikidata properties most frequently associated with art-historical objects are shown with their relative frequency and an indicator of whether each property was included in the tagset.

Layer	Entity Name	Description	Wikidata Item
Metadata	Art Genre	form of art in terms of a medium, format, or theme	Q1792379
	Art Material	substance, raw ingredient, or tool that is utilized by an artist […]	Q15303351
	Art Movement	tendency or style in art with a specific common philosophy […]	Q968159
	Artistic Technique	method by which art is produced	Q11177771
	Type of Work of Art	type of art work based on shared characteristics, functions, […]	Q116474095
	Work of Art	aesthetic item or artistic creation	Q838948

Table 3: The tagset comprises

37

entity types distributed across two annotation layers—the metadata layer and the content layer —with an additional co-reference layer used to link mentions of the same entity. For each type, we provide a brief definition and, where applicable, the corresponding Wikidata item. Types that occur in both layers are annotated independently for each layer.

Panel of Figure 3 — Figure 3: Four-step annotation procedure. (a) First, the text is read in full to obtain an overview, without creating annotations. (b) Second, expressions belonging to the metadata layer are annotated. (c) Third, expressions in the content layer are annotated. (d) Fourth and finally, co-references are annotated.

Named Entity Recognition

Named Entity Recognition identifies and classifies people, places, organizations, and other entities in text, speech, and multimodal data.

See all 110 papers

ExpArt-KG: Artwork Image Description Generation through Iterative Exploration of Knowledge Graphs
Sep 1, 2026Yuta Kato, Shintaro Ozaki, Kazuki Hayashi +4Retrieval-Augmented Generation Image Captioning
The University of Tokyo · Nara Institute of Science and Technology (NAIST)
ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval
Jul 29, 2025Nicola Fanelli, Gennaro Vessio, Giovanna CastellanoMultimodal RAG Multimodal IR
Department of Computer Science, University of Bari Aldo Moro, Bari, Italy
KARAMEL: Knowledge-Aware Ranking for Multimodal Entity Linking in the Arts via Heterogeneous Graphs
Oct 12, 2026Raffaele Scaringi, Alejandro Sierra-Múnera, Gennaro Vessio +2Cultural Heritage Multimodal KGs
Department of Computer Science, University of Bari Aldo Moro, Italy · Hasso Plattner Institute, University of Potsdam, Germany · ZBW - Leibniz Information Centre for Economics, University of Kiel, Germany

Sep 1, 2026J/K move · Enter open · S save

ExpArt-KG: Artwork Image Description Generation through Iterative Exploration of Knowledge Graphs

Yuta Kato, Shintaro Ozaki, Kazuki Hayashi, Yusuke Sakai+3

The University of Tokyo · Nara Institute of Science and Technology (NAIST)

Retrieval-Augmented Generation Image Captioning Visual Reasoning Multimodal KGs Natural Language Explanation Generation

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) achieve strong performance on image-grounded text generation and visual question answering. However, it remains difficult for them to comprehensively and accurately describe the factual relations among the entities and concepts associated with the objects depicted in an image. In this work, we propose a framework that efficiently exploits factual information from a knowledge graph via retrieval-augmented generation (RAG), with the goal of enabling LVLMs to generate detailed and accurate image explanations. Specifically, our method alternates between answer generation and knowledge-graph retrieval, and controls the search using a correctness judgment, thereby acquiring the necessary and sufficient factual information efficiently. We also construct a knowledge graph for the artwork domain (ExpArt-KG), in which the correspondence between images and entities is unambiguous. Applying the proposed method to this knowledge graph, we show experimentally that it improves the level of detail of artwork explanations and reduces the retrieval cost of external knowledge while maintaining generation quality comparable to that of iterating a fixed number of times.

Abstract

Named Entity Recognition

Explore similar work

ExpArt-KG: Artwork Image Description Generation through Iterative Exploration of Knowledge Graphs

ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval

KARAMEL: Knowledge-Aware Ranking for Multimodal Entity Linking in the Arts via Heterogeneous Graphs