ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval

Nicola Fanelli; Gennaro Vessio; Giovanna Castellano

cs.CVJul 29, 2025

ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval

Authors: Nicola Fanelli, Gennaro Vessio, Giovanna Castellano

Organizations: Department of Computer Science, University of Bari Aldo Moro, Bari, Italy

Abstract

Analyzing digitized artworks requires both visual interpretation and deep artistic, contextual, and historical knowledge. We introduce ArtSeek, a multimodal framework for art analysis that combines multimodal large language models with retrieval-augmented generation. Unlike prior work, our pipeline relies only on image input, so it applies to artworks without links to Wikidata or Wikipedia, common in most digitized collections. ArtSeek integrates three components: a late-interaction multimodal retrieval module, a contrastive multitask network predicting artist, genre, style, media, and tags, and an agentic reasoning strategy enabled through in-context examples. Central to this approach is WikiFragments, a dataset of image-text fragments extracted from the entire English Wikipedia to support knowledge-grounded multimodal reasoning. Our multimodal query encoding raises retrieval NDCG@5 from 33.9 to 44.9 over encoding the full image-text query, and fixed-budget token pooling keeps late-interaction search practical over millions of fragments at under 7% of the cost of full multi-vector retrieval. ArtSeek sets a new state of the art in artwork attribute classification, with +8.4% F1 in style classification over GraphCLIP, and gains +7.1 BLEU@1 on ArtPedia captioning. On five knowledge-grounded question answering benchmarks graded by an LLM judge from a different model family, retrieval helps where the answer is documented in our index or lies inside the classifier's label space (+0.19 mean correctness on two of them) and is neutral where it is not; what separates the two regimes is how much of a benchmark our knowledge base covers. The gain transfers to two non-Qwen backbones (+0.22 to +0.28). A blind human study and a qualitative analysis of successes and failures corroborate this picture and expose where retrieval still misleads. Data and code: https://cilab.di.uniba.it/artseek/

Figures & tables

Figure 1: Overview of the ArtSeek pipeline. The retriever’s embedding model doubles as a feature extractor for attribute classification (e.g., artist, style); the resulting artwork card is given to the MLLM, which learns via in-context learning to query the fragment knowledge base, enabling agentic RAG.

Model	Artist			Genre			Style			Media	Tags
	Top-1	Top-2	F1	Top-1	Top-2	F1	Top-1	Top-2	F1	F1	F1
ResNet [ 12 ]	—	—	—	62.51	81.12	57.37	48.46	66.72	47.23	—	—
ContextNet [ 12 , 22 ]	—	—	—	61.77	79.87	56.70	42.61	61.91	41.42	—	—
ViT+GAT [ 12 ]	—	—	—	72.29	86.45	65.29	58.58	76.13	56.58	—	—
ViT-Multitask [ 13 ]	69.93	—	58.63	72.78	—	65.94	59.98	—	57.41	53.55	39.61
GraphCLIP [ 34 ]	—	—	—	72.44	86.62	65.06	61.17	77.81	58.22	—	—

Table 1: Multitask classification results: Top-1/Top-2 accuracy and F1 for the multiclass tasks, F1 only for the multilabel ones. Since we adopt competing methods’ reported results, some metrics are unavailable (“—”). Best in bold , second best underlined .

Dataset	Model	B@1	B@4	M	S	R
ArtPedia	Wu [ 39 ]	24.7	3.06	6.58	—	22.4
	Jiang et al. [ 27 ]	32.6	7.48	9.33	7.68	23.7
	Qwen2.5-VL-32B	36.57	3.93	9.46	6.81	21.22
	ArtSeek (w/o class.)	39.64	6.16	10.52	7.73	23.64
	ArtSeek	39.7	5.26	10.26	7.74	23.0
SemArt v2.0	Bai et al. [ 5 ]	—	9.10	11.4	—	23.1

Table 2: Image captioning results, zero-shot for ArtSeek; some competing metrics are unavailable (“—”). PaintingForm [ 8 ] follows the same pattern ( Tab. S3 ), and BLEU@2–3 are reported in Tab. S4 . Best in bold , second best underlined .B@N = BLEU@N, M = METEOR, S = SPICE, R = ROUGE-L.

Paintings the knowledge base covers
Benchmark	Source	n	KB cov.	Qwen2.5-VL-32B	ArtSeek (w/o class.)	ArtSeek	ArtSeek $-$ backbone [95% CI]
ArtPedia-VQA	ours	651	85.8%	0.535	0.774	0.722	+0.186 [+0.119, +0.254]
LICNHeldOut	ours (Wikidata)	762	53.3%	1.010	1.086	1.197	+0.186 [+0.114, +0.257]
Paintings the knowledge base barely covers
AQUA	Garcia et al. [ 23 ]	652	10.4%	0.920	0.910	0.896	$-$ 0.025 [ $-$ 0.095, +0.046]
ArtCurate-AIC	ours (AIC, CC0)	1489	10.3%	1.125	1.074	1.093	$-$ 0.034 [ $-$ 0.071, +0.003]

Table 3: Question answering: mean correctness on a 0–2 rubric. “KB cov.” is the share of a benchmark’s paintings having any page in our index; green marks a difference whose interval excludes zero, grey one that spans it; bold marks the best system in a row and underlining the second best. The table splits by coverage rather than by benchmark family: ArtSeek gains where the knowledge base covers the painting, and is indistinguishable from the bare backbone where it does not.

Panel of Figure 2 — Figure 2: Why the question alone is not a usable retrieval query: it names no painting, no artist, no period and no place, so embedding it returns generic articles about headwear. The image and the artwork card supply what the query needs, and the backbone’s confident “hennin” becomes the correct balzo —even though the card’s artist field is wrong here. Correct identifications are green, incorrect ones red.

Figure S1 : Our multi-stage retrieval pipeline. We design an efficient retrieval method over a large-scale knowledge base $\mathcal{D}=\{d_{i}\}_{i=1}^{N}$ using a multi-vector strategy and multimodal queries to ColQwen2, without compromising efficiency or worsening the modality gap. Both the input image and text are encoded with ColQwen2, but only text-related embeddings are used for late interaction retrieval. The first stage retrieves $N_{1}$ candidates via pooled representations; the second reranks them using full multi-vector representations to return the top $N_{2}$ results ( $N_{2}<N_{1}$ ).

Figure S2 : Overview of the contrastive training process for LICN. Given a batch of images

\{I_{i}\}_{i=1}^{M}

with corresponding labels

\{q_{i}\}_{i=1}^{N}

for multiple classification tasks, we prepend multiple task tokens to each image’s multi-vector representation

E_{I_{i}}

and a single task token to each label encoding

E_{q_{i}}

. Both are processed by a transformer, producing task-aware embeddings by extracting the contextualized task tokens. We then compute a sigmoid loss on the per-task logits matrix

Z_{t}

, whose entry

(i,j)

is the dot product between image

i

’s and label

j

’s task-

t

embeddings; there is one such matrix per task, of which three are drawn here.

Panel of Figure 8 — Figure S3 : Examples from the WikiFragments dataset. (a) A fragment with four images, their captions, and the paragraph text. (b) A text-only fragment. For each fragment we also store metadata about the source Wikipedia page, its hyperlinks, and the paragraph’s position within the page.

Subsequence	Description
$I^{\text{pref}}$	Embeddings for the preamble introducing the image input.
$I^{\text{content}}$	Embeddings for the image patches, surrounded by <\|vision_start\|> and <\|vision_end\|> .
$I^{\text{text}}$	Embeddings of the default prompt “ Describe the image. ” used during training.
$I^{\text{suff}}$	End-of-message special tokens for image input.
$q^{\text{pref}}$	Embeddings for the prefix string “ Query: ”.
$q^{\text{content}}$	Embeddings for the actual user query.

Table S1 : Structure of multimodal input embeddings. Each input (image or text) is represented as a sequence of embeddings composed of functional subsequences.

Store	Query	NDCG@5 (↑)	R@1 (↑)	Avg time (cs) (↓)
CLIP	CLIP	2.66	1.49	0.39
Full (one-stage)	Full ( $E_{q_{multi}}$ )	33.92	26.90	67.92
Full (one-stage)	Filtered ( $E_{q_{multi}}^{{\text{filtered}}}$ )	44.88	38.41	32.26
Pooled (two-stage)	Full ( $E_{q_{multi}}$ )	21.50	17.39	11.14
Pooled (two-stage)	Filtered ( $E_{q_{multi}}^{\text{filtered}}$ )	27.61	23.87	4.66

Table S2: Retrieval results, including NDCG@5 and Recall@1 (R@1), where “@K” indicates evaluation over the top

K

retrieved items, and the average query time required for retrieval. Best in bold , second best underlined .

Model	B@1	B@2	B@3	B@4	M	R
Bin et al. [ 8 ]	—	—	—	21.23	37.62	31.34
Qwen2.5-VL-32B	51.62	33.09	20.96	13.30	21.96	29.56
ArtSeek (w/o classification)	54.35	34.66	21.93	14.01	20.69	29.48
ArtSeek	52.93	33.90	21.58	13.90	21.18	29.30

Table S3 : Captioning results on PaintingForm [ 8 ] , zero-shot. Metric abbreviations as in Tab. 2 . Best in bold , second best underlined .

Dataset	Model	B@2	B@3
ArtPedia	Wu [ 39 ]	—	—
	Jiang et al. [ 27 ]	17.7	10.9
	Qwen2.5-VL-32B	16.52	7.68
	ArtSeek (w/o classification)	20.07	10.62
	ArtSeek	19.2	9.69
SemArt v2.0	Bai et al. [ 5 ]	—	—

Table S4 : BLEU@2 and BLEU@3 for the captioning comparison of Tab. 2 , omitted there for width. Best in bold , second best underlined .

Shared weights			A	G	S	M	T	Avg
Encoder	Task embeddings	Linear projections	F1	F1	F1	F1	F1	F1
✓	✓	✓	64.69	73.21	66.77	58.04	39.82	60.51
✓	✓	✗	61.12	73.34	66.88	57.88	37.32	59.31
✓	✗	✓	65.01	73.25	66.65	59.49	39.59	60.80
✗	✗	✗	60.50	72.94	66.00	57.09	37.16	58.73

Table S5 : Ablation study on weight sharing. The results include the F1-scores for both multiclass and multilabel classifications. Task abbreviations: A = Artist, G = Genre, S = Style, M = Media, T = Tags. Best in bold , second best underlined .

Backbone	Prec.	Base	Full	$\Delta$ [95% CI]
Qwen2.5-VL-32B	AWQ4	0.535	0.785	+0.249 [+0.183, +0.318]
Gemma-3-27B [ 24 ]	bf16	0.535	0.759	+0.224 [+0.163, +0.287]
Mistral-Small-3.1-24B [ 32 ]	w4a16	0.422	0.696	+0.275 [+0.215, +0.337]

Table S6 : ArtPedia-VQA correctness with the generator swapped and everything else held fixed. All runs use seeded retrieval, so the Qwen row is a matched control rather than the learned-policy system of Tab. 3 of the main paper.

Benchmark	n	$\Delta$ [95% CI]
ArtPedia-VQA	0 618	$+0.057$	[ $+0.003$ , $+0.112$ ]
AQUA	0 631	$+0.054$	[ $-0.010$ , $+0.115$ ]
ArtQuest	1167	$+0.077$	[ $+0.041$ , $+0.113$ ]
LICNHeldOut	0 762	$+0.060$	[ $+0.032$ , $+0.089$ ]
ArtCurate-AIC	1451	$+0.187$	[ $+0.148$ , $+0.225$ ]

Table S7 : The policy ablation: ArtSeek with the in-context worked example minus ArtSeek with the policy stated as prose, everything else held fixed. Positive favors the worked example. Green marks an interval excluding zero, grey one that spans it.

n

is the number of items on which both variants produced a gradable answer.

Variant	ArtPedia	AQUA	ArtQuest	LICNHO	ArtCurate	calls
Qwen2.5-VL-32B	0.535	0.920	0.937	1.010	1.125	—
ArtSeek (w/o classification)	0.774	0.910	0.828	1.086	1.074	—
ArtSeek	0.722	0.896	0.914	1.197	1.093	0.97
ArtSeek w/ prompted policy	0.687	0.838	0.852	1.136	0.913	1.32

Table S8 : Full policy grid, with mean retrieval calls per item on ArtPedia-VQA. LICNHO abbreviates LICNHeldOut and ArtCurate abbreviates ArtCurate-AIC. Cells left blank were not run. Best in bold , second best underlined .

		n	ev. rec.	absent	docs
ArtPedia	ArtSeek	562	0.407	51.4%	11.0
	prompted policy	568	0.360	55.3%	13.2
	oracle query	651	0.830	10.0%	10.0
AQUA	ArtSeek	533	0.171	81.6%	10.4
	prompted policy	561	0.149	84.0%	12.8
	oracle query	652	0.320	64.6%	10.0

Table S9 : Evidence recall: how much of what the reference answer needs is present in the documents actually retrieved. Evidence judging was run on ArtPedia and AQUA only. Best in bold , second best underlined .

Question type	PrefixLM	base	w/o class.	ArtSeek
	(FT)
title (open vocab.)	11.5%	11.5%	0 9.0%	12.0%
type (genre head)	83.2%	75.0%	75.4%	82.0%
artist	34.1%	13.5%	12.8%	14.6%
technique (media head)	61.2%	0 8.0%	12.5%	26.0%
school (no head)	69.4%	14.0%	22.5%	11.0%

Table S10 : ArtQuest by question type, contains against the gold span. PrefixLM [ 9 ] is fine-tuned on the benchmark; ArtSeek columns are zero-shot. † Timeframe reads 0.0% for every zero-shot system because the gold is a bracket ( 1601-1650 ) while the model answers “the early seventeenth century”—right, and scored zero. Under the judge the same rows are 54.0% (base) and 50.0% (ArtSeek), which is the figure to read for this question type. Best in bold , second best underlined .

Variant	median	p95	mean
Qwen2.5-VL-32B	0 6.5	10.0	0 6.7
ArtSeek (w/o classification)	11.1	22.0	11.8
ArtSeek	11.8	20.1	11.9
ArtSeek w/ prompted policy	18.0	37.5	19.5

Table S11 : Wall-clock seconds per item on ArtPedia-VQA, on a shared A100 at max_num_seqs

=4

. These are upper bounds, not tuned deployment figures.

System	subj.	fig.	plac.	evid.	$\Sigma$ hum.	$\Sigma$ phi-4
Tier A — 10 paintings, Wikipedia article exists
Qwen2.5-VL-32B	1.00	1.03	1.20	0.87	3.23	2.65
ArtSeek	1.23	1.30	1.20	1.10	3.73	3.35
GPT-5.5	1.37	1.40	1.27	1.27	4.03	3.40
Tier B — 4 gallery paintings, no article
Qwen2.5-VL-32B	0.58	0.92	1.00	0.67	2.50	1.25

Table S12 : Human evaluation by tier: mean per-axis scores (0–2) over three blind raters,

\Sigma

-correctness (0–6), and the phi-4 rehearsal of the same rubric. On Tier B, ArtSeek

-

base is

+1.17

[

+0.50

+2.00

], the only human difference whose interval excludes zero; ArtSeek

-

GPT-5.5 there is

+0.17

[

-2.17

+2.50

] and the forced choice splits 6–6, so we read the two as level on these works and claim the margin over the backbone. Best in bold , second best underlined .

Statistic	value	chance
mean human–human $\rho$ ( $\Sigma$ )	$+0.651$	0
mean judge–human $\rho$	$+0.761$	0
phi-4 vs. pooled human mean	$+0.860$	0
expert vs. lay mean	$+0.782$	0
Krippendorff $\alpha$ , humans	$+0.463$	0
Krippendorff $\alpha$ , humans $+$ phi-4	$+0.489$	0

Table S13 : Rater agreement on the human study. Spearman

\rho

is computed on

\Sigma

-correctness;

\alpha

is ordinal and axis-level. The phi-4 rehearsal scored the same items over both presentation orders, and the per-item value is the mean of the two.

Panel of Figure 22 — Figure S4 : Example conversation given to the MLLM. Our manually annotated example contains a visual-only question and a complex query that requires iterative reasoning and retrieval, to demonstrate how the MLLM should perform the thinking process and how to utilize the retriever. The second input to the retriever function specifies whether the retrieval should be multimodal, using both image and text, or purely textual.

Panel of Figure 23 — Figure S5 : First qualitative example. ArtSeek correctly recognized the author of this painting and searched for information about the setting of the painting, finding a similar artwork by Renoir.

Panel of Figure 24 — Figure S6 : Second qualitative example. ArtSeek correctly identifies the author of the painting using the LICN module and retrieves information about the woman’s identity based on one of the (incorrectly) predicted tags, in order to provide additional context to the user.

Panel of Figure 25 — Figure S7 : Question answering examples. ArtSeek is capable of answering user queries involving the identification and description of people or events. In these examples, we compare ArtSeek’s responses with those from ChatGPT and the base model Qwen2.5-VL, demonstrating the improvements introduced by our pipeline. Correct identifications are marked in green, while incorrect ones are highlighted in red.

Panel of Figure 26 — Figure S8 : Two ArtPedia-VQA questions on paintings the backbone can identify neither by title nor by artist. Left: the base model invents a rope and misses the butterflies that the reference answer turns on; retrieval finds the painting’s own page. The artwork card’s artist field is wrong here and the answer is correct regardless. Right: the base model correctly refuses—which our rubric scores as wrong—and retrieval converts the refusal into a specific correct answer, alongside two irrelevant bridges.

Panel of Figure 27 — Figure S9 : Left: retrieval supplies iconographic context—Maurus’s staff and Giles’s doe—that neither the image nor the (wrong) artist prediction could provide, alongside two unrelated altarpieces. Right: the corpus does not contain the answer, the mis-attributed surname carries retrieval out of the art domain, and ArtSeek declines. Both the base model and ArtSeek score zero on this item, for opposite reasons. The Kefermarkt altarpiece photograph is reproduced under CC BY-SA 4.0; all other images here are in the public domain.

Panel of Figure 28 — Figure S10 : Retrieval overturns a correct answer. The base model names the right museum; the artwork card is right too, but the retrieved document is a different Bellotto held in Moscow, and the model transfers that collection to the painting in front of it.

Panel of Figure 29 — Figure S11 : The largest margin in each direction in the human study. Left: the largest ArtSeek-over-backbone margin ( $+4.00$ in $\Sigma$ ); all three systems see the same costume, and only the retrieval-augmented one reads it correctly. Right: the largest ArtSeek-below-backbone margin ( $-2.33$ ); retrieval added hedging without adding identification.

Multimodal RAG

Multimodal retrieval-augmented generation grounds AI answers in retrieved text, images, video, audio, and structured visual evidence.

See all 155 papers

Apr 21, 2026

A-MAR: Agent-based Multimodal Art Retrieval for Fine-Grained Artwork Understanding

Understanding artworks requires multi-step reasoning over visual content and cultural, historical, and stylistic context. While recent multimodal large language models show promise in artwork explanation, they rely on implicit reasoning and internalized knowl- edge, limiting interpretability and explicit evidence grounding. We propose A-MAR, an Agent-based Multimodal Art Retrieval framework that explicitly conditions retrieval on structured reasoning plans. Given an artwork and a user query, A-MAR first decomposes the task into a structured reasoning plan that specifies the goals and evidence requirements for each step. Retrieval is then conditionedon this plan, enabling targeted evidence selection and supporting step-wise, grounded explanations. To evaluate agent-based multi- modal reasoning within the art domain, we introduce ArtCoT-QA. This diagnostic benchmark features multi-step reasoning chains for diverse art-related queries, enabling a granular analysis that extends beyond simple final answer accuracy. Experiments on SemArt and Artpedia show that A-MAR consistently outperforms static, non planned retrieval and strong MLLM baselines in final explanation quality, while evaluations on ArtCoT-QA further demonstrate its advantages in evidence grounding and multi-step reasoning ability. These results highlight the importance of reasoning-conditioned retrieval for knowledge-intensive multimodal understanding and position A-MAR as a step toward interpretable, goal-driven AI systems, with particular relevance to cultural industries. The code and data are available at: https://github.com/ShuaiWang97/A-MAR.

Shuai Wang, Hongyi Zhu, Jia-Hong Huang +6

University of Amsterdam · Part of this work was done during interning at Amazon. · Also with Amazon AGI. +2

Multimodal RAG Multimodal Grounding Multimodal QA Multimodal CoT Reasoning Multimodal Agents Multimodal Understanding

Ask PDF

Sep 1, 2026

ExpArt-KG: Artwork Image Description Generation through Iterative Exploration of Knowledge Graphs

Large Vision-Language Models (LVLMs) achieve strong performance on image-grounded text generation and visual question answering. However, it remains difficult for them to comprehensively and accurately describe the factual relations among the entities and concepts associated with the objects depicted in an image. In this work, we propose a framework that efficiently exploits factual information from a knowledge graph via retrieval-augmented generation (RAG), with the goal of enabling LVLMs to generate detailed and accurate image explanations. Specifically, our method alternates between answer generation and knowledge-graph retrieval, and controls the search using a correctness judgment, thereby acquiring the necessary and sufficient factual information efficiently. We also construct a knowledge graph for the artwork domain (ExpArt-KG), in which the correspondence between images and entities is unambiguous. Applying the proposed method to this knowledge graph, we show experimentally that it improves the level of detail of artwork explanations and reduces the retrieval cost of external knowledge while maintaining generation quality comparable to that of iterating a fixed number of times.

Yuta Kato, Shintaro Ozaki, Kazuki Hayashi +4

The University of Tokyo · Nara Institute of Science and Technology (NAIST)

Retrieval-Augmented Generation Image Captioning Visual Reasoning Multimodal KGs Natural Language Explanation Generation

Ask PDF

Jul 2, 2026

EduArt: An educational-level benchmark for evaluating art history knowledge in large language models

Large language models now score near ceiling on general benchmarks, but these aggregate measures reveal little about how models behave within single disciplines. Existing art-focused evaluations rely on synthetic questions and rarely report item-level properties. This paper introduces EduArt, an educational-level benchmark for art-historical knowledge and visual reasoning in multimodal LLMs. EduArt comprises 871 human-authored questions from Italian secondary-school exercises and US Advanced Placement Art History exams, spanning two languages and seven formats from multiple choice to in-text word placement and error identification. Twelve models from six provider families were evaluated under a default answer-only condition and a motivation condition requiring written justification, and characterized using Classical Test Theory and a logistic regression isolating the effects of format, language, image presence, and model. The benchmark showed strong psychometric properties (mean discrimination 0.514, 82.3 percent good discriminators), while multiple-choice accuracy saturated near ceiling for six models, showing recognition formats alone cannot distinguish frontier models. Format was a strong independent predictor of accuracy: models exceeding 94 percent on multiple choice fell to 23.9 percent on open completion (Claude Opus 4.6) and 6.2 percent on error identification (Claude Sonnet 4.6). The motivation condition changed accuracy in a predominantly negative, family-dependent direction. These dissociations indicate that art-historical knowledge and the ability to deploy it are distinct capabilities, and that single-format benchmarks overestimate what models can reliably do. Mapping this capability profile is a precondition for responsible use of multimodal LLMs in art-historical scholarship, where tasks demand producing and manipulating content rather than selecting from fixed options.

Gianmarco Spinaci, Lukas Klic, Giovanni Colavizza

Department of Classical Philology and Italian Studies, University of Bologna, Bologna, Italy · Villa i Tatti – The Harvard University Center for Italian Renaissance Studies, Firenze, Italy · Center for Computational and Digital Humanities, University of Copenhagen, Copenhagen, Denmark

Educational Assessment Multimodal QA Multimodal Model Evaluation Education

Ask PDF

Abstract

Multimodal RAG

Explore similar work

A-MAR: Agent-based Multimodal Art Retrieval for Fine-Grained Artwork Understanding

ExpArt-KG: Artwork Image Description Generation through Iterative Exploration of Knowledge Graphs

EduArt: An educational-level benchmark for evaluating art history knowledge in large language models