M-ArtAgent: Evidence-Based Multimodal Agent for Implicit Art Influence Discovery

Hanyi Liu; Zhonghao Jiu; Minghao Wang; Yuhang Xie; Heran Yang

cs.AIApr 8, 2026

M-ArtAgent: Evidence-Based Multimodal Agent for Implicit Art Influence Discovery

Authors: Hanyi Liu, Zhonghao Jiu, Minghao Wang, Yuhang Xie, Heran Yang

Organizations: China Electronics Technology Group Co., Ltd., Beijing 100043, China · School of Information Science and Engineering, Southeast University, Nanjing 211189, China · Department of Chemical and Biological Engineering, Hong Kong University of Science and Technology, Hong Kong SAR, China · University of California San Diego, La Jolla, CA 92093 USA · Northeastern University, Boston, MA 02115 USA

Abstract

Implicit artistic influence, although visually plausible, is often undocumented and thus poses a historically constrained attribution problem: resemblance is necessary but not sufficient evidence. Most prior systems reduce influence discovery to embedding similarity or label-driven graph completion, while recent multimodal large language models (LLMs) remain vulnerable to temporal inconsistency and unverified attributions. This paper introduces M-ArtAgent, an evidence-based multimodal agent that reframes implicit influence discovery as probabilistic adjudication. It follows a four-phase protocol consisting of Investigation, Corroboration, Falsification, and Verdict governed by a Reasoning and Acting (ReAct)-style controller that assembles verifiable evidence chains from images and biographies, enforces art-historical axioms, and subjects each hypothesis to adversarial falsification via a prompt-isolated critic. Two theory-grounded operators, StyleComparator for Wolfflin formal analysis and ConceptRetriever for ICONCLASS-based iconographic grounding, ensure that intermediate claims are formally auditable. On the balanced WikiArt Influence Benchmark-100 (WIB-100) of 100 artists and 2,000 directed pairs, M-ArtAgent achieves 83.7% positive-class F1, 0.666 Matthews correlation coefficient (MCC), and 0.910 area under the receiver operating characteristic curve (ROC-AUC), with leakage-control and robustness checks confirming that the gains persist when explicit influence phrases are masked. By coupling multimodal perception with domain-constrained falsification, M-ArtAgent demonstrates that implicit influence analysis benefits from historically grounded adjudication rather than pattern matching alone.

Figures & tables

Symbol	Meaning
$A_{i}$ , $\mathcal{A}_{i}$ , $B_{i}$	Artist entity $i$ , artwork portfolio, and biographical text corpus
$T_{i}$	Lifespan interval $[\tau_{\mathrm{birth}}^{(i)},\tau_{\mathrm{death}}^{(i)}]$
$\Delta$	Formative-window tolerance in the temporal gate
$a$	Artwork instance, with components $a=(I,M)$
$\mathbf{z}(I)$ , $\bm{\phi}(I,x)$ , $\Omega$	Global visual embedding, patch-level embedding field, and token index set ( $x\in\Omega$ )
$\alpha_{x}$ , $s_{\mathrm{v}}(\cdot,\cdot)$ , $s_{\mathrm{v}}^{\star}$ , $\gamma_{\mathrm{v}}$ , $K$	Patch aggregation weights, visual similarity, seed similarity passed from candidate generation, threshold, and top- $K$ retrieval size

TABLE I: Summary of key mathematical notations.

Panel of Figure 2 — Fig. 1: Visual summaries of the primary mathematical formalisms in M-ArtAgent. (a) The Observation Formalism (Eq. 1–4): Raw multimodal embeddings are funneled through grammatical projection into an orthogonal 5D Wölfflin manifold $\mathcal{W}$ , transforming implicit clustering into explicit formal coordinates. (b) The Adjudication Formalism (Eq. 10 ): Rather than blindly accepting visual similarity, the ReAct critic subjects the initial confidence $c_{0}$ to systematic downward pressure from alternative historical hypotheses ( $\mathcal{H}_{2},\mathcal{H}_{3},\mathcal{H}_{4}$ ), actively penalizing and clipping the score to yield a conservative $c_{\mathrm{final}}$ .

Panel of Figure 3 — Fig. 2: M-ArtAgent overview. The framework combines four principal processing layers, with candidate generation integrated into Layer 2, and a four-phase epistemic protocol (Investigation, Corroboration, Falsification, Verdict) that mirrors art-historical scholarship.

Section	Component	Configuration
Network Architecture: Layer 2 Perception + Candidate Generation
Network Architecture	Visual encoder	CLIP ViT-B/32 [ 8 ] (512-dim), patch $32\times 32$ , input $224\times 224$
Network Architecture	Text encoder	SBERT all-MiniLM-L6-v2 [ 25 ] (384-dim), MaxSeq=512, $\ell_{2}$ normalization
Network Architecture	Vector index	FAISS [ 26 ] + HNSW [ 27 ] , $M=32$ , $efConstruction=200$
Network Architecture	Candidate filter	Top- $K=10$ , cosine threshold $0.70$ , timeline gate ( 6 ); integrated high-recall proposal module within Layer 2
Agent Configuration: ReAct + Critic

TABLE II: Implementation details and hyperparameters.

Operator	Evidence produced	Grounding mechanism
VisualAnalyzer	Cross-portfolio visual candidates; motif and compositional alignment scores	FAISS-HNSW retrieval in CLIP space together with motif-overlap and spatial-consistency aggregation
BiographyReader	Transmission cues such as co-location, institutions, and explicit references	SBERT retrieval with rule-based cue extraction
TimelineGate	Chronological feasibility certificate	Rule-based temporal constraints from the chronology axiom
StyleComparator	Hereditary formal-profile proximity on $\mathcal{W}$	Projection onto the Wölfflin manifold $\mathcal{W}$ using contrastive prompt axes
ConceptRetriever	ICONCLASS-aligned motif codes and topology-aware semantic distances	Retrieval and alignment on the ICONCLASS concept graph $\mathcal{G}_{\mathrm{IC}}$

TABLE III: Structured evidence operators used by the ReAct controller.

Axiom rule	Methodological source	Computational implementation	Consequence
Chronological intersection	Biographical chronology	TimelineGate	Immediate rejection, forced negative
Pathway accessibility	Social-historical context	SBERT overlap plus transmission cues	Confidence reduction if absent
Marker specificity	Connoisseurship-style diagnosis	Style taxonomy plus motif checks	Require diagnostic markers
Convergence risk	Comparative stylistic analysis	Critic hypothesis $\mathcal{H}_{3}$	Penalize spurious similarity

TABLE IV: Historical constraints that bound the generative agent.

Item	WIB-100 design
Artist pool	100 artists selected from 129 WikiArt candidates; region/era balanced and backward compatible with the original 50 artists
Evaluation pairs	2,000 directed artist pairs
Class balance	1,000 positives + 1,000 negatives (50:50)
Negative tiers	300 hard, 300 medium, 200 easy, and 200 temporal-impossible pairs
Protocol	5-fold stratified cross-validation; each round uses 400 pairs for development and 1,600 for held-out evaluation
Ground truth	Positives cross-validated from art-historical literature and curated influence records; negatives actively verified rather than treated as unlabeled pairs

TABLE V: Summary of WIB-100.

Method	Year	Type	Description
GalleryGPT [ 3 ]	2024	MLLM	Art-specialized multimodal large language model
ComplEx (Text + Vis) [ 30 ]	2016	KG	Complex-valued knowledge-graph embedding baseline augmented with multimodal artist descriptors
LLM Zero-Shot [ 31 ]	2022	LLM	Zero-shot reasoning protocol instantiated on a GPT-4-class backbone without tailored prompting
LLM + Simple CoT [ 32 ]	2022	LLM	Chain-of-thought (CoT) prompting protocol instantiated on a GPT-4-class backbone
WP-CLIP [ 18 ]	2025	CLIP	Wölfflin–Panofsky aligned CLIP fine-tuning
TransE (DeepWalk) [ 29 ]	2013	KG	Translation-based knowledge-graph embedding baseline with graph-walk artist features

TABLE VI: Representative baselines used in the WIB-100 benchmark. Type abbreviation: multimodal large language model (MLLM).

Method	Type	Prec.	Rec.	Spec.	F1 pos	Mac-F1	MCC
M-ArtAgent	Agent	81.5 $\pm$ 1.5%	86.0 $\pm$ 1.4%	80.5 $\pm$ 1.8%	83.7 $\pm$ 1.2%	83.2 $\pm$ 1.1%	0.666 $\pm$ 0.021
GalleryGPT [ 3 ]	MLLM	65.1 $\pm$ 2.2%	82.0 $\pm$ 2.1%	56.0 $\pm$ 3.4%	72.6 $\pm$ 2.0%	68.5 $\pm$ 1.8%	0.394 $\pm$ 0.032
ComplEx (Text+Vis) [ 30 ]	KG	62.1 $\pm$ 2.6%	75.5 $\pm$ 3.4%	54.0 $\pm$ 4.1%	68.2 $\pm$ 2.3%	64.3 $\pm$ 2.1%	0.302 $\pm$ 0.039
LLM + Simple CoT [ 32 ]	LLM	61.5 $\pm$ 2.4%	80.0 $\pm$ 2.5%	50.0 $\pm$ 4.1%	69.6 $\pm$ 2.2%	64.2 $\pm$ 1.9%	0.314 $\pm$ 0.038
LLM Zero-Shot [ 31 ]	LLM	59.5 $\pm$ 2.8%	88.0 $\pm$ 2.8%	40.0 $\pm$ 4.5%	71.0 $\pm$ 2.5%	61.8 $\pm$ 2.2%	0.319 $\pm$ 0.045
WP-CLIP [ 18 ]	CLIP	56.8 $\pm$ 3.5%	83.0 $\pm$ 3.0%	37.0 $\pm$ 4.2%	67.5 $\pm$ 2.6%	57.8 $\pm$ 2.5%	0.225 $\pm$ 0.040

TABLE VII: Main performance comparison on the balanced WIB-100 benchmark.

Fig. 3: Main results visualization on the balanced WIB-100 benchmark for selected baselines. (a) Positive-class F1 comparison; the dashed line marks the Always-YES baseline. (b) Confusion matrix of M-ArtAgent on the full 2,000-pair benchmark. (c) ROC-AUC comparison for score-producing systems.

(a) Critic-penalty tuning
Penalty ( $\gamma$ )	MCC	F1 pos	Rec.	Spec.	Behavior Profile
$\gamma=0$ (No falsification)	0.463	76.1%	94.5%	46.0%	Permissive regime with weak rejection
$\gamma=1.0$ (Moderate)	0.619	82.0%	89.0%	72.0%	Good recall–rejection trade-off
$\gamma=2.0$ (Optimal)	0.666	83.7%	86.0%	80.5%	Best balance on hard negatives
$\gamma=4.0$ (Over-penalized)	0.582	77.0%	71.0%	86.5%	Over-skeptical; suppresses true positives

TABLE VIII: Ablation on critic falsification strength and theory-grounded connectors.

(a) Biography leakage control
Condition	Mac-F1	MCC	Rec.	Spec.
Full biography	83.2%	0.666	86.0%	80.5%
Masked biography	81.8%	0.637	83.2%	80.5%
Text-only (no visual)	71.2%	0.452	85.5%	58.0%
No BiographyReader	69.7%	0.395	68.5%	71.0%
(b) Module-level architectural ablation

TABLE IX: Leakage-control and module-level analyses on WIB-100.

Fig. 4: Diagnostic analysis on the WIB-100 benchmark. (a) Stratified rejection rates for hard, medium, easy, and temporal-impossible negatives. (b) Effect of the critic penalty factor

\gamma

on MCC. (c) Impact of replacing rigid formal connectors with generic prompts.

(a) ConceptRetriever alignment (100 images)
Retrieval Level	Prec.	Rec.	F1
Exact match (Level 5+)	68.4%	52.1%	59.1%
Ancestor match (Level 3+)	89.2%	81.5%	85.2%

TABLE X: Additional transparency checks for the theory-grounded operators.

Fig. 5: Case studies. (a–b) True positive: Hokusai

\rightarrow

Van Gogh. (c–d) True negative: Dalí

\rightarrow

Escher, where high visual similarity is overturned by failed pathway evidence and counter-verification.

Explore similar work

A-MAR: Agent-based Multimodal Art Retrieval for Fine-Grained Artwork Understanding
Apr 21, 2026Shuai Wang, Hongyi Zhu, Jia-Hong Huang +6Multimodal RAG Multimodal Grounding
University of Amsterdam · Part of this work was done during interning at Amazon. · Also with Amazon AGI. +2
MAIA: Multi-Agent Intent Articulation for Requirement Discovery in Art Commissions
Sep 10, 2026Yu-Chao Wang, Yanhong Lu, Yingjie Victor Chen +1Requirements Engineering Human-in-the-Loop AI
School of Applied and Creative Computing Purdue University West Lafayette, Indiana, USA
ArtSociety: Multi-Agent Multimodal Collaboration for Art Emotion Understanding
Sep 3, 2026Jian Li, Fanfan Ji, Jinxiang Lai +5Multimodal Emotion Recognition Affective Computing
Nanjing University Suzhou, China · CSE The Hong Kong University of Science and Technology Hong Kong, China · Tencent YoutuLab Shanghai, China

Apr 21, 2026J/K move · Enter open · S save

A-MAR: Agent-based Multimodal Art Retrieval for Fine-Grained Artwork Understanding

Shuai Wang, Hongyi Zhu, Jia-Hong Huang, Yixian Shen+5

University of Amsterdam · Part of this work was done during interning at Amazon. · Also with Amazon AGI.+2

Multimodal RAG Multimodal Grounding Multimodal QA Multimodal CoT Reasoning Multimodal Agents Multimodal Understanding

Open paper AskPDF arXiv

Understanding artworks requires multi-step reasoning over visual content and cultural, historical, and stylistic context. While recent multimodal large language models show promise in artwork explanation, they rely on implicit reasoning and internalized knowl- edge, limiting interpretability and explicit evidence grounding. We propose A-MAR, an Agent-based Multimodal Art Retrieval framework that explicitly conditions retrieval on structured reasoning plans. Given an artwork and a user query, A-MAR first decomposes the task into a structured reasoning plan that specifies the goals and evidence requirements for each step. Retrieval is then conditionedon this plan, enabling targeted evidence selection and supporting step-wise, grounded explanations. To evaluate agent-based multi- modal reasoning within the art domain, we introduce ArtCoT-QA. This diagnostic benchmark features multi-step reasoning chains for diverse art-related queries, enabling a granular analysis that extends beyond simple final answer accuracy. Experiments on SemArt and Artpedia show that A-MAR consistently outperforms static, non planned retrieval and strong MLLM baselines in final explanation quality, while evaluations on ArtCoT-QA further demonstrate its advantages in evidence grounding and multi-step reasoning ability. These results highlight the importance of reasoning-conditioned retrieval for knowledge-intensive multimodal understanding and position A-MAR as a step toward interpretable, goal-driven AI systems, with particular relevance to cultural industries. The code and data are available at: https://github.com/ShuaiWang97/A-MAR.

Abstract

Explore similar work

A-MAR: Agent-based Multimodal Art Retrieval for Fine-Grained Artwork Understanding

MAIA: Multi-Agent Intent Articulation for Requirement Discovery in Art Commissions

ArtSociety: Multi-Agent Multimodal Collaboration for Art Emotion Understanding