How AI Experiences Art: Emergent Aesthetic Structure in a Self-Supervised Multimodal Embedding Space

Corey D. C. Heath

cs.MMAug 27, 2026

How AI Experiences Art: Emergent Aesthetic Structure in a Self-Supervised Multimodal Embedding Space

Authors: Corey D. C. Heath

Organizations: Independent Researcher Phoenix, AZ, USA

Abstract

Aesthetics are an important part of the symbolism of artistic works. Although subjective, humans categorize art based on the emotion evoked regardless of modality. What remains under-explored is how AI models form their own aesthetic categorization of human-produced media without explicit labels or cross-modal supervision. We present a self-supervised framework that projects four modalities (text, audio, image and video) into a shared 256-dimensional embedding space and applies iterative clustering to discover aesthetic structure. We discuss the divergence between AI-generated cluster assignments and human affective register labels on a weakly supervised multimodal dataset. This work has applications in understanding how AI structures cross-modal similarity, organizing heterogeneous media collections for Retrieval-Augmented Generation (RAG), and automated data labeling.

Figures & tables

Figure 1. Architectural flow diagram depicting training process. Block diagram showing the flow from the original sample through the backbone encoder model into the modality MLP and through the clustering process. The flow is iterative, terminating when cluster assignments are stable.

Affective Register	Text	Audio	Image	Video
Elegiac	840	389	523	380
Sublime	789	553	469	521
Grotesque	967	421	378	384
Uncanny	706	333	408	380
Idyllic	791	342	553	381
Pastoral	725	268	345	164

Table 1. The number of samples in the weakly supervised dataset by affective register (rows) and modality (columns).

Modality	Model	Citation
Text	E5-large-v2	Wang et al. ( Wang et al., 2022 )
Audio	CLAP HTSAT	Wu et al. ( Wu et al., 2023 )
Image	CLIP ViT-L/14	Radford et al. ( Radford et al., 2021 )
Video	V-JEPA 2 ViT-L	Assran et al. ( Assran et al., 2025 )

Table 2. Models utilized to extract embeddings from each modality.

Figure 2. UMAP Comparison of AI-discovered and Human Registry Embeddings Clusters, with 12,010 samples, using cosine metric. Two side by side cluster diagrams. The left side shows the 28 different clusters created by the AI-discovery categorization. The right side show the weakly assigned human label clusters. Both sides show realitve uniformity in cluster asignments except for an area of the left side graph showing a cluster with heterogeneous samples.

Figure 3. AI concept composition by human aesthetic register. Map is normalized by row and sorted by dominant register and cluster purity. Heat map with 28 rows and 6 columns comparing AI concept assignments and human-register labels. The map shows 26 of the 28 AI concepts align with 1 human register except 2 concepts: timeless-nostalgia and soulful-contemplation.

Figure 4. Modality composition of AI concepts clusters. Dashed lines indicate the dataset baseline. Bar chart depicting the AI concept assignments based on modality. Timeless-nostalgia and Soulful-contemplation are both overwhelmingly represented by text.

Multimodal Embedding

Multimodal embedding maps text, images, audio, video, and other data into representations that support comparison and retrieval across modalities.

See all 160 papers

ArtSociety: Multi-Agent Multimodal Collaboration for Art Emotion Understanding
Sep 3, 2026Jian Li, Fanfan Ji, Jinxiang Lai +5Multimodal Emotion Recognition Affective Computing
Nanjing University Suzhou, China · CSE The Hong Kong University of Science and Technology Hong Kong, China · Tencent YoutuLab Shanghai, China
ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval
Jul 29, 2025Nicola Fanelli, Gennaro Vessio, Giovanna CastellanoMultimodal RAG Multimodal IR
Department of Computer Science, University of Bari Aldo Moro, Bari, Italy
Attribute-Grounded Selective Reasoning for Artwork Emotion Understanding with Multimodal Large Language Models
May 15, 2026Cheng Zhang, Yuer Liu, Zhiyu Zhou +2Feature Attribution Multimodal Large Language Models
Department of Computer Science and Technology, Jilin University, Changchun, China · Department of Computer Science, National Taiwan University, Taipei, Taiwan

Sep 3, 2026J/K move · Enter open · S save

ArtSociety: Multi-Agent Multimodal Collaboration for Art Emotion Understanding

Jian Li, Fanfan Ji, Jinxiang Lai, Ying Tai+4

Nanjing University Suzhou, China · CSE The Hong Kong University of Science and Technology Hong Kong, China · Tencent YoutuLab Shanghai, China

Multimodal Emotion Recognition Affective Computing Multi-Agent Collaboration

Open paper AskPDF arXiv

The AffectiveArt Multidimensional Art Emotion Understanding task asks to jointly predict an artwork's fine-grained emotion (12 classes, 1549:1 head-to-tail ratio), binary valence/arousal, and five attribute-grounded descriptions -- sub-tasks that exhibit strong empirical trade-offs, so the single-model solutions we tried do not jointly optimize all of them well. We present ArtSociety, a multi-agent framework that assembles heterogeneous multimodal experts -- a DINOv2-Giant vision agent (A1), a scene-grounded CoT fine-tuned MLLM (A2), and three closed-source reasoning agents (A3-A5) -- and coordinates them with two training-free controllers: (i) a rare-class-aware voting arbiter that lowers the agreement threshold for tail emotions, exploiting decorrelated error patterns across agent families; and (ii) a description-first reasoning agent whose DESCRIBE-then-CLASSIFY chain of thought forces visual evidence before label commitment, yielding near-perfect grounded descriptions. A task-routing policy directs the hard emotion task to the full five-agent ensemble while assigning the near-saturated valence/arousal and generative description tasks to the single strongest reasoning agent. On the official test set (1,000 artworks), ArtSociety achieves an Overall Score of 0.8870 (Classification 0.7789, Description 0.9952). An eleven-variant ablation study reveals that, once method and scale saturate at around 0.76, the decisive gains come from agent collaboration and data-side supervision -- a 30B MoE model trained on older data does not outperform an 8B model trained on better data. Code is available at https://github.com/swordlidev/ArtSociety

Abstract

Multimodal Embedding

Explore similar work

ArtSociety: Multi-Agent Multimodal Collaboration for Art Emotion Understanding

ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval

Attribute-Grounded Selective Reasoning for Artwork Emotion Understanding with Multimodal Large Language Models