ProFocus: Interpreting Affective Experience in Artistic Images with Progressive Visual Focusing

Zhiyan Zhang; Zicheng Yan; Jianqi Chen; Peipei Song; Shanshan Wang; Xun Yang

cs.CVAug 14, 2026

ProFocus: Interpreting Affective Experience in Artistic Images with Progressive Visual Focusing

Authors: Zhiyan Zhang, Zicheng Yan, Jianqi Chen, Peipei Song, Shanshan Wang, Xun Yang

Organizations: University of Science and Technology of China Hefei, Anhui, China · Anhui University Hefei, Anhui, China

Abstract

Interpreting the emotional responses triggered by images is central to achieving emotional intelligence. Compared with natural images, visual art is intentionally created to elicit emotional responses from its viewers through abstract concepts and visual metaphors, making affective interpretation particularly challenging. However, most existing methods rely on general-purpose visual embeddings (e.g., CLIP), failing to capture the nuanced cues underlying artistic emotion. To address this gap, we propose ProFocus, a novel framework that models affective experience in artistic images via progressive visual focusing. The key idea is to model visual representation learning inspired by a hierarchical cognitive theory of human aesthetic appreciation. Technically, ProFocus contains two core components: a Hierarchical Art Critic (HAC) and a Progressive Hint Fusion (PHF) module. HAC leverages multimodal large language models to generate structured linguistic priors at three cognitive levels--atmospheric style, narrative subjects, and concrete details--thereby translating artistic perception into coherent semantic guidance. Building upon these priors, PHF departs from conventional cross-modal fusion by sequentially injecting the hierarchical hints into visual features, enabling a progressive focusing process that mirrors human perception. This design allows the model to capture subtle affective cues and produce more faithful explanations. Extensive experiments on the ArtEmis v1.0 and v2.0 datasets demonstrate that ProFocus consistently outperforms state-of-the-art methods in both emotion recognition and affective explanation. Project page: https://github.com/Zhang-Zhiyan/ProFocus.

Figures & tables

Figure 1. An example of interpreting affective experience in artistic images. The previous state-of-the-art SEVLM relies on generalist visual features. Our method enhances emotion reasoning by introducing a hierarchical art critic that analyses artwork at three explainable levels.

Figure 2. Pipeline comparison between (a) SEVLM and (b) the proposed ProFocus. The key innovations are the Hierarchical Art Critic (HAC) and Progressive Hint Fusion (PHF) modules.

Figure 3. Overview of the proposed ProFocus framework. (1) The Hierarchical Art Critic explicitly decouples abstract visual aesthetics into structured linguistic priors across three cognitive levels. (2) The Progressive Hint Fusion module injects these priors into visual features via semantic-guided visual focusing in a coarse-to-fine sequence. (3) The hierarchically refined features are fed into a language decoder to bridge the affective gap and generate subjective affective explanations.

Figure 4. Detailed architecture of the Progressive Hint Fusion. Visual tokens act as queries to retrieve information from hierarchical textual keys and values across three stages.

Method	Backbone	ArtEmis v1.0							ArtEmis v2.0
Method	Backbone	ACC	B@1	B@2	B@3	B@4	M	R	ACC	B@1	B@2	B@3	B@4	M	R
Emotion-Llama ( Cheng et al., 2024 )	Hybrid-LLaMA2	33.4	20.4	9.8	2.7	1.4	20.1	15.6	16.3	14.2	7.4	3.5	1.7	19.3	15.6
AffectGPT ( Lian et al., 2025 )	CLIP-Qwen2.5	35.0	33.5	11.6	2.8	1.6	22.4	16.0	15.4	27.6	9.6	3.6	1.8	20.5	13.8
VidEmo ( Zhang et al., 2025 )	Qwen2.5-VL	45.9	34.4	13.0	4.9	2.2	20.2	19.5	23.2	28.5	10.9	4.3	2.1	18.0	17.4
M2 ( Achlioptas et al., 2021 )	Trans-Trans	60.2	51.1	28.2	15.4	9.0	13.7	28.6	-	-	-	-	-	-	-
SAT ( Achlioptas et al., 2021 )	CNN-LSTM	60.2	52.0	28.0	14.6	7.9	13.4	29.4	43.3	48.7	25.3	13.2	7.3	12.8	27.2

Table 1. Performance comparison with state-of-the-art methods on the ArtEmis v1.0 and v2.0 datasets. The best results are highlighted in bold . The first three methods are emotion-focused multimodal large language models adapted to the image-based setting, whereas the remaining four methods are task-specific models trained on the ArtEmis datasets. For Emotion-Llama, the backbone is denoted as Hybrid-LLaMA2 , where Hybrid refers to a multi-view multimodal encoder composed of HuBERT, a MAE-based local ViT encoder, VideoMAE, and EVA, and LLaMA2 serves as the decoder.

Figure 5. Ablation study on the ArtEmis v1.0 test set. We evaluate three single-prior variants ( Style-only , Subject-only , and Detail-only ), a Concatenation strategy that replaces the proposed attention-based fusion, and a Single-stage variant that removes the hierarchical multi-stage design. The full model consistently performs best on ACC, BLEU@1–4, METEOR, and ROUGE-L, which confirms the value of complementary semantic priors and progressive coarse-to-fine fusion.

Fusion Order	ACC	B@1	B@2	B@3	B@4	M	R
Style $\rightarrow$ Detail $\rightarrow$ Subject	65.7	53.9	30.1	16.3	9.1	13.6	30.3
Subject $\rightarrow$ Style $\rightarrow$ Detail	64.9	54.8	30.5	16.5	9.3	14.0	30.4
Detail $\rightarrow$ Style $\rightarrow$ Subject	66.0	54.5	30.3	16.2	9.0	14.0	30.4
Subject $\rightarrow$ Detail $\rightarrow$ Style	64.9	54.3	30.1	16.1	9.0	13.7	30.5
Detail $\rightarrow$ Subject $\rightarrow$ Style	64.5	54.2	30.0	16.3	9.2	13.4	30.5
Style $\rightarrow$ Subject $\rightarrow$ Detail	66.1	55.3	31.2	17.1	9.8	14.2	30.8

Table 2. Ablation study on the order of semantic text injection in the three-stage fusion module. The proposed coarse-to-fine order yields the best results.

Figure 6. Attention visualization across the three fusion stages. Guided by hierarchical textual priors, the visual focus dynamically aligns with the corresponding semantic regions.

Figure 7. t-SNE visualization of visual feature distributions across nine emotion categories. (a) Implicitly entangled raw visual features from the baseline. (b) Semantically fused features ( $\mathbf{V}_{fused}$ ) by ProFocus.

Figure 8. Qualitative comparison of generated explanations on two representative artworks. Red text denotes incorrect emotion predictions or hallucinated content, whereas green text highlights visually grounded cues that support the final explanation.

Method	EG	VR	DR	LF	OP	Avg.
NLX-GPT2	3.35	3.47	3.15	4.18	3.30	3.49
SEVLM	3.51	3.65	3.26	4.16	3.55	3.63
ProFocus	3.91	4.05	3.78	4.20	3.82	3.95

Table 3. Human evaluation results on the ArtEmis v1.0 test set in Mean Opinion Scores on a 1–5 scale. Avg. denotes the average over EG, VR, DR, LF, and OP.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration / Metric	Value
Hardware Setup
- Training	2 $\times$ NVIDIA Tesla V100 (32GB)
- Inference	1 $\times$ NVIDIA Tesla V100 (32GB)
Total Model Parameters	$\sim$ 212.0 M
- Frozen CLIP ViT-B/16 Encoder	86.2 M
- Custom GPT-2 Decoder (6-layer)	100.2 M

Appendix

Table 4. Summary of model complexity, hardware resources, and inference efficiency.

Panel of Figure 9 — Figure 9. Qualitative examples of the hierarchical semantic priors extracted by the Hierarchical Art Critic. For each representative artwork, the implicitly entangled visual aesthetics are explicitly decoupled into structured linguistic cues across three cognitive levels.

HAC prior source	ACC	B@1	B@2	B@3	B@4	M	R
InternVL-2.5-8B	65.9	54.8	30.7	16.7	9.4	14.0	30.5
Qwen3.5-9B	66.5	55.1	31.2	16.9	9.7	14.0	30.6
ProFocus (LLaVA-1.6-7B)	66.1	55.3	31.2	17.1	9.8	14.2	30.8

Appendix

Table 5. Sensitivity to the MLLM used for generating HAC priors on ArtEmis v1.0.

Retained Stages	ACC	B@1	B@2	B@3	B@4	M	R
Style + Subject (w/o Detail)	64.0	54.3	30.3	16.5	9.3	13.5	30.5
Detail + Style (w/o Subject)	65.4	54.1	30.3	16.4	9.1	13.8	30.3
Subject + Detail (w/o Style)	64.0	54.3	30.4	16.4	9.2	13.7	30.4
Full (Style + Subject + Detail)	66.1	55.3	31.2	17.1	9.8	14.2	30.8

Appendix

Table 6. Ablation study on pairwise combinations of semantic priors. We omit one specific prior stage to evaluate the contribution of the remaining two.

Figure 10. Qualitative comparison of generated explanations. Top : A success case where ProFocus accurately captures the aesthetic elements without hallucination. Bottom : A failure case where ProFocus predicts the correct emotion and subject, but hallucinates a specific action (“crying”) to justify the abstract sadness.

Hyperparameter	Variant	ACC	B@1	B@2	B@3	B@4	M	R
Cross-Attention Heads	Heads = 8	64.5	54.7	30.7	16.6	9.3	13.8	30.7
Cross-Attention Heads	Heads = 16	64.2	54.5	30.6	16.2	9.3	14.0	30.7
Loss Weights	1.5 : 1.0 : 1.0	64.3	54.1	30.1	16.3	9.2	14.0	30.4
Loss Weights	1.0 : 1.0 : 1.5	64.6	54.5	30.4	16.7	9.5	14.1	30.5
Fusion Layers	2 Layers	66.1	54.5	30.6	16.8	9.5	14.0	30.6
Fusion Layers	3 Layers	65.9	54.7	30.6	16.6	9.1	13.8	30.4

Appendix

Table 7. Hyperparameter sensitivity experiments on attention heads, loss coefficients, and fusion layer depth. The last row denotes the default configuration of ProFocus.

Emotion Recognition

Emotion recognition uses language, voice, face, body movement, and physiological signals to infer emotional categories or dimensions such as valence and arousal.

See all 129 papers

Beyond Vision-Language Models: Visual-Semantic Affect Graphs for Emotion Recognition in Artworks
Oct 26, 2026Ngoc-Duy Mai, Ha-Thanh Nguyen, Hideaki TakedaMultimodal Emotion Recognition Valence-Arousal Modeling
National Institute of Informatics, Tokyo, Japan · The Graduate University for Advanced Studies, SOKENDAI, Kanagawa, Japan · Research and Development Center for Large Language Models, NII, Tokyo, Japan +1
EmoStyle: Affective Conditioning of Style-Specialist Experts for Emotional Image Generation
Jul 11, 2026Dexiang Hong, Yijie Guo, Weidong Chen +4Conditional Image Generation T2I Generation
University of Science and Technology of China HeFei, China · Harbin Institute of Technology, WeiHai WeiHai, China
Attribute-Grounded Selective Reasoning for Artwork Emotion Understanding with Multimodal Large Language Models
May 15, 2026Cheng Zhang, Yuer Liu, Zhiyu Zhou +2Feature Attribution Multimodal Large Language Models
Department of Computer Science and Technology, Jilin University, Changchun, China · Department of Computer Science, National Taiwan University, Taipei, Taiwan

Oct 26, 2026J/K move · Enter open · S save

Beyond Vision-Language Models: Visual-Semantic Affect Graphs for Emotion Recognition in Artworks

Ngoc-Duy Mai, Ha-Thanh Nguyen, Hideaki Takeda

National Institute of Informatics, Tokyo, Japan · The Graduate University for Advanced Studies, SOKENDAI, Kanagawa, Japan · Research and Development Center for Large Language Models, NII, Tokyo, Japan+1

Multimodal Emotion Recognition Valence-Arousal Modeling

Open paper AskPDF arXiv

Emotion recognition in artworks remains challenging because affective meaning is often subjective, implicit, and shaped by relationships between visual form and semantic associations. While vision-language models provide strong multimodal representations, direct image-to-emotion prediction may overlook explicit relations among formal, semantic, and affective cues that support emotional interpretation. This paper proposes Visual-Semantic Affect Graphs (VSAG), a graph-based framework that represents artworks through interpretable feature nodes, including color, brushstroke, composition, light-shadow, line quality, and extracted visual entities. The graph learns feature-emotion associations for dominant emotion prediction, with valence-arousal structure modeled through an optional hierarchical auxiliary affect module. We evaluate VSAG and VSAG-TopK against direct VLM inference using Qwen3-VL and InternVL3.5 on EmoArt-5k. Results show that direct VLMs provide useful but modest baselines, while graph-based models improve dominant emotion prediction. In particular, the hierarchical valence-arousal variant of VSAG achieves the best overall performance, improving accuracy, macro-F1, and weighted-F1 over both direct VLM inference and the emotion-only VSAG baseline. Per-class results indicate that graph-based models and VLMs capture partly complementary signals, while valence-arousal structure is most useful when integrated as a coarse-to-fine affect hierarchy rather than as a simple post-hoc correction. These findings suggest that explicit visual-semantic structure can support both prediction and interpretation in artwork emotion recognition.

Abstract

Emotion Recognition

Explore similar work

Beyond Vision-Language Models: Visual-Semantic Affect Graphs for Emotion Recognition in Artworks

EmoStyle: Affective Conditioning of Style-Specialist Experts for Emotional Image Generation

Attribute-Grounded Selective Reasoning for Artwork Emotion Understanding with Multimodal Large Language Models