EmoGenie: Benchmarking and Evaluating Expressed and Evoked Emotion in T2I Generation with MLLMs

Zixuan Xing; Lancheng Gao; Shengyan Li; Huiyu Duan; Zhaolin Cai; Guangtao Zhai; Xiongkuo Min

cs.CVOct 13, 2026

EmoGenie: Benchmarking and Evaluating Expressed and Evoked Emotion in T2I Generation with MLLMs

Authors: Zixuan Xing, Lancheng Gao, Shengyan Li, Huiyu Duan, Zhaolin Cai, Guangtao Zhai, Xiongkuo Min

Organizations: Shanghai Jiao Tong University

Abstract

Rapid advances in artificial intelligence-generated content (AIGC) have significantly promoted using text-to-image (T2I) models to generate images with specific affective intentions. However, the emotional effects conveyed by generated images frequently deviate from target expectations, and it remains unclear whether multimodal large language models (MLLMs) can reliably interpret the emotion of AI-generated visual content. Existing affective benchmarks are predominantly built on natural images, while affective studies of generated images have largely focused on emotions expressed in portraits, lacking a unified benchmark for studying the emotions expressed and evoked by generated images. We therefore introduce EmoGenie, a benchmark of emotion for AI-generated images that evaluates capabilities in T2I generation and MLLM understanding across expressed and evoked emotions. It comprises 6,400 images generated by eight T2I models using 800 emotion-targeted prompts, annotated with question-answer (QA) pairs, termed EmoGenie-QA, and continuous human ratings, termed EmoGenie-Assess. EmoGenie-QA contains 90,000 closed-set questions in single-image and image-pair settings, as well as 1,057 open-ended questions for emotion description. EmoGenie-Assess benchmarks continuous affective and aesthetic prediction using 13,175 mean opinion scores (MOSs) aggregated from human ratings in terms of valence, arousal, dominance, and aesthetic quality. We further develop EmoProbe, which combines emotion question answering with response-aligned, dimension-specific regression heads for continuous score prediction. Experimental results show that EmoProbe achieves superior results on emotion-related QA tasks, while its affective and aesthetic predictions correlate better with human ratings, demonstrating its state-of-the-art capabilities of unified coarse-to-fine affective understanding for AI-generated images.

Figures & tables

Resource	Images	QA Pairs	Source	Expressed	Evoked	VAD	MOS	Image Pairs	Open-Ended
ArtPhoto ( Machajdik & Hanbury, 2010 )	807	$\times$	Collected	$\times$	$\checkmark$	$\times$	$\times$	$\times$	$\times$
Abstract-8 ( Machajdik & Hanbury, 2010 )	280	$\times$	Collected	$\times$	$\checkmark$	$\times$	$\times$	$\times$	$\times$
Emotion6 ( Peng et al., 2015 )	1,980	$\times$	Collected	$\times$	$\checkmark$	$\times$	$\times$	$\times$	$\times$
EEmo-Bench ( Gao et al., 2025 )	1,960	6,773	Collected	$\times$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$
MEMO-Bench ( Zhou et al., 2026 )	7,145	$\times$	Generated	$\checkmark$	$\times$	$\times$	$\checkmark$	$\times$	$\times$
EmoNet-Face ( Schuhmann et al., 2026 )	2,500	$\times$	Generated	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\times$

Table 1: Emotion-analysis datasets and benchmarks. A check indicates an explicitly supported annotation or question format; a cross indicates that the resource does not provide it. QA Pairs reports the number of question–answer pairs. VAD denotes valence, arousal, and dominance; MOS denotes an aesthetic mean opinion score.

Figure 1: EmoGenie construction pipeline. (a) Designs prompts for expressed and evoked emotion. (b) Generates $6{,}400$ images from $800$ prompts with eight T2I models. (c) Collects human annotations of emotion, intensity or VAD, aesthetic quality, and supporting visual evidence. (d) Constructs EmoGenie-QA with $90$ K closed-set and $1{,}057$ open-ended QA pairs, and EmoGenie-Assess with $13{,}175$ affective and aesthetic scores.

Type	Label	T2I model	Year	Resolution
Open-source	PXA	PixArt- $\Sigma$ ( Chen et al., 2024 )	$2024$	$1024\times 1024$
Open-source	SD3	Stable Diffusion 3 ( Esser et al., 2024 )	$2024$	$1024\times 1024$
Open-source	SD35	Stable Diffusion 3.5 ( Stability AI, 2024 )	$2024$	$1024\times 1024$
Open-source	FLX2	FLUX.2 ( Labs, 2025 )	$2025$	$1360\times 1360$
Open-source	ZIM	Z-Image ( Cai et al., 2025 )	$2025$	$1024\times 1024$
Open-source	QWI	Qwen-Image ( Wu et al., 2025 )	$2025$	$1328\times 1328$

Table 2: Details of T2I models used for the generation. Year denotes the model release year, and resolution denotes the generation setting used for benchmark construction. Open-source models provide publicly released checkpoints.

Figure 2: Overview of EmoProbe training. Stage $1$ instruction-tunes the vision encoder, multimodal projector, and language model for single-image and image-pair emotion perception. In Stage $2$ , the projector is frozen, the vision and language LoRA parameters remain trainable, and four dimension-specific regression heads map response-aligned representations to continuous affective and aesthetic scores. The Stage $2$ objective combines response cross-entropy with masked regression losses.

Figure 3: Performance of T2I models and MLLMs from the expressed- and evoked-emotion perspectives. Panels (a) and (b) compare emotion alignment and aesthetic quality across the eight T2I models for expressed and evoked emotion, respectively. Panels (c) and (d) compare MLLM performance for expressed and evoked emotion, respectively, across single-image, image-pair, and overall perception and description scores. The aggregate Overall axis denotes the mean of the perception and description scores.

Perspectives	Expressed emotion									Evoked emotion										Overall
Dimensions	Emotion alignment								Aesthetics	Emotion alignment									Aesthetics	Aggregate [-1pt] alignment	Aesthetics
Models / Metrics	ANG	DIS	FEA	HAP	NEU	SAD	SUR	Overall	(1–5)	AMU	ANG	AWE	CON	DIS	EXC	FEA	SAD	Overall	(1–5)		(1–5)
PixArt- $\Sigma$	77.4%	18.3%	11.7%	95.7%	72.7%	93.9%	98.6%	66.8%	2.02	42.0%	0.0%	66.0%	90.0%	80.0%	74.0%	70.0%	66.0%	61.0%	3.45	63.9%	2.74
SD3	39.6%	20.0%	0.0%	100.0%	81.8%	30.3%	80.0%	48.0%	2.68	56.0%	8.0%	76.0%	88.0%	54.0%	80.0%	56.0%	54.0%	59.0%	3.42	53.5%	3.05
SD3.5	71.7%	25.0%	23.3%	95.7%	84.1%	66.7%	98.6%	65.5%	2.54	58.0%	2.0%	76.0%	94.0%	68.0%	74.0%	70.0%	54.0%	62.0%	3.52	63.8%	3.03
FLUX.2	49.1%	43.3%	43.3%	100.0%	86.4%	92.4%	98.6%	73.2%	3.32	58.0%	12.0%	80.0%	96.0%	86.0%	84.0%	74.0%	90.0%	72.5%	3.52	72.9%	3.42

Table 3: Target-emotion alignment and aesthetic quality for eight T2I models on EmoGenie. Alignment is soft accuracy (%): the target emotion shares the highest human vote count. Avg. pools the

400

images from each perspective, whereas Aggregate pools all

800

images. Aesthetics is the mean human rating on a

1

–

5

scale. Best results are in bold , and second-best results are underlined .

Categories	Expressed emotion						Evoked emotion						Aggregate
Tasks	Perception			Description			Perception			Description			Perception	Description	Overall
Models	Single $\uparrow$	Pair $\uparrow$	Overall $\uparrow$	Single $\uparrow$	Pair $\uparrow$	Overall $\uparrow$	Single $\uparrow$	Pair $\uparrow$	Overall $\uparrow$	Single $\uparrow$	Pair $\uparrow$	Overall $\uparrow$			Overall
Random guess	35.98%	50.00%	44.71%	–	–	–	38.37%	50.00%	44.20%	–	–	–	44.43%	–	–
General-purpose open-source MLLMs
Janus-Pro-7B ( Chen et al., 2025 )	44.95%	59.68%	54.12%	55.77%	68.75%	57.39%	39.94%	53.72%	46.85%	58.73%	56.80%	58.59%	50.09%	58.13%	54.11%
LLaVA-OneVision-1.5-8B-Instruct ( An et al., 2025 )	52.81%	58.60%	56.42%	58.68%	70.45%	60.14%	44.39%	57.38%	50.90%	62.02%	60.10%	61.87%	53.36%	61.22%	57.29%

Table 4: MLLM evaluation on the EmoGenie-QA subset. Perception reports exact-match accuracy (%), while description reports a normalized evaluation score (%). Aggregate Overall is the mean of Aggregate Perception and Aggregate Description. Random-guess accuracy is computed by averaging the reciprocal of the number of answer choices across perception questions.

Dimensions	Valence		Arousal		Dominance		Aesthetics		Overall
Models / Metrics	SRCC $\uparrow$	PLCC $\uparrow$	SRCC $\uparrow$	PLCC $\uparrow$	SRCC $\uparrow$	PLCC $\uparrow$	SRCC $\uparrow$	PLCC $\uparrow$	SRCC $\uparrow$	PLCC $\uparrow$
Janus-Pro-7B ( Chen et al., 2025 )	0.727	0.577	0.380	0.294	0.089	0.085	$-0.072$	$-0.044$	0.281	0.228
LLaVA-OneVision-1.5-8B-Instruct ( An et al., 2025 )	0.783	0.760	0.394	0.393	$-0.083$	$-0.061$	0.326	0.318	0.355	0.353
Qwen2.5-VL-7B-Instruct ( Bai et al., 2025b )	0.646	0.625	0.276	0.284	0.203	0.163	0.173	0.152	0.325	0.306
Qwen3-VL-8B-Instruct ( Bai et al., 2025a )	0.675	0.732	0.346	0.324	$-0.172$	$-0.130$	0.317	0.337	0.291	0.316
Qwen3.5-VL-9B ( Qwen Team, 2026 )	0.758	0.761	0.232	0.255	0.075	0.039	0.169	0.202	0.308	0.314

Table 5: MLLM evaluation on the EmoGenie-Assess subset. Results report SRCC and PLCC correlations with human ratings; Overall is the unweighted mean across the four dimensions.

No.

Backbone

Training components

Perception

Description

Assessment

Vision

Projector

LLM

Reg.

Single

\uparrow

Pair

\uparrow

Overall

\uparrow

Single

\uparrow

Pair

\uparrow

Overall

\uparrow

Valence

\uparrow

Arousal

\uparrow

Dominance

\uparrow

Aesthetics

\uparrow

Overall

\uparrow

(1)

InternVL3.5-8B

\times

\times

\checkmark

\times

71.80%

81.10%

76.97%

68.03%

72.05%

68.41%

0.746/0.738

0.417/0.437

0.319/0.377

0.261/0.295

0.436/0.462

(2)

InternVL3.5-8B

\checkmark

\checkmark

\checkmark

\times

77.08%

81.84%

79.72%

68.69%

72.60%

69.06%

0.729/0.677

0.404/0.430

0.142/0.260

0.350/0.331

0.406/0.424

(3)

InternVL3.5-8B

\checkmark

\checkmark

\checkmark

\checkmark

77.70%

81.77%

79.96%

70.88%

73.30%

71.11%

0.836 / 0.855

0.481 / 0.499

0.628 / 0.616

0.637 / 0.756

0.645 / 0.681

(4)

Qwen3.5-VL-9B

\checkmark

\checkmark

\checkmark

\times

69.90%

81.13%

76.14%

38.12%

41.73%

38.47%

0.790/0.833

0.250/0.312

0.348/0.337

0.558/0.637

0.486/0.530

(5)

Qwen3.5-VL-9B

\checkmark

\checkmark

\checkmark

\checkmark

71.30%

81.49%

76.96%

44.82%

56.30%

45.91%

0.835 / 0.862

0.451 / 0.466

0.623 / 0.609

0.625 / 0.748

0.634 / 0.671

Table 6: EmoProbe ablations on EmoGenie. Perception reports exact-match accuracy (%), description reports a normalized evaluation score (%), and assessment reports SRCC / PLCC. Variants without regression heads are evaluated on assessment using Q-BENCH, whereas two-stage variants use the learned regression heads.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Distribution of aesthetic quality across T2I models for (a) expressed-emotion and (b) evoked-emotion images. Each violin summarizes the human-aggregated ratings of $400$ images from one model. Internal boxes indicate the interquartile range, white lines denote the median, and red points denote the mean. All ratings use the original $1$ – $5$ scale.

Figure 5: Structured annotation interfaces. (a) Expressed-emotion annotation records the emotion conveyed by the depicted person, its expression intensity, and aesthetic quality. (b) Evoked-emotion annotation records the emotion elicited in the viewer, valence, arousal, dominance, and aesthetic quality. The displayed selections illustrate the interface layout.

Panel of Figure 6 — Figure 6: Evidence-grounded description interfaces. (a) Expressed-emotion annotation elicits visual evidence for the category and intensity judgments and records contextual or additional emotional cues. (b) Evoked-emotion annotation characterizes scene-level evidence, elicits rationales for the VAD ratings, and records mixed emotional responses. (c) An expressed-emotion image pair requires a comparison of the facial and bodily evidence underlying an intensity difference. (d) An evoked-emotion image pair requires the visual factors underlying a difference in continuous affective ratings to be identified.

Perspective	Dimension	Agreement measure	Estimate
Expressed emotion	Emotion category	Gwet’s AC1	$0.682$
	Expression intensity	Quadratic-weighted AC2	$0.734$
	Aesthetic quality	Quadratic-weighted AC2	$0.809$
Evoked emotion	Emotion category	Gwet’s AC1	$0.449$
	Valence	Quadratic-weighted AC2	$0.784$
	Arousal	Quadratic-weighted AC2	$0.798$

Appendix

Table 7: Inter-annotator agreement for the principal annotation dimensions. AC1 is used for nominal emotion categories, and quadratic-weighted AC2 is used for ordered integer ratings.

Figure 7: Category-level confusion matrices for the emotion-generation capability of eight T2I models. Rows denote generation prompts and columns denote emotions assigned by human annotators. For expressed emotion, A–G correspond to anger, disgust, fear, happiness, neutral, sadness, and surprise. For evoked emotion, A–H correspond to amusement, anger, awe, contentment, disgust, excitement, fear, and sadness. Each row is normalized to sum to $1$ ; diagonal entries indicate target-emotion alignment, and off-diagonal entries show the distribution of emotion substitutions.

Model	Single image					Image pair
Model	Evoked emotion	Valence	Arousal	Dominance	Aesthetics	Target alignment	Valence	Arousal	Dominance	Aesthetics
Janus-Pro-7B	56.99%	37.26%	40.51%	24.06%	42.64%	60.81%	57.77%	52.29%	48.18%	51.80%
LLaVA-OneVision-1.5-8B-Instruct	73.10%	50.10%	32.02%	28.33%	42.21%	56.46%	64.26%	59.59%	51.17%	54.93%
Qwen2.5-VL-7B-Instruct	74.41%	51.54%	33.43%	31.46%	49.30%	78.79%	72.30%	62.11%	49.39%	62.24%
Qwen3-VL-8B-Instruct	74.05%	57.05%	44.34%	27.60%	42.91%	78.65%	72.67%	63.05%	56.77%	60.91%
Qwen3.5-VL-9B	77.13%	50.48%	35.35%	23.33%	37.83%	75.14%	72.49%	56.22%	42.95%	64.42%

Appendix

Table 8: Dimension-wise closed-set QA accuracy for evoked emotion (%). Single-image and image-pair questions occupy separate column groups. The best result in each column is shown in bold; the bottom row gives the number of questions.

Model	Single image			Image pair
Model	Emotion	Emotion intensity	Aesthetics	Target alignment	Emotion intensity	Aesthetics
Janus-Pro-7B	69.44%	32.80%	32.66%	71.06%	63.69%	49.60%
LLaVA-OneVision-1.5-8B-Instruct	87.50%	36.56%	34.68%	69.52%	64.00%	47.45%
Qwen2.5-VL-7B-Instruct	87.60%	35.42%	46.67%	76.62%	79.03%	62.31%
Qwen3-VL-8B-Instruct	86.61%	30.30%	34.24%	81.16%	79.35%	62.96%
Qwen3.5-VL-9B	87.10%	30.75%	32.92%	84.17%	80.08%	66.37%

Appendix

Table 9: Dimension-wise closed-set QA accuracy for expressed emotion (%). Single-image and image-pair questions occupy separate column groups. The best result in each column is shown in bold; the bottom row gives the number of questions.

Figure 8: Qualitative comparison of continuous affective and aesthetic assessment. Each row presents an example for valence, arousal, dominance, or aesthetic quality and compares the human-aggregated score with predictions from EmoProbe and the evaluated baselines. EmoProbe yields predictions close to the human scores across all four illustrated dimensions.

Figure 9: Challenging closed-set perception cases for evoked emotion. The examples cover single-image judgments of valence and dominance and image-pair comparisons of arousal and dominance. Green and red markers indicate correct and incorrect responses, respectively; EmoProbe answers all four illustrated questions correctly despite systematic errors among the comparison models.

Figure 10: Challenging closed-set perception cases for expressed emotion. The examples include single-image recognition of emotion category and expression intensity, together with image-pair comparisons of expression intensity and target-emotion alignment. Green and red markers indicate correct and incorrect responses, respectively; EmoProbe provides the correct response in each case.

Explore similar work

Multimodal Language Models as Text-to-Image Model Evaluators
May 1, 2025Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat +5LLM-as-a-Judge Benchmark Design
1FAIR at Meta · University of Texas at Austin · ∗Work done at Meta +3
Evaluating the Evaluators: Diagnosing Large Multimodal Models for AI-Generated Image Assessment
Sep 29, 2026Yu Zhao, Jiarui Wang, Huiyu Duan +5LLM-as-a-Judge Image Generation Evaluation
Shanghai Jiao Tong University · Dalian University of Technology
EmoStyle: Affective Conditioning of Style-Specialist Experts for Emotional Image Generation
Jul 11, 2026Dexiang Hong, Yijie Guo, Weidong Chen +4Conditional Image Generation T2I Generation
University of Science and Technology of China HeFei, China · Harbin Institute of Technology, WeiHai WeiHai, China

May 1, 2025J/K move · Enter open · S save

Multimodal Language Models as Text-to-Image Model Evaluators

Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat, Koustuv Sinha+4

1FAIR at Meta · University of Texas at Austin · ∗Work done at Meta+3

LLM-as-a-Judge Benchmark Design Automated Evaluation Vision-Language Model-as-a-Judge T2I Generation Evaluation

Open paper AskPDF arXiv

The steady improvements of text-to-image (T2I) generative models lead to slow deprecation of automatic evaluation benchmarks that rely on static datasets, motivating researchers to seek alternative ways to evaluate T2I progress. We present Multimodal Text-to-Image Eval (MT2IE), an evaluation framework in which a single multimodal large language model (MLLM) acts as an evaluator agent, iteratively generating the evaluation prompts and scoring the resulting images. We show that MT2IE's image-text consistency scores have higher correlation with human judgment than metrics previously introduced in the literature. MT2IE generates prompts that are efficient at probing T2I model performance: closely recovering the official T2I model rankings of three structurally distinct benchmarks from just 20 generated evaluation prompts, 28-105x fewer than the benchmarks' own prompt sets. When compared to existing evaluation metrics such as CLIPScore, VIEScore, and VQAScore, MT2IE's T2I model rankings are more faithful and far more consistent across multiple evaluation seeds when using the same number of prompts. MT2IE can also adapt evaluation to the model being tested: rewriting each prompt based on the model's own measured performance to produce a bespoke per-model benchmark that still recovers the official rankings and keeps the evaluated model in an informative scoring range. We hope that these results will encourage the development of dynamic and interactive evaluation frameworks, and mitigate the deprecation of automatic evaluation benchmarks.

Abstract

Explore similar work

Multimodal Language Models as Text-to-Image Model Evaluators

Evaluating the Evaluators: Diagnosing Large Multimodal Models for AI-Generated Image Assessment

EmoStyle: Affective Conditioning of Style-Specialist Experts for Emotional Image Generation