AesCanvas: A Large-Scale Dataset and Benchmark for Aesthetic Critique and Contextual Suitability

Xuanwei Hu; Haoyu Dong; Kejun Wu; Tianyi Liu; Jianjun Gao

cs.CVAug 27, 2026

AesCanvas: A Large-Scale Dataset and Benchmark for Aesthetic Critique and Contextual Suitability

Authors: Xuanwei Hu, Haoyu Dong, Kejun Wu, Tianyi Liu, Jianjun Gao

Organizations: School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, China · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have extended Image Aesthetic Assessment (IAA) beyond scalar scores toward interpretable critique and guidance. Yet existing benchmarks mainly assess intrinsic visual quality or fixed domain criteria, leaving open whether an appealing image is appropriate for a specific purpose, audience, cultural setting, or domain convention. We introduce AesCanvas, a unified suite with two complementary components: CritiqueCanvas with 519,136 instruction-response pairs from 54,300 images supports long-form, multi-dimensional critique across photography, painting, and virtual imagery, whereas ContextCanvas with 301 expert-reviewed use scenarios evaluates contextual aesthetic suitability in realistic use scenarios. Under a unified protocol, we evaluate closed-source frontier, open-weight general, and aesthetic-specific MLLMs. Results reveal a clear separation between critique generation and context-sensitive judgment: reference-based lexical and semantic metrics only partially capture critique quality, while aesthetic specialists remain competitive on selected critique metrics yet substantially lag strong general-purpose MLLMs on ContextCanvas. Further analyses show that aesthetic specialization does not reliably transfer to contextual suitability and that model decisions may fail to track or ground themselves in decisive contextual visual cues. These findings establish culturally situated, evidence-grounded suitability as a distinct objective for aesthetic modeling.

Figures & tables

Figure 1: Fluent aesthetic critique does not guarantee contextually valid judgment. The top example shows an evidence-grounded critique, whereas the bottom shows a suitability error in which visually appealing attributes are mistaken for appropriateness in a mismatched use context. This contrast motivates evaluating critique grounding and contextual suitability as complementary capabilities.

Figure 2: Four recurring failure modes in practical aesthetic evaluation. Models may generate critiques that are visually ungrounded or rely on inappropriate domain priors, and may judge real-world suitability from surface aesthetics or stylistic plausibility rather than communicative and contextual fit.

Figure 3: Construction of AesCanvas . (a) A shared multi-domain image pool is collected and filtered. (b) CritiqueCanvas is generated through dimension-aware prompt planning, MLLM-assisted annotation, and quality filtering. (c) ContextCanvas is constructed from realistic aesthetic-use scenarios followed by expert review.

Task	Metric Family	Metrics
Critique	Lexical overlap	BLEU, ROUGE-L, METEOR
	Semantic similarity	BERT-F1, SBERT-Cos
	Image–text alignment	CLIPScore
	Descriptive statistics	Avg. Length, Avg. Dim. Hits, Top-3 Dimensions
Context	Prediction quality	Accuracy, Macro-F1
	Prediction tendency	Yes Rate

Table 1: Metrics used for the two evaluation tasks.

Model	Acc.	Macro-F1	Yes (%)
Closed-source MLLMs
Claude Opus 5	91.69	90.45	30.24
Gemini 3.1 Pro	85.71	83.06	26.12
GPT-5.5	83.06	80.13	27.49
Claude Opus 4.6	72.76	67.96	27.15
GPT-5.2	71.43	64.51	20.96

Table 2: Contextual aesthetic suitability results. Acc. is exact-match accuracy on all 301 cases. Macro-F1 and Yes (%) are computed on the 291 binary cases; the gold Yes rate is 38.14%. Best Acc. and Macro-F1 are bolded.

\dagger

denotes the development model used for difficulty screening.

Closed-source MLLMs
Model	BLEU	ROUGE-L	METEOR	BERT-F1	SBERT-Cos	CLIPScore
Gemini 3.1 Pro	7.15	0.250	0.281	0.699	0.848	0.303
GPT-5.2	2.84	0.210	0.209	0.669	0.818	0.301
Open-weight General MLLMs
GLM-4.6-Flash ( GLM-V Team et al. 2026 )	5.50	0.241	0.273	0.676	0.844	0.317
LLaVA-OneVision ( An et al. 2025 )	10.32	0.285	0.275	0.724	0.839	0.319

Table 3: Quantitative evaluation of long-form aesthetic critique generation on CritiqueCanvas. The reported metrics assess complementary aspects of generated critiques, including reference alignment through lexical and semantic similarity, and image–text relevance.

Rating	GPT	Gem.	Qwen3-VL FT	AM	AQ
Human	3.9	4.1	2.5	2.6	2.3
MLLM	5.0	4.5	2.5	2.6	2.2

Table 4: Human and MLLM ratings of overall critique quality.

Figure 4: Cross-task analysis. (a) Base–specialist transfer on ContextCanvas; * denotes Bonferroni-corrected McNemar

p<.05

. (b) ContextCanvas accuracy against the mean percentile over six CritiqueCanvas metrics.

Metric	Gem.	GPT	Qwen+	AM
Accuracy $\uparrow$	83	65	42	21
EGA $\uparrow$	79	55	31	3
Unsupported $\downarrow$	4.82	15.38	26.19	85.71

Table 6: Decisive-cue attribution on 100 balanced cases.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Scale	Photo- graphy	Painting / art	AIGC / virtual	Long-form critique	Structured dimensions	Grounded explanation	Use-context judgment
AVA	$>$ 250K images	$\checkmark$	–	–	–	$\triangle$	–	–
AADB	10K images	$\checkmark$	–	–	–	$\checkmark$	–	–
PARA	31,220 images	$\checkmark$	–	–	–	$\checkmark$	–	–
AGIQA-3K	2,982 AGIs	–	–	$\checkmark$	–	$\triangle$	–	–
AIGCIQA- 2023	2,400 AGIs	–	–	$\checkmark$	–	$\checkmark$	–	–
AesBench/ EAPD	2,800 images; 11,200 annotations	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	–

Appendix

Table S1 : Comparison of media coverage and aesthetic supervision.

Field	CritiqueCanvas	ContextCanvas
Primary task	Long-form, multi-dimensional aesthetic critique	Contextual aesthetic suitability judgment
Data unit	Image, instruction, selected dimensions, and response	Image(s), use scenario, options, gold label, rationale, and provenance
Scale	519,136 instruction–response pairs	301 cases using 302 unique images
Output	Open-ended critique	291 two-option and 10 three-option decisions
Coverage	Photography, painting, and virtual imagery	Painting, sculpture, illustration/comics, animation/digital imagery, photography, and film
Evaluation role	6,000-case generation evaluation	Fixed-order diagnostic benchmark

Appendix

Table S2 : Suite-level data card for the two AesCanvas components.

Figure S1 : Representative CritiqueCanvas examples across photography, illustration, and painting. Each example pairs an image with its rendered query and the corresponding aesthetic focus. The examples include both broad image-appreciation prompts and prompts targeting specific dimensions such as light and shadow, atmosphere, tonal treatment, and formal analysis.

Figure S2 : Representative retained ContextCanvas cases. Top row, from left to right: HB208 (Mid-Autumn cultural fit), HB014 (historical military identity), and HB159 (restorative return versus the Lotus-Eaters’ withdrawal). Bottom row: HB155 (fiction–reality confusion), HB181 (teamwork under visual tension), and HB186 (addiction in a refined setting). Each card separates visible form, use scenario, contextual evidence, decision, and gold label.

Figure S3 : Human-drafted candidates excluded after expert review. From left to right: R41Q05 was insufficiently challenging because conspicuous negative affect enabled a coarse shortcut; R41Q08 weakened visual necessity because the decisive betrayal occurs outside the depicted moment; and R69Q06 lacked a uniquely compelled gold label for a contemporary campaign.

Model	Requested model/checkpoint	Precision	Generation	Adapter details
GPT-5.2	openai/gpt-5.2	API	$T=0$ , 256	OpenRouter; reasoning excluded
Gemini 3.1 Pro	google/gemini-3.1-pro-preview	API	$T=0$ , 256	Google provider preference; minimal reasoning excluded
GLM-4.6-Flash	GLM-4.6-Flash	bfloat16	greedy, 256	Native; thinking disabled and stripped
LLaVA-OneVision	LLaVA-OneVision-1.5-8B-Instruct	float16	greedy, 256	Native
InternVL3.5-8B	InternVL3_5-8B	float16	greedy, 256	Native InternVL chat API
Qwen3-VL	Qwen3-VL-8B-Instruct	bfloat16	greedy, 256	Qwen processor; SDPA

Appendix

Table S3 : CritiqueCanvas inference configurations. “Native” means the checkpoint’s own processor and chat template after the common

448\times 448

RGB resize.

Model	Yes F1	No F1	Pred. Yes
Claude Opus 5	87.44	93.47	30.24
Gemini 3.1 Pro	77.01	89.11	26.12
GPT-5.5	73.30	86.96	27.49
Claude Opus 4.6	56.84	79.08	27.15
GPT-5.2	50.00	79.02	20.96
GLM-5V-Turbo	47.06	63.37	43.64

Appendix

Table S4 : Class-specific behavior on the 291 two-option ContextCanvas cases. All values are percentages. The gold Yes rate is 38.14%.

Figure S4 : Cross-model outputs for ContextCanvas case HB208 (gold: No). GPT-5.2 and Gemini 3.1 Pro reject the image, whereas Qwen3.7-Plus, ArtiMuse, and ArtQuant-APDD accept it. The displayed responses illustrate whether models recognize the culturally specific visual identity rather than relying only on surface festival cues.

Figure S5 : ArtQuant-APDD returns a bare Yes response without an explanatory rationale;

Figure S6 : Three CritiqueCanvas cases compared across five models. From top to bottom: CC001 (illustration), CC003 (photograph), and CC002 (painting). The panel preserves each prompt and representative output excerpt, exposing differences in visual grounding, medium specificity, and analytical focus.

Figure S7 : Response-level scoring diagnostics on E004 (top) and HB208 (bottom). E004 contrasts a grounded GPT-5.2 rationale with an unsupported ArtQuant-APDD label. HB208 contrasts fluent but incorrect Qwen3.7-Plus reasoning with an ArtQuant-APDD free-form response that illustrates output-format and parser sensitivity.

Model	Length	Len/Ref	DimHits	Density	Prompt Align	Generic Top 3 Dimensions
Closed-source Models
Gemini 3.1 Pro	166.58	1.241	6.53	11.87	90.8%	Composition, Emotion, Lighting
GPT-5.2	141.52	1.054	5.71	8.60	93.8%	Composition, Emotion, Lighting
General MLLMs
GLM-4.6-Flash	189.37	1.410	6.50	9.44	94.2%	Composition, Content, Emotion
InternVL3.5-8B	142.78	1.090	5.17	11.18	92.5%	Content, Composition, Emotion

Appendix

Table S5: Generation style and aesthetic-dimension coverage on CritiqueCanvas. Len/Ref is average output length relative to the reference; DimHits is the average number of explicitly addressed dimensions; aesthetic-term density is the number of matched lexical/phrase occurrences per 100 words. Prompt Align requires every dimension detected in a focused prompt to be detected in the response. Generic Top 3 excludes prompts that explicitly request a dimension.

Model	Human	MLLM judge
GPT-5.2	3.9	5.0
Gemini 3.1 Pro	4.1	4.5
Qwen3-VL-FT	2.5	2.5
ArtiMuse	2.6	2.6
ArtQuant-APDD	2.3	2.2

Appendix

Table S6 : Aggregate human and MLLM ratings of critique quality on 100 randomly sampled CritiqueCanvas cases.

Model	Full	Orig.	Edit	Pair	C/R	NCU	95% CI
GPT-5.2	71.43	72.22	58.33	36.11	13/2	30.56	[11.11, 50.00]
Gemini 3.1 Pro	85.71	86.11	38.89	27.78	10/1	25.00	[8.33, 41.67]
Qwen3.7-Plus	51.83	52.78	75.00	30.56	11/1	27.78	[11.11, 44.44]
ArtQuant-APDD	29.57	30.56	72.22	8.33	3/2	2.78	[-8.33, 13.89]
ArtiMuse	19.60	19.44	97.22	16.67	6/0	16.67	[5.56, 30.56]

Appendix

Table S7 : Counterfactual audit on 36 pairs. Full and Orig. are accuracies on the complete benchmark and selected incompatible originals; Edit is accuracy on compatible edits. Pair requires both labels to be correct. C/R are correct and reverse directional updates. NCU confidence intervals are paired bootstrap intervals.

Figure S8 : Four No-to-Yes counterfactual interventions. From left to right: CF20/HB163 (coercion to reassurance), CF18/HB084 (isolation to inclusion), CF28/E108 (incomplete to completed journey), and CF35/HB075 (incompatibility to friendliness). Each column shows the original and edited image, the targeted intervention, and each model’s before–after prediction; colored boxes indicate agreement or disagreement with the corresponding gold label.

Model	Accuracy	EGA	Unsupported (%)
Gemini 3.1 Pro	83	79	4.82
GPT-5.2	65	55	15.38
Qwen3.7-Plus	42	31	26.19
ArtiMuse	21	3	85.71
ArtQuant-APDD	27	0	100.00

Appendix

Table S8 : Decisive-cue grounding on 100 ContextCanvas cases.

Model	Text	Neutral	Evidence	Image
GPT-5.2	45	77	84	71
Gemini 3.1 Pro	58	86	90	86
Qwen3.7-Plus	44	78	88	52
ArtQuant-APDD	16	20	19	27
ArtiMuse	16	32	47	20

Appendix

Table S9 : Accuracy under different visual-information conditions on 100 ContextCanvas cases.

Base $\rightarrow$ specialist	Base	Spec.	$\Delta$	$b/c$	95% CI	$p/p_{\mathrm{adj}}$
mPLUG-Owl2 $\rightarrow$ ArtQuant	20.27	29.57	+9.30	29/57	[3.32, 15.28]	.0034/.0134
InternVL3-8B $\rightarrow$ ArtiMuse	23.26	19.60	-3.65	27/16	[-7.97, 0.33]	.1263/.5052
LLaVA-v1.5 $\rightarrow$ AesExpert	12.62	18.27	+5.65	10/27	[1.66, 9.63]	.0076/.0305
LLaVA-OV $\rightarrow$ Q-SiT	19.93	19.60	-0.33	12/11	[-3.32, 2.66]	1.000/1.000

Appendix

Table S10 : Paired base-to-specialist ContextCanvas transfer.

b

counts cases correct only for the base and

c

cases correct only for the specialist. Raw McNemar

p

-values and Bonferroni-adjusted values are both shown.

Multimodal Model Evaluation

Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.

See all 772 papers

Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models
Jun 29, 2026Sajjad Ghiasvand, Maryam Amirizaniani, Haniyeh Ehsani Oskouie +2Language Model Generation Evaluation Multimodal Reasoning
UCSB · University of Washington · UCLA
Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?
May 12, 2026Yichen Feng, Yuetai Li, Chunjiang Liu +14Multimodal Large Language Models Image Quality Assessment
PortraitAes: Intent-Conditioned Structured Portrait Aesthetic Assessment
Oct 4, 2026Junzhou Xie, Haozhong Xiong, Xunyun Tian +8Image Quality Assessment
Southeast University · Qwen Business Unit of Alibaba

Jun 29, 2026J/K move · Enter open · S save

Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models

Sajjad Ghiasvand, Maryam Amirizaniani, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh+1

UCSB · University of Washington · UCLA

Language Model Generation Evaluation Multimodal Reasoning Multimodal Model Evaluation Generative AI Evaluation

Open paper AskPDF arXiv

Open-ended aesthetic critique is a challenge for multimodal large language models (MLLMs): it has no single correct answer, and most aesthetic evaluation measures models against numeric scores rather than the written critiques people actually give. We ask whether MLLM critiques are close to human ones, scoring eight open-weight MLLMs from

7

B to

397

B, plus GPT-5.5, against multiple ranked human critiques for each of

1{,}227

r/photocritique posts under eight prompt conditions. Reference-based similarity gives a misleading picture. In absolute terms the stricter lexical and learned metrics align only weakly with human critiques while a coarse embedding cosine reports broad topical overlap, yet requesting shorter critiques raises those scores and withholding the image barely changes them: the similarity reflects length, the post text, and a stable critiquing style more than image-specific observation. An LLM judge sharpens the question rather than settling it: in the primary condition all four judges prefer the frontier models' critiques to the human ones, but on the

7

8

B models they diverge wildly, from

9\%

81\%

preference on identical pairs. Asked instead how similar each pair is in substance, those judges and two human annotators agree, rating every model between

1.81

and

2.59

on a

1

5

scale, close to ``mostly different''. Behaviorally, the models diverge in ways the scores do not surface: they cover nearly every aesthetic aspect where humans are selective and repeat themselves across critiques of one photo, even when prompted to write at human length. We argue that reference-based similarity rewards a fluent, comprehensive critique style rather than the selectivity and specificity of human critique.

Abstract

Multimodal Model Evaluation

Explore similar work

Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models

Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?

PortraitAes: Intent-Conditioned Structured Portrait Aesthetic Assessment