E-comIQ-ZH: A Human-Aligned Dataset and Benchmark for Fine-Grained Evaluation of E-commerce Posters with Chain-of-Thought

Meiqi Sun; Mingyu Li; Junxiong Zhu

cs.CVFeb 25, 2026

E-comIQ-ZH: A Human-Aligned Dataset and Benchmark for Fine-Grained Evaluation of E-commerce Posters with Chain-of-Thought

Authors: Meiqi Sun, Mingyu Li, Junxiong Zhu

Organizations: Taobao & Tmall Group, Alibaba Group

Abstract

Generative AI is widely used to create commercial posters. However, rapid advances in generation have outpaced automated quality assessment. Existing models emphasize generic esthetics or low level distortions and lack the functional criteria required for e-commerce design. It is especially challenging for Chinese content, where complex characters often produce subtle but critical textual artifacts that are overlooked by existing methods. To address this, we introduce E-comIQ-ZH, a framework for evaluating Chinese e-commerce posters. We build the first dataset E-comIQ-18k to feature multi dimensional scores and expert calibrated Chain of Thought (CoT) rationales. Using this dataset, we train E-comIQ-M, a specialized evaluation model that aligns with human expert judgment. Our framework enables E-comIQ-Bench, the first automated and scalable benchmark for the generation of Chinese e-commerce posters. Extensive experiments show our E-comIQ-M aligns more closely with expert standards and enables scalable automated assessment of e-commerce posters. All datasets, models, and evaluation tools will be released to support future research in this area.Code will be available at https://github.com/4mm7/E-comIQ-ZH.

Figures & tables

Figure 1: Qualitative comparison of E-comIQ-M with leading MLLMs on a challenging e-commerce image. While other powerful models like Gemini 2.5 Pro [ 10 ] and Q-Insight [ 24 ] overlook critical flaws, our E-comIQ-M accurately identifies the subtle stroke-level corruption . This leads to a more human-aligned low score for the text dimension (1.0), demonstrating its superior fine-grained diagnostic capabilities.

Figure 2: Overview of the E-comIQ-ZH framework. (a–c) E-comIQ-Dataset : multi-dimensional expert annotations with Chain-of-Thought rationales. (d–e) E-comIQ-M : two-stage training via Supervised Fine-Tuning (SFT) and Generative Reranking Policy Optimization (GRPO). (f) E-comIQ-Bench : evaluation of generative models on e-commerce image generation capabilities.

Figure 3: An illustration of our human-AI collaborative pipeline for generating diagnostic Chain-of-Thought (CoT) rationales.

Figure 4: Distribution of image sources.

	Overall	Object	Background	Text	Layout
$\alpha$	0.858	0.745	0.721	0.765	0.877
Acc. (%)	96.4	92.2	94.6	93.2	96.6

Table 1: Inter-annotator agreement for E-comIQ-18k, measured by Krippendorff’s Alpha (

\alpha

) and loose accuracy (Acc., within a 0.5 margin), confirming the substantial reliability of our annotations.

Dataset	Domain	Purpose	# Images	Ref.	Annotation Type	Score Range	CoT
AIGC Quality / Preference
ImageRewardDB [ 53 ]	AIGC	Preference	137k pairs	NR	Pairwise Pref.	N/A	✗
AGIQA-3K [ 25 ]	AIGC	General	3,000	NR	Multi-dim. Score	[1, 5]	✗
General “In-the-Wild”
KonIQ-10k [ 17 ]	General	General	10,073	NR	Single MOS	[1, 5]	✗
SPAQ [ 11 ]	General	Aesthetics	11,125	NR	Single MOS	[1, 100]	✗

Table 2: Comparison of E-comIQ-18k with representative image quality, preference, and e-commerce evaluation datasets. Most existing datasets target general aesthetics, distortion fidelity, or holistic AIGC preference, while AIGuard is the only e-commerce functional dataset but relies on binary labels without multidimensional scoring or CoT explanations. E-comIQ-18k uniquely provides e-commerce focused functional multidimensional scores together with expert verified CoT rationales.

Panel of Figure 5 — Figure 5: Statistical Profile of E-comIQ-18k. (a) The multi-modal distribution of overall scores highlights sample diversity. (b) The distribution of CoT rationale lengths . (c) The correlation matrix reveals a semi-orthogonal dimensional structure. (d) A ‘weakest link’ analysis pinpoints common diagnostic challenges.

Model	Overall		Background		Object		Text		Layout
	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC
Traditional NR-IQA Models
MUSIQ [ 21 ]	0.074	0.081	0.143	0.158	0.064	0.066	0.026	0.066	0.074	0.081
SPAQ [ 11 ]	-0.174	-0.172	-0.271	-0.279	-0.069	-0.119	-0.047	-0.046	-0.161	-0.167
General-Purpose MLLMs
GPT-4o [ 33 ]	0.242	0.219	0.368	0.413	0.105	0.122	0.126	0.148	0.297	0.282

Table 3: Correlation performance against state-of-the-art models on the E-comIQ-18k test set. Each cell reports PLCC / SRCC . The best result is in bold , and the second-best is underlined .

Model	Overall		Background		Object		Text		Layout
	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]
General-Purpose MLLMs
GPT-4o [ 33 ]	32.4	59.0	45.8	70.4	47.8	70.8	34.3	55.0	48.8	74.8
Gemini 2.5 Pro [ 10 ]	26.6	51.0	50.4	73.8	42.2	63.4	29.2	45.0	46.0	68.0
Claude-Sonnet-4.5 [ 1 ]	28.4	57.2	53.4	77.8	49.4	71.4	27.8	45.2	50.1	74.8
Grok-4 [ 51 ]	33.3	57.3	46.3	69.5	42.9	65.7	26.9	41.1	43.9	73.0

Table 4: Accuracy performance against state-of-the-art models on the E-comIQ-18k test set. Each cell reports [email protected] / [email protected] (in %) . The best result is in bold , and the second-best is underlined .

Config.	Overall		Background		Object		Text		Layout
	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC
GRPO only	0.158	0.154	0.174	0.218	0.013	0.022	0.059	0.077	0.228	0.192
SFT only	0.346	0.346	0.458	0.530	0.272	0.238	0.272	0.283	0.390	0.418
SFT+GRPO (Simple)	0.352	0.360	0.466	0.509	0.287	0.261	0.368	0.385	0.413	0.445
SFT+GRPO (Complex)	0.413	0.410	0.509	0.540	0.320	0.274	0.391	0.405	0.457	0.466

Table 5: Ablation studies on the E-comIQ-18k test set. We analyze the impact of different training stages and reward function designs. Each cell reports PLCC / SRCC .

Figure 6: Qualitative comparison of images generated by leading models on E-comIQ-Bench.

E-comIQ-M	Overall		Background		Object		Text		Layout
	Human	E-comIQ-M	Human	E-comIQ-M	Human	E-comIQ-M	Human	E-comIQ-M	Human	E-comIQ-M
SeeDream [ 38 ]	3.65	3.53	4.70	4.40	3.78	4.08	4.05	3.98	4.51	4.37
Qwen [ 47 ]	3.26	3.36	4.71	4.51	3.84	4.11	3.17	2.87	4.55	4.27
GPT-4o [ 5 ]	2.76	2.76	4.67	4.19	3.49	4.12	2.83	3.49	4.36	4.36
Gemini [ 10 ]	1.92	2.91	4.65	4.26	3.72	3.73	1.45	2.41	4.52	4.30
Flux [ 3 ]	1.89	2.72	4.60	4.19	3.66	3.76	1.48	1.98	4.56	4.40

Table 6: Benchmark results for leading generative E-comIQ-Ms on E-comIQ-Bench.

Model	Subject Fidelity			Text Content Accuracy
	DINO Sim $\uparrow$	LPIPS $\downarrow$	CLIP Score $\uparrow$	Phrase F1 $\uparrow$	Char Sim $\uparrow$
SeeDream [ 38 ]	0.74	0.62	0.81	0.88	0.92
Qwen [ 47 ]	0.81	0.57	0.84	0.49	0.54
GPT-4o [ 5 ]	0.73	0.67	0.81	0.86	0.91
Gemini [ 10 ]	0.67	0.69	0.78	0.56	0.62
Flux [ 3 ]	0.76	0.64	0.82	0.10	0.10

Table 7: Objective metrics for subject fidelity and text content accuracy on E-comIQ-Bench.

Panel of Figure 7 — Figure 7: (a) Multi-model performance comparison across five dimensions via radar chart. (b) Distribution of image categories.

Figure 8: AI-edited posters via Flux. Given a product cutout and its category (left), we retrieve a matching scene from a predefined background library (middle) and feed both into Flux to compose a subject–background image. The final poster (right) is obtained by adding Chinese marketing copy using predefined text templates.

Source	Train	Val	Test
Merchant HQ	4166	555	279
Merchant LQ	4166	555	279
Open-Source	2500	333	167
AI-generated	1666	222	112
AI-edited	1666	222	112
Prof. design	833	111	56

Table 8: Train/val/test splits of E-comIQ-18k.

Figure 9: Mean expert scores by source on E-comIQ-18k.

Dimension / Issue tags

Background

\square

Color clash with product or brand;

\square

weak scene or context;

\square

irrelevant scene;

\square

cluttered or noisy background;

\square

strong “AI-generated” artefacts;

Table 9: Annotation checklist and tag taxonomy.

Figure 11: Example of our CoT editing interface. Given a poster image (left), the annotator reviews the LLM-generated Chinese rationale (middle) and performs span-level edits to correct errors (shown in red/orange), producing an English translated version (right) that remains faithful to expert judgement.

Panel of Figure 12 — Figure 12: Word frequency analysis of model reasoning traces. We visualize the top frequent words appearing in the model’s chain-of-thought (CoT) across 18k e-commerce samples. The Chinese word cloud (left) is directly computed from the original CoT supervision, while the English version (right) is obtained by carefully mapping the top 200 Chinese terms to semantically aligned English phrases. Both visualizations reflect consistent semantic emphasis on background, object clarity, textual quality, composition, and visual communication.

Figure 13: Instruction and prompt template for E-comIQ-M. We use a fixed system prompt and a fixed user prompt that ask the model to first provide a <think> Chain-of-Thought and then output a JSON object with five scores in the <answer> block.

Metric	1k	2k	3k (default)	4k	5k
PLCC	0.412	0.421	0.425	0.426	0.423
SRCC	0.423	0.429	0.433	0.431	0.430

Table 10: Effect of hard-subset size on GRPO performance. Performance is reported as PLCC and SRCC on the validation split. The default configuration (3k) is highlighted.

\tau=0.5

\tau=0.4

\tau=0.3

\tau=0.2

\tau=0.1

\lambda_{\text{score}}=1.00

\lambda_{\text{score}}=0.85

\lambda_{\text{score}}=0.75

\lambda_{\text{score}}=0.65

\lambda_{\text{score}}=0.55

\lambda_{\text{score}}=0.45

\lambda_{\text{score}}=0.35

Setting		Overall		Background		Object		Text		Layout
lr	$\boldsymbol{\beta}$	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC
$1\times 10^{-6}$	0.10	0.425	0.433	0.496	0.520	0.391	0.361	0.364	0.392	0.483	0.506
$5\times 10^{-5}$	0.10	0.267	0.277	0.383	0.462	0.189	0.223	0.270	0.270	0.375	0.403
$1\times 10^{-6}$	0.05	0.329	0.348	0.441	0.508	0.263	0.239	0.342	0.346	0.431	0.450
$5\times 10^{-5}$	0.05	0.326	0.302	0.417	0.454	0.265	0.237	0.226	0.311	0.455	0.488
$1\times 10^{-6}$	0	0.346	0.346	0.458	0.530	0.272	0.238	0.272	0.283	0.390	0.418

Table 11: Effect of GRPO learning rate and KL coefficient on correlation performance. Each cell reports PLCC / SRCC on the E-comIQ-18k test set.

Panel of Figure 16 — Figure 16: Additional qualitative examples.

Figure 17: Prompt templates for generating Chinese e-commerce poster instructions. We provide the white-background cutout, the original merchant poster, and structured attributes (category, short title, and selling points) to Qwen2.5-VL-72B. The model rewrites the information into a professional, stylistic generation prompt that follows detailed typography, layout, and text-quality constraints. Five templates are used to encourage stylistic diversity.

Figure 18: Prompt–generation showcase across product categories (1/2). For each cutout–prompt pair, several commercial systems and research models are queried to generate one poster per model. The merchant poster is shown as the human-designed reference. Categories on this page include Daily Tools, Home Living , and 3C Accessories .

Figure 19: Prompt–generation showcase across product categories (2/2). The same protocol applies to the remaining categories: Fresh Food, Daily Tools (additional cases), Footwear & Apparel , and Pet & Toys .

	Overall	Object	Background	Text	Layout
$\alpha$	0.741	0.780	0.699	0.818	0.930

Table 12: Inter-annotator agreement for E-comIQ-Bench

Text Rendering in T2I Generation

See all 18 papers

PosterIQ: A Design Perspective Benchmark for Poster Understanding and Generation
Mar 25, 2026Yuheng Feng, Wen Zhang, Haodong Duan +1Image Generation T2I Generation Evaluation
The Hong Kong Polytechnic University · Snapchat Inc. · ByteDance Seed
InnoAds-Composer: Efficient Condition Composition for E-Commerce Poster Generation
Mar 6, 2026Yuxin Qin, Ke Cao, Haowei Liu +13Image Generation Compositional Image Generation
JD.com, Inc., Beijing, China · Chongqing University of Posts and Telecommunications, Chongqing, China · The Hong Kong University of Science and Technology, Hong Kong, China +1
BizGenEval: A Systematic Benchmark for Commercial Visual Content Generation
Mar 26, 2026Yan Li, Zezi Zeng, Ziwei Zhou +13Image Generation Evaluation Image Generation
Shanghai Jiao Tong University · Xi’an Jiaotong University · Fudan University +1

Mar 25, 2026J/K move · Enter open · S save

PosterIQ: A Design Perspective Benchmark for Poster Understanding and Generation

Yuheng Feng, Wen Zhang, Haodong Duan, Xingxing Zou

The Hong Kong Polytechnic University · Snapchat Inc. · ByteDance Seed

Image Generation T2I Generation Evaluation Document Understanding

Open paper AskPDF arXiv

We present PosterIQ, a design-driven benchmark for poster understanding and generation, annotated across composition structure, typographic hierarchy, and semantic intent. It includes 7,765 image-annotation instances and 822 generation prompts spanning real, professional, and synthetic cases. To bridge visual design cognition and generative modeling, we define tasks for layout parsing, text-image correspondence, typography/readability and font perception, design quality assessment, and controllable, composition-aware generation with metaphor. We evaluate state-of-the-art MLLMs and diffusion-based generators, finding persistent gaps in visual hierarchy, typographic semantics, saliency control, and intention communication; commercial models lead on high-level reasoning but act as insensitive automatic raters, while generators render text well yet struggle with composition-aware synthesis. Extensive analyses show PosterIQ is both a quantitative benchmark and a diagnostic tool for design reasoning, offering reproducible, task-specific metrics. We aim to catalyze models' creativity and integrate human-centred design principles into generative vision-language systems.

Abstract

Text Rendering in T2I Generation

Explore similar work

PosterIQ: A Design Perspective Benchmark for Poster Understanding and Generation

InnoAds-Composer: Efficient Condition Composition for E-Commerce Poster Generation

BizGenEval: A Systematic Benchmark for Commercial Visual Content Generation