E-comIQ-ZH: A Human-Aligned Dataset and Benchmark for Fine-Grained Evaluation of E-commerce Posters with Chain-of-Thought

Meiqi Sun; Mingyu Li; Junxiong Zhu

cs.CVFeb 25, 2026

E-comIQ-ZH: A Human-Aligned Dataset and Benchmark for Fine-Grained Evaluation of E-commerce Posters with Chain-of-Thought

Authors: Meiqi Sun, Mingyu Li, Junxiong Zhu

Organizations: Taobao & Tmall Group, Alibaba Group

Abstract

Generative AI is widely used to create commercial posters. However, rapid advances in generation have outpaced automated quality assessment. Existing models emphasize generic esthetics or low level distortions and lack the functional criteria required for e-commerce design. It is especially challenging for Chinese content, where complex characters often produce subtle but critical textual artifacts that are overlooked by existing methods. To address this, we introduce E-comIQ-ZH, a framework for evaluating Chinese e-commerce posters. We build the first dataset E-comIQ-18k to feature multi dimensional scores and expert calibrated Chain of Thought (CoT) rationales. Using this dataset, we train E-comIQ-M, a specialized evaluation model that aligns with human expert judgment. Our framework enables E-comIQ-Bench, the first automated and scalable benchmark for the generation of Chinese e-commerce posters. Extensive experiments show our E-comIQ-M aligns more closely with expert standards and enables scalable automated assessment of e-commerce posters. All datasets, models, and evaluation tools will be released to support future research in this area.Code will be available at https://github.com/4mm7/E-comIQ-ZH.

Figures & tables

Figure 1: Qualitative comparison of E-comIQ-M with leading MLLMs on a challenging e-commerce image. While other powerful models like Gemini 2.5 Pro [ 10 ] and Q-Insight [ 24 ] overlook critical flaws, our E-comIQ-M accurately identifies the subtle stroke-level corruption . This leads to a more human-aligned low score for the text dimension (1.0), demonstrating its superior fine-grained diagnostic capabilities.

Figure 2: Overview of the E-comIQ-ZH framework. (a–c) E-comIQ-Dataset : multi-dimensional expert annotations with Chain-of-Thought rationales. (d–e) E-comIQ-M : two-stage training via Supervised Fine-Tuning (SFT) and Generative Reranking Policy Optimization (GRPO). (f) E-comIQ-Bench : evaluation of generative models on e-commerce image generation capabilities.

Figure 3: An illustration of our human-AI collaborative pipeline for generating diagnostic Chain-of-Thought (CoT) rationales.

Figure 4: Distribution of image sources.

	Overall	Object	Background	Text	Layout
$\alpha$	0.858	0.745	0.721	0.765	0.877
Acc. (%)	96.4	92.2	94.6	93.2	96.6

Table 1: Inter-annotator agreement for E-comIQ-18k, measured by Krippendorff’s Alpha (

\alpha

) and loose accuracy (Acc., within a 0.5 margin), confirming the substantial reliability of our annotations.

Dataset	Domain	Purpose	# Images	Ref.	Annotation Type	Score Range	CoT
AIGC Quality / Preference
ImageRewardDB [ 53 ]	AIGC	Preference	137k pairs	NR	Pairwise Pref.	N/A	✗
AGIQA-3K [ 25 ]	AIGC	General	3,000	NR	Multi-dim. Score	[1, 5]	✗
General “In-the-Wild”
KonIQ-10k [ 17 ]	General	General	10,073	NR	Single MOS	[1, 5]	✗
SPAQ [ 11 ]	General	Aesthetics	11,125	NR	Single MOS	[1, 100]	✗

Table 2: Comparison of E-comIQ-18k with representative image quality, preference, and e-commerce evaluation datasets. Most existing datasets target general aesthetics, distortion fidelity, or holistic AIGC preference, while AIGuard is the only e-commerce functional dataset but relies on binary labels without multidimensional scoring or CoT explanations. E-comIQ-18k uniquely provides e-commerce focused functional multidimensional scores together with expert verified CoT rationales.

Panel of Figure 5 — Figure 5: Statistical Profile of E-comIQ-18k. (a) The multi-modal distribution of overall scores highlights sample diversity. (b) The distribution of CoT rationale lengths . (c) The correlation matrix reveals a semi-orthogonal dimensional structure. (d) A ‘weakest link’ analysis pinpoints common diagnostic challenges.

Model	Overall		Background		Object		Text		Layout
	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC
Traditional NR-IQA Models
MUSIQ [ 21 ]	0.074	0.081	0.143	0.158	0.064	0.066	0.026	0.066	0.074	0.081
SPAQ [ 11 ]	-0.174	-0.172	-0.271	-0.279	-0.069	-0.119	-0.047	-0.046	-0.161	-0.167
General-Purpose MLLMs
GPT-4o [ 33 ]	0.242	0.219	0.368	0.413	0.105	0.122	0.126	0.148	0.297	0.282

Table 3: Correlation performance against state-of-the-art models on the E-comIQ-18k test set. Each cell reports PLCC / SRCC . The best result is in bold , and the second-best is underlined .

Model	Overall		Background		Object		Text		Layout
	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]
General-Purpose MLLMs
GPT-4o [ 33 ]	32.4	59.0	45.8	70.4	47.8	70.8	34.3	55.0	48.8	74.8
Gemini 2.5 Pro [ 10 ]	26.6	51.0	50.4	73.8	42.2	63.4	29.2	45.0	46.0	68.0
Claude-Sonnet-4.5 [ 1 ]	28.4	57.2	53.4	77.8	49.4	71.4	27.8	45.2	50.1	74.8
Grok-4 [ 51 ]	33.3	57.3	46.3	69.5	42.9	65.7	26.9	41.1	43.9	73.0

Table 4: Accuracy performance against state-of-the-art models on the E-comIQ-18k test set. Each cell reports [email protected] / [email protected] (in %) . The best result is in bold , and the second-best is underlined .

Config.	Overall		Background		Object		Text		Layout
	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC
GRPO only	0.158	0.154	0.174	0.218	0.013	0.022	0.059	0.077	0.228	0.192
SFT only	0.346	0.346	0.458	0.530	0.272	0.238	0.272	0.283	0.390	0.418
SFT+GRPO (Simple)	0.352	0.360	0.466	0.509	0.287	0.261	0.368	0.385	0.413	0.445
SFT+GRPO (Complex)	0.413	0.410	0.509	0.540	0.320	0.274	0.391	0.405	0.457	0.466

Table 5: Ablation studies on the E-comIQ-18k test set. We analyze the impact of different training stages and reward function designs. Each cell reports PLCC / SRCC .

Figure 6: Qualitative comparison of images generated by leading models on E-comIQ-Bench.

E-comIQ-M	Overall		Background		Object		Text		Layout
	Human	E-comIQ-M	Human	E-comIQ-M	Human	E-comIQ-M	Human	E-comIQ-M	Human	E-comIQ-M
SeeDream [ 38 ]	3.65	3.53	4.70	4.40	3.78	4.08	4.05	3.98	4.51	4.37
Qwen [ 47 ]	3.26	3.36	4.71	4.51	3.84	4.11	3.17	2.87	4.55	4.27
GPT-4o [ 5 ]	2.76	2.76	4.67	4.19	3.49	4.12	2.83	3.49	4.36	4.36
Gemini [ 10 ]	1.92	2.91	4.65	4.26	3.72	3.73	1.45	2.41	4.52	4.30
Flux [ 3 ]	1.89	2.72	4.60	4.19	3.66	3.76	1.48	1.98	4.56	4.40

Table 6: Benchmark results for leading generative E-comIQ-Ms on E-comIQ-Bench.

Model	Subject Fidelity			Text Content Accuracy
	DINO Sim $\uparrow$	LPIPS $\downarrow$	CLIP Score $\uparrow$	Phrase F1 $\uparrow$	Char Sim $\uparrow$
SeeDream [ 38 ]	0.74	0.62	0.81	0.88	0.92
Qwen [ 47 ]	0.81	0.57	0.84	0.49	0.54
GPT-4o [ 5 ]	0.73	0.67	0.81	0.86	0.91
Gemini [ 10 ]	0.67	0.69	0.78	0.56	0.62
Flux [ 3 ]	0.76	0.64	0.82	0.10	0.10

Table 7: Objective metrics for subject fidelity and text content accuracy on E-comIQ-Bench.

Panel of Figure 7 — Figure 7: (a) Multi-model performance comparison across five dimensions via radar chart. (b) Distribution of image categories.

Figure 8: AI-edited posters via Flux. Given a product cutout and its category (left), we retrieve a matching scene from a predefined background library (middle) and feed both into Flux to compose a subject–background image. The final poster (right) is obtained by adding Chinese marketing copy using predefined text templates.

Source	Train	Val	Test
Merchant HQ	4166	555	279
Merchant LQ	4166	555	279
Open-Source	2500	333	167
AI-generated	1666	222	112
AI-edited	1666	222	112
Prof. design	833	111	56

Table 8: Train/val/test splits of E-comIQ-18k.

Figure 9: Mean expert scores by source on E-comIQ-18k.

Dimension / Issue tags

Background

\square

Color clash with product or brand;

\square

weak scene or context;

\square

irrelevant scene;

\square

cluttered or noisy background;

\square

strong “AI-generated” artefacts;

Table 9: Annotation checklist and tag taxonomy.

Figure 11: Example of our CoT editing interface. Given a poster image (left), the annotator reviews the LLM-generated Chinese rationale (middle) and performs span-level edits to correct errors (shown in red/orange), producing an English translated version (right) that remains faithful to expert judgement.

Panel of Figure 12 — Figure 12: Word frequency analysis of model reasoning traces. We visualize the top frequent words appearing in the model’s chain-of-thought (CoT) across 18k e-commerce samples. The Chinese word cloud (left) is directly computed from the original CoT supervision, while the English version (right) is obtained by carefully mapping the top 200 Chinese terms to semantically aligned English phrases. Both visualizations reflect consistent semantic emphasis on background, object clarity, textual quality, composition, and visual communication.

Figure 13: Instruction and prompt template for E-comIQ-M. We use a fixed system prompt and a fixed user prompt that ask the model to first provide a <think> Chain-of-Thought and then output a JSON object with five scores in the <answer> block.

Metric	1k	2k	3k (default)	4k	5k
PLCC	0.412	0.421	0.425	0.426	0.423
SRCC	0.423	0.429	0.433	0.431	0.430

Table 10: Effect of hard-subset size on GRPO performance. Performance is reported as PLCC and SRCC on the validation split. The default configuration (3k) is highlighted.

\tau=0.5

\tau=0.4

\tau=0.3

\tau=0.2

\tau=0.1

\lambda_{\text{score}}=1.00

\lambda_{\text{score}}=0.85

\lambda_{\text{score}}=0.75

\lambda_{\text{score}}=0.65

\lambda_{\text{score}}=0.55

\lambda_{\text{score}}=0.45

\lambda_{\text{score}}=0.35

Setting		Overall		Background		Object		Text		Layout
lr	$\boldsymbol{\beta}$	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC
$1\times 10^{-6}$	0.10	0.425	0.433	0.496	0.520	0.391	0.361	0.364	0.392	0.483	0.506
$5\times 10^{-5}$	0.10	0.267	0.277	0.383	0.462	0.189	0.223	0.270	0.270	0.375	0.403
$1\times 10^{-6}$	0.05	0.329	0.348	0.441	0.508	0.263	0.239	0.342	0.346	0.431	0.450
$5\times 10^{-5}$	0.05	0.326	0.302	0.417	0.454	0.265	0.237	0.226	0.311	0.455	0.488
$1\times 10^{-6}$	0	0.346	0.346	0.458	0.530	0.272	0.238	0.272	0.283	0.390	0.418

Table 11: Effect of GRPO learning rate and KL coefficient on correlation performance. Each cell reports PLCC / SRCC on the E-comIQ-18k test set.

Panel of Figure 16 — Figure 16: Additional qualitative examples.

Figure 17: Prompt templates for generating Chinese e-commerce poster instructions. We provide the white-background cutout, the original merchant poster, and structured attributes (category, short title, and selling points) to Qwen2.5-VL-72B. The model rewrites the information into a professional, stylistic generation prompt that follows detailed typography, layout, and text-quality constraints. Five templates are used to encourage stylistic diversity.

Figure 18: Prompt–generation showcase across product categories (1/2). For each cutout–prompt pair, several commercial systems and research models are queried to generate one poster per model. The merchant poster is shown as the human-designed reference. Categories on this page include Daily Tools, Home Living , and 3C Accessories .

Figure 19: Prompt–generation showcase across product categories (2/2). The same protocol applies to the remaining categories: Fresh Food, Daily Tools (additional cases), Footwear & Apparel , and Pet & Toys .

	Overall	Object	Background	Text	Layout
$\alpha$	0.741	0.780	0.699	0.818	0.930

Table 12: Inter-annotator agreement for E-comIQ-Bench

Text Rendering in T2I Generation

See all 18 papers

Mar 25, 2026

PosterIQ: A Design Perspective Benchmark for Poster Understanding and Generation

We present PosterIQ, a design-driven benchmark for poster understanding and generation, annotated across composition structure, typographic hierarchy, and semantic intent. It includes 7,765 image-annotation instances and 822 generation prompts spanning real, professional, and synthetic cases. To bridge visual design cognition and generative modeling, we define tasks for layout parsing, text-image correspondence, typography/readability and font perception, design quality assessment, and controllable, composition-aware generation with metaphor. We evaluate state-of-the-art MLLMs and diffusion-based generators, finding persistent gaps in visual hierarchy, typographic semantics, saliency control, and intention communication; commercial models lead on high-level reasoning but act as insensitive automatic raters, while generators render text well yet struggle with composition-aware synthesis. Extensive analyses show PosterIQ is both a quantitative benchmark and a diagnostic tool for design reasoning, offering reproducible, task-specific metrics. We aim to catalyze models' creativity and integrate human-centred design principles into generative vision-language systems.

Yuheng Feng, Wen Zhang, Haodong Duan +1

The Hong Kong Polytechnic University · Snapchat Inc. · ByteDance Seed

Image Generation T2I Generation Evaluation Document Understanding

Ask PDF

Mar 6, 2026

InnoAds-Composer: Efficient Condition Composition for E-Commerce Poster Generation

E-commerce product poster generation aims to automatically synthesize a single image that effectively conveys product information by presenting a subject, text, and a designed style. Recent diffusion models with fine-grained and efficient controllability have advanced product poster synthesis, yet they typically rely on multi-stage pipelines, and simultaneous control over subject, text, and style remains underexplored. Such naive multi-stage pipelines also show three issues: poor subject fidelity, inaccurate text, and inconsistent style. To address these issues, we propose InnoAds-Composer, a single-stage framework that enables efficient tri-conditional control tokens over subject, glyph, and style. To alleviate the quadratic overhead introduced by naive tri-conditional token concatenation, we perform importance analysis over layers and timesteps and route each condition only to the most responsive positions, thereby shortening the active token sequence. Besides, to improve the accuracy of Chinese text rendering, we design a Text Feature Enhancement Module (TFEM) that integrates features from both glyph images and glyph crops. To support training and evaluation, we also construct a high-quality e-commerce product poster dataset and benchmark, which is the first dataset that jointly contains subject, text, and style conditions. Extensive experiments demonstrate that InnoAds-Composer significantly outperforms existing product poster methods without obviously increasing inference latency.

Yuxin Qin, Ke Cao, Haowei Liu +13

JD.com, Inc., Beijing, China · Chongqing University of Posts and Telecommunications, Chongqing, China · The Hong Kong University of Science and Technology, Hong Kong, China +1

Image Generation Compositional Image Generation Conditional Diffusion Models

Ask PDF

Mar 26, 2026

BizGenEval: A Systematic Benchmark for Commercial Visual Content Generation

Recent advances in image generation models have expanded their applications beyond aesthetic imagery toward practical visual content creation. However, existing benchmarks mainly focus on natural image synthesis and fail to systematically evaluate models under the structured and multi-constraint requirements of real-world commercial design tasks. In this work, we introduce BizGenEval, a systematic benchmark for commercial visual content generation. The benchmark spans five representative document types: slides, charts, webpages, posters, and scientific figures, and evaluates four key capability dimensions: text rendering, layout control, attribute binding, and knowledge-based reasoning, forming 20 diverse evaluation tasks. BizGenEval contains 400 carefully curated prompts and 8000 human-verified checklist questions to rigorously assess whether generated images satisfy complex visual and semantic constraints. We conduct large-scale benchmarking on 26 popular image generation systems, including state-of-the-art commercial APIs and leading open-source models. The results reveal substantial capability gaps between current generative models and the requirements of professional visual content creation. We hope BizGenEval serves as a standardized benchmark for real-world commercial visual content generation.

Yan Li, Zezi Zeng, Ziwei Zhou +13

Shanghai Jiao Tong University · Xi’an Jiaotong University · Fudan University +1

Image Generation Evaluation Image Generation T2I Generation Evaluation Controllable Image Generation

Ask PDF

Abstract

Text Rendering in T2I Generation

Explore similar work

PosterIQ: A Design Perspective Benchmark for Poster Understanding and Generation

InnoAds-Composer: Efficient Condition Composition for E-Commerce Poster Generation

BizGenEval: A Systematic Benchmark for Commercial Visual Content Generation