On Semiotic-Grounded Interpretive Evaluation of Generative Art

Ruixiang Jiang; Changwen Chen

cs.CVApr 9, 2026

On Semiotic-Grounded Interpretive Evaluation of Generative Art

Authors: Ruixiang Jiang, Changwen Chen

Organizations: The Hong Kong Polytechnic University, Hong Kong SAR, China

Abstract

Interpretation is essential to deciphering the language of art: audiences communicate with artists by recovering meaning from visual artifacts. However, current Generative Art (GenArt) evaluators remain fixated on surface-level image quality or literal prompt adherence, failing to assess the deeper symbolic or abstract meaning intended by the creator. We address this gap by formalizing a Peircean computational semiotic theory that models Human-GenArt Interaction (HGI) as cascaded semiosis. This framework reveals that artistic meaning is conveyed through three modes - iconic, symbolic, and indexical - yet existing evaluators operate heavily within the iconic mode, remaining structurally blind to the latter two. To overcome this structural blindness, we propose SemJudge. This evaluator explicitly assesses symbolic and indexical meaning in HGI via a Hierarchical Semiosis Graph (HSG) that reconstructs the meaning-making process from prompt to generated artifact. Extensive quantitative experiments show that SemJudge aligns more closely with human judgments than prior evaluators on an interpretation-intensive fine-art benchmark. User studies further demonstrate that SemJudge produces deeper, more insightful artistic interpretations, thereby paving the way for GenArt to move beyond the generation of "pretty" images toward a medium capable of expressing complex human experience. Project page: https://github.com/songrise/SemJudge.

Figures & tables

Figure 1. HGI as cascaded semiosis. We model HGI as a chain of meaning-making steps: a creator encodes an intention into a prompt, which the model interprets to generate an artifact. A viewer then interprets this artifact to reconstruct the meaning, which may differ from the original intention.

Figure 2. HSG of generated artifact. We show the image with bounding boxes (top-left), its global semiosis (top-right), and sub-semioses (bottom), constructed by an MLLM in zero-shot. The HSG provides a structured interpretation of the Annunciation motif in the abstract painting. Best viewed in color.

Figure 3. SemiosisArt Construction. Top: we construct a prompt from canonical motifs, and generate images from various models. Bottom: We use two task formats: 2AFC for relative judgment and VQA for fine-grained interpretation.

Conventional Scorers
Group	Method	Backbone	Correlation			Interpretation
Group	Method	Backbone	KRCC $\uparrow$	SRCC $\uparrow$	CCC $\uparrow$	Acc (%) $\uparrow$
–	Random Guess	–	-0.023	-0.077	0.007	25.2
Alignment Scoring	CLIPScore ( Radford et al., 2021 )	CLIP	0.041	0.059	0.106	–
Quality Scoring	CLIP-IQA ( Wang et al., 2023 )	CLIP	0.080	0.250	0.088	–
Quality Scoring	DeQA-Score ( You et al., 2025 )	Tuned MLLM	0.023	-0.112	-0.118	–

Table 1. Correlation analysis of all compared evaluation metrics. KRCC (Kendall’s

\tau

), SRCC (Spearman’s

\rho

), CCC (Lin’s

\rho_{c}

), and VQA accuracy measure alignment with human judgment on semiosis quality of HGI. Human (Non-expert) denotes crowdsourced majority-vote judgment. Gemini-Flash stands for Gemini-3.1-Flash-Lite. (

\dagger

): Re-implemented with the same Qwen-9B backbone as SemJudge for fairness.

Figure 4. Subjective Interpretation Quality Experiment on Four Dimensions. We show the user ( $m=70$ ) feedback distribution on a 5-point Likert rating, with the mean score for each bar. SemJudge (w/o HSG): SemJudge with only root artifact semiosis. Base MLLM: Prompt MLLM to generate art interpretation.

Evaluator	$\Delta$	95% CI	Cohen’s $d$	Sig.
ImageReward	0.086	$[\,0.039,\ \infty)$	0.306	**
PickScore	0.126	$[\,0.047,\ \infty)$	0.595	**
DSG	0.087	$[\,0.006,\ \infty)$	0.402
ArtCoT	0.182	$[\,0.090,\ \infty)$	0.848	**
SemJudge	-0.010	$[\,-0.157,\ \infty)$	-0.047

Table 2. Iconicity-bias hypothesis test across evaluators. We report

\Delta

, bootstrap 95% confidence intervals, and Cohen’s

d

. Sig. indicates one-sided permutation-test significance:

{}^{*}p{<}0.05

{}^{**}p{<}0.01

. Conventional evaluators are biased towards iconic signs, while SemJudge remains robust for symbolic / indexical signs.

HSG Setting	HSG Builder	Judge (2AFC)	KRCC $\uparrow$	VQA Acc $\uparrow$
(A) Same judge, vary HSG complexity
No HSG	–	Qwen-9B	0.48	82.0
Standard HSG	Qwen-9B	Qwen-9B	0.55	86.1
Complex HSG	Qwen-9B	Qwen-9B	0.51	84.3
(B) Strong HSGs can lift weak judges
No HSG	–	Qwen-2B	-0.04	24.1

Table 3. Controlled ablation of SemJudge. We isolate three questions: (A) whether introducing a standard or more complex HSG structure helps under a fixed judge, (B) whether a strong HSG can lift a weak judge, and (C) how much judge scaling still matters once a strong HSG is available.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5. Net Iconicity Distribution (Jittored and normalized, with outliers ignored in plotting) of SemiosisArt.

Symbol	Description
Peircean Semiotics (Section 3)
$\xi$	An atomic semiosis, defined as the tuple $(o,s,i)$ .
$s\in\mathcal{S}$	The Sign (or Representamen); the perceptible form (e.g., prompt, image).
$i\in\mathcal{I}$	The Interpretant ; the meaning constructed by an interpreter.
$o\in\mathcal{O}$	The Dynamic Object ; the underlying meaning or intent not directly observable.
$\hat{o}\in\mathcal{O}$	The Immediate Object ; the underlying meaning or intent as interpreted from sign.

Appendix

Table 4. Glossary of Notations

Figure 6. HSG Visualization for Artifact Sign - 1. Best viewed in color. The prompt associated with the image is : Create an artistic-conception illustration inspired by Jiang Jie’s “Yu Meiren · Listening to the Rain” in the style of freehand ink-wash painting, using traditional Chinese artistic techniques to highlight the contrasts expressed in the poem. Top: Output HSG from SemJudge: Bottom: art analysis from compared models.

Figure 7. HSG Visualization for Artifact Sign - 2. Best viewed in color. The prompt associated with the image is: Render the story of Matthew 2:11 in a millennial-era video game style, such as Half-Life 1. The clothing and environment still reflect the historical period. Top: Output HSG from SemJudge: Bottom: art analysis from compared models.

Figure 8. HSG Visualization for Artifact Sign - 3. Best viewed in color. The prompt associated with the image is: Modern vector art illustration style for Farid al-Din Attar’s Ilahinama (Book of God), respecting the classical symbolism. Top: Output HSG from SemJudge: Bottom: art analysis from compared models.

Figure 9. HSG Visualization for Artifact Sign - 4 Best viewed in color. The prompt associated with the image is: Jain manuscript painting style (in the tradition of Kalpasutra illustrations) depicting the philosophical concept of Anekāntavāda (the many-sidedness of truth) — the parable of the blind men and the elephant reimagined with Jain symbolic figures in distinct colored zones each perceiving a fragment of a radiant liberated Jiva — with characteristic red-orange ground and flat-profile faces, no text. Top: Output HSG from SemJudge: Bottom: art analysis from compared models.

Figure 10. HSG Visualization for User Sign. Best viewed in color.

Figure 11. 2AFC tasks (prompt, pair of images) with net iconicity annotation. The image with a red border means the winner in human annotation. Note that this is not equal to the iconicity of the image(s) themselves.

Figure 12. 2AFC User Annotation Interface. Users are forced to choose the best image in a pairwise comparison. The initial input prompts are provided. The image will be zoomed in when clicking on the option card for a better display.

Figure 13. User Interface for fine-grained interpretation quality annotation. User views the pairwise comparison, the model judges the winner, and the interpretation produced by the model. In this case, the model is SemJudge, and we render the HSGs on the web front end. The User can click a node to view the detailed semiosis (e.g., interpretant, object). The User may annotate the quality on the bottom left panel.

User Prompt (After translation, if necessary)	Iconicity / Symbolism / Indexicality
Portrait of Power from Chainsaw Man, wearing a casual oversized sweater, making a peace sign gesture, anime style.	6.4 / 3.4 / 1.8
HD graphite drawing of a robotic hand holding a reflective metal sphere in first person perspective. The ball reflects the android’s body, the surrounding warehouse environment.	4.0 / 3.0 / 5.8
The Garden of Earthly Delights by Hieronymus Bosch, but the three panels use different art styles. Left panel use the common style in Renaissance. The center panel use Fauvism by Henry Martisse, and right one use Picasso’s Cubism. The story should be still recognizable. Keep the structure of Triptych.	5.6 / 5.8 / 2.0
It turns out that long ago, when Nüwa refined stones to mend the sky, she forged at the Crag of Inaction on Mount Great Desolation a total of 36,501 stones—each twelve zhang high and twenty - four zhang wide. Of these, the Divine Empress Nüwa used 36,500 stones, leaving one single stone unused, which she discarded beneath Green Ridge Peak of this mountain. Who would have thought that after undergoing refinement, this stone had already gained spiritual awareness? Seeing that all the other stones were chosen to repair the heavens while it alone was deemed unfit and left behind, it fell into deep self - pity and lamentation, crying out in shame and sorrow day and night.’ Based on the above story, create a traditional Chinese lianhuanhua (illustrated serial comic). The narrative should be complete yet concise.	3.6 / 5.4 / 2.2
The Return of the Prodigal Son happened in Minecraft world. The story should be recognizable.	4.8 / 4.6 / 1.8
Create an illustrated step-by-step diagram showing how this sculpture is made in five stages, starting from a solid marble cube and ending with the finished sculpture displayed in a gallery. The diagram should be clear, simple, and easy for a 12-year-old to understand. [IMAGE]	5.8 / 3.0 / 5.2

Appendix

Table 5. Random sample of 15 prompts from the SemiosisArt.

Instruction Type	Instruction Summary
Question generation instructions
Global generation instruction	Generate exactly one image-grounded multiple-choice QA item for iconographic interpretation evaluation. The final question must be answerable from the visible image and bbox references alone, without access to the original prompt or hidden metadata. The stem and choices must avoid directly describing image contents; if bounding boxes are used, refer only to IDs such as bbox_1 . Questions should be difficult for viewers without relevant iconographic or art-historical knowledge. All distractors must be intra-tradition plausible. Return valid JSON with keys question , choices , answer , and rationale ; choices must be exactly A , B , C , and D .
Pair-comparison generation rule	For paired images, the stem must ask why the known winner is better than the other panel, rather than asking which image wins. The two panels should be referenced only as image a and image b, with only a minimal high-level topical hint. The correct answer should identify the strongest visible comparative reason, and distractors should be hard near-miss comparative alternatives in the same semantic family.
Negative-sample generation rule	For negative samples, the stem must paraphrase the intended user prompt or requested scene and ask what is wrong with the image relative to that request. The correct answer should diagnose the primary mismatch, inconsistency, or failure, while distractors should be plausible competing diagnoses rather than generic claims that the image is fine.
Follow-up question generation	A second question for the same image or pair must be meaningfully different from the first one. It should not reuse the same queried relation, answer target, or near-duplicate wording, and it must continue to follow bbox-only references and the non-literal wording constraints used in the initial authoring prompt.
JSON repair prompt	If the model response does not satisfy the required schema, the pipeline issues a repair prompt asking for valid JSON only. This step is a formatting-recovery instruction that preserves the original authoring task while enforcing the required output structure.

Appendix

Table 6. Instructions used for VQA question generation and quality control.

Component	Prompt Content
System instruction	You are an expert Computational Semiotician acting as an interpreter. Your task is to analyze a sign, namely the user input prompt (text or text+image), and infer the user’s intention as a structured Hierarchical Semiosis Graph (HSG) whose nodes represent triadic semiosis: sign, object, and interpretant.
Theoretical framework	The prompt follows Peircean triadic semiosis. For the root and each child node, identify: (i) the sign as the relevant textual or multimodal feature, (ii) the object as the target reality or conceptual subject, (iii) the interpretant as the target effect or mental conception, and (iv) the expected grounds connecting sign to object: iconic, indexical, or symbolic. During input analysis, expected grounds are inferred guidelines rather than hard constraints.
Task instructions	1. Analyze the prompt holistically to identify the global object, dominant interpretant, and expected grounds. 2. Decompose the prompt into 3–5 critical sub-signs using concise descriptions. 3. State how each sub-sign contributes to the root node, such as elaboration, contextualization, or stylization.
Output format	Return a valid JSON object only. The root node is hsg_root with node_id , a semiosis object containing sign_description , inferred_object , interpretant , and expected_grounds , plus a children list of sub-sign nodes and their relation_to_root .
Input placeholder	The sign is: [$PROMPT] .

Appendix

Table 7. System prompt for HSG construction from the user sign.

Component	Prompt Content
System instruction	You are an expert Computational Semiotician acting as a visual interpreter. Your task is to analyze a pair of generated images produced from the same prompt and decode their semiotic structure as two HSGs.
Theoretical framework	As in the input-sign analysis, follow Peircean triadic semiosis. For the root node and each visual child node, identify sign, object, interpretant, and grounds.
Task instructions	1. Analyze each generated image holistically to identify the overarching object, dominant interpretant, and primary grounds. 2. Decompose each image into 3–5 visual sub-signs and analyze their triadic semiosis. 3. When a sub-sign is localizable, provide up to three bounding boxes in image coordinates [x_min, y_min, x_max, y_max] relative to the full image. 4. State how each sub-sign contributes to the global meaning-making, such as contextualization, contrast, or thematic reinforcement.
Output format	Return two valid JSON objects only, one for image A and one for image B. Each should contain an hsg_root with root-level semiosis fields, child nodes, optional bounding_box entries for localizable elements, and relation_to_root .
Input placeholders	The sign is: A: [ $IMAGE_A] B: [$ IMAGE_B] .

Appendix

Table 8. System prompt for HSG construction from generated artifacts.

Component	Prompt Content
Judgment instruction	Given the raw input and the reconstructed HSGs for the user input and the two model outputs, decide which generated image better fulfills the user’s intended object in the input semiosis. The HSGs are used as structured evidence for comparison.
Input placeholders	[ $INPUT_HSG] [$ OUTPUT_HSG_A] [$OUTPUT_HSG_B] .
Output format	Return a valid JSON object only with fields discussion and winner . The discussion should give the verbatim decision process with reference to the input and output HSGs, and winner must be either ‘‘A’’ or ‘‘B’’ .

Appendix

Table 9. System prompt for 2AFC judgment from reconstructed HSGs.

Image Generation Evaluation

Image generation evaluation measures the quality, fidelity, diversity, and instruction-following ability of AI-generated images.

See all 113 papers

QQJ: Quantifying Qualitative Judgment for Scalable and Human-Aligned Evaluation of Generative AI
May 17, 2026Marjan Veysi, Pirooz Shamsinejadbabaki, Mohammad Zare +1LLM-as-a-Judge Human-in-the-Loop Evaluation
AI Lab, Arioobarzan Engineering Team, Shiraz, Iran · Department of Computer Engineering and Information Technology, Shiraz University of Technology, Shiraz, Iran · Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genoa, Genoa, Italy
Beyond Explicit Intent: Discovering and Shaping Artistic Intent through Human-AI Co-Creation
Oct 15, 2026Renye Yan, Jialun Zhong, Jikang Cheng +3Human-AI Co-Creation Human-in-the-Loop AI
Peking University, China
The Silent Brush: Evaluating Artistic Style Leakage in AI Art Generation
May 17, 2026Ninad Joshi, Ashutosh Ranjan, Vivek Srivastava +1T2I Generation T2I Generation Evaluation
TCS Research, India

May 17, 2026J/K move · Enter open · S save

QQJ: Quantifying Qualitative Judgment for Scalable and Human-Aligned Evaluation of Generative AI

Marjan Veysi, Pirooz Shamsinejadbabaki, Mohammad Zare, Mohammad Sabouri

AI Lab, Arioobarzan Engineering Team, Shiraz, Iran · Department of Computer Engineering and Information Technology, Shiraz University of Technology, Shiraz, Iran · Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genoa, Genoa, Italy

LLM-as-a-Judge Human-in-the-Loop Evaluation Rubric-Based Evaluation Image Generation Evaluation Generative AI Evaluation

Open paper AskPDF arXiv

The rapid progress of generative artificial intelligence has exposed fundamental limitations in existing evaluation methodologies, particularly for open-ended, creative, and human-facing tasks. Traditional automatic metrics rely on surface-level statistical similarity and often fail to reflect human perceptions of quality, while purely human evaluation, although reliable, is costly, subjective, and difficult to scale. Recent approaches using large language models as evaluators offer improved scalability but frequently lack explicit grounding in human-defined evaluation principles, leading to bias and inconsistency. In this paper, we introduce Quantifying Qualitative Judgment (QQJ), a scalable and human-centric evaluation framework that explicitly bridges the gap between human judgment and automated assessment. QQJ separates the definition of quality from its execution by anchoring evaluation in expert-designed, multi-dimensional rubrics and calibrating large language model evaluators to align with expert reasoning using a small, high-quality annotation set. This design enables consistent, interpretable, and scalable evaluation across diverse generative tasks and modalities. Extensive experiments on text and image generation demonstrate that QQJ achieves substantially stronger alignment with human judgment than traditional automatic metrics and unconstrained LLM-based evaluators. Moreover, QQJ exhibits improved stability across repeated evaluations and superior diagnostic capability in identifying critical failure modes such as hallucination and intent mismatch. These results indicate that structured qualitative judgment can be operationalized at scale without sacrificing interpretability or human alignment, positioning QQJ as a practical foundation for reliable evaluation of modern generative AI systems.

Abstract

Image Generation Evaluation

Explore similar work

QQJ: Quantifying Qualitative Judgment for Scalable and Human-Aligned Evaluation of Generative AI

Beyond Explicit Intent: Discovering and Shaping Artistic Intent through Human-AI Co-Creation

The Silent Brush: Evaluating Artistic Style Leakage in AI Art Generation