Automatic evaluation of artistic style transfer remains an open problem: existing metrics score an image pair in isolation, without seeing an example of the intended style transformation, and correlate poorly with human judgment. We introduce StyleJudge, a compact vision-language evaluator conditioned on a small set of reference source-to-stylized pairs before scoring a query pair, mirroring how a human rater calibrates to a style before judging a candidate. On a human-annotated benchmark of 878 rated pairs across 12 artistic styles (728 from 10 training/validation styles, 150 from 2 unseen styles), our efficient 0.8B-parameter StyleJudge reaches a Pearson correlation of 0.291 (Spearman 0.325) with human ratings on seen styles and 0.381 on unseen ones. On seen styles, it achieves the highest positive correlation among tested metrics: Gram-matrix style loss and CLIP similarity correlate weakly (r=0.041, r=0.168), while LPIPS (scored higher when closer to the source) correlates negatively (r=−0.404). We achieve this by distilling evaluation knowledge from a 72B-parameter vision-language teacher used only during training, which reaches r=0.327, confirming that most of its human alignment survives compact distillation. For training, we build a dataset of source-to-stylized pairs across the 12 styles via LoRA-adapted diffusion, densified with graded intermediate examples from a high-resolution image-morphing pipeline so the evaluator recognizes partial and complete stylization. On two held-out validation styles, StyleJudge reaches 28.2% exact-match agreement with its teacher's structured scores versus 1.1% for the untuned base model, with 67.8% shorter reasoning traces, and scores a pair in 39 s (median) using 2.5 GB of GPU memory on one NVIDIA T4. We release the human-rating benchmark, annotation guideline, SDXL-adapted FreeMorph code, and model weights.
Figures & tables
Figure 1. Overview of StyleJudge . (a) Training: stylized targets from FLUX.1 Kontext and a style LoRA are expanded into graded ten-frame strips (plus degraded variants), annotated by a 72B teacher and distilled into a 0.8B student. (b) Inference: two exemplar pairs, the text style markers and the query pair yield SF, CP and RQ, combined into Sfinal . Two-row diagram. Top row, training: a source portrait is stylized into watercolor by FLUX.1 Kontext with a style LoRA; SDXL-FreeMorph expands it into a graded strip of frames; the Qwen2.5-VL 72B teacher annotates each item with XML reasoning and SF, CP, RQ scores; supervised fine-tuning with a score-weighted loss yields the StyleJudge student. Bottom row, inference: a prompt containing two mosaic exemplar pairs, the mosaic style markers and a query pair is scored by StyleJudge, which outputs SF, CP, RQ and a rationale combined into the final score.
Figure 2. Morph strips from the original SD 2.1 FreeMorph (top) and our SDXL adaptation (bottom), frames 1 to 10 (different input pairs). SD 2.1 changes objects and clothing alongside the style; SDXL changes the rendering while keeping identity and layout stable. More strips are in the project repository ( Lokesh et al., 2026 ) . Two rows of ten frames each. The top row, from SD 2.1 FreeMorph, turns a black-and-white photo of two women with a bicycle into a pop-art image, with objects and clothing changing between frames. The bottom row, from the SDXL adaptation, gradually turns a group photo into a painterly rendering while the people and layout stay the same.
Style
Style markers
Style
Style markers
Abstract
non-representational geometric forms, bold flat primary color blocks, and elimination of figurative elements
Impressionist
short broken dappled brushstrokes, chromatic color juxtaposition without blending, and soft atmospheric hazy edges
discrete tile fragments with visible grout lines, angular hard-edge color segmentation, and tesserae patterns
Cubist
simultaneous multiple-viewpoint planes, fractured interlocking geometric facets, and a muted palette
Pencil
parallel graphite hatching lines, linear tonal gradation from light to dark, and visible paper grain texture
Expressive
raw emotionally distorted brushwork, unblended aggressive color strokes, and angular forms
Pop Art †
flat bold primary poster colors, Ben-Day halftone dot patterns, and thick black outline contour lines
Gouache
opaque chalky flat color areas, matte non-reflective finish, and clean sharp color edges
Watercolor
transparent layered wash glazes, soft wet-edge bloom blending, and paper granulation in wash areas
Table 1. The twelve styles and the text style markers included in every prompt (written with web search and an LLM; also given to the human raters as style definitions). Acrylic and Pop Art ( † ) are held out from training and used only as unseen test styles (§ 4.2 ).
Metric
Result
Mean score (VLM / human)
5.86±1.66 / 6.62±1.95
Pearson correlation ( r )
0.19 ( p=0.18 )
Spearman correlation ( ρ )
0.11 ( p=0.46 )
Human inter-rater ( r )
0.25 (95% CI [−0.03,0.49] , p=0.08 )
VLM systematic bias
Under-estimated in 62% of cases (31/50)
Table 2. Quantitative evaluation metrics: Zero-Shot VLM vs. Human annotators.
Source
Contribution
HQ-50K ( Yang et al., 2023 )
Indoor, outdoor, artificial-domain coverage
LAION ( Schuhmann et al., 2022 ; OpenDiffusionAI, 2024 )
Table 3. Source collections used to build the curated 10K-image pool.
Figure 3. Source images (top) and FLUX.1-dev Kontext stylized outputs (bottom) for each of the twelve style LoRAs. † Unseen test styles. A row of twelve image pairs, one per style, each showing a source photograph above its stylized version.
Split
Samples
Styles
Note
Train
5,614
8
Teacher rationales
Val
1,326
2
Expressive, mosaic
Test
800
2
Unseen : Acrylic, Pop Art
Table 4. Splits of the teacher-annotated data used to train the K=2 student.
Mean score
Pearson r
Style
n
Human
Student
Student
Teacher
Abstract
70
5.2
4.3
0.34
0.40
Charcoal
74
4.8
6.7
0.30
0.23
Cubist
74
5.3
5.1
0.57
0.45
Expressive
75
5.6
6.3
0.51
0.42
Gouache
70
6.1
6.7
0.45
0.38
Table 5. Per-style Pearson r with human ratings (all 878 pairs) and mean scores; higher r per row in bold; per-style 95% intervals are about ±0.2 . † Unseen styles.
Figure 4. Score distributions on the 878 benchmark pairs. Human ratings spread across the scale; the student’s scores pile up at 9 (39%) and at 1–3 (43%), with few in the middle; the teacher’s are flat by construction, because items were sampled uniformly over its score. Grouped bar chart of how often each score from 1 to 10 occurs for humans, the student and the teacher. Human ratings are spread out with a peak at 8; the student's scores are concentrated at 9 and at 1 to 3; the teacher's are about 100 per score level from 1 to 9.
Metric
Pearson r
Spearman ρ
Gram loss (negated) ( Gatys et al., 2016 )
−0.041
−0.049
LPIPS (negated) ( Zhang et al., 2018 )
−0.404
−0.394
CLIP similarity ( Radford et al., 2021 )
−0.168
−0.178
Untuned base (0.8B, zero-shot) ‡
−0.019
−0.017
StyleJudge (ours, SFT, 0.8B)
−0.291
−0.325
StyleJudge (ours, teacher, 72B)
−0.327
−0.337
Table 6. Correlation with human ratings on the 728 seen-style pairs of § 4.2 . Distances (Gram, LPIPS) are negated, so higher is better in every row.
Style
Base
SFT
Teacher
Match?
Acrylic (m8)
8/9/8 →8
7/8/8 →7
8
×
Pop Art (m4)
8/8/8 →8
5/7/6 →5
6
×
Acrylic (m2)
9/9/8 →9
3/8/7 →3
3
✓
Expressive (m1)
10/10/10 →10
2/9/6 →2
2
✓
Table 7. Base vs. SFT predictions (SF/CP/RQ → composite by Eq. 1 ); m k is morph frame k . Teacher is the 72B score, not a human rating.
Figure 5. Attention rollout for a mosaic query given two exemplar pairs (Ex A, Ex B). Six-panel figure showing two in-context exemplar pairs and a query source/output pair for a mosaic style transfer, overlaid with attention rollout heatmaps.
K
MAE ↓
Exact (%)
±1 (%)
Avg. Length
0
5.179
1.1
12.4
8,088 chars
1
2.842
22.8
43.1
2,689 chars
2
2.109
28.2
54.4
2,606 chars
3
2.600
19.6
44.0
3,527 chars
Table 8. Ablation on the number of exemplar pairs K (the same 800 validation-style items for every K , all scored against the teacher’s labels in the K=3 annotation set). K=0 is the untuned base model, K=2 the StyleJudge student; each K≥1 has its own adapter.
Vision-language model-as-a-judge uses multimodal models to assess images, video, and visual-agent outputs against instructions, rubrics, or human preferences.
Style transfer lacks a reliable evaluation standard: ground truth is inherently ill-defined, and existing automatic metrics often fail to reflect human preference. This paper introduces ASTRA (Assessment of Style TRansfer Algorithms), an approach for automatic evaluation of style transfer algorithms; it contains two components, ASTRA-Data and ASTRA-Score. ASTRA-Data consists of a benchmark image set of content and style references, a collection of style transfer results generated on the benchmark set, and user study data capturing human judgements through a two-stage pairwise comparison protocol. From these annotations, we derive ranking-based ground truth for content preservation, style fidelity, and overall preference. Based on ASTRA-Data, we construct ASTRA-Score, a learnt evaluator that predicts preference-aligned scores from content-style-stylization image triplets, enabling automatic and scalable evaluation of new models applied to the benchmark set. Experimental results demonstrate that ASTRA-Score achieves substantially higher correlation with human rankings compared to prior metrics. Overall, ASTRA establishes a robust mechanism for standardised evaluation of style transfer methods.
Yang Deng, Eleftherios Ioannou, David Mould +3
Cardiff University, UK · The Cyprus Institute, Cyprus · Carleton University, Canada +1
Pairwise preferences and pointwise ratings are the two dominant annotation protocols in image aesthetic assessment (IAA), yet existing benchmarks adopt only one, leaving their complementarity unmeasured under controlled conditions. We introduce PPaint, a matched dual-protocol benchmark in which 15 domain experts, 5 per category, annotate 150 Chinese paintings under both protocols across five aesthetic dimensions, collecting 45,900 pairwise expert judgments through a locally dense preference design alongside the matched ratings. The matched design reveals complementary strengths: preferences yield more consistent ordinal rankings, while ratings anchor the absolute score scale. Fusing both signals via two independent preference-to-score methods yields a fused expert ground truth on which the two constructions converge to nearly identical scores. The same preference-to-score principle extends to label-free VLM training. PSDistill converts VLM pairwise judgments into calibrated pseudo-scores via an Elo reference pool, and trains the same VLM with confidence-weighted ranking optimization to produce a single-pass aesthetic scorer. Trained on a single painting category, the distilled Qwen3-VL-8B improves mean SRCC from 0.504 to 0.709 across all three categories, outperforming all open-source baselines including the dedicated aesthetic model ArtiMuse and matching closed-source Gemini-3.1-Pro within 0.04 SRCC at single-pass inference cost, with cross-domain transfer further validated on APDDv2. We will release the full PPaint dataset and training code.
Yuanpei Zhao, Jie Lin, Chao Zhang +5
1Sichuan University · 2NetEase Fuxi AI Lab · †Work done during an internship at NetEase Fuxi AI Lab. +1
The color, texture, and structure of an image can come from different references, but reference-guided generators transfer each reference as a single style signal. We find that in a diffusion transformer, every pathway that brings in a reference transfers multiple attributes, yet each attribute has one representation in which it is most directly controllable. Guided by this finding, we design StyleComposer to route color through the chromatic coordinates of the latent, texture through the attention keys and values, and structure through the queries, and to limit what else each route admits from its reference. Without training or inversion, StyleComposer follows the references and the prompt together more closely than stylization methods and multi-image generators, and it sets the strength of each attribute separately.