Automatic evaluation of artistic style transfer remains an open problem: existing metrics score an image pair in isolation, without seeing an example of the intended style transformation, and correlate poorly with human judgment. We introduce StyleJudge, a compact vision-language evaluator conditioned on a small set of reference source-to-stylized pairs before scoring a query pair, mirroring how a human rater calibrates to a style before judging a candidate. On a human-annotated benchmark of 878 rated pairs across 12 artistic styles (728 from 10 training/validation styles, 150 from 2 unseen styles), our efficient 0.8B-parameter StyleJudge reaches a Pearson correlation of 0.291 (Spearman 0.325) with human ratings on seen styles and 0.381 on unseen ones. On seen styles, it achieves the highest positive correlation among tested metrics: Gram-matrix style loss and CLIP similarity correlate weakly (r=0.041, r=0.168), while LPIPS (scored higher when closer to the source) correlates negatively (r=−0.404). We achieve this by distilling evaluation knowledge from a 72B-parameter vision-language teacher used only during training, which reaches r=0.327, confirming that most of its human alignment survives compact distillation. For training, we build a dataset of source-to-stylized pairs across the 12 styles via LoRA-adapted diffusion, densified with graded intermediate examples from a high-resolution image-morphing pipeline so the evaluator recognizes partial and complete stylization. On two held-out validation styles, StyleJudge reaches 28.2% exact-match agreement with its teacher's structured scores versus 1.1% for the untuned base model, with 67.8% shorter reasoning traces, and scores a pair in 39 s (median) using 2.5 GB of GPU memory on one NVIDIA T4. We release the human-rating benchmark, annotation guideline, SDXL-adapted FreeMorph code, and model weights.
Figures & tables
Figure 1. Overview of StyleJudge . (a) Training: stylized targets from FLUX.1 Kontext and a style LoRA are expanded into graded ten-frame strips (plus degraded variants), annotated by a 72B teacher and distilled into a 0.8B student. (b) Inference: two exemplar pairs, the text style markers and the query pair yield SF, CP and RQ, combined into Sfinal . Two-row diagram. Top row, training: a source portrait is stylized into watercolor by FLUX.1 Kontext with a style LoRA; SDXL-FreeMorph expands it into a graded strip of frames; the Qwen2.5-VL 72B teacher annotates each item with XML reasoning and SF, CP, RQ scores; supervised fine-tuning with a score-weighted loss yields the StyleJudge student. Bottom row, inference: a prompt containing two mosaic exemplar pairs, the mosaic style markers and a query pair is scored by StyleJudge, which outputs SF, CP, RQ and a rationale combined into the final score.
Figure 2. Morph strips from the original SD 2.1 FreeMorph (top) and our SDXL adaptation (bottom), frames 1 to 10 (different input pairs). SD 2.1 changes objects and clothing alongside the style; SDXL changes the rendering while keeping identity and layout stable. More strips are in the project repository ( Lokesh et al., 2026 ) . Two rows of ten frames each. The top row, from SD 2.1 FreeMorph, turns a black-and-white photo of two women with a bicycle into a pop-art image, with objects and clothing changing between frames. The bottom row, from the SDXL adaptation, gradually turns a group photo into a painterly rendering while the people and layout stay the same.
Style
Style markers
Style
Style markers
Abstract
non-representational geometric forms, bold flat primary color blocks, and elimination of figurative elements
Impressionist
short broken dappled brushstrokes, chromatic color juxtaposition without blending, and soft atmospheric hazy edges
discrete tile fragments with visible grout lines, angular hard-edge color segmentation, and tesserae patterns
Cubist
simultaneous multiple-viewpoint planes, fractured interlocking geometric facets, and a muted palette
Pencil
parallel graphite hatching lines, linear tonal gradation from light to dark, and visible paper grain texture
Expressive
raw emotionally distorted brushwork, unblended aggressive color strokes, and angular forms
Pop Art †
flat bold primary poster colors, Ben-Day halftone dot patterns, and thick black outline contour lines
Gouache
opaque chalky flat color areas, matte non-reflective finish, and clean sharp color edges
Watercolor
transparent layered wash glazes, soft wet-edge bloom blending, and paper granulation in wash areas
Table 1. The twelve styles and the text style markers included in every prompt (written with web search and an LLM; also given to the human raters as style definitions). Acrylic and Pop Art ( † ) are held out from training and used only as unseen test styles (§ 4.2 ).
Metric
Result
Mean score (VLM / human)
5.86±1.66 / 6.62±1.95
Pearson correlation ( r )
0.19 ( p=0.18 )
Spearman correlation ( ρ )
0.11 ( p=0.46 )
Human inter-rater ( r )
0.25 (95% CI [−0.03,0.49] , p=0.08 )
VLM systematic bias
Under-estimated in 62% of cases (31/50)
Table 2. Quantitative evaluation metrics: Zero-Shot VLM vs. Human annotators.
Source
Contribution
HQ-50K ( Yang et al., 2023 )
Indoor, outdoor, artificial-domain coverage
LAION ( Schuhmann et al., 2022 ; OpenDiffusionAI, 2024 )
Table 3. Source collections used to build the curated 10K-image pool.
Figure 3. Source images (top) and FLUX.1-dev Kontext stylized outputs (bottom) for each of the twelve style LoRAs. † Unseen test styles. A row of twelve image pairs, one per style, each showing a source photograph above its stylized version.
Split
Samples
Styles
Note
Train
5,614
8
Teacher rationales
Val
1,326
2
Expressive, mosaic
Test
800
2
Unseen : Acrylic, Pop Art
Table 4. Splits of the teacher-annotated data used to train the K=2 student.
Mean score
Pearson r
Style
n
Human
Student
Student
Teacher
Abstract
70
5.2
4.3
0.34
0.40
Charcoal
74
4.8
6.7
0.30
0.23
Cubist
74
5.3
5.1
0.57
0.45
Expressive
75
5.6
6.3
0.51
0.42
Gouache
70
6.1
6.7
0.45
0.38
Table 5. Per-style Pearson r with human ratings (all 878 pairs) and mean scores; higher r per row in bold; per-style 95% intervals are about ±0.2 . † Unseen styles.
Figure 4. Score distributions on the 878 benchmark pairs. Human ratings spread across the scale; the student’s scores pile up at 9 (39%) and at 1–3 (43%), with few in the middle; the teacher’s are flat by construction, because items were sampled uniformly over its score. Grouped bar chart of how often each score from 1 to 10 occurs for humans, the student and the teacher. Human ratings are spread out with a peak at 8; the student's scores are concentrated at 9 and at 1 to 3; the teacher's are about 100 per score level from 1 to 9.
Metric
Pearson r
Spearman ρ
Gram loss (negated) ( Gatys et al., 2016 )
−0.041
−0.049
LPIPS (negated) ( Zhang et al., 2018 )
−0.404
−0.394
CLIP similarity ( Radford et al., 2021 )
−0.168
−0.178
Untuned base (0.8B, zero-shot) ‡
−0.019
−0.017
StyleJudge (ours, SFT, 0.8B)
−0.291
−0.325
StyleJudge (ours, teacher, 72B)
−0.327
−0.337
Table 6. Correlation with human ratings on the 728 seen-style pairs of § 4.2 . Distances (Gram, LPIPS) are negated, so higher is better in every row.
Style
Base
SFT
Teacher
Match?
Acrylic (m8)
8/9/8 →8
7/8/8 →7
8
×
Pop Art (m4)
8/8/8 →8
5/7/6 →5
6
×
Acrylic (m2)
9/9/8 →9
3/8/7 →3
3
✓
Expressive (m1)
10/10/10 →10
2/9/6 →2
2
✓
Table 7. Base vs. SFT predictions (SF/CP/RQ → composite by Eq. 1 ); m k is morph frame k . Teacher is the 72B score, not a human rating.
Figure 5. Attention rollout for a mosaic query given two exemplar pairs (Ex A, Ex B). Six-panel figure showing two in-context exemplar pairs and a query source/output pair for a mosaic style transfer, overlaid with attention rollout heatmaps.
K
MAE ↓
Exact (%)
±1 (%)
Avg. Length
0
5.179
1.1
12.4
8,088 chars
1
2.842
22.8
43.1
2,689 chars
2
2.109
28.2
54.4
2,606 chars
3
2.600
19.6
44.0
3,527 chars
Table 8. Ablation on the number of exemplar pairs K (the same 800 validation-style items for every K , all scored against the teacher’s labels in the K=3 annotation set). K=0 is the untuned base model, K=2 the StyleJudge student; each K≥1 has its own adapter.
Vision-language model-as-a-judge uses multimodal models to assess images, video, and visual-agent outputs against instructions, rubrics, or human preferences.