cs.CVOct 7, 2026

Perceptually Aligned Evaluation of Style Transfer

Authors: Yang Deng, Eleftherios Ioannou, David Mould, Steve Maddock, Paul L. Rosin, Yu-Kun Lai

Organizations: Cardiff University, UK · The Cyprus Institute, Cyprus · Carleton University, Canada · University of Sheffield, UK

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Style transfer lacks a reliable evaluation standard: ground truth is inherently ill-defined, and existing automatic metrics often fail to reflect human preference. This paper introduces ASTRA (Assessment of Style TRansfer Algorithms), an approach for automatic evaluation of style transfer algorithms; it contains two components, ASTRA-Data and ASTRA-Score. ASTRA-Data consists of a benchmark image set of content and style references, a collection of style transfer results generated on the benchmark set, and user study data capturing human judgements through a two-stage pairwise comparison protocol. From these annotations, we derive ranking-based ground truth for content preservation, style fidelity, and overall preference. Based on ASTRA-Data, we construct ASTRA-Score, a learnt evaluator that predicts preference-aligned scores from content-style-stylization image triplets, enabling automatic and scalable evaluation of new models applied to the benchmark set. Experimental results demonstrate that ASTRA-Score achieves substantially higher correlation with human rankings compared to prior metrics. Overall, ASTRA establishes a robust mechanism for standardised evaluation of style transfer methods.

Figures & tables

Figure 1 : Examples of selected images produced by different algorithms.
Figure 2 : Examples of the two evaluation interfaces used in our user study. ( Top ) In Stage 1, participants compare two stylised results generated from the same content-style pair by different methods, together with the corresponding content and style references. ( Bottom ) In Stage 2, participants compare two stylised results that come from different content-style pairs.
MethodStage 1Stage 2Total
AAMS [ YRX∗19 ]288627125598
AdaAttN [ LLH∗21 ]298328545837
AdaIN [ HB17 ]302628455871
ArtFlow [ AHS∗21 ]296028555815
ChatGPT [ AAA∗23 ]283627715607
Gatys [ GEB16 ]305228595911
Table 1 : Number of appearances of each method in Stage 1 and Stage 2 pairwise comparisons.
Figure 3 : Top-5 (top row) and bottom-5 (bottom row) stylised results selected from the top-10 and bottom-10 samples ranked by their mean Rank Centrality (RC) scores for overall preference. The style references for each column, arranged from left to right and top to bottom, are: Kandinsky, Carr, Picasso, Ptolemaic Egyptian, Chinese landscape, Australian Aboriginal, Australian Aboriginal, Rembrandt, Hokusai, and Picasso.
Figure 4 : Scatter plots of (a) overall effectiveness vs. content preservation; (b) overall effectiveness vs. style fidelity.
Figure 5 : Trade-off between style similarity and content preservation measured from Stage 1 pairwise comparisons. The dashed curve indicates the Pareto frontier of the two objectives, which highlights the optimal trade-off boundary between the two criteria.
Figure 6 : Content–style preference trade-off under different style references.
Figure 7 : Human preference rankings derived from Rank Centrality. Each point shows the mean score and the error bar denotes bootstrap uncertainty. “Overall preference” refers to the subjects’ holistic assessment of the effectiveness of the style transfer.
Figure 8 : Overview of the proposed evaluation model. A frozen VGG11 extracts content features ( fcg,fc\mathbf{f}_{c}^{g},\mathbf{f}_{c} ) and style features ( fsg,fs\mathbf{f}_{s}^{g},\mathbf{f}_{s} ) from the generated image IgI_{g} , content reference IcI_{c} , and style reference IsI_{s} . Three independent heads are employed: the Content Head uses ϕ(fcg,fc)\phi(\mathbf{f}_{c}^{g},\mathbf{f}_{c}) , the Style Head uses ψ(fsg,fs)\psi(\mathbf{f}_{s}^{g},\mathbf{f}_{s}) , and the Overall Head combines both relation features to predict an overall quality score.
MetricSpearmanKendall
Content
CFSD [ CHH24 ]0.281 ±\pm 0.2120.196 ±\pm 0.151
NMI [ SHH98 ]0.415 ±\pm 0.1410.288 ±\pm 0.100
SIFID [ SDM19 ]0.429 ±\pm 0.1700.300 ±\pm 0.129
GMSD [ XZMB13 ]0.456 ±\pm 0.1760.323 ±\pm 0.133
FSIM [ ZZMZ11 ]0.515 ±\pm 0.1600.368 ±\pm 0.118
Table 2 : Correlation between automatic metrics and study-derived rankings across the evaluated methods. Baseline metrics are evaluated using leave-one-method-out (LOMO) evaluation and reported as mean ±\pm standard deviation.
MetricSpearmanKendall
Content
CFSD [ CHH24 ]0.440 ±\pm 0.0550.310 ±\pm 0.044
Qwen [ BBC∗23 ]0.455 ±\pm 0.1840.357 ±\pm 0.145
NMI [ SHH98 ]0.492 ±\pm 0.0430.345 ±\pm 0.033
GMSD [ XZMB13 ]0.498 ±\pm 0.1030.361 ±\pm 0.081
FSIM [ ZZMZ11 ]0.544 ±\pm 0.0860.395 ±\pm 0.074
Table 3 : Correlation between automatic metrics and study-derived rankings across the evaluated methods. Baseline metrics are evaluated using leave-one-content-out (LOCO) evaluation and reported as mean ±\pm standard deviation.
MetricSpearmanKendall
Content
CFSD [ CHH24 ]0.250 ±\pm 0.1310.178 ±\pm 0.091
GMSD [ XZMB13 ]0.391 ±\pm 0.1790.280 ±\pm 0.133
Qwen [ BBC∗23 ]0.417 ±\pm 0.1910.325 ±\pm 0.143
FSIM [ ZZMZ11 ]0.435 ±\pm 0.1680.312 ±\pm 0.125
SIFID [ SDM19 ]0.467 ±\pm 0.1520.334 ±\pm 0.114
Table 4 : Correlation between automatic metrics and study-derived rankings across the evaluated methods. Baseline metrics are evaluated using leave-one-style-out (LOSO) evaluation and reported as mean ±\pm standard deviation.
Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9 : Top-3 and bottom-3 stylised results for each method ranked by overall human preference scores. In each row, the left three images show the top-3 results and the right three show the bottom-3 results. Methods are listed in alphabetical order: AAMS, AdaAttN, AdaIN, ArtFlow, ChatGPT, NST-Ghiasi, Gatys, LLIP, OmniStyle, SANET, StyTR-2, and USO.
Figure 10 : Illustration of content consistency: the same scene rendered in different styles.
Figure 11 : Illustration of style consistency: different contents rendered in the same style.
Figure 12 : Example of style transfer: content from one image rendered in the style of another.
Figure 13 : Ranking stability under simulated within-group comparisons (Stage 1). The Spearman correlation between the recovered ranking and the assumed ground-truth ranking increases as the number of comparisons grows. The curve rapidly converges and exceeds ρ=0.99\rho=0.99 at approximately 1.2×1041.2\times 10^{4} comparisons, indicating that the estimated method ranking becomes highly stable beyond this point. Each point represents the mean correlation over multiple simulation runs.
Figure 14 : Examples of the two illustrative examples used in our user study.
Figure 15 : Global ranking recovery under simulated cross-group comparisons (Stage 2). Additional comparisons between items belonging to different content-style pairs gradually improve the accuracy of the recovered global ranking. The Spearman correlation steadily increases as more cross-group comparisons are introduced, demonstrating that sufficient Stage 2 comparisons enable reliable global ranking estimation.
StyleAAMSAdaAttNAdaINArtFlowChatGPTGatysLLIPOmniStyleSANETStyTR-2USONST-Ghiasi
Australian Aboriginal236288230276177238255228268213276241
Bierstadt229249280225248281201241276263248227
Carr225243266258254274223310230217237195
Hokusai215268321272221256232263209221214232
Gris246224240231322257271210190268204257
Picasso236251291225268251212213290253230208
Appendix
Table 5: Exposure counts between reference styles and style transfer methods in the Stage 1 user study. Each entry reports the number of pairwise comparisons in which a given method appears under the corresponding style reference.
ContentAAMSAdaAttNAdaINArtFlowChatGPTGatysLLIPOmniStyleSANETStyTR-2USONST-Ghiasi
angel516493544469500491480453512522425437
athletes509522479474432428515496486543548402
barn389520549498497494437473525479510489
berries507506459562462523443516458496431491
daisy506494476489468549496521469485489456
mac459448519468477567498474453509516460
Appendix
Table 6: Exposure counts between content images and style transfer methods in the Stage 1 user study. Each entry indicates the number of pairwise comparisons in which a given method appears for the corresponding content image.
StyleAAMSAdaAttNAdaINArtFlowChatGPTGatysLLIPOmniStyleSANETStyTR-2USONST-Ghiasi
Australian Aboriginal214254264246232237254228229202250269
Bierstadt249231245231245255241234251209265247
Carr241219229219223239223243235228225251
Hokusai238248233223210219239237236252217250
Gris215245240251243242234255259221239269
Picasso209237241228232237252223222248257241
Appendix
Table 7: Exposure counts between style references and methods in Stage 2 comparisons.
ContentAAMSAdaAttNAdaINArtFlowChatGPTGatysLLIPOmniStyleSANETStyTR-2USONST-Ghiasi
Angel442477490528468505473471454465480490
Athletes456493461441503502494504444452482520
Barn473508478500486470510454497452480466
Berries446437492470423445472474438460459519
Daisy441476503475419459466504491456515481
Mac446459436442475461470484493457471496
Appendix
Table 8: Exposure counts between content images and methods in Stage 2 comparisons.
Figure 16 : Content-style preference across different style references. Each subplot corresponds to a specific style, arranged from left to right and top to bottom as: Mondrian, Carr, Chinese Landscape, Kandinsky, Bierstadt, Rembrandt, Picasso, Australian Aboriginal, Hokusai, Ptolemaic Egyptian, Gris, and Veronese. Each point represents a stylisation method positioned by its content and style win rates. The red curve indicates the Pareto frontier of the two objectives, which highlights the optimal trade-off boundary between the two criteria. The relationship between content preservation and style similarity varies significantly across styles, highlighting strong style-dependent behaviour.
Figure 17 : Content-style preference across different content images. Each subplot corresponds to a specific content image, arranged from left to right and top to bottom as: Angel, Athletes, Barn, Berries, Daisy, and Mac. Each point represents a stylisation method positioned by its content and style win rates. The red curve indicates the Pareto frontier of the two objectives, which highlights the optimal trade-off boundary between the two criteria. Compared to the per-style analysis, the distributions remain largely consistent across different contents, indicating that the trade-off is relatively stable with respect to the input image.
 ContentStyleOverall
BackboneSpearmanKendallSpearmanKendallSpearmanKendall
ResNet180.658 ±\pm 0.2090.489 ±\pm 0.1640.674 ±\pm 0.0760.497 ±\pm 0.0620.580 ±\pm 0.1560.418 ±\pm 0.118
ResNet340.628 ±\pm 0.1980.464 ±\pm 0.1570.672 ±\pm 0.0770.492 ±\pm 0.0630.566 ±\pm 0.1790.410 ±\pm 0.135
ResNet500.656 ±\pm 0.1730.488 ±\pm 0.1350.707 ±\pm 0.0880.521 ±\pm 0.0740.608 ±\pm 0.1560.442 ±\pm 0.123
VGG190.671 ±\pm 0.1380.491 ±\pm 0.1150.659 ±\pm 0.0820.480 ±\pm 0.0670.622 ±\pm 0.1190.453 ±\pm 0.095
ViT-B/160.691 ±\pm 0.1520.513 ±\pm 0.1270.648 ±\pm 0.1360.475 ±\pm 0.1150.505 ±\pm 0.1640.360 ±\pm 0.128
Appendix
Table 9 : Backbone ablation for ASTRA-Score under leave-one-method-out (LOMO) evaluation. Results are reported as the mean ±\pm standard deviation of Spearman’s ρ\rho and Kendall’s τ\tau across held-out stylisation methods. The best result for each criterion and correlation metric is shown in bold.

Explore similar work

CardsList

  1. Towards In-Context Tone Style Transfer with A Large-Scale Triplet Dataset

    Apr 17, 2026Yuhai Deng, Huimin She, Wei Shen +4Photorealistic Style TransferStyle

    VCIP, School of Computer Science, Nankai University · OPPO AI Center, OPPO Inc. · Nankai International Advanced Research Institute (Shenzhen Futian)

  2. CLeaR: A Unified Framework for Resolving the Leakage-Degradation Dilemma in Style Transfer

    Sep 29, 2026Teng Zhou, Yunhao ChenPhotorealistic Style TransferStyle

    Zhejiang University · Fudan University

  3. CDST: Color Disentangled Style Transfer for Universal Style Reference Customization

    May 22, 2025Shiwen Zhang, Zhuowei Chen, Lang Chen +1Photorealistic Style TransferStyle

    ByteDance