We identify a previously unreported phenomenon in CLIP representations: human and AI-generated paintings spontaneously separate along the dominant principal directions of their joint embedding distribution, without any supervised objective designed to distinguish the two classes. Rather than exploiting this phenomenon for detection, our objective is to interpret it: we seek to identify the visual information underlying the separation and to trace it back from the embedding space to the image domain. We pursue this objective through a progressive investigation combining interpretable image representations with gradient-based inversion, used systematically as an experimental probe of the relationships identified in feature space. Robustness experiments and increasingly expressive statistical descriptors progressively rule out several intuitive explanations based on global image properties and simple local statistics, and point instead to distributed multiscale image structure. Multiscale scattering provides the most informative interpretable representation considered, but offers only a partial account of the phenomenon. Direct inversion provides a complementary and striking observation: substantial displacements along the dominant CLIP directions can be induced by image perturbations that remain nearly imperceptible to human observers, showing that the directions involved in the separation are highly sensitive to image variations with very low perceptual salience for humans. Taken together, these results reveal a significant difference between the visual evidence reflected in CLIP representations and that readily accessible to human perception, raising broader questions about the relationship between artificial and human vision and, ultimately, between artificial and human aesthetic judgment.
Figures & tables
Figure 1: Two-dimensional PCA projection of CLIP embeddings for AI-generated paintings (red) and human paintings (blue).
Figure 2: Two-dimensional PCA projection of CLIP embeddings for a random subset of AI-generated paintings (yellow) and human paintings (orange) borrowed from the AI-WikiArt dataset.
Figure 3: Conceptualization of the relative position of the manifolds of human and AI-generated paintings in CLIP’s embedding space.
Figure 4: In the left column the original image, in the middle column the modified version, and in the right column the difference between them.
Figure 5: Movements in the 2D latent space induced by gradient ascent through CLIP along the main PCA component. Opposite movements for reproductions of human paintings can be easily generated too.
image id
x init
y init
x final
y final
distance
max perturb.
354
-0.36
-0.20
-0.15
-0.18
0.21
0.049
336
0.03
-0.35
0.43
-0.08
0.47
0.044
758
-0.45
-0.10
-0.14
-0.13
0.30
0.046
729
-0.11
-0.41
0.28
-0.23
0.43
0.049
Table 1: Movements in the 2D PCA space induced by the gradient ascent along the main component. We provide initial and final coordinates, and euclidean distance. The perturbation is measured as max absolute difference on images, normalized in the [0,1] interval.
Figure 6: Projection of grayscale AI-Pastiche (dark gray) and NGA (light gray) images onto the PCA space computed from the original CLIP embeddings.
168×168
112×112
48×48
24×24
AI-Pastiche
0.93±0.02
0.86±0.03
0.72±0.06
0.63±0.07
NGA
0.93±0.02
0.87±0.03
0.67±0.06
0.56±0.08
Table 2: Cosine similarity between the original CLIP embeddings and the embeddings obtained after resolution degradation.
Figure 7: Projection of images degraded to 112×112 pixels onto the original PCA space. Although the embeddings are displaced, the separation between AI-generated and human paintings remains largely preserved.
Figure 8: Projection of images degraded to 48×48 pixels. The two distributions become more entangled, although a substantial fraction of AI-generated images remains separated from the human paintings.
Figure 9: Projection of images degraded to 24×24 pixels. At this resolution the two datasets become substantially entangled along the original PCA directions.
Crop size
Accuracy
Precision
Recall
F1
10%
0.516
0.516
0.515
0.515
20%
0.763
0.767
0.763
0.762
40%
0.943
0.946
0.945
0.945
Table 3: AI vs. human classification performance using CLIP embeddings computed from random image crops. Classification is performed directly in the embedding space, before PCA projection.
Figure 10: Projection of random image crops onto the PCA space defined by the original CLIP embeddings. Although crops introduce additional variability, the separation between AI-generated and human paintings remains largely preserved for sufficiently large crop sizes.
Figure 11: Distribution of AI-artworks for the different generative models of AI-Pastiche.
Figure 12: Distribution of AI-craftworks for generative models of the latest generation.
AI-Pastiche
min
max
mean
std
red
5.7±9.3
249.5±9.2
116.7±63.3
63.4±13.2
green
6.6±9.9
242.1±12.9
105.7±88.2
57.4±14.3
blue
4.0±8.1
229.5±21.7
116.7±63.4
54.4±14.3
NGA
min
max
mean
std
red
20.2±26.5
239.8±18.3
118.6±43.1
48.0±15.6
green
20.9±22.7
224.0±26.7
104.1±42.8
42.9±14.6
Table 4: Basic statistics about AI-Pastiche and NGA data.
AI-pastiche
NGA
entropy
7.43±0.52
7.05±0.63
Table 5: Entropy of the empirical distribution of pixel intensities for AI-pastiche and NGA.
Figure 13: Histograms of distribution of pixel intensities for AI-pastiche and NGA.
Figure 14: Empirical color palette distribution for the AI-Pastiche and NGA datasets.
red
orange
yellow
lime
green
cyan
blue
violet
magenta
pink
brown
gray
AI
.002
.016
.002
.001
.000
.001
.002
.000
.000
.126
.424
.423
NGA
.001
.012
.002
.000
.000
.000
.000
.000
.000
.113
.472
.400
Table 6: Relative frequency of dominant color categories in the two datasets.
Metric
AI-Pastiche
Human
AI-Pastiche
Human
AI-Pastiche
Human
336×336
168×168
84×84
Chroma
19.7±9.8
19.1±8.6
19.4±9.6
19.0±8.6
19.43±9.58
18.87±8.58
Saturation HSL
0.36±0.15
0.32±0.13
0.34±0.14
0.32±0.13
0.34±0.14
0.31±0.13
Luminance mean
44.7±14.7
44.2±16.8
44.7±14.7
44.2±16.8
44.7±14.7
44.2±16.8
Luminance std
22.6±4.3
17.2±5.6
22.0±4.3
16.8±5.6
21.3±4.2
16.3±5.6
local L contrast
6.8±2.2
4.6±1.8
9.8±2.7
6.9±2.4
9.84±2.7
6.8±2.4
Table 7: Basic luminance and color statistics at different scales.
Figure 22
feature
size
PC1 R2
PC2 R2
HOG field
23×23×2×2×9
0.30
0.32
field of HOG av.
23×23×2×2
0.24
0.25
field of HOG av.
23×23
0.10
0.11
HOG spatial av.
9
0.03
0.02
HOG spatial av.
2×2×9
0.06
0.05
Table 8: R2 similarity of different aggregations of 24×24 patch based hog features to PC1 and PC2.
Figure 16: Visualization of HOG statistics.
model
features
PC1 R2
PC2 R2
MLP
23×23×2×2×9
0.39
0.41
Table 9: R2 similarity over a field of HOG features processed by a shallow MLP.
block size
1
2×2
4×4
8×8
23
PC1
0.39
0.36
0.30
0.27
0.04
PC2
0.41
0.37
0.29
0.22
0.16
Table 10: HOG field shuffling, processed through MLP. The block size is the area being shuffled.
Covariance descriptor
Correlated representation
PC1 R2
PC2 R2
Coarse HOG covariance
block-level 36 -D HOG descriptors
0.07
0.04
Local HOG neighbor corr.
neighboring 36 -D HOG cells
0.18
0.33
Luminance mean neighbor corr.
neighboring scalar luminance cells
0.04
0.02
Luminance-std neighbor corr.
neighboring scalar contrast cells
0.02
0.05
Raw patch neighbor corr.
neighboring raw patch vectors
0.01
0.01
Table 11: Regression performance obtained using pairwise spatial descriptors computed from different local image representations. Neighboring regions were compared through correlation or covariance statistics, while the coarse HOG descriptor was obtained by partitioning the image into a 4×4 grid and computing covariance statistics between the corresponding block-level HOG descriptors.
J
Max scale 2J
L
Features
PC1 R2
PC2 R2
α
3
8
4
61
0.33
0.40
1
3
8
8
217
0.33
0.44
5
4
16
4
113
0.32
0.44
1
4
16
8
417
0.37
0.47
5
5
32
4
181
0.35
0.46
1
5
32
8
681
0.40
0.48
10
Table 12: Ridge regression from order-two scattering descriptors at different maximum scales and different orientations.
J
L
features
PC1 R2
PC2 R2
3
4
61
0.54
0.57
3
8
217
0.55
0.59
4
4
113
0.56
0.60
4
8
417
0.58
0.60
5
4
181
0.58
0.61
5
8
681
0.60
0.61
Table 13: Non linear (MLP) predictors from scattering descriptors. Results measured with Kfolds over 5 splits. Dropout 0.3, 200 epochs.
R2 PC1
R2 PC2
transformer
attention
transformer
attention
patch dim
J
features
CL
pos
CL
pos
CL
pos
CL
pos
28×28
3
61×12×12
0.50
0.56
0.43
0.50
0.50
0.54
0.47
0.49
56×56
4
113×6×6
0.51
0.57
0.47
0.52
0.53
0.55
0.52
0.53
84×84
4
113×4×4
0.51
0.57
0.50
0.52
0.51
0.54
0.48
0.52
112×112
5
181×3×3
0.52
0.58
0.52
0.58
0.51
0.53
0.52
0.52
Table 14: PC1 and PC2 regression values for a lightweight model acting residually over a linear baseline. We tested a single layer Transform and a pure multi-head attention layer, both with four head. The two models have been tested in two configurations: with a classification token, and with positional encoding with averaging.
PC1 displacement
features
model
mean
std
CLIP embeddinggs
CLIP
0.079
0.056
hog field
ridge
0.010
0.09
mlp
0.09
0.07
global scattering J3 L4
ridge
0.016
0.017
mlp
0.020
0.021
Table 15: Summary of movements in the 2D PCA space induced by inversion of different regression techniques over different extracted features. The mean is computed over all images in AI-Pastiche.
Figure 17: Movements in 2D space induced through scattering inversion. Movements are governed by the x-axis (PC1).
Figure 18: Distribution of scattering coefficients along aggregated groups at J=3 for works of AI-Pastiche (AI-generated) and NGA (human).
Figure 19: Distribution of scattering coefficients along aggregated groups at J=4 for works of AI-Pastiche (AI-generated) and NGA (human).
The classic duck-rabbit illusion reveals that when visual evidence is ambiguous, the human brain must decide what it sees. But where exactly do human observers draw the line between ''duck'' and ''rabbit'', and do machine classifiers draw it in the same place? We use semantically ambiguous images as interpretability probes to expose how vision models represent the boundaries between concepts. We present a psychophysically-informed framework that interpolates between concepts in the CLIP embedding space to generate continuous spectra of ambiguous images, allowing us to precisely measure where and how humans and machine classifiers place their semantic boundaries. Using this framework, we show that machine classifiers are more biased towards seeing ''rabbit'', whereas humans are more aligned with the CLIP embedding used for synthesis, and the guidance scale seems to affect human sensitivity more strongly than machine classifiers. Our framework demonstrates how controlled ambiguity can serve as a diagnostic tool to bridge the gap between human psychophysical analysis, image classification, and generative image models, offering insight into human-model alignment, robustness, model interpretability, and image synthesis methods.
Yuqi Hu, Vasha DuTell, Ahna R. Girshick +1
University of California, Berkeley · Massachusetts Institute of Technology
Verifying the authenticity of AI-generated images presents a growing challenge on social media platforms these days. While vision-language models (VLMs) like CLIP outdo in multimodal representation, their capacity for AI-generated image classification is underexplored due to the absence of such labels during the pre-training process. This work investigates whether CLIP embeddings inherently contain information indicative of AI generation. A proposed pipeline extracts visual embeddings using a frozen CLIP model, feeds its embeddings to lightweight networks, and fine-tunes only the final classifier. Experiments on the public CIFAKE benchmark show the performance reaches 95% accuracy without language reasoning. Few-shot adaptation to curated custom with 20% of the data results in performance to 85%. A closed-source baseline (Gemini-2.0) has the best zero-shot accuracy yet fails on specific styles. Notably, some specific image types, such as wide-angle photographs and oil paintings, pose significant challenges to classification. These results indicate previously unexplored difficulties in classifying certain types of AI-generated images, revealing new and more specific questions in this domain that are worth further investigation.
Ziyang Ou
Department of Electrical and Computer Engineering University of Rochester Rochester, NY 14627, USA
Iterative prompt refinement is central to reproducing target images with text to image generative models. Previous studies have incorporated image similarity metrics (ISMs) as additional feedback to human users. Existing ISMs such as LPIPS and CLIP provide objective measures of image likeness but often fail to align with human judgments, particularly in context specific or user driven tasks. In this paper, we introduce Customized Learned Perceptual Image Patch Similarity (CLPIPS), a customized extension of LPIPS that adapts a metric's notion of similarity directly to human judgments. We aim to explore whether lightweight, human augmented fine tuning can meaningfully improve perceptual alignment, positioning similarity metrics as adaptive components for human in the loop workflows with text to image tools. We evaluate CLPIPS on a human subject dataset in which participants iteratively regenerate target images and rank generated outputs by perceived similarity. Using margin ranking loss on human ranked image pairs, we fine tune only the LPIPS layer combination weights and assess alignment via Spearman rank correlation and Intraclass Correlation Coefficient. Our results show that CLPIPS achieves stronger correlation and agreement with human judgments than baseline LPIPS. Rather than optimizing absolute metric performance, our work emphasizes improving alignment consistency between metric predictions and human ranks, demonstrating that even limited human specific fine tuning can meaningfully enhance perceptual alignment in human in the loop text to image workflows.
Khoi Trinh, Jay Rothenberger, Scott Seidenberger +2