Perceptual misalignment of texture representations in convolutional neural networks

Ludovica de Paolis; Fabio Anselmi; Alessio Ansuini; Eugenio Piasini

cs.CVApr 1, 2026

Perceptual misalignment of texture representations in convolutional neural networks

Authors: Ludovica de Paolis, Fabio Anselmi, Alessio Ansuini, Eugenio Piasini

Organizations: Neuroscience area, International School for Advanced Studies (SISSA), Trieste, Italy · Department of Mathematics, Informatics and Geosciences, Università degli Studi di Trieste, Trieste, Italy · Department of Data Engineering, Area Science Park, Trieste, Italy

Abstract

Mathematical modeling of visual textures traces back to Julesz's intuition that texture perception in humans is based on local correlations between image features. An influential approach for texture analysis and generation generalizes this notion to linear correlations between the nonlinear features computed by convolutional neural networks (CNNs), compiled into Gram matrices. Given that CNNs are often used as models for the visual system, it is natural to ask whether such "texture representations" spontaneously align with the textures' perceptual content, and in particular whether those CNNs that are regarded as better models for the visual system also possess more human-like texture representations. Here we quantify the perceptual content captured by feature correlations computed for a diverse pool of CNNs, and we compare it to the models' perceptual alignment with the mammalian visual system as measured by Brain-Score. Surprisingly, we find that there is no connection between conventional measures of CNN quality as a model of the visual system and its alignment with human texture perception. We conclude that texture perception involves mechanisms that are distinct from those that are commonly modeled using approaches based on CNNs trained on object recognition, possibly depending on the integration of contextual information.

Figures & tables

Family	Model
	AlexNet ( Krizhevsky et al., 1994 )
DenseNet ( Huang et al., 2017 )	DenseNet-121
	DenseNet-169
	DenseNet-201
InceptionNet ( Howard et al., 2017 )	Inception-v3 ( Szegedy et al., 2016 )
ResNet ( He et al., 2016 )	ResNet18

Table 1: Convolutional Neural Network architectures grouped by family.

Panel of Figure 1 — Figure 1: Representational Dissimilarity Matrices obtained from the five layers analyzed for VGG-19. The size of each matrix is $5640\times 5640$ , and each entry corresponds to the Euclidean distance – relative to each matrix, as in the colorbars – between the Gram matrix representation of one pair of images in DTD. The entries are sorted such that images belonging to each of the 47 ground-truth categories are placed next to each other.

Figure 2: MI values across CNN layers identified by index (1-5). Each line corresponds to one of the 12 models as color-coded in the legend.

Brain-Score metric	Pearson’s r	p-value
average vision	-0.130	0.68
neural vision	-0.165	0.60
behavior vision	-0.029	0.92
V1	-0.229	0.47
V2	-0.262	0.41
V4	-0.063	0.84

Table 2: Correlation between texture representation quality, expressed as MI between the clusters in the Gram representation of textures and the ground-truth labels in DTD, and quality of the networks as models of the ventral visual stream, measured by Brain-Score. For more details on Brain-Score see Section 4.7 .

Figure 3: Plots of the correlations with Pearson’s p . Each subplot represents the correlation of Brain-Score benchmarks average vision, neural vision, behavior vision, V1, V2, V4, IT respectively vs. the layer with the highest MI for each of the 12 CNNs. Each color identifies a model as reported in the legend.

Panel of Figure 4 — Figure 4: Textures generated with the Gatys algorithm applied to a subsample of CNNs (rows). The 4 original images (top row) belong to 4 different classes in DTD (columns: blotchy; striped; matted; scaly ). The last column represents one of the images originally used by Gatys et al., which we synthesized as a reference. Textures generated by the whole pool of 12 models can be found in Appendix (Supplementary Figures 1 , 2 )

Model	Extracted layers	Gram dimensionality
AlexNet	layer idx=[0, 3, 6, 8, 10]	L0= $64(64+1)/2=2080$
		L3= $192(192+1)/2=18520$
		L6= $384(384+1)/2=73920$
		L8= $256(256+1)/2=32896$
		L10= $256(256+1)/2=32896$
DenseNet-121	bn_0=features.norm0	bn0= $64(64+1)/2=2080$

Table 3: Extracted layers from each CNN and relative Gram matrix dimensionality. The names refer to the terminology of the models’ versions in Torchvision. For models whose layers are not associated with a name, we report the absolute index.

Model	Extracted layers	Gram matrix dimensionality
Inception-v3	bn_0=Conv2d_1a_3x3.bn	bn0= $32(32+1)/2=561$
	bn_1=Conv2d_2a_3x3.bn	bn1= $32(32+1)/2=561$
	bn_2=Conv2d_2b_3x3.bn	bn2= $64(64+1)/2=2080$
	bn_3=Conv2d_3b_1x1.bn	bn3= $80(80+1)/2=3240$
	bn_4=Conv2d_4a_3x3.bn	bn4= $192(192+1)/2=185208$
ResNet18	bn_0=layer1.0.bn1	bn0= $64(64+1)/2=2080$

Table 4: (Continued from previous page)

Model	Model coordinates	average vision	neural vision	behavior vision	V1	V2	V4	IT
Benchmark coordinates		$D$ 2	$E$ 2	$DB$ 2	$F$ 2	$AM$ 2	$AP$ 2	$AV$ 2
AlexNet	$B$ 290	0.171	0.281	0.06	0.338	0.176	0.355	0.255
Densenet-121	$B$ 128	0.311	0.344	0.278	0.388	0.275	0.396	0.315
DenseNet-169	$B$ 158	0.276	0.305	0.248	0.346	0.178	0.384	0.312
DenseNet-201	$B$ 94	0.318	0.293	0.343	0.34	0.19	0.388	0.253
Inception-v3	$B$ 84	0.33	0.3	0.358	0.321	0.169	0.388	0.324

Table 5: Summary of the values from Brain-Score benchmarks that we selected for our study (columns), for the 12 CNNs from our pool (rows). In the second column and second row, for models and benchmarks respectively, we report the .csv coordinates of the leaderboard. Our Brain-Score values were exported from the leaderboard website on September 8th 2025.

Panel of Table 10 — Supplementary Figure 1: Textures generated with the Gatys algorithm with the 12 CNNs of choice (rows). The 4 original images (top row) belong to 4 different classes in DTD (columns: blotchy; striped; matted; scaly ). The last column represents one of the images originally used by Gatys et al., which we synthesized as a reference.

Panel of Table 11 — Supplementary Figure 2: (continued from previous page)

Explore similar work

Texture Representations in Deep Vision Models: Comparing CNNs, Vision Transformers, and Human Perception
Jul 9, 2026Ludovica de Paolis, Marco Baroni, Alessandro Laio +1Visual Representation Learning Vision Transformer
Department of Neuroscience, International School for Advanced Studies (SISSA), Trieste, Italy · Department of Language and Translation Sciences, Pompeu Fabra University, Barcelona, Spain · ICREA, Barcelona, Spain +1
Do Vision Model See Like the Brain? A Comparison Across EEG Encoding Model
Sep 22, 2026Shashank Baghel, Kshitij Dwivedi, Dinesh Singh +1Vision Transformer Visual Neuroscience
Centre for Human-Computer Interaction, Indian Institute of Technology, Mandi, Mandi, 175075, Himachal Pradesh, India · Freie Universität, Berlin, Germany · Goethe University, Frankfurt, Germany
Denoising Models Develop Human-Like Perceptual Illusion Representations Across Architectures
Jul 19, 2026Gautam Ranka, Paras ChopraVisual Representation Learning
Lossfunk

Jul 9, 2026J/K move · Enter open · S save

Texture Representations in Deep Vision Models: Comparing CNNs, Vision Transformers, and Human Perception

Ludovica de Paolis, Marco Baroni, Alessandro Laio, Eugenio Piasini

Department of Neuroscience, International School for Advanced Studies (SISSA), Trieste, Italy · Department of Language and Translation Sciences, Pompeu Fabra University, Barcelona, Spain · ICREA, Barcelona, Spain+1

Visual Representation Learning Vision Transformer Convolutional Neural Networks

Open paper AskPDF arXiv

In computational vision science, Convolutional Neural Networks (CNNs) have emerged as a popular model of biological vision because of the alignment they can exhibit with neural and behavioral data in humans and animals. However, it remains unclear to what extent this alignment persists for visual tasks that extend beyond the canonical object recognition paradigm based on well defined semantic content. In this study, we diverge from the common object-centric view by focusing on another aspect of vision: texture perception. We consider textures of different complexity generated with three different algorithms from the same source images. Using a rank-based statistic, we quantify the information encoded in the internal representations of a CNN and three Vision Transformers (ViTs), and we compare the similarity of these representations to those inferred from human psychophysics data. We find that the representation of textures is aligned in different ViTs, but not between the ViTs and the CNN; that ViTs form similar representations for textures of different complexity; that human performance in recognizing textures can be better predicted from ViTs representations rather than CNN representations. Taken together, these results suggest that ViTs may capture more faithfully than CNNs how texture patterns are visually processed by humans, and that the representations of texture stimuli in computational models may be driven by the network architecture.

Abstract

Explore similar work

Texture Representations in Deep Vision Models: Comparing CNNs, Vision Transformers, and Human Perception

Do Vision Model See Like the Brain? A Comparison Across EEG Encoding Model

Denoising Models Develop Human-Like Perceptual Illusion Representations Across Architectures