Evaluating Graphical Perception Capabilities of Vision Transformers

Poonam Poonam; Pere-Pau Vázquez; Timo Ropinski

cs.CVFeb 20, 2026

Evaluating Graphical Perception Capabilities of Vision Transformers

Authors: Poonam Poonam, Pere-Pau Vázquez, Timo Ropinski

Organizations: Ulm University, Ulm, 89081, Germany · Viscom Group, Institute of Media Informatics, Ulm University · ViRVIG Group, Universitat Politècnica de Catalunya - BarcelonaTech

Abstract

Vision Transformers, ViTs, have emerged as a powerful alternative to convolutional neural networks, CNNs, in a variety of image-based tasks. While CNNs have previously been evaluated for their ability to perform graphical perception tasks, which are essential for interpreting visualizations, the perceptual capabilities of ViTs remain largely unexplored. In this work, we investigate the performance of ViTs in elementary visual judgment tasks inspired by the foundational studies of Cleveland and McGill, which quantified the accuracy of human perception across different visual encodings. Inspired by their study, we benchmark ViTs against CNNs and human participants in a series of controlled graphical perception tasks. Our results reveal that, although ViTs demonstrate strong performance in general vision tasks, their alignment with human-like graphical perception in the visualization domain is limited. This study highlights key perceptual gaps and points to important considerations for the application of ViTs in visualization systems and graphical perceptual modeling.

Figures & tables

Panel of Figure 1 — Fig. 1: Illustration of the tested ViT architectures: vViT (a), CvT (b), and Swin (c). Input image patches ( green ) can be projected linearly (vViT) or through convolution (CvT). Transformer blocks with self-attention ( blue ) process the projected patches, before feed forward networks and MLPs ( orange ) generate the output. CvT additionally uses convolutional embedding ( pink ). Swin transformer blocks ( gray ) employ shifted windowing. All models conclude with class label outputs (labels are parameters used during the generation of the image.)

Model	Parameters	Patch Style	Attention Mechanism
vViT	$\sim$ 5.7M	Fixed	Global
CvT	$\sim$ 19.6M	Convolutional	Global
Swin	$\sim$ 27.5M	Hierarchical	Windowed

Table 1: Comparison of the three tested ViTs architectures: vViT, CvT, and Swin. The table highlights key architectural distinctions, including the approximate number of trainable parameters, patch embedding, and the type of attention mechanism employed. While vViT adopts a simple fixed patch embedding with global attention, CvT introduces convolutional embeddings to better preserve spatial locality. Swin further departs from the global attention paradigm by employing a hierarchical structure with windowed attention.

Fig. 2: Comparison of elementary perceptual task rankings across humans, CNNs, and ViTs. Rankings are derived based MLAE, with lower positions indicating better perceptual accuracy. The shading intensity of each box reflects rank: darker tones indicate higher accuracy (that is, rank 1), while lighter tones correspond to lower ranks. Human rankings are based on Cleveland and McGill [ 12 ] , while CNNs randking from Haehn et al. [ 26 ] , and ViT rankings are based on our retrained models. Lines connect identical perceptual tasks across the three groups. Notably, ViTs exhibit most deviation—particularly in their treatment of position and length—highlighting greater ambiguity and weaker perceptual alignment with human judgments.

Task Category	Humans	VGG19 [ 26 ]	Swin
Position-Angle
Bar		2.18	4.22
Pie without outline		3.30	4.21
Pie		3.30	4.21
Average Error	2.05	2.93	4.21
Position-Length

Table 2: Average error (MLAE) for humans, and the best-performing CNN (VGG19) from Haehn et al. [ 26 ] , and our best ViT model (Swin) different perceptual experiments. All models were trained from scratch under a unified setup for fair comparison. Lower values indicate better perceptual accuracy.

Fig. 3: Cross-generalization performance of the Swin Transformer across different parameterizations of elementary perceptual tasks. Each matrix corresponds to a specific task (for example, Position, Length, Area, Shading). Rows represent the parameterization used for training, while columns represent the parameterization used for testing. The top row in each matrix shows within-task (in-parameter) performance, with diagonal entries indicating baseline accuracy when training and testing on the same parameter configuration. Off-diagonal entries indicate the model’s ability to generalize to new parameter settings (that is, changes in object position, size, or shape). Higher values off-diagonal reflect poor generalization and sensitivity to unseen visual variations.

Fig. 4: Comparison of Regression Errors for Elementary Perceptual Tasks: elementary perceptual tasks experiment. Comparison of visual stimuli (left) and MLAE scores with 95 $\%$ confidence intervals (lower is better) for various ViTs and humans (right). Swin generally aligns more closely with human performance for simpler encodings, while all ViTs diverge on complex features like curvature and shading. Human scores derived from Cleveland and McGill, and Hahen et al.’s studies [ 12 , 26 ] .

Fig. 5: Performance in Position-Angle Estimation: Comparison of visual stimuli (left) and MLAE scores with 95 $\%$ confidence intervals (lower is better) for various ViTs and humans (right). Human scores derived from Cleveland and McGill, and Hahen et al.’s studies [ 12 , 26 ] . All ViTs underperform compared to human baselines.

Fig. 6: Accuracy in Position-Length Estimation Across Chart Types: Comparison of visual stimuli (left) and MLAE scores with 95 $\%$ confidence intervals (lower is better) for various ViTs and humans (right). Human scores derived from Cleveland and McGill, and Hahen et al.’s studies [ 12 , 26 ] . Swin outperforms other ViTs but still lags behind human perception.

Fig. 7: Performance in Length Estimation with Framed Rectangles and Bars: Comparison of visual stimuli (left) and MLAE scores with 95 $\%$ confidence intervals (lower is better, best result is marked as boldface) for various ViTs and humans (right). Human scores derived from Cleveland and McGill, and Hahen et al.’s studies [ 12 , 26 ] . Like in previous cases, humans perform better than ViTs.

Fig. 8: Perception of Quantity in Point Cloud Estimation. Comparison of visual stimuli (left) and MLAE scores with 95 $\%$ confidence intervals (lower is better) for various ViTs and humans (right). Human scores derived from Cleveland and McGill, and Hahen et al.’s studies [ 12 , 26 ] . Note how ViTs struggle significantly with estimating dot quantities.

Fig. 9: Analysis of mean performance for CNNs from the study by Hahen et al. [ 26 ] across five experiments. The lowest mean score indicates the best-performing CNN.

Fig. 10: Analysis of mean performance for our trained ViTs across five experiments. Swin shows the best overall performance among ViTs, yet remains inconsistent compared to CNNs in several key encodings.

Network	Position common scale	Position non-aligned scale	Length	Direction	Angle	Area	Volume	Curvature	Shading		Average Error
MLP [ 26 ]	3.84	3.61	1.99	4.65	4.61	2.01	2.38	2.34	3.04		3.16
LeNet [ 26 ]	1.36	1.35	3.19	3.07	3.33	2.21	1.91	1.81	2.23		2.27
VGG19 [ 26 ]	-0.04	0.26	-0.14	0.92	0.66	-0.17	0.87	0.28	0.14		0.30
Xception [ 26 ]	1.04	1.02	1.11	1.57	1.69	1.38	2.10	1.13	1.82		1.42
ResNet-18	2.33	3.22	1.65	2.21	3.71	3.01	2.46	1.71	2.73		2.55
										Average CNNs Error	1.64

Table 3: Performance comparison on elementary perceptual tasks for various CNN and ViT models. The comparison is based on the MLAE scores (lower is better, best results encoded in boldface). VGG19 consistently achieves the lowest error among CNNs, while Swin leads the ViTs, yet with noticeably higher errors in complex visual encodings.

Network	Position common scale	Position non-aligned scale	Length	Direction	Angle	Area	Volume	Curvature	Shading		Average Error
VGG19* [ 26 ]	1.02	1.09	0.87	2.84	2.31	0.49	1.16	0.71	0.73		1.24
Xception* [ 26 ]	1.65	1.71	1.59	3.46	2.60	0.80	2.03	1.17	1.57		1.84
										Average CNNs Error	1.54
CvT*	2.55	3.64	-0.38	1.18	0.88	0.62	0.62	4.03	2.38		1.70
Swin*	2.10	2.16	1.93	1.38	0.55	1.53	3.54	1.94	3.55		2.15
vViT*	0.97	3.28	0.03	1.99	2.39	1.33	4.25	0.90	2.98		2.01

Table 4: Performance comparison of CNNs and ViTs (* indicates pretrained on ImageNet [ 45 ] ) on elementary perceptual tasks (lower is better, boldface indicates the best performing model). Pretrained CvT shows improved perceptual accuracy compared to models trained from scratch.

Experiments	Base			Image Resolution			Large Data			Pretraining			Small Patch
	CvT	Swin	vViT	CvT	Swin	vViT	CvT	Swin	vViT	CvT	Swin	vViT	vViT-8
Elementary Perceptual Tasks	4.42	0.95	2.77	4.44	2.60	3.05	4.02	1.99	2.58	1.70	2.15	2.01	4.61
Position-Length	5.34	4.72	5.05	5.15	4.73	5.09	5.05	4.72	5.03	4.77	4.70	5.00	2.14
Positron-Angle	4.62	4.21	5.41	4.93	4.22	5.54	4.66	4.21	5.30	4.47	4.32	4.72	5.51
Bars and Framed Rectangles	4.72	4.75	5.31	4.80	4.77	5.42	4.78	4.76	5.13	4.78	4.73	4.99	5.35
Point Cloud	9.09	6.02	6.83	35.21	4.80	7.01	42.41	6.77	6.60	4.39	3.94	5.86	7.45

Table 5: Average MLAE scores (lower is better) of three ViT models (CvT, Swin, vViT) across various experimental conditions in our ablation study: base architecture, increased image resolution, expanded training data, and smaller patch sizes (vViT-8). Results show minimal performance gains from increased data or resolution. Swin consistently outperforms others, especially in elementary perceptual tasks, while vViT-8 shows improvement only in specific configurations like position-length.

Category	Human (mean $\pm$ std) [ 26 ]	CvT (mean $\pm$ std)	Swin (mean $\pm$ std)	vViT (mean $\pm$ std)
Position common scale	3.30 $\pm$ 1.08	4.68 $\pm$ 0.06	2.81 $\pm$ 1.02	3.77 $\pm$ 0.53
Position non-aligned scale	3.14 $\pm$ 1.48	4.88 $\pm$ 0.26	2.61 $\pm$ 2.46	3.57 $\pm$ 0.47
Length	3.49 $\pm$ 1.08	3.77 $\pm$ 0.25	1.84 $\pm$ 1.04	1.23 $\pm$ 0.42
Direction	3.75 $\pm$ 0.90	4.14 $\pm$ 0.16	0.72 $\pm$ 0.24	2.12 $\pm$ 0.13
Angle	3.28 $\pm$ 1.00	4.16 $\pm$ 0.14	0.88 $\pm$ 0.61	3.48 $\pm$ 0.28
Area	3.63 $\pm$ 0.79	4.80 $\pm$ 0.51	2.25 $\pm$ 1.08	3.88 $\pm$ 0.11

Table 6: Comparison of mean MLAE and standard deviation between human observers and ViT models across nine elementary perceptual tasks. While ViTs like Swin achieve low average errors in certain tasks (e.g., shading, direction), they exhibit higher variance and inconsistency across encodings compared to humans, highlighting their limited reliability in generalizing perceptual judgments.

Task	Human (mean $\pm$ std) [ 26 ]	CvT (mean $\pm$ std)	Swin (mean $\pm$ std)	vViT (mean $\pm$ std)
Bar chart	2.05 $\pm$ 0.12	4.72 $\pm$ 0.19	4.21 $\pm$ 0.01	5.45 $\pm$ 0.03
Pie chart	2.05 $\pm$ 0.12	4.51 $\pm$ 0.06	4.22 $\pm$ 0.01	5.49 $\pm$ 0.03
Pie chart without outline	2.05 $\pm$ 0.12	4.68 $\pm$ 0.15	4.21 $\pm$ 0.01	5.44 $\pm$ 0.05

Table 7: Comparison of mean MLAE and standard deviation between human observers and ViT models across position-angle tasks. While ViTs like Swin achieve low average errors in each position angle task but they exhibit higher errors compared to humans, highlighting their limited reliability in generalizing perceptual judgments.

Category	Human (mean $\pm$ std) [ 26 ]	CvT (mean $\pm$ std)	Swin (mean $\pm$ std)	vViT (mean $\pm$ std)
Position length type 1	1.40 $\pm$ 0.14	4.79 $\pm$ 0.05	4.71 $\pm$ 0.02	4.77 $\pm$ 0.04
Position length type 2	1.72 $\pm$ 0.20	4.95 $\pm$ 0.17	4.76 $\pm$ 0.04	5.31 $\pm$ 0.06
Position length type 3	1.84 $\pm$ 0.16	5.16 $\pm$ 0.64	4.72 $\pm$ 0.01	4.82 $\pm$ 0.07
Position length type 4	2.35 $\pm$ 0.18	5.52 $\pm$ 0.10	4.72 $\pm$ 0.01	5.10 $\pm$ 0.05
Position length type 5	2.72 $\pm$ 0.16	5.37 $\pm$ 0.01	4.74 $\pm$ 0.01	5.13 $\pm$ 0.09

Table 8: Comparison of mean MLAE and standard deviation between human observers and ViT models across position-length tasks. While ViTs like Swin achieve low average errors in each position length type and they exhibit higher errors compared to humans, highlighting their limited reliability in generalizing perceptual judgments.

Task	Human (mean $\pm$ std)	CvT (mean $\pm$ std)	Swin (mean $\pm$ std)	vViT (mean $\pm$ std)
Framed Rectangle	3.33 $\pm$ 0.83	4.76 $\pm$ 0.03	4.75 $\pm$ 0.02	5.46 $\pm$ 0.05
Bar	3.93 $\pm$ 0.52	4.74 $\pm$ 0.05	4.75 $\pm$ 0.01	5.26 $\pm$ 0.07

Table 9: Comparison of mean MLAE and standard deviation between human observers and ViT models for bar and framed rectangle tasks. Swin achieve low average error for framed rectangle tasks and CvT for Bar tasks but they exhibit higher errors compared to humans, highlighting their limited reliability in generalizing perceptual judgments.

Task	Human (mean $\pm$ std) [ 26 ]	CvT (mean $\pm$ std)	Swin (mean $\pm$ std)	vViT (mean $\pm$ std)
Base 10	4.00 $\pm$ 0.52	9.12 $\pm$ 0.21	6.62 $\pm$ 1.23	8.22 $\pm$ 0.07
Base 100	5.39 $\pm$ 0.25	9.00 $\pm$ 0.00	6.03 $\pm$ 2.08	7.36 $\pm$ 0.08
Base 1000	5.46 $\pm$ 0.35	7.13 $\pm$ 0.82	5.73 $\pm$ 0.23	4.79 $\pm$ 0.00

Table 10: Comparison of mean MLAE and standard deviation between human observers and ViT models across three point cloud estimation tasks. vViT achieve low average error in Base 1000 task but overall exhibit higher errors compared to humans, highlighting their limited reliability in generalizing perceptual judgments.

Explore similar work

Texture Representations in Deep Vision Models: Comparing CNNs, Vision Transformers, and Human Perception
Jul 9, 2026Ludovica de Paolis, Marco Baroni, Alessandro Laio +1Visual Representation Learning Vision Transformer
Department of Neuroscience, International School for Advanced Studies (SISSA), Trieste, Italy · Department of Language and Translation Sciences, Pompeu Fabra University, Barcelona, Spain · ICREA, Barcelona, Spain +1
VEIL: How Visual Encoding Hijacking Induces Bias In Vision Models
Jul 6, 2026Suranjana Sooraj, Xuyang Chen, Madhumitha Venkatesan +1Scientific Visualization Data Visualization
University of California, Davis
Visual graphs for image classification: does the structure affect performance?
Jul 7, 2026Alessandra IbbaGraph Structure Learning Graph Neural Networks
University of Sassari - Department of Engineering, Sassari, Italy

Jul 9, 2026J/K move · Enter open · S save

Texture Representations in Deep Vision Models: Comparing CNNs, Vision Transformers, and Human Perception

Ludovica de Paolis, Marco Baroni, Alessandro Laio, Eugenio Piasini

Department of Neuroscience, International School for Advanced Studies (SISSA), Trieste, Italy · Department of Language and Translation Sciences, Pompeu Fabra University, Barcelona, Spain · ICREA, Barcelona, Spain+1

Visual Representation Learning Vision Transformer Convolutional Neural Networks

Open paper AskPDF arXiv

In computational vision science, Convolutional Neural Networks (CNNs) have emerged as a popular model of biological vision because of the alignment they can exhibit with neural and behavioral data in humans and animals. However, it remains unclear to what extent this alignment persists for visual tasks that extend beyond the canonical object recognition paradigm based on well defined semantic content. In this study, we diverge from the common object-centric view by focusing on another aspect of vision: texture perception. We consider textures of different complexity generated with three different algorithms from the same source images. Using a rank-based statistic, we quantify the information encoded in the internal representations of a CNN and three Vision Transformers (ViTs), and we compare the similarity of these representations to those inferred from human psychophysics data. We find that the representation of textures is aligned in different ViTs, but not between the ViTs and the CNN; that ViTs form similar representations for textures of different complexity; that human performance in recognizing textures can be better predicted from ViTs representations rather than CNN representations. Taken together, these results suggest that ViTs may capture more faithfully than CNNs how texture patterns are visually processed by humans, and that the representations of texture stimuli in computational models may be driven by the network architecture.

Abstract

Explore similar work

Texture Representations in Deep Vision Models: Comparing CNNs, Vision Transformers, and Human Perception

VEIL: How Visual Encoding Hijacking Induces Bias In Vision Models

Visual graphs for image classification: does the structure affect performance?