Do Machines Fail Like Humans? A Human-Centred Out-of-Distribution Spectrum for Mapping Error Alignment
Authors: Binxia Xu, Xiaoliang Luo, Luke Dickens, Robert M. Mok
Organizations: School of Data Science, Fudan University, Shanghai · Independent Researcher, London · Department of Information Studies, University College London, London · Center for Information and Neural Networks, National Institute of Information and Communications Technology, University of Osaka, Osaka
Determining whether AI systems process information similarly to humans is central to cognitive science and trustworthy AI. While modern AI models can match human accuracy on standard tasks, such parity does not guarantee that their underlying decision-making strategies resemble those of humans. Assessing performance using error alignment metrics to compare how humans and models fail, and how this changes for distorted, or otherwise more challenging, stimuli, provides a viable pathway toward a finer characterization of model-human alignment. However, existing out-of-distribution (OOD) analyses for challenging stimuli are limited due to methodological choices: they define OOD shift relative to model training data or use arbitrary distortion-specific parameters with little correspondence to human perception, hindering principled comparisons. We propose a human-centred framework that redefines the degree of OOD as a spectrum of human perceptual difficulty. By quantifying how much a collection of stimuli deviates from an undistorted reference set based on human accuracy, we construct an OOD spectrum and identify four distinct regimes of perceptual challenge. This approach enables principled model-human comparisons at calibrated difficulty levels. We apply this framework to object recognition and reveal unique, regime-dependent model-human alignment rankings and profiles across deep learning architectures. Vision-language models are most consistently human aligned across near- and far-OOD conditions, but convolutional neural networks (CNNs) are more aligned than vision transformers (ViTs) for near-OOD and ViTs are more aligned than CNNs for far-OOD. Our work demonstrates the critical importance of accounting for cross-condition differences, such as perceptual difficulty, for a principled assessment of model-human alignment.
Figures & tables
Figure 1: A human-centered OOD spectrum. (A) Example images from two distortion types (left column: low-pass filtering; right column: high-pass filtering) from the modelvshuman datase [ 25 ] at different parameter levels. Images within the same row share the same parameter rank within each distortion type. (B) Illustration of the process for measuring the degree of human performance deviation on a specific distortion condition relative to undistorted images. (C) OOD scores for all levels of corruption for each distortion type in modelvshuman dataset. The x-axis represents the OOD score, indicating the degree of deviation from the performance baseline on undistorted images, with greater negative values corresponding to larger distributional shifts. (D) Four-component Gaussian Mixture Model (GMM) fitted to OOD scores across all distortion conditions, defining the four OOD groups. Color reflects different OOD regimes. Boundaries illustrated by dashed lines. (E) OOD spectrum with example images from selected distortions.
Figure 2: Human-human error alignment A. Dot plots for Error Consistency (EC; top), and Misclassification Agreement (MA; bottom) across distortion types and OOD levels. Each dot represents the mean error alignment value between each pair of participants for a specific level of a distortion, coloured by its OOD category, with vertical bars indicating ±1 standard deviation. Transparent violin plots show the distributions of the alignment scores across distortion levels within distortion type. B. t-SNE plot illustrating similarity of human-human error patterns across distortion domains based on the human-human class-level error divergence (CLED) matrix. Each point represents a distortion condition, labeled by distortion type (e.g., CT: contrast, HP: high-pass). Colors indicate OOD regime they belong to. Spatial proximity reflects similarity in human error patterns across conditions.
Cohens’ D
Permutation Test
p -value
Effect Size
Distortion Type
-0.161
0.025
-2.331
OOD-Level
-0.599
0.000
-12.555
Table 1: Comparison of distortion type and OOD level effects on class-level error divergence (CLED). Reported are Cohen’s D and permutation test results. Negative D indicates lower within-group CLED (greater error similarity). OOD level shows a markedly stronger effect than distortion type in both measures.
Figure 3: Model-human alignment across distortion levels. Accuracy (ACC, top row), Error Consistency (EC, middle row), and Misclassification Agreement (MA, bottom row) for humans and models across three representative distortion types: High-pass (left), Low-pass (middle), and Contrast (right). Each curve shows the mean: for humans, the mean across participant pairs (orange); for models, the mean across model–human pairs within a subfamily. Curve colours denote superfamilies (CNNs: purple, ViTs: yellow, VLMs: pink), while marker shapes distinguish subfamilies within each superfamily. Shaded regions indicate ±1 standard deviation. Curves are plotted against corruption parameters that control the severity, with background shading marking OOD regimes.
Model
near-OOD
far-OOD
Family
Families
p-value
Effect Size
p-value
Effect Size
All
CNN, ViT, VLM
0.001
3.721
< 5e-4
14.923
CNN
VGG, ResNet, DenseNet, Inception*
0.005
3.072
< 5e-4
4.593
ViT
ViT (l or b), Swin, MaxVit*, EfficientNet*
0.336
0.424
0.057
2.037
VLM
CLIP-ResNet, CLIP-ViT, CLIP-Vit (small), BLIP*
0.014
2.908
0.005
3.323
Table 2: Results of inter- vs intra-family alignment permutation test for near-OOD and far-OOD. Families marked with an asterisk (*) contain one model so do not contribute to sub-family differences.
Figure 4: Radar plots of mean model–human error alignment for each superfamily across distortion types, separated by OOD regime (near-OOD, EC (A), MA (B); far-OOD, EC (C), MA (D)). Each axis represents a distortion type, and the radial coordinate indicates model–human alignment scores. Each curve shows the mean EC or MA value for a model superfamily, with error bars indicating ±1 standard deviation. Human–human alignment scores are denoted in orange.
Figure 5: Model–human error alignment rankings. (A) Model–human error alignment values normalised by human–human alignment ordered from most or least aligned in near- (top) and far-OOD (bottom) regimes. Values are the ratio of model–human alignment to human–human alignment, where higher values indicate that the error patterns between those models and humans are closer to the baseline agreement among humans. Each dot represents a model, with vertical bars showing ±1 SD confidence interval. Colours denote model families (CNNs in purple, ViTs in yellow, VLMs in pink). (B) Scatter plot of alignment ratio near-OOD (x-axis) and far-OOD (y-axis) conditions. (C) Pairwise statistical ranking of model superfamilies under near- and far-OOD regimes. Each plot summarises Mann–Whitney U tests comparing the ranks between model families; “ > ” indicates a significant rank advantage ( p<0.01 ), while “ ≥ ” denotes a non-significant difference. Median alignment ratios are shown only to aid visual interpretation.
Table 2: Mann–Whitney U test p -values (raw and BH-corrected) for all distortion conditions. Each p -value indicates whether the distribution of human accuracies under the distortion differs significantly from the non-distorted baseline. Lower values (e.g., p<0.01 ) suggest a statistically significant shift in prediction behavior. The adjusted p values (corrected by the Benjamini–Hochberg procedure) are used to reject conditions under false discovery rate control at α=0.01 . Bold rows denote conditions with no distortion applied (reference condition/baseline). Asterisks (*) mark conditions with raw p>0.01 (all of them also rejected the null hypothesis after the BH-procedure).
Table 3: Results of binomial tests under the null hypothesis P(1/16) .
Figure 6: Histogram of accuracy on the uncorrupted datasets before and after logit transformation. (a) Accuracy values lie in the bounded interval [0, 1], and those values for the uncorrupted set are usually very high, leading to a left-skewed distribution near 1. The fitted Beta distribution illustrates this skew. (b) After applying the logit transformation, the distribution becomes more symmetric, resembling a normal distribution. This transformation allows standardised comparisons using parametric methods.
Shapiro–Wilk
D’Agostino–Pearson
Lilliefors
Accuracy ( a )
0.004
0.001
0.021
Logit-transformed Accuracy ( la )
0.194
0.135
0.192
Table 4: P-values from three normality tests (Shapiro–Wilk, D’Agostino–Pearson, and Lilliefors) applied to both raw accuracy values and logit-transformed accuracy. While the raw accuracy values significantly deviate from normality ( p<0.05 ), the logit-transformed accuracies yield p-values above the 0.05 threshold across all tests. This indicates that we cannot reject the null hypothesis that la is drawn from a normal distribution.
Figure 7: Model selection using the Bayesian Information Criterion (BIC) and corrected Akaike Information Criterion (AICc). Both criteria indicate four components as the optimal model complexity, with BIC and AICc reaching their minimum values at this point.
Figure 8: Accuracy (ACC, top row), Error Consistency (EC, middle row), and Misclassification Agreement (MA, bottom row) for humans and models across three distortion types: EidolonI (left), EidolonII (middle), EidolonIII (right).
Figure 9: Accuracy (ACC, top row), Error Consistency (EC, middle row), and Misclassification Agreement (MA, bottom row) for humans and models across three distortion types: Uniform Noise (left), Phase Scrambling (middle), Rotation (right).
Figure 10: The ranking for each metric.
Figure 11: Radar plots of mean model–human error alignment for each subfamily across distortion types, separated by OOD regime (near-OOD, EC (left top), MA (right top); far-OOD, EC (left bottom), MA (right bottom)). Human–human alignment scores are denoted in orange.
Human-model alignment is critical for trustworthy AI-assisted decision-making systems. Yet, most work evaluates model predictions against single ground-truth labels, overlooking that humans themselves often disagree on labels, a signal of genuine ambiguity. We investigate whether models struggle on the same instances that humans find difficult. We measure this on two vision datasets (FER+ and CIFAR-10H) where multiple human annotations per image capture human disagreement patterns. We evaluate eight pretrained models across three architectures (ResNet, EfficientNet, MobileNetV3) in two parts: first, whether model uncertainty (softmax confidence, entropy) correlates with human disagreement, and second, whether predictive multiplicity measures (inter-model disagreement, Jensen-Shannon divergence) do. We find that it does not: alignment is weak in both dimensions. At the discrete label level, 50.4% of CIFAR-10H images and 33.5% of FER+ images receive multiple valid classifications from humans, while the models converge on only one. These instances represent a critical failure case where humans perceive ambiguity and would request expert review, yet models decide confidently. At the continuous score level, single-model uncertainty correlates weakly with human disagreement (ρ=0.24−−0.55), and predictive multiplicity provides only modest improvement. Widely-used uncertainty quantification methods do not reliably identify instances humans find ambiguous. Model uncertainty should not be treated as a trustworthy signal by default for decision-making in high-stakes scenarios.
Manya Singh, Arjun Pakrashi
School of Computer Science, University College Dublin, Ireland
Evaluating the perceptual alignment between Contrastive Vision-Language Models (CVLMs) and humans is typically constrained by traditional benchmarks that overlook fine-grained semantic and cultural nuances. In this work, we propose a novel evaluation framework that leverages the gamified, discrete color space of the board game Hues and Cues. By mapping the board's 480 color cells to the CIE xy chromaticity diagram, we calculate empirical perceptual distances across a carefully curated 100-word vocabulary spanning seven semantic categories. To properly contextualize model performance, we establish an empirical lower bound of expected error-the Human Consistency baseline-calculated via Leave-One-Out (LOO) cross-validation on a dense dataset of color associations collected from 325 human observers through a custom digital interface. We evaluate 162 models across multiple architectural families and pre-training datasets to assess their semantic color grounding. Our results demonstrate that while CVLMs successfully replicate human cognitive biases, such as idealized memory colors for concrete physical referents (e.g., food and plants), they systematically diverge from the human baseline in abstract, subjective, and pop-culture domains. We identify two distinct failure modes in severely misaligned concepts: semantic misclassification and a systematic uncertainty collapse into a default blue coordinate. Furthermore, we reveal that highly curated pre-training datasets are significantly more effective than massive, uncurated corpora in mitigating these severe misalignments. Ultimately, this work highlights that despite their broad categorization capabilities, current CVLMs still fail to capture the nuanced, localized consensus of human color memory, emphasizing the value of gamified tasks in exposing underlying model biases. The data and code are publicly available to test other metrics.
Nuria Alabau-Bosque, Jorge Vila-Tomás, Paula Daudén-Oliver +3
Image Processing Lab, Universitat de València, Carrer del Catedrátic José Beltrán Martinez, Paterna, 46980, Spain
Modern machine learning models typically represent inputs as fixed points in a high-dimensional embedding space. While this approach has been proven powerful for a wide range of downstream tasks, it fundamentally differs from the way humans process information. Because humans are constantly adapting to their environment, they represent objects and their relationships in a highly context-sensitive manner. To address this gap, we propose a method for context-sensitive similarity computation from neural network embeddings, applied to modeling a triplet odd-one-out task with an anchor image serving as simultaneous context. Modeling context enables us to achieve up to a 15% improvement in odd-one-out accuracy over a context-insensitive model. We find that this improvement is consistent across both original and "human-aligned" vision foundation models.
Frieda Born, Tom Neuhäuser, Lukas Muttenthaler +6
Technische Universität Berlin · BIFOLD · Max Planck Institute for Human Development +9