Organizations: Integrated Program in Neuroscience, McGill University, Montréal, Canada · Mila, Université de Montréal, Montréal, Canada · Department of Physiology, McGill University, Montréal, Canada
Computational models of cognition have traditionally relied on abstract symbolic inputs that approximate sensory experiences with simplified vector representations. This simplification reflects historical computational constraints and a long-held assumption that the details of sensory realism are incidental to the study of cognitive processes. Recent advances have enabled the development of sensory-realistic models capable of performing diverse cognitive tasks, yet whether sensory realism meaningfully shapes cognition has not been formally established. We address this gap through a large-scale empirical comparison of abstract and sensory-realistic neural network models trained on a battery of memory-dependent decision-making tasks. Our results show that sensory-realistic models mirror human behavioral response patterns significantly more than their abstract counterparts, though a substantial gap to human internal behavioral consistency remains even for the best tested models. By analyzing the internal dynamics of these systems, we find that the learned recurrent weights are not the main source of this divergence. In fact, frozen abstract-trained weights driven by sensory-realistic inputs recover the human consistency of sensory-realistic models. The resulting dynamics are also within the seed-to-seed variability of sensory-realistic models. Together, our findings argue that across a broad set of visual working-memory tasks, sensory realism accounts for a substantial portion of the model-human behavioral agreement, more than the choice of sequence architecture and comparable to the choice of vision encoder. These results suggest that findings obtained from abstract task models in neuroscience should be more carefully interpreted in light of their potential behavioral limitations.
Figures & tables
Figure 1: Schematic of the Experimental Setup. Models receive both task and stimulus information in their inputs. Task indicator is provided in either textual or one-hot while stimuli are presented in their naturalistic or abstract formats. In the abstract format, information about the primary object in the scene is provided as a combination of one-hot (category, object identity, position) and scalar values. Three groups are studied: humans, neural network models trained on abstract or natural inputs, and pretrained vision-language models. Behavioral responses are collected from each group and used to estimate the choice probabilities for each task trial. These probabilities are then used to calculate behavioral consistency with humans using different metrics.
Figure 2: Comparison of Abstract and Natural models across sequential architectures. (a) Correlation with human behavior. Gray line and zone indicates human internal behavioral consistency, ρ(h,h) in Eq. 1 . Bars show ρ(m,h) averaged over 20 random split-halves in Fisher- z space; error bars show the standard deviation across split-halves. (b) Task accuracy. Mean and standard deviation reported as error bars, 6 seeds for models and 20 participants for humans. VLMs (left to right): nemotron-nano-12b-v2 , nemotron-nano-12b-v2-no-thinking , qwen3-vl-30b , qwen3-vl-235b , and gemini-3.1-pro-preview (surpasses human-level accuracy).
Figure 3: Behavioral metrics of Abstract and Natural models with substituted components. From left to right, warm-colored bars show θAbs , θNat→Abs , (θ+Wout)Nat→Abs and untrained θ ; cold-colored bars show θNat , θAbs→Nat , (θ+Wout)Abs→Nat and untrained θ . Abstract models use final checkpoints. (a) Correlation with human behavior. The gray line and band indicate human internal behavioral consistency, ρ(h,h) in Eq. 1 . Bars show ρ(m,h) averaged over 20 random split-halves in Fisher- z space; error bars show the standard deviation across split-halves. On Abstract input, substitution improves human consistency only marginally (orange triangle and square vs. red circle: 0.138 and 0.188 vs. 0.082), barely above the untrained θ (0.126). On natural input, θAbs→Nat recovers the human consistency of the native model (blue triangle vs. circle: 0.446 vs. 0.452), well above the untrained θ (0.274), whereas (θ+Wout)Abs→Nat reaches only 0.258 (blue square). (b) Task accuracy; mean and standard deviation over 6 seeds for models and 20 participants for humans. With an untrained θ , models still reach 0.78 MTurk accuracy on abstract input but only 0.61 on natural input.
Figure 4: Dynamic distances of retrained models to native models. (a). In models that receive abstract input, θ and θ+Wout substitution lie above the noise floor at every dimension but well below the untrained θ . (b). In models that receive ResNet embedding input, from 5 dimensions on, θ substitution is at the noise floor, while θ+Wout and the untrained θ sit far above it, with θ+Wout farthest. Distances are between sequence-core hidden states. Gray bands and dashed lines indicate native models’ seed-to-seed distances, which is the noise floor. Error bars indicate standard error of mean over tasks and seed pairs.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Family
Description
# Tasks
Task IDs
NBack
n -back task for each feature
9
1–9
InterDMS
Two interleaved DMS sub-tasks (patterns: AABB, ABAB, ABBA)
27
10–36
CtxDMS
Contextual DMS: first comparison gates the feature to use in the second comparison
4
37–40
DMS
Simple delayed match-to-sample
3
41–43
Appendix
Table 1: Summary of the 43 DMS tasks.
Figure 5: Examples of the HvM stimuli seen by sensory-realistic models at 3 variation levels. Variation 0 (var-0): No variation. Objects are centered, with a fixed neutral view angle and fixed scale. Variation 3 (var-3): Medium variation. Objects undergo moderate translation, rotation, and scaling. Variation 6 (var-6): High variation. Objects undergo large translation, rotation, and scaling, presenting the most challenging visual condition.
Figure 6: An example trial of 1-Back Category task
Figure 7: An example trial of InterDMS - ABAB - Position - Identity task
Figure 8: An example trial of CtxDMS - Category - Identity - Position task
Figure 9: An example trial of DMS-position task
Figure 10: Correlation of behavioral metrics. Across models, human consistency ρ(m,h) is almost linearly related to error consistency κ (Pearson r=0.941 ) and much less to accuracy ( r=0.484 ), which motivates measuring model behavior beyond accuracy.
Figure 11: Cross-task and cross-stimulus RSA. (a) Human cross-task RSA matrix shows clusters of high correlation in tasks using the same stimuli features, e.g. 1back position and InterDMS position. Since identity tasks are related to category tasks, we also see correlation between the response vectors in these cross-task comparisons. (b) Correlation between cross-task RSA and human consistency. Abstract models (orange) have much lower alignment values than natural models (blue, green, and purple colors match the colors used in Fig. 17 ). (c) Human cross-stimulus RSA matrix. If two stimuli are similar in category, identity, and position, then they would have high correlation across tasks. (d) Correlation between cross-stimulus RSA and human consistency. Cross-stimuli RSA alignment predicts human consistency better than cross-task RSA.
Figure 12: Feature-level linear-SVM accuracy and behavioral metric correlation.
Figure 13: Comparison of Abstract models with matched accuracy and final accuracy. (a) Correlation with human behavior. (b) Task accuracy. Empty orange bars are model checkpoints with 75% validation accuracy, orange bars with backward slashes are 80% matched accuracy, and orange bars with forward slashes are final model checkpoints. The blue bar is the ResNet-CLIP LSTM natural model. Although accuracy on the MTurk trials increases (right figure), the gap in behavioral alignment persists (left figure).
Figure 14: Behavioral metrics of accuracy-matched Abstract and Natural models with substituted components. Same conditions and colors as Fig. 3 , but with accuracy-matched Abstract checkpoints. Models receiving abstract input (orange) are evaluated at the checkpoint closest to 80% validation accuracy, and the Abs→Nat substitutions (blue) load the sequence module of the 80%-matched Abstract model. (a) Human consistency ρ(m,h) . (b) Task accuracy on the MTurk trials. All trained models fall within 0.03 of each other in accuracy, except the untrained- θ Natural baseline. With accuracy controlled, the input still dominates behavior alignment. Every model receiving naturalistic input, including the untrained- θ baseline, is more consistent with humans than every model receiving abstract input. A natural θ barely changes abstract-input models: θNat→Abs falls slightly below the native Abstract model (0.147 vs. 0.183), to the level of an untrained θ . Keeping the natural readout as well raises consistency only to 0.221, below every model receiving naturalistic input. The 80%-matched abstract θ supports higher consistency than an untrained θ but lower than θNat , whereas the fully trained module in Fig. 3 matches θNat , so the contribution of an abstract sequence module grows with its training.
Figure 15: Comparison of Abstract models with dimensionality and perceptual uncertainty matching. (a) Correlation with human behavior decreases as more noise is injected. (b) Task accuracy also decreases as more noise is injected. Each level shows 6 seeds at their final checkpoints, including seeds that did not learn the task. Responses are averaged over 20 noise draws.
Figure 16: Behavioral metrics of dimensionality-matched Abstract and Natural models with substituted components. Same conditions and colors as Fig. 3 , but with dimensionality-matched Abstract models and θ+Win substitution (diamond markers). Bars below zero are not drawn.
ρ(m,h)
Encoder
Val. acc.
MTurk acc.
All
Category
Identity
Position
Pretrained
0.771±0.029
0.706±0.021
0.452
0.329
0.534
0.381
Random
0.596±0.009
0.516±0.012
0.025
0.000
0.000
0.101
Appendix
Table 2: Natural LSTM with a frozen pretrained or a frozen randomly initialized ResNet-CLIP encoder (6 seeds). Accuracy: mean ± SD over seeds; ρ(m,h) : mean over 20 split-halves.
Figure 17: Comparison of behavioral alignment across vision encoders, sequential architectures (LSTM vs. Transformer), hidden dimension sizes, and vision encoder sizes. Hatch patterns indicate pretraining method: empty bars for supervised ImageNet [ 36 ] , forward slash for CLIP, backslash for DINOv1 [ 111 ] , crossed diagonal lines for DINOv3 [ 89 ] , and horizontal lines for masked image modeling (MIM) [ 112 ] . (a) Correlation with human behavior. (b) Task accuracy. (c) Effect of sequential module hidden dimension size on human alignment. (d) Effect of hidden size on task accuracy. Increasing model representation size increases accuracy, but does not increase behavioral alignment. For natural transformers, the human alignment declines steadily. (e) Effect of vision encoder size (number of parameters, log scale) on human alignment, for Natural LSTMs with ConvNeXt (supervised ImageNet-22k; Tiny to XLarge) and EVA02-CLIP (Base, Large, Enormous) encoders. (f) Effect of vision encoder size on task accuracy. Neither behavioral alignment nor accuracy increases consistently with encoder size.
Figure 18: Feature-level human consistency and accuracy of trained task models and VLMs. For every architecture, the Natural model is more human-consistent than the Abstract on every feature, and both model types are most human-consistent on identity trials. Among Natural models, BERT and the Transformer lead on identity, the Transformer on category, and the LSTM on position. VLMs are the most human-consistent models on category trials (0.39–0.58) but fall below every Natural model on identity trials. On position trials, the human consistency of Abstract models is 0 for every architecture except BERT ( 0.01 ). We hypothesize that Abstract models solve position tasks from the one-hot position code, whereas Natural models, VLMs and humans judge position from the object’s location over an ambiguous 3×3 grid on a naturalistic background. Abstract models are evaluated at the 80%-matched checkpoint; markers, error bars and the human reference follow Fig. 2 .
Figure 19: Task-family human consistency and accuracy by sequence architecture. (a, b) NBack, (c, d) InterDMS, (e, f) CtxDMS and (g, h) DMS; left of each pair ρ(m,h) , right MTurk accuracy. For every architecture, the Natural model is more human-consistent than the Abstract in every task family. Among Natural models, the LSTM leads on InterDMS and DMS, and the Transformer on CtxDMS, while four architectures are within 0.015 of each other on NBack. Among Abstract models, BERT leads in every family. On DMS trials (the DMS-position task only) every Abstract model has zero correlation with humans. Abstract models are evaluated at the 80%-matched checkpoint; markers, error bars and the human reference follow Fig. 2 .
Covariate
R2
Unregressed
Residualized
Null
Effect
ResNet-CLIP
0.029
0.229
0.249
0.252±0.001
−0.003
DINOv3
0.042
0.229
0.233
0.252±0.002
−0.019
Appendix
Table 3: Natural minus Abstract difference in ρ(m,h) (mean over the five sequence models; Abstract models at the 80%-matched checkpoint) before and after regressing out visual discriminability. Null: mean ± SD over 10 within-condition permutations of the covariate. Effect: residualized minus null.
Figure 20: Representational similarity analysis (RSA) of substituted models. The y-axis shows the Spearman correlation between the upper-triangular entries of the model and human representational dissimilarity matrices (RDMs).
Figure 21: Cross-task response structure of humans and LSTM models. (a) Human cross-task matrix Ctask (Appendix A.2.7 ); rows and columns are the 18 task conditions, in the same order in every panel. (b, d) Ctask of the Natural LSTM (ResNet-CLIP) and of the Abstract LSTM at its 80%-matched checkpoint, on the same scale as (a). (c, e) Model minus human, on the shared scale at the right. Green cells are condition pairs the model correlates more strongly than humans do, pink cells less strongly. The diagonal is the split-half reliability of each condition, not 1.
Figure 22: Dynamic distances of retrained models to native models in the projected input. Each panel shows the DSA angular distance between each retrained model and the native models of the same input modality, computed on the projected input zprojt , against the number of PCA components. (a). In models that receive abstract input, θ is at the noise floor at every dimension, while θ+Wout and the untrained θ lie just above it. (b). In models that receive ResNet embedding input, θ+Wout lies at or just above the noise floor, θ is well above it, and the untrained θ is farthest. Closeness in zprojt does not predict closeness in the hidden state ht (Fig. 4 ). With abstract input, θ is at the noise floor here but not in ht . With ResNet input, θ is far from native here but at the noise floor in ht , while θ+Wout shows the reverse.
Figure 23: Dynamic distances of retrained dimensionality-matched models to native models. Each panel shows the DSA angular distance between each retrained model and the native models of the same input modality, against the number of PCA components, computed on the projected input zprojt (a, b) or the hidden state ht (c, d). (a). In models that receive dimensionality-matched abstract input, θ is at the noise floor in zprojt at every dimension and θ+Wout lies above it, while the untrained θ and θ+Win are far above it, with θ+Win farthest. (b). In models that receive ResNet embedding input, θ and θ+Wout are at the noise floor in zprojt from 5 dimensions on, while θ+Win and the untrained θ are far above it, with the untrained θ farthest. (c). In ht of models that receive abstract input, θ and θ+Wout lie above the noise floor at every dimension but well below the untrained θ , and θ+Win is farthest. (d). In ht of models that receive ResNet embedding input, θ and θ+Wout are at the noise floor from 5 dimensions on, while the untrained θ sits far above it and θ+Win is farthest.
Figure 24: Pairwise DSA distances between native and substituted models. (a) DSA pairwise distance heatmap on zprojt . (b) DSA pairwise distance heatmap on ht . (c) MDS [ 113 ] applied to the pre-computed pairwise distance matrix of zprojt . (d) MDS applied to the pairwise distance matrix of ht .
Comparison
Feature
μ
Delays
Rank
Distance
Total
2–50
1, 3
2, 4
Eucl./Wass.
Natural input
θAbs→Nat within floor
ht
4/5
1/2
1/2
2/2
8/11
Untrained θ farther than θAbs→Nat
ht
5/5
2/2
1/2
1/2
9/11
θAbs→Nat above floor
zprojt
5/5
2/2
0/2
2/2
9/11
(θ+Wout)Abs→Nat within floor
zprojt
3/5
1/2
1/2
1/2
6/11
Appendix
Table 4: DSA sensitivity, number of configurations in which each comparison holds. Each column varies one setting around the default setting of the main text ( μ=10 , 2 delays, rank 3, angular distance), which is counted once, in the μ column. The floor is the mean distance between seeds of the native model of the input modality; “within” means not significantly above it (one-sided paired Wilcoxon signed-rank test over the 12 tasks, p<0.05 ). The 8 optimizer and seed configurations give the default outcome in every row and are not shown.
Item
Value
Input dimension
171 for natural models (image embedding of size 128 + one-hot task embedding of size 43 ), 128 for abstract models
Hidden dimension
RNN: 256 , All other models: 128
Layers
1
Learning rate
D∈{64,128,256} : 3e-4 for Natural models, 3e-3 for Abstract models D∈{512,1024} : 1e-4 for Natural models, 3e-4 for Abstract models
Label smoothing
0.2 for natural models, none for abstract
Attention head
8
Appendix
Table 5: Training configuration.
Family
# of frames
Param.
Action frame
ID
Comparison dimension
NBack
6
n=1
2–6
1
position
2
identity
3
category
n=2
3–6
4
position
5
identity
6
category
Appendix
Table 6: Complete listing of all 43 tasks. Each task is defined by a family-specific parameter (memory lag n for NBack, interleaving pattern for InterDMS) and its comparison dimension(s); for CtxDMS the arrows give the feature ordering. Raw frames are the natural sequence length; all tasks are padded to 6 frames in multi-task training. Action frames are the frames that require a response (1-based).