q-bio.NCNov 7, 2026

Pixels over Symbols: Sensory Realism Improves Behavioral Alignment in Models of Cognition

Authors: Hao Yuan Bai, Xiaoxuan Lei, Zihan Weng, Motahareh Pourrahimi, Lucas Gomez, Pouya Bashivan

Organizations: Integrated Program in Neuroscience, McGill University, Montréal, Canada · Mila, Université de Montréal, Montréal, Canada · Department of Physiology, McGill University, Montréal, Canada

Abstract

Computational models of cognition have traditionally relied on abstract symbolic inputs that approximate sensory experiences with simplified vector representations. This simplification reflects historical computational constraints and a long-held assumption that the details of sensory realism are incidental to the study of cognitive processes. Recent advances have enabled the development of sensory-realistic models capable of performing diverse cognitive tasks, yet whether sensory realism meaningfully shapes cognition has not been formally established. We address this gap through a large-scale empirical comparison of abstract and sensory-realistic neural network models trained on a battery of memory-dependent decision-making tasks. Our results show that sensory-realistic models mirror human behavioral response patterns significantly more than their abstract counterparts, though a substantial gap to human internal behavioral consistency remains even for the best tested models. By analyzing the internal dynamics of these systems, we find that the learned recurrent weights are not the main source of this divergence. In fact, frozen abstract-trained weights driven by sensory-realistic inputs recover the human consistency of sensory-realistic models. The resulting dynamics are also within the seed-to-seed variability of sensory-realistic models. Together, our findings argue that across a broad set of visual working-memory tasks, sensory realism accounts for a substantial portion of the model-human behavioral agreement, more than the choice of sequence architecture and comparable to the choice of vision encoder. These results suggest that findings obtained from abstract task models in neuroscience should be more carefully interpreted in light of their potential behavioral limitations.

Figures & tables

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. More Accurate, Less Human: Gestalt Grouping in Vision Models

    Aug 10, 2026Sudhanva Manjunath Athreya, Sai Phani Kumar MalladiVLM EvaluationData Visualization

  2. Alignment of visual brain encoding models with individual perceptual similarity judgments

    Oct 15, 2026Daniel Chong, Haomiao Chen, Keith W Jamison +4fMRI Encoding ModelsNeural Encoding