Encoding models that predict brain activation responses to images have traditionally been trained to maximize accuracy when predicting observed brain data. However, their internal representations may not reflect individual human perception. In this work, we investigate whether we can fine-tune brain encoding models to better align their representations with individuals' similarity judgments. Using similarity judgments of 100 images from 8 participants in the Natural Scenes Dataset (NSD), we adapt a generic human similarity judgment model (DreamSim) separately for each participant to obtain personalized perceptual similarity models. These models are then used to obtain subject-specific similarity structure across images in their encoding model training sets. We fine-tune each participant's baseline encoder in two conditions, aligning its representations with either generic or personalized similarity judgments, with the same initialization and objective. We find that baseline brain encoders reflect within-subject perceptual similarity more strongly than measured fMRI responses and preserve intersubject correlations present in measured fMRI, but fail to reflect those of human perceptual similarity judgments. Fine-tuning with generic, population-level perceptual similarity information results in better alignment with human judgments than baseline encoders, but these models still fail to capture intersubject correlations of human judgments. However, fine-tuning with personalized perceptual similarity information results in models that are the most aligned with human perceptual similarity judgments and also reflect intersubject correlations of human judgments. Our framework provides an approach for bootstrapping human similarity judgment data into more individualized, human-aligned brain encoding models of vision.
Figures & tables
Figure 1: Overview of the human image similarity judgment-brain encoding model alignment framework. (a) Each of the eight NSD participants arranged an identical set of 100 images according to their perceived similarity, producing a subject-specific human similarity judgment matrix. (b) A generic model of human perception (here called Population DreamSim) was adapted to each participant’s observed similarity judgments to create perceptual similarity models that reflect that person’s judgments (here called personalized DreamSim). Population and personalized DreamSim models were used to generate similarity values between each pair of images in a given participant’s approximately 9,000 training set images. (c) Each participant’s brain encoder was fine-tuned using both population and personalized DreamSim estimates of image similarity to increase alignment of the brain encoder’s predictions of brain responses with human perception. (d) The resulting representations were evaluated by their alignment with human similarity judgments and their preservation of across-subject similarity structure.
Figure 2: (a) Within subject alignment of an individual’s perceptual similarity judgments for pairs of images in the set of 100 images used in the perceptual task and i) measured fMRI (left, purple) and ii) baseline brain encoder predictions(right, pink) for the same pairs of images. (b) Intersubject similarity matrices derived from human similarity judgments, measured fMRI, and Baseline Brain Encoder predictions for image-pairs in the set of 100 images across the eight NSD subjects. (c) Spearman correlation between the 28 unique subject-pair values of each intersubject similarity matrix across the modalities in panel b). Intersubject baseline encoder structure strongly matches measured fMRI ( ρ=0.82 , p=0.008 ), but neither measured fMRI nor the baseline encoder matches behavioral intersubject similarity judgment structure ( ρ=−0.19 and −0.03 , both p>0.05 ).
Figure 3: (a) Within-subject alignment of an individuals’ image-pair similarity judgments for the 100 images in the perceptual task compared to the representational similarity of: population DreamSim, personalized DreamSim, baseline brain encoder, brain encoder + population DreamSim, and brainencoder + personalized DreamSim, for the same image-pairs. Bars above pairs of boxplots indicate those with significant differences ( p<0.05 ). (b) Human similarity judgment alignment (Spearman ρ ) between each subject-specific model (rows) and each subject’s human similarity judgments (columns); red boxes mark the maximum value for each column. Boxes on the diagonal indicate that person’s model best matched to their observed similarity judgments compared to others’ similarity judgments. (c) Intersubject correlations of representational spaces for human similarity judgments vs intersubject correlations of representational spaces the same modalities listed in panel (b). Each of the 28 points represents a unique pair of individuals, where the x-axis is the correlation of those two individuals’ human similarity judgment RSMs and the y-axis is the correlation of those two individuals’ RSMs for that specific modality.
Figure 4: (a) A Sankey plot showing changes in image-pair similarity agreement between human judgments and the brain encoder before (left column) and after (right column) Personalized DreamSim fine-tuning. Red vertical bars indicate humans and encoders did not agree in their similarity assessments for that image-pair, while green vertical bars indicate the two models did agree. (b) Representative image-pairs illustrating the different group agreement transition types (from baseline brain encoder to brain encoder + Personalized DreamSim), where the color behind each vertical label matches the image-pair group in panel (a). Image-pair similarity scores for a specific human’s judgments (i.e. “S7” indicates subject 7), baseline brain encoder, and brain encoder + Personalized DreamSim are above each image pair.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Subject-specific relative change in brain encoder accuracy plotted against that same individuals’ change in human similarity judgment alignment, with each subject a different color. Baseline Brain Encoders are marked as stars, Brain Encoder + Population DreamSim are marked as circles, and Brain Encoder + Personalized DreamSim are marked as diamonds. Dashed lines with arrows emphasize the flow from Baseline Brain Encoder → Brain Encoder + Population DreamSim → Brain Encoder + Personalized DreamSim.
Figure 6: Region-wise correlation between brain and human representational similarity matrices, for the 19 ROIs common to all eight subjects and ordered early → late along the visual hierarchy. Each region’s RSM is built from that region’s voxels alone and correlated with that same individual’s human similarity judgment RSM; boxes summarize the eight subjects and the overlaid points are those same eight values. Measured fMRI is shown in purple, Baseline Brain Encoders in gray, Brain Encoder + Population DreamSim in orange, and Brain Encoder + Personalized DreamSim in blue. All four are scored on the pooled held-out pairs of 5-fold cross-validation (950 pairs per subject), averaged over ten fold assignments.
Artificial vision models are often evaluated against the human visual cortex by measuring how accurately their internal representations predict brain responses. However, prediction accuracy alone does not indicate which dimensions of the target brain's response space are recovered. Here, we introduce a unified framework for evaluating both model-brain and brain-brain alignment by identifying the response dimensions recovered by prediction. Using repeated fMRI measurements, we first identify target-brain response dimensions that can be reproducibly predicted across independent trial splits. We then predict target-brain responses from either another subject's brain responses or a vision model's internal representations, and quantify how strongly each of these reproducible response dimensions is recovered. Applying this framework to a subset of the Natural Scenes Dataset, in which eight subjects viewed the same natural images during fMRI, we find that the early-to-intermediate visual-cortex responses contain a low-dimensional set of reproducible dimensions. Brain-to-brain comparisons identify which of these dimensions are consistently recoverable from other subjects' brains, providing a diagnostic human reference rather than only a scalar benchmark. In some cases, pretrained and randomly initialized models achieve similar prediction accuracy while showing distinct recovery profiles across these response dimensions. These results show that prediction accuracy alone can mask model-brain mismatches. By making explicit which reproducible brain response dimensions are recovered by prediction, our framework provides a more diagnostic evaluation of alignment between artificial vision models and the human visual cortex.
Ken Nakamura, Tomoya Nakai, Ryuto Yashiro +2
1The University of Tokyo · 2Osnabrück University and Freie Universität Berlin · 3Kobe University
Artificial neural networks trained on visual tasks develop internal representations resembling those of the primate visual system, a discovery that has guided a decade of computational neuroscience. Research on building brain-aligned models has progressively embraced finer-grained learning ob- jectives, from object classification to contrastive self-supervised objectives that maximize distinc- tions among individual images. Yet the effect of learning-signal granularity on brain alignment remains largely unexamined. Here we systematically investigate how the granularity of a learning signal shapes representational alignment with human vision. We parametrically vary the number of training classes using a data-driven approach that partitions a set of training images into differ- ent numbers of categories via PCA-based splits of pretrained embeddings. We train hundreds of neural networks across convolutional and transformer architectures on these coarse classification tasks and compare their representations with human fMRI responses, macaque electrophysiology recordings, and human behavior. We find that networks trained to distinguish as few as eight broad categories learn representations that match or exceed the neural alignment of models distinguishing 1,000 classes. Even more strikingly, these coarsely trained networks align more closely with hu- man perceptual similarity judgments than all other models evaluated, including networks trained with fine-grained supervision or self-supervision as well as leading large-scale vision models. These results demonstrate that human-like visual representations can emerge from surprisingly simple learning objectives, reframing what learning signals vision may require and opening a path toward building AI systems that are more aligned with human perception.
Yash Mehta, Michael F. Bonner
Department of Cognitive Science, Johns Hopkins University
Most knowledge of graphical perception comes from behavioral studies. Understanding from a neural perspective is much more limited due in part to neuroimaging studies' expensiveness and difficulty to conduct. In this paper, we evaluate whether Meta's Tribe V2 neural encoding model can recover neural contrasts from a visualization fMRI study. Specifically, we evaluate Tribe V2 through a conceptual replication of the visualization-viewing component of a prior comparison of Bubble charts and three-dimensional Surface charts in color and grayscale. We generate TRIBE-predicted cortical responses for the original stimuli and compare the resulting contrasts with those reported in the human study. The model reproduced the direction of 11 of 14 reported cortical effects, with agreement concentrated in visual-processing regions. This agreement characterizes the model's alignment with the prior human-generated fMRI results rather than independently confirming them. We discuss the limitations encountered when working with this model for in-silico replication and hope to encourage future work exploring this new avenue for neuroimaging studies in visualization. Supplemental materials are available at https://osf.io/8a96x/.
Erfan Nasirzadeh Orang, Zack While
Department of Computer Science and Information Systems Youngstown State University