Encoding models that predict brain activation responses to images have traditionally been trained to maximize accuracy when predicting observed brain data. However, their internal representations may not reflect individual human perception. In this work, we investigate whether we can fine-tune brain encoding models to better align their representations with individuals' similarity judgments. Using similarity judgments of 100 images from 8 participants in the Natural Scenes Dataset (NSD), we adapt a generic human similarity judgment model (DreamSim) separately for each participant to obtain personalized perceptual similarity models. These models are then used to obtain subject-specific similarity structure across images in their encoding model training sets. We fine-tune each participant's baseline encoder in two conditions, aligning its representations with either generic or personalized similarity judgments, with the same initialization and objective. We find that baseline brain encoders reflect within-subject perceptual similarity more strongly than measured fMRI responses and preserve intersubject correlations present in measured fMRI, but fail to reflect those of human perceptual similarity judgments. Fine-tuning with generic, population-level perceptual similarity information results in better alignment with human judgments than baseline encoders, but these models still fail to capture intersubject correlations of human judgments. However, fine-tuning with personalized perceptual similarity information results in models that are the most aligned with human perceptual similarity judgments and also reflect intersubject correlations of human judgments. Our framework provides an approach for bootstrapping human similarity judgment data into more individualized, human-aligned brain encoding models of vision.
Figures & tables
Figure 1: Overview of the human image similarity judgment-brain encoding model alignment framework. (a) Each of the eight NSD participants arranged an identical set of 100 images according to their perceived similarity, producing a subject-specific human similarity judgment matrix. (b) A generic model of human perception (here called Population DreamSim) was adapted to each participant’s observed similarity judgments to create perceptual similarity models that reflect that person’s judgments (here called personalized DreamSim). Population and personalized DreamSim models were used to generate similarity values between each pair of images in a given participant’s approximately 9,000 training set images. (c) Each participant’s brain encoder was fine-tuned using both population and personalized DreamSim estimates of image similarity to increase alignment of the brain encoder’s predictions of brain responses with human perception. (d) The resulting representations were evaluated by their alignment with human similarity judgments and their preservation of across-subject similarity structure.
Figure 2: (a) Within subject alignment of an individual’s perceptual similarity judgments for pairs of images in the set of 100 images used in the perceptual task and i) measured fMRI (left, purple) and ii) baseline brain encoder predictions(right, pink) for the same pairs of images. (b) Intersubject similarity matrices derived from human similarity judgments, measured fMRI, and Baseline Brain Encoder predictions for image-pairs in the set of 100 images across the eight NSD subjects. (c) Spearman correlation between the 28 unique subject-pair values of each intersubject similarity matrix across the modalities in panel b). Intersubject baseline encoder structure strongly matches measured fMRI ( ρ=0.82 , p=0.008 ), but neither measured fMRI nor the baseline encoder matches behavioral intersubject similarity judgment structure ( ρ=−0.19 and −0.03 , both p>0.05 ).
Figure 3: (a) Within-subject alignment of an individuals’ image-pair similarity judgments for the 100 images in the perceptual task compared to the representational similarity of: population DreamSim, personalized DreamSim, baseline brain encoder, brain encoder + population DreamSim, and brainencoder + personalized DreamSim, for the same image-pairs. Bars above pairs of boxplots indicate those with significant differences ( p<0.05 ). (b) Human similarity judgment alignment (Spearman ρ ) between each subject-specific model (rows) and each subject’s human similarity judgments (columns); red boxes mark the maximum value for each column. Boxes on the diagonal indicate that person’s model best matched to their observed similarity judgments compared to others’ similarity judgments. (c) Intersubject correlations of representational spaces for human similarity judgments vs intersubject correlations of representational spaces the same modalities listed in panel (b). Each of the 28 points represents a unique pair of individuals, where the x-axis is the correlation of those two individuals’ human similarity judgment RSMs and the y-axis is the correlation of those two individuals’ RSMs for that specific modality.
Figure 4: (a) A Sankey plot showing changes in image-pair similarity agreement between human judgments and the brain encoder before (left column) and after (right column) Personalized DreamSim fine-tuning. Red vertical bars indicate humans and encoders did not agree in their similarity assessments for that image-pair, while green vertical bars indicate the two models did agree. (b) Representative image-pairs illustrating the different group agreement transition types (from baseline brain encoder to brain encoder + Personalized DreamSim), where the color behind each vertical label matches the image-pair group in panel (a). Image-pair similarity scores for a specific human’s judgments (i.e. “S7” indicates subject 7), baseline brain encoder, and brain encoder + Personalized DreamSim are above each image pair.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Subject-specific relative change in brain encoder accuracy plotted against that same individuals’ change in human similarity judgment alignment, with each subject a different color. Baseline Brain Encoders are marked as stars, Brain Encoder + Population DreamSim are marked as circles, and Brain Encoder + Personalized DreamSim are marked as diamonds. Dashed lines with arrows emphasize the flow from Baseline Brain Encoder → Brain Encoder + Population DreamSim → Brain Encoder + Personalized DreamSim.
Figure 6: Region-wise correlation between brain and human representational similarity matrices, for the 19 ROIs common to all eight subjects and ordered early → late along the visual hierarchy. Each region’s RSM is built from that region’s voxels alone and correlated with that same individual’s human similarity judgment RSM; boxes summarize the eight subjects and the overlaid points are those same eight values. Measured fMRI is shown in purple, Baseline Brain Encoders in gray, Brain Encoder + Population DreamSim in orange, and Brain Encoder + Personalized DreamSim in blue. All four are scored on the pooled held-out pairs of 5-fold cross-validation (950 pairs per subject), averaged over ten fold assignments.