cs.CVAug 28, 2026
SaveRelational Knowledge Distillation Brings DNN Representations Close Enough to Humans to Be Aligned Without Supervision
Organizations: Graduate School of Arts and Sciences, The University of Tokyo, 3-8-1 Komaba, 153-8902, Meguro, Tokyo, Japan · Graduate School of Engineering Science, The University of Osaka, 1-3 Machikaneyama-cho, 56-8531, Toyonaka, Osaka, Japan
Abstract
Linking the internal representations of deep neural networks (DNNs) to human mental representations is important for using DNNs as computational models of human vision. Existing DNN representations remain insufficiently similar to human mental representations, which are not directly observable and are therefore commonly measured through large-scale similarity judgments of object images. A natural approach to narrowing this gap is to directly transfer the relational structure of human representations into DNNs, and previous studies have reported improved human-DNN representational similarity. However, whether this improvement holds under stricter evaluation remains untested in two respects: fine-grained alignment at the individual-object level, and generalization to a human embedding derived from a dataset independent of the training data. Here, we employ an unsupervised comparison method, Gromov-Wasserstein optimal transport (GWOT), which estimates human-DNN correspondences from the internal distance structure alone and thereby tests fine-grained alignment. We further assess generalization on a curated test set of concepts non-overlapping with the training data. We show that fine-tuning pre-trained DNNs with Relational Knowledge Distillation (RKD), an established relational transfer method, brings DNNs close enough to humans to be aligned at the individual-object level on this test set. We also show that this improvement is driven by a more human-like global structure, as reflected in the ordering of distances among coarse categories, while the local human-DNN nearest-neighbor overlap rate remains largely unchanged. These findings indicate that relational transfer from humans brings the global structure of pre-trained DNNs close enough to the human structure to enable fine-grained human-DNN alignment without supervision.
Figures & tables
Figure 1: Evaluating relational transfer from humans to DNNs by unsupervised comparison at the fine-grained, individual-object level. (a) Supervised comparison evaluates a human representation (teacher, ) and a DNN representation (student, ) under a pre-specified stimulus pairing ( ): the object correspondence is given, not inferred. (b) Unsupervised comparison assumes no such pairing; the object-to-object correspondence is estimated from each space’s internal distance structure alone. (c) Unsupervised comparison evaluates the estimated correspondence at the fine-grained, individual-object level. Each representation is summarized as a dissimilarity matrix, and the method estimates a transport plan between the human and DNN dissimilarity matrices. Scoring this plan against the ground-truth identity of each individual object yields fine-grained matching. Scoring the same plan instead against the coarse category of each object yields coarse-grained matching (in this example, and , respectively). (d) The evaluation dataset has two levels of granularity: each individual object at the fine-grained level belongs to one coarse-grained category. (e) When relational transfer brings the human and DNN representational structures closer, the transport plan reflects this correspondence as a pronounced identity diagonal, so individual objects become matchable.
Figure 2: Pipeline for relational transfer from humans to DNNs and its evaluation. (a) Training. We performed relational transfer using ImageNet stimuli. We obtained a human psychological embedding (teacher) from human similarity judgments on ImageNet images. We used a pre-trained DNN embedding, taken from the vision encoder’s projection head, as the student. We then fine-tuned this DNN using Relational Knowledge Distillation (RKD; 40 ) to bring the relations among objects in the DNN embedding closer to those in the human psychological embedding. Specifically, we minimized a distance-wise loss ( ) and an angle-wise loss ( ) defined over the relations among objects. (b) Evaluation. To test generalization, we evaluated relational transfer using THINGS stimuli, after removing concepts that overlapped with the ImageNet training data. We then compared the relational structure of the human, pre-trained-DNN, and fine-tuned-DNN embeddings using an unsupervised comparison method, Gromov–Wasserstein optimal transport. To investigate what drove the fine-grained alignment, we further probed the underlying representational change. One analysis addressed the local neighborhood structure, and the other addressed the global structure. Within the global structure, we focused on how the coarse categories were arranged relative to one another.
| Pre-training data | RSA correlation (Spearman ) | GWOT matching accuracy (%) | ||||
| Fine-grained | Coarse-grained | |||||
| Top-1 | Top-5 | Top-1 | Top-5 | |||
| LAION-2B | Before | 0.560 | 2.64 | 34.8 | 60.4 | |
| After | 19.5 | 42.5 | 61.2 | 90.6 | ||
| LAION-400M | Before | 1.28 | 6.49 | 37.3 | 62.8 | |
| After | 16.1 | 37.7 | 58.7 | 89.7 | ||
Table 1: RSA and GWOT results for five CLIP ViT-B/16 variants before and after RKD on the 1,249-object THINGS test set. Values in the RSA column are Spearman correlations ( ) between human and DNN dissimilarity matrices. Values in the GWOT columns are matching accuracies. For each human object, the top-1 and top-5 columns consider, respectively, the single DNN object and the five DNN objects that receive the most transport mass from it. Fine-grained (individual-object-level) matching counts only the identical object as correct, whereas coarse-grained (coarse-category-level) matching counts any object from the same coarse category as correct. Chance-level values are listed for reference.
Figure 3: Unsupervised comparison between the human and DNN representations before and after RKD, for the CLIP ViT-B/16 (LAION-2B) model on the 1,249 THINGS test objects. In (a–e), the 1,249 objects follow the same ordering, in which objects are sorted by coarse category. This ordering is annotated on both axes of (a–e), where the color bars indicate coarse-category membership and the grid lines mark the category boundaries (legend at right). (a–c) Dissimilarity matrices for CLIP before RKD (a), the human embedding (b), and CLIP after RKD (c). (d, e) GWOT transport plans between the human and DNN dissimilarity matrices, before RKD (d) and after RKD (e). Rows are human source objects, and columns are DNN target objects. (f, g) For four randomly selected human source objects (identical in both panels), the five DNN target objects receiving the largest transport mass in : before RKD (f) and after RKD (g). In each row, the source object is shown on the left, and the five targets follow in order of decreasing transport mass.
Figure 4: -nearest-neighbor overlap rate between the human and DNN representations on the 1,249-object THINGS test set, before and after RKD. (a, b) Mean overlap rates for and , respectively. In each panel, each pair of bars gives the mean overlap rate for one CLIP ViT-B/16 variant, named on the horizontal axis; the blue bar (Before) shows the rate before RKD and the red bar (After) shows the rate after RKD. Error bars denote 95% confidence intervals of the mean across the 1,249 query objects ( ). The dotted horizontal line marks the chance overlap rate ( for and for ). (c–f) Five-nearest-neighbor examples for four randomly selected THINGS query objects, the same four objects as in Fig. 3 f, g. In each panel, the query object is shown on the left. The three rows give the five nearest neighbors retrieved under the human embedding (Human), under CLIP ViT-B/16 (LAION-2B) before RKD (Before), and under the same model after RKD (After). Within each row, neighbors are ordered from left to right by increasing dissimilarity from the query (Top-1 to Top-5).
Figure 5: Arrangement among the coarse categories in the human and DNN representations before and after RKD, for the CLIP ViT-B/16 (LAION-2B) model. (a–c) Three-dimensional projections of the 794 THINGS test objects across 20 coarse categories, excluding the heterogeneous others category: (a) CLIP before RKD, (b) human psychological embedding, and (c) CLIP after RKD. The projections were obtained with principal component analysis and placed in a shared coordinate frame by Procrustes alignment. Each object is colored by its coarse category according to the legend at the right of the top row. (d–f) inter-coarse-category dissimilarity matrices: (d) CLIP before RKD, (e) human psychological embedding, and (f) CLIP after RKD. The color bars along the left and bottom edges indicate the coarse category of each row and column, using the same legend as in (a–c). Brackets below the matrices indicate Spearman correlations between the DNN and human inter-coarse-category dissimilarity matrices before and after RKD.
Figure S1 : Data-driven cluster-level GWOT matching accuracy and inter-cluster arrangement, before and after RKD. (a) Sankey diagram relating the 21 human-annotated THINGS categories, including others , to the 21 data-driven clusters. The categories on the left are ordered alphabetically, with others last. Each cluster on the right is then placed at the average position of the categories to which its objects belong, which reduces the crossing of the flows. Flow widths are proportional to the number of objects. (b) Data-driven cluster-level top-1 GWOT matching accuracy, before and after RKD, for the five CLIP ViT-B/16 variants labeled in the panel. The dotted line marks chance ( ). (c–e) Three-dimensional projections of the 1,249 test objects for CLIP ViT-B/16 (LAION-2B) as an example, with each object colored by its data-driven cluster: CLIP before RKD (c), human psychological embedding (d), and CLIP after RKD (e). (f–h) inter-cluster dissimilarity matrices for the same model: CLIP before RKD (f), human psychological embedding (g), and CLIP after RKD (h). The color bars along the left and bottom edges indicate cluster identity, shown in the legend at the right of (c–e). Brackets below the matrices indicate Spearman correlations between the CLIP and human inter-cluster dissimilarity matrices before and after RKD.
| Architecture | Pre-training data | Learning paradigm | RSA correlation (Spearman ) | GWOT matching accuracy (%) | ||||
| Fine-grained | Coarse-grained | |||||||
| Top-1 | Top-5 | Top-1 | Top-5 | |||||
| ResNet-50 | OpenAI | Image–text contrastive (CLIP) | Before | 0.400 | 1.52 | 36.7 | 61.2 | |
| After | 10.3 | 31.7 | 53.8 | 87.3 | ||||
| ResNet-50 | ImageNet-1k | Supervised | Before | 1.92 | 8.17 | 38.8 | 71.8 | |
| After | 4.40 | 15.3 | 45.1 | 78.1 | ||||
Table S1 : RSA and GWOT results for seven additional DNN models before and after RKD on the 1,249-object THINGS test set. Values in the RSA column are Spearman correlations ( ) between human and DNN dissimilarity matrices. Values in the GWOT columns are matching accuracies. For each human object, the top-1 and top-5 columns consider, respectively, the single DNN object and the five DNN objects that receive the most transport mass from it. Fine-grained (individual-object-level) matching counts only the identical object as correct, whereas coarse-grained (coarse-category-level) matching counts any object from the same coarse category as correct. Chance-level values are listed for reference.
| Pre-training data | RSA correlation (Spearman ) | GWOT matching accuracy (%) | ||||
| Fine-grained | Coarse-grained | |||||
| Top-1 | Top-5 | Top-1 | Top-5 | |||
| LAION-2B | Before | 1.33 | 4.00 | 23.7 | 47.7 | |
| After | 65.0 | 92.7 | 84.7 | 99.0 | ||
| LAION-400M | Before | 0.00 | 2.67 | 16.0 | 43.0 | |
| After | 62.3 | 87.3 | 80.7 | 97.7 | ||
Table S2 : RSA and GWOT results for five CLIP ViT-B/16 variants before and after RKD under the within-dataset split of THINGS. Values in the RSA column are Spearman correlations ( ) between human and DNN dissimilarity matrices. Values in the GWOT columns are matching accuracies. For each human object, the top-1 and top-5 columns consider, respectively, the single DNN object and the five DNN objects that receive the most transport mass from it. Fine-grained (individual-object-level) matching counts only the identical object as correct, whereas coarse-grained (coarse-category-level) matching counts any object from the same coarse category as correct. Chance-level values are listed for reference.
| Pre-training data | Nearest-neighbor overlap rate (%, mean 95% CI) | |
| LAION-2B | ||
| LAION-400M | ||
| DataComp-L | ||
| DataComp-XL | ||
| OpenAI | ||
Table S3 : -nearest-neighbor overlap rate between the pre-RKD and post-RKD CLIP ViT-B/16 representations on the 1,249-object THINGS test set. For each object, the value is the percentage of its nearest neighbors that are shared between the pre-RKD and post-RKD embeddings. Values are the mean across all 1,249 objects, shown with the 95% confidence interval (CI).
| Pre-training data | Few-shot accuracy (%, mean SD) | ||||
| THINGS | ImageNet | CIFAR-100 | CIFAR-20 | ||
| LAION-2B | Before | ||||
| After | |||||
| LAION-400M | Before | ||||
| After | |||||
| DataComp-L | Before | ||||
Table S4 : Few-shot classification accuracy for the five CLIP ViT-B/16 variants before and after RKD. Values are accuracy (%), mean SD across five random draws of the five support examples per class, reported for four datasets: THINGS, ImageNet, CIFAR-100, and its coarse-label variant CIFAR-20. The same five draws were used before and after RKD. Chance-level values are listed for reference.
More in this topic
Relational Knowledge Distillation
Relational knowledge distillation transfers a teacher’s structure among samples, classes, tokens, or features to train more capable student models.
See all 27 papers