Organizations: Graduate School of Arts and Sciences, The University of Tokyo, 3-8-1 Komaba, 153-8902, Meguro, Tokyo, Japan · Graduate School of Engineering Science, The University of Osaka, 1-3 Machikaneyama-cho, 56-8531, Toyonaka, Osaka, Japan
Linking the internal representations of deep neural networks (DNNs) to human mental representations is important for using DNNs as computational models of human vision. Existing DNN representations remain insufficiently similar to human mental representations, which are not directly observable and are therefore commonly measured through large-scale similarity judgments of object images. A natural approach to narrowing this gap is to directly transfer the relational structure of human representations into DNNs, and previous studies have reported improved human-DNN representational similarity. However, whether this improvement holds under stricter evaluation remains untested in two respects: fine-grained alignment at the individual-object level, and generalization to a human embedding derived from a dataset independent of the training data. Here, we employ an unsupervised comparison method, Gromov-Wasserstein optimal transport (GWOT), which estimates human-DNN correspondences from the internal distance structure alone and thereby tests fine-grained alignment. We further assess generalization on a curated test set of concepts non-overlapping with the training data. We show that fine-tuning pre-trained DNNs with Relational Knowledge Distillation (RKD), an established relational transfer method, brings DNNs close enough to humans to be aligned at the individual-object level on this test set. We also show that this improvement is driven by a more human-like global structure, as reflected in the ordering of distances among coarse categories, while the local human-DNN nearest-neighbor overlap rate remains largely unchanged. These findings indicate that relational transfer from humans brings the global structure of pre-trained DNNs close enough to the human structure to enable fine-grained human-DNN alignment without supervision.
Figures & tables
Figure 1: Evaluating relational transfer from humans to DNNs by unsupervised comparison at the fine-grained, individual-object level. (a) Supervised comparison evaluates a human representation (teacher, Rl ) and a DNN representation (student, Rm ) under a pre-specified stimulus pairing ( ti↔si ): the object correspondence is given, not inferred. (b) Unsupervised comparison assumes no such pairing; the object-to-object correspondence is estimated from each space’s internal distance structure alone. (c) Unsupervised comparison evaluates the estimated correspondence at the fine-grained, individual-object level. Each representation is summarized as a dissimilarity matrix, and the method estimates a transport plan between the human and DNN dissimilarity matrices. Scoring this plan against the ground-truth identity of each individual object yields fine-grained matching. Scoring the same plan instead against the coarse category of each object yields coarse-grained matching (in this example, 1/6 and 4/6 , respectively). (d) The evaluation dataset has two levels of granularity: each individual object at the fine-grained level belongs to one coarse-grained category. (e) When relational transfer brings the human and DNN representational structures closer, the transport plan reflects this correspondence as a pronounced identity diagonal, so individual objects become matchable.
Figure 2: Pipeline for relational transfer from humans to DNNs and its evaluation. (a) Training. We performed relational transfer using ImageNet stimuli. We obtained a human psychological embedding (teacher) from human similarity judgments on ImageNet images. We used a pre-trained DNN embedding, taken from the vision encoder’s projection head, as the student. We then fine-tuned this DNN using Relational Knowledge Distillation (RKD; 40 ) to bring the relations among objects in the DNN embedding closer to those in the human psychological embedding. Specifically, we minimized a distance-wise loss ( LD ) and an angle-wise loss ( LA ) defined over the relations among objects. (b) Evaluation. To test generalization, we evaluated relational transfer using THINGS stimuli, after removing concepts that overlapped with the ImageNet training data. We then compared the relational structure of the human, pre-trained-DNN, and fine-tuned-DNN embeddings using an unsupervised comparison method, Gromov–Wasserstein optimal transport. To investigate what drove the fine-grained alignment, we further probed the underlying representational change. One analysis addressed the local neighborhood structure, and the other addressed the global structure. Within the global structure, we focused on how the coarse categories were arranged relative to one another.
Pre-training data
RSA correlation (Spearman ρ )
GWOT matching accuracy (%)
Fine-grained
Coarse-grained
Top-1
Top-5
Top-1
Top-5
LAION-2B
Before
0.407
0.560
2.64
34.8
60.4
After
0.568
19.5
42.5
61.2
90.6
LAION-400M
Before
0.423
1.28
6.49
37.3
62.8
After
0.548
16.1
37.7
58.7
89.7
Table 1: RSA and GWOT results for five CLIP ViT-B/16 variants before and after RKD on the 1,249-object THINGS test set. Values in the RSA column are Spearman correlations ( ρ ) between human and DNN dissimilarity matrices. Values in the GWOT columns are matching accuracies. For each human object, the top-1 and top-5 columns consider, respectively, the single DNN object and the five DNN objects that receive the most transport mass from it. Fine-grained (individual-object-level) matching counts only the identical object as correct, whereas coarse-grained (coarse-category-level) matching counts any object from the same coarse category as correct. Chance-level values are listed for reference.
Figure 3: Unsupervised comparison between the human and DNN representations before and after RKD, for the CLIP ViT-B/16 (LAION-2B) model on the 1,249 THINGS test objects. In (a–e), the 1,249 objects follow the same ordering, in which objects are sorted by coarse category. This ordering is annotated on both axes of (a–e), where the color bars indicate coarse-category membership and the grid lines mark the category boundaries (legend at right). (a–c) Dissimilarity matrices for CLIP before RKD (a), the human embedding (b), and CLIP after RKD (c). (d, e) GWOT transport plans Γ between the human and DNN dissimilarity matrices, before RKD (d) and after RKD (e). Rows are human source objects, and columns are DNN target objects. (f, g) For four randomly selected human source objects (identical in both panels), the five DNN target objects receiving the largest transport mass in Γ : before RKD (f) and after RKD (g). In each row, the source object is shown on the left, and the five targets follow in order of decreasing transport mass.
Figure 4: k -nearest-neighbor overlap rate between the human and DNN representations on the 1,249-object THINGS test set, before and after RKD. (a, b) Mean overlap rates for k=5 and k=10 , respectively. In each panel, each pair of bars gives the mean overlap rate for one CLIP ViT-B/16 variant, named on the horizontal axis; the blue bar (Before) shows the rate before RKD and the red bar (After) shows the rate after RKD. Error bars denote 95% confidence intervals of the mean across the 1,249 query objects ( ±1.96×SEM ). The dotted horizontal line marks the chance overlap rate ( 0.4% for k=5 and 0.8% for k=10 ). (c–f) Five-nearest-neighbor examples for four randomly selected THINGS query objects, the same four objects as in Fig. 3 f, g. In each panel, the query object is shown on the left. The three rows give the five nearest neighbors retrieved under the human embedding (Human), under CLIP ViT-B/16 (LAION-2B) before RKD (Before), and under the same model after RKD (After). Within each row, neighbors are ordered from left to right by increasing dissimilarity from the query (Top-1 to Top-5).
Figure 5: Arrangement among the coarse categories in the human and DNN representations before and after RKD, for the CLIP ViT-B/16 (LAION-2B) model. (a–c) Three-dimensional projections of the 794 THINGS test objects across 20 coarse categories, excluding the heterogeneous others category: (a) CLIP before RKD, (b) human psychological embedding, and (c) CLIP after RKD. The projections were obtained with principal component analysis and placed in a shared coordinate frame by Procrustes alignment. Each object is colored by its coarse category according to the legend at the right of the top row. (d–f) 20×20 inter-coarse-category dissimilarity matrices: (d) CLIP before RKD, (e) human psychological embedding, and (f) CLIP after RKD. The color bars along the left and bottom edges indicate the coarse category of each row and column, using the same legend as in (a–c). Brackets below the matrices indicate Spearman correlations between the DNN and human inter-coarse-category dissimilarity matrices before and after RKD.
Figure S1 : Data-driven cluster-level GWOT matching accuracy and inter-cluster arrangement, before and after RKD. (a) Sankey diagram relating the 21 human-annotated THINGS categories, including others , to the 21 data-driven clusters. The categories on the left are ordered alphabetically, with others last. Each cluster on the right is then placed at the average position of the categories to which its objects belong, which reduces the crossing of the flows. Flow widths are proportional to the number of objects. (b) Data-driven cluster-level top-1 GWOT matching accuracy, before and after RKD, for the five CLIP ViT-B/16 variants labeled in the panel. The dotted line marks chance ( 5.8% ). (c–e) Three-dimensional projections of the 1,249 test objects for CLIP ViT-B/16 (LAION-2B) as an example, with each object colored by its data-driven cluster: CLIP before RKD (c), human psychological embedding (d), and CLIP after RKD (e). (f–h) 21×21 inter-cluster dissimilarity matrices for the same model: CLIP before RKD (f), human psychological embedding (g), and CLIP after RKD (h). The color bars along the left and bottom edges indicate cluster identity, shown in the legend at the right of (c–e). Brackets below the matrices indicate Spearman correlations between the CLIP and human inter-cluster dissimilarity matrices before and after RKD.
Architecture
Pre-training data
Learning paradigm
RSA correlation (Spearman ρ )
GWOT matching accuracy (%)
Fine-grained
Coarse-grained
Top-1
Top-5
Top-1
Top-5
ResNet-50
OpenAI
Image–text contrastive (CLIP)
Before
0.500
0.400
1.52
36.7
61.2
After
0.501
10.3
31.7
53.8
87.3
ResNet-50
ImageNet-1k
Supervised
Before
0.351
1.92
8.17
38.8
71.8
After
0.458
4.40
15.3
45.1
78.1
Table S1 : RSA and GWOT results for seven additional DNN models before and after RKD on the 1,249-object THINGS test set. Values in the RSA column are Spearman correlations ( ρ ) between human and DNN dissimilarity matrices. Values in the GWOT columns are matching accuracies. For each human object, the top-1 and top-5 columns consider, respectively, the single DNN object and the five DNN objects that receive the most transport mass from it. Fine-grained (individual-object-level) matching counts only the identical object as correct, whereas coarse-grained (coarse-category-level) matching counts any object from the same coarse category as correct. Chance-level values are listed for reference.
Pre-training data
RSA correlation (Spearman ρ )
GWOT matching accuracy (%)
Fine-grained
Coarse-grained
Top-1
Top-5
Top-1
Top-5
LAION-2B
Before
0.451
1.33
4.00
23.7
47.7
After
0.822
65.0
92.7
84.7
99.0
LAION-400M
Before
0.460
0.00
2.67
16.0
43.0
After
0.803
62.3
87.3
80.7
97.7
Table S2 : RSA and GWOT results for five CLIP ViT-B/16 variants before and after RKD under the within-dataset split of THINGS. Values in the RSA column are Spearman correlations ( ρ ) between human and DNN dissimilarity matrices. Values in the GWOT columns are matching accuracies. For each human object, the top-1 and top-5 columns consider, respectively, the single DNN object and the five DNN objects that receive the most transport mass from it. Fine-grained (individual-object-level) matching counts only the identical object as correct, whereas coarse-grained (coarse-category-level) matching counts any object from the same coarse category as correct. Chance-level values are listed for reference.
Pre-training data
Nearest-neighbor overlap rate (%, mean ± 95% CI)
k=5
k=10
LAION-2B
41.9±1.3
42.8±1.0
LAION-400M
41.7±1.3
42.9±1.0
DataComp-L
44.6±1.3
45.5±1.0
DataComp-XL
40.8±1.2
41.0±1.0
OpenAI
33.8±1.2
35.0±1.0
Table S3 : k -nearest-neighbor overlap rate between the pre-RKD and post-RKD CLIP ViT-B/16 representations on the 1,249-object THINGS test set. For each object, the value is the percentage of its k nearest neighbors that are shared between the pre-RKD and post-RKD embeddings. Values are the mean across all 1,249 objects, shown with the 95% confidence interval (CI).
Pre-training data
Few-shot accuracy (%, mean ± SD)
THINGS
ImageNet
CIFAR-100
CIFAR-20
LAION-2B
Before
87.8±0.3
61.8±0.2
67.4±0.7
65.1±1.3
After
84.9±0.3
53.5±0.3
55.1±0.7
58.1±1.4
LAION-400M
Before
86.7±0.3
58.7±0.5
61.1±0.3
60.0±1.2
After
82.2±0.1
49.9±0.1
50.1±0.5
54.3±0.6
DataComp-L
Before
85.6±0.4
55.6±0.3
66.4±0.5
65.6±0.9
Table S4 : Few-shot classification accuracy for the five CLIP ViT-B/16 variants before and after RKD. Values are accuracy (%), mean ± SD across five random draws of the five support examples per class, reported for four datasets: THINGS, ImageNet, CIFAR-100, and its coarse-label variant CIFAR-20. The same five draws were used before and after RKD. Chance-level values are listed for reference.
Knowledge distillation transfers representational knowledge from a large teacher network to a compact student. Existing relational and contrastive distillation methods treat all samples uniformly, spending equal gradient on well-matched and poorly-matched regions of the representation space alike. We propose Representational Difference Distillation (RDD), which diagnoses where teacher and student representations structurally disagree and concentrates the distillation signal on those regions. RDD uses two asymmetric affinity matrices that capture opposing directions of misalignment: one identifies samples the student incorrectly considers similar, the other identifies teacher relationships the student has yet to learn. These affinities determine which sample pairs the distillation loss aligns and how strongly it corrects each misaligned sample. On CIFAR-100 with resnet32x4 to resnet8x4, RDD reaches 75.58% test accuracy in our runs, on par with CRD (75.45%) and above KD and RKD. We additionally explore curriculum-based and sampling-based variants, finding that weighting the loss is more effective than filtering the data.
Abdulkarim Mugisha, Divin Irakiza
Department of Computing + Mathematical Sciences, California Institute of Technology, Pasadena, US.
Knowledge distillation transfers knowledge from large teacher models to more compact student networks. The standard approach minimizes the Kullback-Leibler (KL) divergence between the probabilistic outputs of the teacher and student, aligning predictions but neglecting the structural relationships encoded within the teacher's internal representations. Recent advances have adopted contrastive learning objectives to address this limitation; however, such instance-discrimination-based methods induce a "class collision problem", in which semantically related samples are inappropriately pushed apart despite belonging to similar classes. To overcome this, we propose Relational Representation Distillation (RRD) that preserves the relative relationships among instances rather than enforcing absolute separation. Our method introduces separate temperature parameters for teacher and student distributions, with a sharper teacher (low τt) emphasizing primary relationships and a softer student (high τs) maintaining secondary similarities. This dual-temperature formulation creates an implicit information bottleneck that preserves fine-grained relational structure while avoiding the over-separation characteristic of contrastive losses. We establish theoretical connections showing that InfoNCE emerges as a limiting case of our objective when τt→0, and empirically demonstrate that this relaxed formulation yields superior relational alignment and generalization across classification and detection tasks.
Knowledge distillation is usually framed as a choice of what to match in the teacher - its logits, hidden features, or sample relations - which presupposes that the teacher's representation has absolute coordinates to match. It does not: a pretrained representation is identifiable only up to an orthogonal-and-isotropic-scaling equivalence class, so a student should learn the teacher's equivalence class, not its features. The organizing fact is that capability is the teacher's output function, a class invariant that factors through the quotient by the class action, so an objective recovers capability exactly when it is defined there. This makes absolute feature matching ill-posed, and admissible supervision a matter of targeting class invariants (Gram structure, CKA, principal subspaces) or aligning coordinates first, unifying feature matching, relational distillation, alignment, and grafting in one geometric account. We validate our framework on Qwen2.5 and Llama-3.1. A restoration study recovers a corrupted model's representation (CKA ~ 0.99) but not its capability, and an ablation isolates the cause: output-function (logit) matching drives capability, while matching hidden representations aligns geometry without restoring function. Recovery is confined to the corpus-covered region, and a graft study confirms that boundary overlap predicts transplant success but is necessary, not sufficient.
Sang Il Han
Korea University of Technology and Education · Biomonster Co.