Comparing internal representations across trained models is important for identifying corresponding layers and selecting components for model reuse. Yet, direct comparisons at inference require evaluating candidate models on known reference inputs and collecting their internal activations. In this paper, we study whether internal representation similarity measured across independently trained models can instead be learned to predict in unseen models from their parameterized computations. We introduce a framework that applies a graph metanetwork for mapping each internal site to an embedding derived from its upstream computations while accounting for hidden-coordinate permutation and internal attention-basis transformations. We analyze the structural properties of our estimators and derive conditional guarantees for layer retrieval and stitching candidate selection. Empirically, we find that the learned predictors support relation estimation and low-regret layer retrieval. They also select useful stitching connections while not being trained on stitching outcomes. This opens up a new way to study representation similarity directly from trained neural networks: representational relations learned across a collection of trained models can generalize to new computations, and that the predicted relations can guide candidate selection before direct measurement or adapter fitting for functional evaluations.
Figures & tables
Figure 1: Learning a connection between trained computations and representation geometry. A representation site is determined by its upstream computation, but symmetry-related parameterizations can induce equivalent computations. Representational correspondence can also extend beyond these parameter symmetries: different computations may produce the same representation geometry under a chosen relation.
Figure 2: Overview of PARIS . We consider a population of trained models. Each site is described by a graph of its upstream computation. PARIS -Ind encodes each site independently and compares reusable embeddings using relation-specific distance heads. PARIS -Joint allows the two site graphs to interact before predicting their relation. Both are trained with internal representation similarity and rank candidate sites in unseen models without collecting new base-model activations. Our metanetworks can guide the selection of sites and be used for adapter fitting.
Figure 3: Same function, different parameterization. The graph only changes by a relabeling.
Figure 5: Predicted correspondence between networks of different depths: between 27 ResNet56 sites (columns) and nine ResNet20 sites (rows), ordered from earlier to later sites. Darker green indicates stronger correspondence.
Figure 6: Spearman correlation ( ↑ ) between predicted and ground-truth representation relations across ResNet and ViT populations on CIFAR-10 and CIFAR-100.
Figure 7: Layer retrieval across model populations, measured by Mean Regret@5 ( ↓ ) for retrieving representation sites from a candidate bank spanning multiple test models.
Figure 8: Stitching candidate selection for ResNet-to-ResNet and ViT-to-ViT connections on CIFAR-10 and CIFAR-100. Performance is measured by the stitching penalty ( ↓ ).
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Representation-score prediction performance of various model population sizes.
Figure 10: Cross graph interaction analysis. PARIS -Ind-MLP replaces the independent estimator’s calibrated metric head with a symmetric MLP while retaining partner-independent site embeddings. The MLP improves prediction scores but does not close the gap to PARIS -Joint.
Figure 11: Complementary regression and ranking supervision. Representation-score prediction ( ↑ ) for PARIS -Ind and PARIS -Joint using combined regression and ranking pair supervision, regression alone, or ranking alone.
Figure 12: Effect of reference-sample size on predicted-reference correlation. Correlation between predicted and activation-derived reference dissimilarities for PARIS -Ind and PARIS -Joint.
Figure 13: Per-representation score correlation between predicted and reference dissimilarities.
Figure 14: Per-representation score MAE between predicted and reference dissimilarities.
Figure 15: Per-representation score RMSE between predicted and reference dissimilarities.
Figure 16: Representation-score prediction under same-family and diverse meta-training. Mean Spearman correlations for DMC, Graph Metanetwork, PARIS -Ind, and PARIS -Joint on held-out ResNet and ViT populations trained on CIFAR-10 and CIFAR-100.
Figure 17: Out-of-distribution generalization across architecture configurations.
Figure 18: Cross-task correlation under partially overlapping label spaces. We partition each parent dataset’s classes into 30% task-A-only, 40% shared, and 30% task-B-only subsets. Each task contains 70% of the original classes. For pairs containing one site from each task-specific base-model population, entries report aggregate correlation between predicted and activation-derived reference dissimilarities.
Figure 19: Score-prediction performance under compensated positive ReLU rescaling.
Figure 19
Figure 22: Representation similarity prediction across modalities using pretrained CLIP models.
Figure 23: Scalability of stitching-candidate selection under same-family and diverse-family meta-training.
Figure 24: Representation-score prediction on pretrained CLIP vision encoders evaluated on Flowers-102.