Manifold Approximation leads to Robust Kernel Alignment

Mohammad Tariqul Islam; Du Liu; Deblina Sarkar

cs.LGOct 27, 2025

Manifold Approximation leads to Robust Kernel Alignment

Authors: Mohammad Tariqul Islam, Du Liu, Deblina Sarkar

Organizations: MIT

Abstract

Centered kernel alignment (CKA) is a popular metric for comparing representations, determining equivalence of networks, and conducting neuroscience research. However, CKA does not account for the underlying manifold and relies on numerous heuristics that cause it to behave differently at different scales of data. In this work, we propose Manifold-approximated Kernel Alignment (MKA), which incorporates manifold geometry (i.e., local neighborhood) into the alignment task. We derive a theoretical framework for MKA. Our construction removes representation-dependent degree weighting, yields a closed-form alignment score, and gives each row a probabilistic interpretation as a local Markov transition distribution. We further establish kernel- and score-level stability with respect to the neighborhood size. We perform empirical evaluations on synthetic datasets and real-world examples to characterize and compare MKA to its contemporaries. In controlled synthetic experiments, MKA is consistently less sensitive to hyperparameters and captures the selected topology-preserving relationships. On the large-scale benchmark, MKA achieves its strongest aggregate performance in vision, is competitive on graphs, and is weaker than CKA in several NLP evaluations. Our findings suggest that manifold-aware kernel alignment provides a robust foundation for measuring representations and is potentially useful when local neighborhood geometry is important, with possible applications in representation learning.

Figures & tables

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

\displaystyle d_{i(1)}\leq d_{i(2)}\leq\cdots\leq d_{i(k)},

Appendix

Algorithm 1 Computation of the row-wise bandwidth

\sigma_{i}

using binary search

\mathbf{K}\leftarrow\textsc{ConstructMKA-Kernel}(\mathbf{X},k)

Appendix

Algorithm 2 Manifold-approximated Kernel Alignment (MKA)

\mathbf{K}\leftarrow\textsc{KNNKernel}(\mathbf{X},k)

Appendix

Algorithm 3

k

-nearest-neighbor Centered Kernel Alignment (

k

CKA)

	Method	Distance Matrix	kNN	Kernel/Weight	Score Computation	Residual
$1,000$	CKA	0.0120	–	0.0214	0.0036	0.0007
	kCKA	–	0.0592	0.0081	0.0037	0.0007
	MKA	–	0.0412	0.3950	0.0016	0.0008
$10,000$	CKA	0.6522	–	1.9994	0.5585	0.0116
	kCKA	–	0.6778	0.6142	0.5657	0.0069
	MKA	–	0.6275	3.9232	0.2890	0.0070

Appendix

Table 1: Timing decomposition (in seconds) for CKA, kCKA (

k=100

), and MKA (

k=100

). ‘Residual’ denotes the difference between total time and sum of the individual components reported in the table.

Figure 12: Alignment for the “clusters” data. (a) Point clouds used in the clusters experiment. (b) Alignment using various methods, along with Kendall’s rank correlation (higher is better). (c-e) Alignment by varying nearest neighbors, $k$ , in (c) IMD, (d) kCKA, and (e) MKA. MKA shows the greatest robustness to parameters.

Test	Accuracy Correlation
Dataset	IN100
Architecture	RNet18	RNet34	RNet101	VGG11	VGG19	ViT B32	ViT L32
CKA	0.33	-0.09	0.08	0.02	-0.22	-0.23	0.02
CKA ( $\delta=0.45$ )	0.29	-0.06	0.01	0.00	-0.10	-0.28	0.09
CKA ( $\delta=0.2$ )	0.02	0.19	-0.08	-0.14	-0.23	-0.15	0.02
kCKA ( $k=100$ )	0.00	-0.01	0.15	0.09	0.58	-0.16	0.10

Appendix

Table 2: Results of Test 1 (Correlation to Accuracy Difference) for the vision domain on ImageNet-100

Test	JSD Correlation							Disagreement Correlation
Dataset	IN100							IN100
Architecture	RNet18	RNet34	RNet101	VGG11	VGG19	ViT B32	ViT L32	RNet18	RNet34	RNet101	VGG11	VGG19	ViT B32	ViT L32
CKA	0.26	0.02	0.30	-0.09	0.04	0.02	-0.12	0.36	0.00	0.29	-0.01	-0.25	0.00	-0.05
CKA ( $\delta=0.45$ )	0.20	-0.13	0.30	-0.08	0.19	0.03	-0.21	0.33	-0.14	0.35	-0.00	-0.21	0.04	-0.16
CKA ( $\delta=0.2$ )	0.07	-0.03	-0.26	0.02	-0.13	0.32	-0.61	0.35	-0.18	0.08	0.39	0.13	0.35	-0.48
kCKA ( $k=100$ )	-0.34	0.35	0.43	-0.02	0.24	0.32	-0.32	-0.56	0.45	0.18	-0.18	-0.17	0.22	-0.17

Appendix

Table 3: Results of Test 2 (Correlation to Output Difference) for the vision domain on ImageNet-100

Evaluation	AUPRC							Conformity Rate
Dataset	IN100							IN100
Architecture	RNet18	RNet34	RNet101	VGG11	VGG19	ViT B32	ViT L32	RNet18	RNet34	RNet101	VGG11	VGG19	ViT B32	ViT L32
CKA	1.00	1.00	1.00	1.00	0.77	0.52	0.81	1.00	1.00	1.00	1.00	0.77	0.65	0.83
CKA ( $\delta=0.45$ )	1.00	1.00	1.00	1.00	0.77	0.78	0.86	1.00	1.00	1.00	1.00	0.75	0.81	0.87
CKA ( $\delta=0.2$ )	0.65	0.70	0.82	0.80	0.76	0.77	0.97	0.71	0.74	0.87	0.84	0.73	0.75	0.99
kCKA ( $k=100$ )	1.00	1.00	1.00	0.88	0.73	0.57	0.90	1.00	1.00	1.00	0.95	0.70	0.73	0.97

Appendix

Table 4: Results of Test 3 (Label Randomization) for the vision domain on ImageNet-100

Evaluation	AUPRC							Conformity Rate
Dataset	IN100							IN100
Architecture	RNet18	RNet34	RNet101	VGG11	VGG19	ViT B32	ViT L32	RNet18	RNet34	RNet101	VGG11	VGG19	ViT B32	ViT L32
CKA	1.00	1.00	1.00	1.00	1.00	1.00	0.86	1.00	1.00	1.00	1.00	1.00	1.00	0.96
CKA ( $\delta=0.45$ )	1.00	1.00	1.00	1.00	1.00	1.00	0.90	1.00	1.00	1.00	1.00	1.00	1.00	0.97
CKA ( $\delta=0.2$ )	0.77	0.62	0.66	0.80	0.74	1.00	0.79	0.94	0.85	0.88	0.93	0.94	1.00	0.93
kCKA ( $k=100$ )	0.94	0.92	0.98	1.00	1.00	1.00	0.91	0.99	0.98	1.00	1.00	1.00	1.00	0.97

Appendix

Table 5: Results of Test 4 (Shortcut Affinity) for the vision domain on ImageNet-100

Evaluation	Spearman							Conformity Rate
Dataset	IN100							IN100
Architecture	RNet18	RNet34	RNet101	VGG11	VGG19	ViT B32	ViT L32	RNet18	RNet34	RNet101	VGG11	VGG19	ViT B32	ViT L32
CKA	0.97	0.79	0.97	0.88	0.93	1.00	1.00	0.95	0.98	0.99	0.90	0.92	1.00	1.00
CKA ( $\delta=0.45$ )	0.90	0.73	0.91	0.97	0.94	1.00	1.00	0.94	0.95	0.97	0.95	0.93	1.00	1.00
CKA ( $\delta=0.2$ )	0.97	0.89	0.84	1.00	0.84	0.95	1.00	0.96	0.96	0.96	1.00	0.94	0.93	1.00
kCKA ( $k=100$ )	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00

Appendix

Table 6: Results of Test 6 (Layer Monotonicity) for the vision domain on ImageNet-100

Representation	CLS Token
Test	Accuracy Correlation
Architecture	BERT	ALBERT
CKA	0.11	0.26
CKA ( $\delta=0.45$ )	0.08	0.09
CKA ( $\delta=0.2$ )	-0.17	-0.24
kCKA ( $k=100)$	0.03	-0.24

Appendix

Table 7: Results of Test 1 (Correlation to Accuracy Difference) on MNLI

Representation	CLS Token				Mean-pooled Token
Test	JSD Correlation		Disagreement Correlation		JSD Correlation		Disagreement Correlation
Architecture	BERT	ALBERT	BERT	ALBERT	BERT	ALBERT	BERT	ALBERT
CKA	0.36	0.29	0.15	0.57	0.51	0.06	0.38	0.18
CKA ( $\delta=0.45$ )	0.31	0.05	0.50	0.22	0.53	0.02	0.49	0.16
CKA ( $\delta=0.2$ )	-0.25	-0.11	0.27	-0.04	0.13	-0.06	-0.25	0.05
kCKA ( $k=100$ )	0.27	0.03	0.33	-0.00	0.45	0.05	0.41	0.10

Appendix

Table 8: Results of Test 2 (Correlation to Output Difference) on MNLI

Representation	CLS Token				Mean-pooled Token
Evaluation	AUPRC		Conformity Rate		AUPRC		Conformity Rate
Architecture	BERT	ALBERT	BERT	ALBERT	BERT	ALBERT	BERT	ALBERT
CKA	0.75	0.80	0.89	0.93	0.66	0.45	0.86	0.71
CKA ( $\delta=0.45$ )	0.74	0.79	0.89	0.93	0.58	0.45	0.81	0.68
CKA ( $\delta=0.2$ )	0.56	0.54	0.82	0.75	0.45	0.31	0.66	0.53
kCKA ( $k=100$ )	0.69	0.46	0.83	0.68	0.50	0.43	0.73	0.59

Appendix

Table 9: Results of Test 3 (Label Randomization) on MNLI

Representation	CLS Token				Mean-pooled Token
Evaluation	AUPRC		Conformity Rate		AUPRC		Conformity Rate
Architecture	BERT	ALBERT	BERT	ALBERT	BERT	ALBERT	BERT	ALBERT
CKA	0.59	0.63	0.88	0.67	0.55	0.56	0.85	0.64
CKA ( $\delta=0.45$ )	0.58	0.67	0.87	0.81	0.55	0.56	0.84	0.66
CKA ( $\delta=0.2$ )	0.50	0.43	0.79	0.58	0.55	0.32	0.74	0.52
kCKA ( $k=100$ )	0.57	0.61	0.83	0.65	0.56	0.58	0.82	0.66

Appendix

Table 10: Results of Test 4 (Shortcut Affinity) on MNLI

Representation	CLS Token				Mean-pooled Token
Evaluation	AUPRC		Conformity Rate		AUPRC		Conformity Rate
Architecture	BERT	ALBERT	BERT	ALBERT	BERT	ALBERT	BERT	ALBERT
CKA	0.44	0.85	0.84	0.84	0.36	0.74	0.79	0.78
CKA ( $\delta=0.45$ )	0.45	0.85	0.80	0.86	0.34	0.72	0.77	0.78
CKA ( $\delta=0.2$ )	0.43	0.69	0.71	0.74	0.35	0.42	0.56	0.55
kCKA ( $k=100$ )	0.37	0.76	0.71	0.78	0.35	0.63	0.73	0.73

Appendix

Table 11: Results of Test 5 (Augmentation) on MNLI

Representation	CLS Token				Mean-pooled Token
Evaluation	Spearman		Conformity Rate		Spearman		Conformity Rate
Architecture	BERT	ALBERT	BERT	ALBERT	BERT	ALBERT	BERT	ALBERT
CKA	0.99	0.98	0.98	0.97	1.00	0.99	0.99	0.99
CKA ( $\delta=0.45$ )	0.99	0.98	0.98	0.97	1.00	1.00	0.99	1.00
CKA ( $\delta=0.2$ )	0.92	0.95	0.90	0.93	0.95	1.00	0.96	1.00
kCKA ( $k=100$ )	0.91	0.90	0.90	0.89	1.00	1.00	1.00	1.00

Appendix

Table 12: Results of Test 6 (Layer Monotonicity) on MNLI

Evaluation	Spearman
Dataset	Cora				Flickr			OGBN-Arxiv
Architecture	GCN	SAGE	GAT	PGNN	GCN	SAGE	GAT	GCN	SAGE	GAT
CKA	0.04	-0.15	0.03	-0.01	0.50	0.41	-0.14	-0.07	0.14	-0.03
CKA ( $\delta=0.45$ )	0.09	-0.22	0.00	-0.06	0.52	0.17	-0.16	-0.12	0.18	-0.13
CKA ( $\delta=0.2$ )	0.01	-0.29	0.07	0.03	0.43	0.14	0.11	-0.22	0.17	-0.28
kCKA ( $k=100$ )	0.07	-0.25	-0.01	0.11	0.42	-0.22	-0.28	-0.06	0.23	-0.16

Appendix

Table 13: Results of Test 1 (Correlation to Accuracy Difference) for the graph domain

Type	Grounding by Prediction
Test	JSD Correlation										Disagreement Correlation
Evaluation	Spearman										Spearman
Dataset	Cora				Flickr			OGBN-Arxiv			Cora				Flickr			OGBN-Arxiv
Architecture	GCN	SAGE	GAT	PGNN	GCN	SAGE	GAT	GCN	SAGE	GAT	GCN	SAGE	GAT	PGNN	GCN	SAGE	GAT	GCN	SAGE	GAT
CKA	0.73	0.12	0.59	-0.23	0.54	0.35	0.12	0.18	0.19	0.38	0.65	-0.05	0.44	0.01	0.26	0.32	0.02	0.13	0.14	0.21
CKA ( $\delta=0.45$ )	0.73	0.30	0.52	-0.08	0.56	0.18	0.15	0.22	0.22	0.25	0.64	-0.06	0.34	0.14	0.29	0.15	0.01	0.21	0.18	0.09

Appendix

Table 14: Results of Test 2 (Correlation to Output Difference) for the graph domain

Evaluation	AUPRC										Conformity Rate
Dataset	Cora				Flickr			OGBN-Arxiv			Cora				Flickr			OGBN-Arxiv
Architecture	GCN	SAGE	GAT	PGNN	GCN	SAGE	GAT	GCN	SAGE	GAT	GCN	SAGE	GAT	PGNN	GCN	SAGE	GAT	GCN	SAGE	GAT
CKA	0.48	0.56	0.43	0.25	0.88	0.42	0.31	1.00	1.00	1.00	0.72	0.78	0.54	0.64	0.96	0.50	0.61	1.00	1.00	1.00
CKA ( $\delta=0.45$ )	0.50	0.50	0.52	0.26	0.92	0.45	0.38	1.00	0.96	1.00	0.75	0.77	0.76	0.65	0.98	0.66	0.69	1.00	0.98	1.00
CKA ( $\delta=0.2$ )	0.74	0.72	0.73	0.21	0.78	0.42	0.23	1.00	0.90	0.99	0.93	0.88	0.86	0.52	0.93	0.50	0.57	1.00	0.98	1.00
kCKA ( $k=100$ )	0.43	0.42	0.42	0.24	0.74	0.43	0.37	1.00	0.74	0.92	0.59	0.52	0.52	0.58	0.88	0.61	0.66	1.00	0.88	0.98

Appendix

Table 15: Results of Test 3 (Label Randomization) for the graph domain

Evaluation	AUPRC										Conformity Rate
Dataset	Cora				Flickr			OGBN-Arxiv			Cora				Flickr			OGBN-Arxiv
Architecture	GCN	SAGE	GAT	PGNN	GCN	SAGE	GAT	GCN	SAGE	GAT	GCN	SAGE	GAT	PGNN	GCN	SAGE	GAT	GCN	SAGE	GAT
CKA	0.67	0.82	0.80	0.42	0.41	1.00	0.37	1.00	1.00	1.00	0.85	0.92	0.95	0.62	0.78	1.00	0.75	1.00	1.00	1.00
CKA ( $\delta=0.45$ )	0.73	0.81	0.82	0.42	0.75	1.00	0.53	1.00	1.00	1.00	0.91	0.92	0.96	0.55	0.94	1.00	0.80	1.00	1.00	1.00
CKA ( $\delta=0.2$ )	0.79	0.85	0.78	0.42	0.97	1.00	0.29	1.00	1.00	1.00	0.95	0.96	0.93	0.55	0.99	1.00	0.60	1.00	1.00	1.00
kCKA ( $k=100$ )	0.77	0.86	0.84	0.42	1.00	1.00	0.55	1.00	1.00	1.00	0.94	0.95	0.97	0.56	1.00	1.00	0.84	1.00	1.00	1.00

Appendix

Table 16: Results of Test 4 (Shortcut Affinity) for the graph domain

Evaluation	AUPRC								Conformity Rate
Dataset	Cora			Flickr			OGBN-Arxiv		Cora			Flickr			OGBN-Arxiv
Architecture	GCN	SAGE	GAT	GCN	SAGE	GAT	GCN	SAGE	GCN	SAGE	GAT	GCN	SAGE	GAT	GCN	SAGE
CKA	0.13	0.76	0.91	0.64	0.80	0.57	1.00	1.00	0.37	0.92	0.97	0.85	0.94	0.77	1.00	1.00
CKA ( $\delta=0.45$ )	0.13	0.80	0.98	0.69	0.94	0.58	1.00	1.00	0.38	0.93	0.99	0.86	0.99	0.80	1.00	1.00
CKA ( $\delta=0.2$ )	0.13	0.93	0.93	0.75	0.42	0.55	1.00	0.99	0.38	0.99	0.97	0.90	0.52	0.84	1.00	1.00
kCKA ( $k=100$ )	0.13	0.76	0.96	0.74	0.82	0.60	1.00	0.96	0.37	0.92	0.99	0.89	0.94	0.83	1.00	0.99

Appendix

Table 17: Results of Test 5 (Augmentation) for the graph domain

Evaluation	Conformity Rate										Spearman
Dataset	Cora				Flickr			OGBN-Arxiv			Cora				Flickr			OGBN-Arxiv
Architecture	GCN	SAGE	GAT	PGNN	GCN	SAGE	GAT	GCN	SAGE	GAT	GCN	SAGE	GAT	PGNN	GCN	SAGE	GAT	GCN	SAGE	GAT
CKA	0.99	1.00	1.00	0.99	0.81	0.99	0.63	0.88	0.70	0.90	0.99	1.00	1.00	0.99	0.73	0.99	0.48	0.86	0.58	0.91
CKA ( $\delta=0.45$ )	1.00	1.00	1.00	1.00	0.86	0.99	0.80	0.84	0.80	0.93	1.00	1.00	1.00	1.00	0.79	0.99	0.66	0.75	0.73	0.93
CKA ( $\delta=0.2$ )	1.00	1.00	1.00	1.00	0.97	0.99	0.85	0.91	0.96	1.00	1.00	1.00	1.00	1.00	0.98	0.99	0.64	0.84	0.91	1.00
kCKA ( $k=100$ )	1.00	1.00	1.00	0.91	0.95	0.95	0.96	0.88	0.98	1.00	1.00	1.00	1.00	0.91	0.90	0.97	0.95	0.85	0.99	1.00

Appendix

Table 18: Results of Test 6 (Layer Monotonicity) for the graph domain

Kernel Methods

Kernel methods use similarity functions and reproducing kernel spaces to model nonlinear relationships, quantify uncertainty, and compare data.

See all 233 papers

From Layers to Networks: Comparing Neural Representations via Diffusion Geometry
May 15, 2026Atharva Khandait, Jan E. GerkenRepresentational Similarity Analysis Neural Representation Geometry
Department of Mathematical Sciences, Chalmers University of Technology and the University of Gothenburg, SE-412 96 Gothenburg, Sweden.
TopoAlign: Topology-Aware Visual Representation Alignment
May 25, 2026Xinyuan Yan, Rita Sevastjanova, Mennatallah El-Assady +1Data Visualization Visual Analytics
University of Utah · ETH Zürich
Beyond Activation Alignment: The Geometry of Neural Sensitivity
May 4, 2026Amirhossein Yavari, Farnaz Zamani EsfahlaniNeural Representation Geometry Neural Decoding
Stephenson School of Biomedical Engineering, University of Oklahoma · Data Science and Analytics Institute, University of Oklahoma

May 15, 2026J/K move · Enter open · S save

From Layers to Networks: Comparing Neural Representations via Diffusion Geometry

Atharva Khandait, Jan E. Gerken

Department of Mathematical Sciences, Chalmers University of Technology and the University of Gothenburg, SE-412 96 Gothenburg, Sweden.

Representational Similarity Analysis Neural Representation Geometry Kernel Methods

Open paper AskPDF arXiv

Diffusion geometry is a manifold learning framework that uses random walks defined by Markov transition matrices to characterize the geometry of a dataset at multiple scales. We use diffusion geometry for neural representations, incorporating tools from multi-view learning into this field for the first time. Our key technical observation is that a broad class of similarity measures based on representational similarity matrices (RSMs) admits a closed-form equivalent formulation in terms of row-stochastic Markov matrices, opening the door to manipulations from diffusion geometry. As a first application, we develop multi-scale variants of Centered Kernel Alignment and Distance Correlation, which utilise the

t^{th}

power of the underlying transition matrix to probe the data geometry at adjustable diffusion scales. Going further, we introduce variants of these measures which fuse the Markov matrices of several layers via alternating diffusion into a single operator that captures the network's joint sample geometry, allowing similarity to be computed across multiple layers and shifting the comparison from layer-to-layer to network-to-network. We perform extensive numerical experiments, evaluating our measures on the Representational Similarity (ReSi) benchmark comprising 14 architectures trained on 7 datasets across three different domains. Our methods achieve SoTA results in accuracy and output correlation for both language and vision tasks across different models. We furthermore show SoTA performance on an additional benchmark evaluating on out-of-distribution data.

Abstract

Kernel Methods

Explore similar work

From Layers to Networks: Comparing Neural Representations via Diffusion Geometry

TopoAlign: Topology-Aware Visual Representation Alignment

Beyond Activation Alignment: The Geometry of Neural Sensitivity