Distilling Graph Geometry: Knowledge Gap from GNNs to MLPs

Zhewei Chen; Hao Zhu; Jiaojiao Jiang; Ahad N. Zehmakan

cs.LGOct 7, 2026

Distilling Graph Geometry: Knowledge Gap from GNNs to MLPs

Authors: Zhewei Chen, Hao Zhu, Jiaojiao Jiang, Ahad N. Zehmakan

Organizations: Australian National University · Data61, CSIRO · University of New South Wales

Abstract

GNN-to-MLP distillation aims to retain the predictive accuracy of a message-passing teacher while deploying a graph-free MLP at inference. Existing methods mainly transfer node-wise predictions or use confidence-based reweighting, but they do not specify where the student should preserve the teacher's graph-induced geometry. We show that this omission leads to two spectral failure modes in the student's representation space. On sparse graphs, the student suffers from spectral underfit, missing high-energy teacher directions concentrated near boundary regions. On dense graphs, it suffers from spectral overfit, retaining spurious directions that the teacher has collapsed through aggregation. Motivated by an energy-weighted teacher-student alignment objective, we propose Graph Geometry-aware MLP (G^2MLP), a training-time distillation framework guided by Ollivier-Ricci curvature. Curvature identifies where the two spectral errors concentrate and is used to allocate supervision between prediction-level and representation-level alignment. The deployed model remains a standard MLP and requires no graph access at inference. Across node-classification benchmarks, G^2MLP consistently improves over graph-free distillation baselines, reduces the teacher-student rank gap in both regimes, and transfers without architectural changes to Graph Transformer teachers and link prediction.

Figures & tables

Figure 1: Two spectral failure modes of GNN-to-MLP distillation. (a) Singular-value spectra: GLNN exhibits spectral underfit on sparse graphs and spectral overfit on dense graphs. (b) Rank-direction sketch. (c) Per-dataset rank gap.

Figure 2: Overview of G 2 MLP. A frozen GNN teacher and an MLP student produce embeddings and logits; precomputed Ollivier–Ricci curvature yields per-node weights $\alpha_{v}$ and $\beta_{v}$ that allocate prediction-level and representation-level alignment during training.

	Teacher	Graph-free students
Dataset	SAGE	MLP	GLNN	KRD	FF-G2M	G 2 MLP
Citeseer	70.49 $\pm$ 1.53	58.50 $\pm$ 1.86	71.22 $\pm$ 1.50	72.84 $\pm$ 1.70	72.85 $\pm$ 1.59	74.51 $\pm$ 2.35
Pubmed	75.56 $\pm$ 2.06	68.39 $\pm$ 3.09	75.59 $\pm$ 2.46	77.01 $\pm$ 3.11	76.56 $\pm$ 3.41	78.17 $\pm$ 2.75
Cora	80.64 $\pm$ 1.57	59.18 $\pm$ 1.60	80.26 $\pm$ 1.66	82.27 $\pm$ 1.31	82.38 $\pm$ 1.41	82.54 $\pm$ 1.87
A-computer	82.82 $\pm$ 1.37	67.62 $\pm$ 2.21	82.71 $\pm$ 1.18	82.87 $\pm$ 0.87	83.67 $\pm$ 1.04	83.87 $\pm$ 1.09
A-photo	90.85 $\pm$ 0.87	77.29 $\pm$ 1.79	91.95 $\pm$ 1.04	91.95 $\pm$ 1.44	92.51 $\pm$ 0.56	92.85 $\pm$ 0.78

Table 1: Node classification accuracy (%) under the transductive setting. Best student is in bold .

		Teacher	Graph-free students
Dataset	Eval	SAGE	MLP	GLNN	KRD	FF-G2M	G 2 MLP
Citeseer	prod	68.06	58.49	69.08	71.38	71.89	71.67
	ind	69.14 $\pm$ 2.99	59.31 $\pm$ 4.56	68.48 $\pm$ 2.38	69.78 $\pm$ 3.04	69.75 $\pm$ 3.16	71.85 $\pm$ 2.27
	tran	67.79 $\pm$ 2.80	58.29 $\pm$ 1.94	69.23 $\pm$ 2.39	71.77 $\pm$ 2.81	72.12 $\pm$ 2.69	71.62 $\pm$ 1.31
Pubmed	prod	74.77	68.39	74.67	76.00	73.98	77.45
	ind	75.07 $\pm$ 2.89	68.28 $\pm$ 3.25	74.52 $\pm$ 2.95	75.17 $\pm$ 3.11	73.49 $\pm$ 7.91	77.30 $\pm$ 1.48

Table 2: Node classification accuracy (%) under the production setting.

Dataset	$T$	$S$	$S^{\dagger}$	$T$	$S$	$S^{\dagger}$	$T$	$S$	$S^{\dagger}$
	GT			GraphGPS			NAGphormer
Citeseer	71.16 $\pm$ 2.31	71.85 $\pm$ 2.45	72.35 $\pm$ 1.85	68.07 $\pm$ 2.62	69.38 $\pm$ 2.30	69.68 $\pm$ 2.05	70.09 $\pm$ 1.81	71.24 $\pm$ 1.64	71.82 $\pm$ 1.29
Pubmed	74.09 $\pm$ 2.48	74.61 $\pm$ 2.48	74.76 $\pm$ 2.83	71.77 $\pm$ 2.81	71.68 $\pm$ 2.61	72.03 $\pm$ 2.58	76.19 $\pm$ 2.86	76.85 $\pm$ 3.34	77.06 $\pm$ 3.49
Cora	78.53 $\pm$ 2.08	78.98 $\pm$ 1.90	79.21 $\pm$ 2.09	76.21 $\pm$ 1.91	76.06 $\pm$ 1.99	76.77 $\pm$ 1.97	80.88 $\pm$ 1.77	81.00 $\pm$ 1.45	81.48 $\pm$ 1.53
A-computer	83.89 $\pm$ 1.59	84.53 $\pm$ 1.42	84.72 $\pm$ 1.32	78.30 $\pm$ 1.70	78.99 $\pm$ 1.73	80.12 $\pm$ 1.55	82.78 $\pm$ 1.92	83.02 $\pm$ 1.87	83.31 $\pm$ 1.77
A-photo	90.94 $\pm$ 0.87	91.90 $\pm$ 0.56	92.06 $\pm$ 0.82	90.41 $\pm$ 1.10	91.03 $\pm$ 0.95	91.32 $\pm$ 0.93	91.45 $\pm$ 0.82	92.07 $\pm$ 0.74	92.49 $\pm$ 0.65

Table 3: Transformer-teacher to MLP distillation.

T

: teacher;

S

: GLNN student;

S^{\dagger}

: G 2 MLP student. Bold : student value

\geq

teacher. Underline :

S^{\dagger}>S

	Cora		Citeseer		Pubmed
Method	AUC	AP	AUC	AP	AUC	AP
SAGE (teacher, ref)	81.29 $\pm$ 1.99	80.54 $\pm$ 2.69	81.21 $\pm$ 3.15	81.80 $\pm$ 3.87	88.36 $\pm$ 1.65	86.83 $\pm$ 1.54
MLP	77.92 $\pm$ 1.78	76.23 $\pm$ 2.01	76.47 $\pm$ 2.08	76.13 $\pm$ 2.74	90.29 $\pm$ 0.38	89.34 $\pm$ 0.44
GLNN	78.61 $\pm$ 1.68	77.05 $\pm$ 1.76	83.32 $\pm$ 2.34	83.71 $\pm$ 2.53	90.03 $\pm$ 0.38	89.12 $\pm$ 0.32
KRD	79.20 $\pm$ 1.65	77.89 $\pm$ 2.31	83.31 $\pm$ 2.23	83.81 $\pm$ 2.42	90.32 $\pm$ 0.42	89.38 $\pm$ 0.51
G 2 MLP (Ours)	83.90 $\pm$ 1.10	83.36 $\pm$ 0.99	83.62 $\pm$ 2.33	84.32 $\pm$ 2.43	90.36 $\pm$ 1.29	89.96 $\pm$ 1.07

Table 4: Link prediction (multi-task regime). SAGE row: teacher reference. Bold : best graph-free student per column. Underline : exceeds the teacher.

Figure 3: Inference time on Arxiv ( $\sim$ 170K nodes).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	#Nodes	#Edges	#Feat.	#Cls.	$\bar{d}$	$h$
Cora	2,708	5,429	1,433	7	3.9	0.81
Citeseer	3,327	4,732	3,703	6	2.7	0.74
Pubmed	19,717	44,338	500	3	4.5	0.80
A-computer	13,752	245,861	767	10	35.8	0.78
A-photo	7,650	119,081	745	8	31.1	0.83
ogbn-arxiv	169,343	1,166,243	128	40	13.7	0.65

Appendix

Table 5: Benchmark statistics of the full graphs as published.

\bar{d}

is the mean degree;

h

is edge homophily. Following CPF, node-classification experiments on the five CPF datasets use the largest connected component (Cora 2,485, Citeseer 2,110, Pubmed 19,717, A-computer 13,381, A-photo 7,487 nodes).

Dataset	$\eta$	weight-decay	dropout
Cora	$10^{-2}$	$5{\times}10^{-3}$	0.6
Citeseer	$10^{-2}$	$10^{-3}$	0.1
Pubmed	$5{\times}10^{-3}$	0	0.4
A-computer	$10^{-3}$	$2{\times}10^{-3}$	0.3
A-photo	$5{\times}10^{-3}$	$2{\times}10^{-3}$	0.3
ogbn-arxiv	$10^{-2}$	0	0.2

Appendix

Table 6: Student MLP optimizer hyperparameters.

Variant	Cora	Citeseer	Pubmed
GLNN baseline	80.26 $\pm$ 1.66	71.22 $\pm$ 1.50	75.59 $\pm$ 2.46
$+\;W_{2}$ uniform (no curvature)	80.86 $\pm$ 2.30	71.07 $\pm$ 2.19	75.22 $\pm$ 2.15
$+\;$ curvature, logit-side only	81.78 $\pm$ 1.55	73.87 $\pm$ 2.09	77.27 $\pm$ 2.92
$+\;$ curvature, feature-side only	81.34 $\pm$ 1.88	73.99 $\pm$ 2.02	77.14 $\pm$ 2.93
Full G 2 MLP (both sides)	82.54 $\pm$ 1.87	74.51 $\pm$ 2.35	78.17 $\pm$ 2.75

Appendix

Table 7: Component ablation on transductive node classification. Baseline reproduced from Table 1 .

	GraphGPS (dense, canonical)			GraphGPS (sparse, edge-softmax)
Dataset	$T$	$S$	$S^{\dagger}$	$T$	$S$	$S^{\dagger}$
Cora	76.21 $\pm$ 1.91	76.06 $\pm$ 1.99	76.77 $\pm$ 1.97	78.12 $\pm$ 2.12	77.86 $\pm$ 2.09	78.74 $\pm$ 1.87
Citeseer	68.07 $\pm$ 2.62	69.38 $\pm$ 2.30	69.68 $\pm$ 2.05	68.88 $\pm$ 2.25	70.29 $\pm$ 2.13	70.44 $\pm$ 2.13
Pubmed	71.77 $\pm$ 2.81	71.68 $\pm$ 2.61	72.03 $\pm$ 2.58	75.61 $\pm$ 2.13	76.08 $\pm$ 2.34	75.97 $\pm$ 2.21
A-computer	78.30 $\pm$ 1.70	78.99 $\pm$ 1.73	80.12 $\pm$ 1.55	81.84 $\pm$ 0.88	82.05 $\pm$ 1.18	82.55 $\pm$ 0.86
A-photo	90.41 $\pm$ 1.10	91.03 $\pm$ 0.95	91.32 $\pm$ 0.93	90.34 $\pm$ 1.21	91.12 $\pm$ 1.04	91.26 $\pm$ 1.02

Appendix

Table 8: G 2 MLP vs. GLNN with the GraphGPS teacher under both canonical dense (

N\times N

) and sparse (edge-softmax) attention. Conventions as in Table 3 :

T

teacher,

S

GLNN student,

S^{\dagger}

G 2 MLP student. Bold : student value

\geq

teacher. Underline :

S^{\dagger}>S

	Cora		Citeseer		Pubmed
Method	AUC	AP	AUC	AP	AUC	AP
SAGE (teacher, ref)	81.29 $\pm$ 1.99	80.54 $\pm$ 2.69	81.21 $\pm$ 3.15	81.80 $\pm$ 3.87	88.36 $\pm$ 1.65	86.83 $\pm$ 1.54
GLNN	56.18 $\pm$ 5.79	56.81 $\pm$ 4.55	83.99 $\pm$ 2.03	84.23 $\pm$ 2.16	88.62 $\pm$ 1.57	87.17 $\pm$ 1.58
KRD	57.23 $\pm$ 5.53	57.41 $\pm$ 5.02	84.26 $\pm$ 2.56	84.46 $\pm$ 2.59	88.63 $\pm$ 1.56	87.18 $\pm$ 1.57
G 2 MLP (Ours)	81.77 $\pm$ 2.25	80.28 $\pm$ 2.84	82.89 $\pm$ 3.61	83.15 $\pm$ 4.39	88.82 $\pm$ 1.46	87.43 $\pm$ 1.35

Appendix

Table 9: Link prediction (pure-distillation regime, no edge-BCE). Conventions as in Table 4 .

Explore similar work

Learning Propagation Geometry from Message-Passing Feedback
Sep 28, 2026Yingxu Wang, Kunyu Zhang, Xinwang Liu +4Graph Neural Networks Geometric Machine Learning
The Chinese University of Hong Kong · The Education University of Hong Kong · National University of Defense Technology +3
Spectral Graph Sparsification Preserves Representation Geometry in Graph Neural Networks
May 1, 2026Sanjukta KrishnagopalGraph Edge Sparsification Graph Neural Networks
Department of Computer Science, University of California Santa Barbara
Train Small, Deploy Large: Zero-Shot GNN Transfer Through Geometric Renormalization
Jul 30, 2026Robert Jankowski, Pedro Almagro-Blanco, Marián Boguñá +2Graph Neural Networks Renormalization Group
TU Delft · University of Seville · University of Barcelona +2

Sep 28, 2026, cs.LGJ/K move · Enter open · S save

Learning Propagation Geometry from Message-Passing Feedback

Yingxu Wang, Kunyu Zhang, Xinwang Liu, Mengzhu Wang+3

The Chinese University of Hong Kong · The Education University of Hong Kong · National University of Defense Technology+3

Graph Neural Networks Geometric Machine Learning Geometric Constraints Geogs-Slam Propagation Feedback+1

Open paper AskPDF arXiv

Learning local geometry enables graph neural networks (GNNs) to adapt how they compare and integrate neighborhood information. However, estimating geometry from aggregated representations can overlook variation among individual messages and dependencies across feature dimensions. We propose GeoF, a recurrent framework that jointly evolves node features and propagation geometry through message-passing feedback. Each node maintains a local symmetric positive-definite geometry, initialized from a structure-aware prototype atlas and parameterized in block log-triangular coordinates. At each step, the geometry determines neighborhood weights, while triangular frame transport maps transformed source messages into the target node's local coordinates before aggregation. Weighted second-order statistics of residuals between aligned messages and the transformed target state capture directional variation and within-block dependencies, yielding a geometric update target. A shared controller learns complementary corrections through task supervision. A bounded log-triangular update combines these corrections, the target, and the previous geometric state while preserving positive definiteness. The geometry governs subsequent propagation, closing the feedback loop. With parameters shared across recurrent steps, task-specific readouts support node classification, link prediction, and graph classification. Experiments on benchmark datasets show that GeoF consistently outperforms state-of-the-art GNN baselines.

Abstract

Explore similar work

Learning Propagation Geometry from Message-Passing Feedback

Spectral Graph Sparsification Preserves Representation Geometry in Graph Neural Networks

Train Small, Deploy Large: Zero-Shot GNN Transfer Through Geometric Renormalization