Efficient and Scalable Multi-View Masked Autoencoder for Heterogeneous Graph Self-Supervised Learning
Authors: Hao Geng, Deqing Wang, Lihao Ge, Chenguang Du, Fuzhen Zhuang, Bowen Yang, Yanlin Hu, Songtao Wang
Organizations: Beihang University, Beijing 100191, China · Zhongguancun Laboratory, Beijing 100194, China · Westlake University, Hangzhou 310030, China · National Computer Network Emergency Response Technical Team/Coordination Center of China, Beijing 100029, China
Recent years have witnessed the flourishing of self-supervised learning on heterogeneous graphs, e.g., masked autoencoder-based methods. However, existing methods suffer from three issues, including the prohibitive computational cost of constructing meta-path-induced subgraphs, the limitations of uniform random masking, and the suboptimal objective of reconstructing potentially noisy or uninformative original node features. To address these issues, we propose the Multi-View Masked Auto Encoder (MVMAE), a novel, efficient, and scalable framework for heterogeneous graph self-supervised learning. First, MVMAE introduces a novel multi-view mask-then-reconstruct strategy. Instead of masking individual nodes or features, MVMAE randomly masks entire meta-path views and guides the model to reconstruct their representations from remaining visible views. This approach shifts the learning objective to a high-level and semantic task, providing a more robust and informative supervisory signal that is effective even when initial node features are absent or flawed. Then, to handle the computational demands, MVMAE employs an efficient offline aggregation mechanism that pre-computes and reuses representations for each meta-path view, completely avoiding the costly construction of meta-path-induced subgraphs. Next, to fuse the representations of each view, MVMAE uses a Transformer-based multi-view fusion encoder to capture the complex correlations between views, and a deliberately simple MLP decoder to compel the encoder to learn highly expressive and robust node representations. Extensive experiments on six real-world datasets, including three large-scale graphs, demonstrate that MVMAE consistently outperforms state-of-the-art self-supervised and even semi-supervised baselines, confirming its superior effectiveness and scalability. Our source code is released at https://github.com/hcmdgh/MVMAE.
Figures & tables
Fig. 1: Example of meta-path-induced subgraphs in OAG. GPAP and GPCP are two subgraphs induced by meta-paths P-A-P and P-C-P, respectively.
Fig. 2: Framework of our proposed MVMAE model. The example is based on an academic heterogeneous graph with node types paper (P), author (A), and field (F). The figure demonstrates the aggregation of 1-hop meta-path P-P, and 2-hop meta-paths P-F-P and P-A-P.
Datasets
Node Types
# Nodes
Edge Types
# Edges
Meta-paths
ACM
Paper
4,019
PA/AP
13,407
P-A-P, P-S-P,
Author
7,167
PS/SP
4,019
P-A-P-A-P, P-A-P-S-P,
Subject
60
P-S-P-A-P, P-S-P-S-P
DBLP
Author
4,057
AP/PA
19,645
A-P-A,
Paper
14,328
PT/TP
85,810
A-P-T-P-A,
Term
7,723
PC/CP
14,328
A-P-C-P-A
TABLE I: Statistics of the datasets.
Dataset
Metric
Split
HAN
HGT
DMGI
HDGI
HeCo
HGMAE
RMR
MV-HetGNN
Hetero2Net
MVMAE
ACM
Mic-F1
20
85.11 ± 2.2
86.23 ± 2.1
87.60 ± 0.8
86.53 ± 1.4
88.13 ± 0.8
90.24 ± 0.5
87.58 ± 0.5
89.33 ± 0.4
88.70 ± 0.2
92.46 ± 0.4
40
87.21 ± 1.2
87.46 ± 1.3
86.02 ± 0.9
85.88 ± 1.2
87.45 ± 0.5
90.18 ± 0.6
87.73 ± 0.7
91.33 ± 0.8
90.60 ± 0.8
91.53 ± 0.9
60
88.10 ± 1.2
88.05 ± 1.1
87.82 ± 0.5
87.10 ± 1.3
88.71 ± 0.5
91.34 ± 0.4
88.10 ± 0.6
89.63 ± 0.9
89.90 ± 0.2
92.20 ± 0.2
Mac-F1
20
85.66 ± 2.1
86.47 ± 1.9
87.86 ± 0.2
86.83 ± 0.9
88.56 ± 0.8
90.66 ± 0.4
87.63 ± 0.5
89.18 ± 0.6
88.56 ± 0.2
92.36 ± 0.5
40
87.47 ± 1.1
87.48 ± 1.4
86.23 ± 0.8
86.27 ± 0.8
87.61 ± 0.5
90.15 ± 0.6
88.92 ± 0.4
91.34 ± 0.8
90.57 ± 0.8
91.51 ± 0.9
60
88.41 ± 1.1
88.53 ± 1.3
87.97 ± 0.4
87.14 ± 0.5
89.04 ± 0.5
91.59 ± 0.4
90.33 ± 0.5
89.69 ± 0.8
89.97 ± 0.2
92.31 ± 0.2
TABLE II: Node classification results (%).
Method
Micro-F1
Macro-F1
AP
HAN
OOM
OOM
OOM
HGT
98.25 ± 0.1
91.57 ± 0.2
92.29 ± 0.1
DMGI
OOM
OOM
OOM
HDGI
OOM
OOM
OOM
HeCo
OOM
OOM
OOM
HGMAE
OOM
OOM
OOM
TABLE III: Node classification results on the RCDD dataset (%).
Dataset
ACM
DBLP
Freebase
AMiner
Metric
NMI
ARI
NMI
ARI
NMI
ARI
NMI
ARI
DMGI
46.53
42.16
47.35
46.28
19.72
19.56
OOM
OOM
HDGI
49.76
44.35
59.24
60.43
18.13
19.44
OOM
OOM
HeCo
56.17
56.36
74.71
79.41
21.33
22.99
OOM
OOM
HGMAE
71.20
74.33
75.23
80.63
19.91
21.04
OOM
OOM
RMR
51.86
47.27
9.90
7.21
13.08
11.82
27.42
18.82
TABLE IV: Node clustering results (%).
Datasets
Metrics
w/o mask
w/o aggr
w/o fusion
MVMAE
ACM
Mic-F1
90.24
91.07
88.73
92.46
Mac-F1
90.16
91.33
89.76
92.36
DBLP
Mic-F1
93.12
94.37
92.68
94.96
Mac-F1
93.02
94.16
92.23
94.47
Freebase
Mic-F1
62.76
63.92
60.44
66.40
Mac-F1
55.66
56.15
54.27
58.72
TABLE V: Ablation results of MVMAE on five datasets.
Datasets
Metrics
Learnable Aggregator
Expressive Decoder
MVMAE
ACM
Mic-F1
92.54
92.42
92.46
Mac-F1
92.21
92.38
92.36
DBLP
Mic-F1
94.92
94.75
94.96
Mac-F1
94.35
94.41
94.47
Freebase
Mic-F1
66.25
66.31
66.40
Mac-F1
58.68
58.84
58.72
TABLE VI: Further Ablation results of MVMAE on five datasets.
Method
ACM (N: 11,246)
AMiner (N: 439,448)
OAG (N: 764,914)
# Params
Memory
Time
# Params
Memory
Time
# Params
Memory
Time
HAN
394,499
15,526M
0.30s
OOM
OOM
OOM
OOM
OOM
OOM
HGT
462,231
806M
0.27s
728,114
4,568M
2.3s
1,679,676
23,940M
5.7s
HGMAE
285,404
7,624M
0.32s
OOM
OOM
OOM
OOM
OOM
OOM
RMR
85,216
762M
0.23s
92,352
3,052M
0.2s
174,208
15,996M
0.5s
MV-HetGNN
103,636
976M
0.28s
114,538
4,372M
0.8s
242,446
22,784M
3.8s
TABLE VII: Efficiency comparison of different methods regarding parameter count (# Params), memory footprint, and training time. Memory consumption is reported as the peak GPU memory usage in megabytes (MBs). Training time denotes the average duration per epoch in seconds (s), calculated over ten training epochs. All measurements are conducted on an NVIDIA GeForce RTX 3090 GPU (24-GB memory) and 128-GB RAM.
Fig. 3: Sensitivity analysis of different meta-paths’ contributions to the model performance on the AMiner dataset.
Fig. 4: Sensitivity analysis of the masking ratio on the AMiner dataset.
Fig. 5: Sensitivity analysis of the hop of meta-paths on the ACM and Freebase datasets.
Dataset
ACM
DBLP
Freebase
AMiner
OAG
RCDD
Masking ratio ( ρ )
0.2
0.3
0.5
0.5
0.3
0.5
Embedding dimension ( d )
256
256
256
256
128
128
Encoder depth ( Le )
3
1
2
3
2
2
Decoder depth ( Ld )
1
1
1
1
1
1
Learning rate ( η )
1e-3
1e-3
1e-3
1e-3
1e-3
1e-3
Dropout rate
0.4
0.5
0.4
0.2
0.2
0.2
TABLE VIII: Hyperparameter settings of MVMAE on various datasets.
Unsupervised graph representation learning aims to derive meaningful node embeddings by capturing both structural and attribute information without relying on labeled data. Existing methods, such as GAEs, have demonstrated effectiveness but typically rely on message-passing mechanisms that assume homophily, leading to performance degradation on heterophilous graphs, where connected nodes exhibit dissimilar features. This homophily bias results in the loss of critical high-frequency components that are essential for identifying heterophilous patterns. To address these challenges, we propose AlignGAE, a novel extension of MaskGAE that preserves the full frequency spectrum through complementary view alignment. Our framework introduces a dual-encoder architecture that separately processes structural and attribute information, incorporates node positional encoding to approximate Neighborhood Identity Distribution (NID), and employs dual reconstruction tasks for both edges and node attributes. We further propose theoretically grounded NID alignment strategies that ensure semantic consistency across views while preserving their distinct characteristics. Through comprehensive spectral analysis, we demonstrate that AlignGAE achieves optimal representation properties when the alignment loss converges. Extensive experiments across 12 benchmark datasets validate our approach, showing that AlignGAE outperforms state-of-the-art methods by up to 18.7% on heterophilous graphs in node classification, while maintaining competitive performance on homophilous graphs. Our results establish a new paradigm for frequency-aware graph representation learning.
Zengyi Wo, Shiyu Zhang, Qiyao Peng +2
Nanjing University, Nanjing, China · Tianjin University, Tianjin, China
We propose MABLE (Masked Autoencoding with Bi-Lipschitz Decoding for Embeddings and Graph Metric Learning), a self-supervised framework for learning node and graph embeddings from large, heterogeneous graphs, demonstrated here on geospatial mineral-exploration data. MABLE combines masked reconstruction with fixed cosine-similarity losses that align matched augmented views while keeping unpaired embeddings well spread. A bi-Lipschitz feature decoder ties a low-dimensional reconstruction component of each node embedding to feature similarity, while matched-node consistency shapes the remaining context used by graph pooling. Lipschitz-controlled pooling helps stabilize graph-level representations under perturbations of retained node embeddings, while augmentation alignment trains robustness to masking, node dropping, and sampling variation. Across local copper and regional Arabian Shield studies, MABLE embeddings provide complementary downstream signal and produce coherent embedding-derived layers for hypothesis generation without learned discriminators or hard-negative selection.
Yaniv Shulman, Shaghayegh Akbarpour, Jack B. Muir
Fleet Space Technologies, Adelaide, Australia · Research School of Earth Sciences, Australian National University, Acton, Australia
Self-supervised learning on graphs is largely shaped by contrastive methods that depend on carefully designed augmentations, and by generative methods that reconstruct node attributes in the input space. Both paradigms can entangle representations with low-level input statistics rather than with relational structure. Joint-embedding predictive architectures (JEPA) instead learn by predicting latent targets rather than reconstructing inputs. Recent work has explored this idea for graph-level representation learning, but how to design JEPA-style objectives for node-level tasks, and which structural signals the predictor should condition on, remains less clear. We present NodeJEPA, a joint-embedding predictive architecture for node-level graph self-supervised learning. NodeJEPA masks structure-aware k-hop ego-subgraphs and trains a context encoder to predict the latent representations of the masked nodes. These targets come from an EMA-updated target encoder with stop-gradient. A structure-conditioned predictor integrates spectral and centrality descriptors through cross-attention. Variance, covariance, and Laplacian spectral regularizers help stabilize the embedding geometry, and an optional curriculum gradually increases masking difficulty during training. Because prediction occurs in latent space, NodeJEPA does not rely on input reconstruction or hand-crafted graph augmentations. We evaluate NodeJEPA on standard node classification benchmarks under linear probing and fine-tuning protocols, and conduct ablations on masking, prediction, and regularization design choices. Our study offers a practical recipe for node-level JEPA-style latent prediction on graphs, and clarifies when structural conditioning helps representation learning. Code, configurations, and evaluation scripts are publicly available at https://github.com/OliverZ-dot/Node-Jepa.