Efficient and Scalable Multi-View Masked Autoencoder for Heterogeneous Graph Self-Supervised Learning
Authors: Hao Geng, Deqing Wang, Lihao Ge, Chenguang Du, Fuzhen Zhuang, Bowen Yang, Yanlin Hu, Songtao Wang
Organizations: Beihang University, Beijing 100191, China · Zhongguancun Laboratory, Beijing 100194, China · Westlake University, Hangzhou 310030, China · National Computer Network Emergency Response Technical Team/Coordination Center of China, Beijing 100029, China
Recent years have witnessed the flourishing of self-supervised learning on heterogeneous graphs, e.g., masked autoencoder-based methods. However, existing methods suffer from three issues, including the prohibitive computational cost of constructing meta-path-induced subgraphs, the limitations of uniform random masking, and the suboptimal objective of reconstructing potentially noisy or uninformative original node features. To address these issues, we propose the Multi-View Masked Auto Encoder (MVMAE), a novel, efficient, and scalable framework for heterogeneous graph self-supervised learning. First, MVMAE introduces a novel multi-view mask-then-reconstruct strategy. Instead of masking individual nodes or features, MVMAE randomly masks entire meta-path views and guides the model to reconstruct their representations from remaining visible views. This approach shifts the learning objective to a high-level and semantic task, providing a more robust and informative supervisory signal that is effective even when initial node features are absent or flawed. Then, to handle the computational demands, MVMAE employs an efficient offline aggregation mechanism that pre-computes and reuses representations for each meta-path view, completely avoiding the costly construction of meta-path-induced subgraphs. Next, to fuse the representations of each view, MVMAE uses a Transformer-based multi-view fusion encoder to capture the complex correlations between views, and a deliberately simple MLP decoder to compel the encoder to learn highly expressive and robust node representations. Extensive experiments on six real-world datasets, including three large-scale graphs, demonstrate that MVMAE consistently outperforms state-of-the-art self-supervised and even semi-supervised baselines, confirming its superior effectiveness and scalability. Our source code is released at https://github.com/hcmdgh/MVMAE.
Figures & tables
Fig. 1: Example of meta-path-induced subgraphs in OAG. GPAP and GPCP are two subgraphs induced by meta-paths P-A-P and P-C-P, respectively.
Fig. 2: Framework of our proposed MVMAE model. The example is based on an academic heterogeneous graph with node types paper (P), author (A), and field (F). The figure demonstrates the aggregation of 1-hop meta-path P-P, and 2-hop meta-paths P-F-P and P-A-P.
Datasets
Node Types
# Nodes
Edge Types
# Edges
Meta-paths
ACM
Paper
4,019
PA/AP
13,407
P-A-P, P-S-P,
Author
7,167
PS/SP
4,019
P-A-P-A-P, P-A-P-S-P,
Subject
60
P-S-P-A-P, P-S-P-S-P
DBLP
Author
4,057
AP/PA
19,645
A-P-A,
Paper
14,328
PT/TP
85,810
A-P-T-P-A,
Term
7,723
PC/CP
14,328
A-P-C-P-A
TABLE I: Statistics of the datasets.
Dataset
Metric
Split
HAN
HGT
DMGI
HDGI
HeCo
HGMAE
RMR
MV-HetGNN
Hetero2Net
MVMAE
ACM
Mic-F1
20
85.11 ± 2.2
86.23 ± 2.1
87.60 ± 0.8
86.53 ± 1.4
88.13 ± 0.8
90.24 ± 0.5
87.58 ± 0.5
89.33 ± 0.4
88.70 ± 0.2
92.46 ± 0.4
40
87.21 ± 1.2
87.46 ± 1.3
86.02 ± 0.9
85.88 ± 1.2
87.45 ± 0.5
90.18 ± 0.6
87.73 ± 0.7
91.33 ± 0.8
90.60 ± 0.8
91.53 ± 0.9
60
88.10 ± 1.2
88.05 ± 1.1
87.82 ± 0.5
87.10 ± 1.3
88.71 ± 0.5
91.34 ± 0.4
88.10 ± 0.6
89.63 ± 0.9
89.90 ± 0.2
92.20 ± 0.2
Mac-F1
20
85.66 ± 2.1
86.47 ± 1.9
87.86 ± 0.2
86.83 ± 0.9
88.56 ± 0.8
90.66 ± 0.4
87.63 ± 0.5
89.18 ± 0.6
88.56 ± 0.2
92.36 ± 0.5
40
87.47 ± 1.1
87.48 ± 1.4
86.23 ± 0.8
86.27 ± 0.8
87.61 ± 0.5
90.15 ± 0.6
88.92 ± 0.4
91.34 ± 0.8
90.57 ± 0.8
91.51 ± 0.9
60
88.41 ± 1.1
88.53 ± 1.3
87.97 ± 0.4
87.14 ± 0.5
89.04 ± 0.5
91.59 ± 0.4
90.33 ± 0.5
89.69 ± 0.8
89.97 ± 0.2
92.31 ± 0.2
TABLE II: Node classification results (%).
Method
Micro-F1
Macro-F1
AP
HAN
OOM
OOM
OOM
HGT
98.25 ± 0.1
91.57 ± 0.2
92.29 ± 0.1
DMGI
OOM
OOM
OOM
HDGI
OOM
OOM
OOM
HeCo
OOM
OOM
OOM
HGMAE
OOM
OOM
OOM
TABLE III: Node classification results on the RCDD dataset (%).
Dataset
ACM
DBLP
Freebase
AMiner
Metric
NMI
ARI
NMI
ARI
NMI
ARI
NMI
ARI
DMGI
46.53
42.16
47.35
46.28
19.72
19.56
OOM
OOM
HDGI
49.76
44.35
59.24
60.43
18.13
19.44
OOM
OOM
HeCo
56.17
56.36
74.71
79.41
21.33
22.99
OOM
OOM
HGMAE
71.20
74.33
75.23
80.63
19.91
21.04
OOM
OOM
RMR
51.86
47.27
9.90
7.21
13.08
11.82
27.42
18.82
TABLE IV: Node clustering results (%).
Datasets
Metrics
w/o mask
w/o aggr
w/o fusion
MVMAE
ACM
Mic-F1
90.24
91.07
88.73
92.46
Mac-F1
90.16
91.33
89.76
92.36
DBLP
Mic-F1
93.12
94.37
92.68
94.96
Mac-F1
93.02
94.16
92.23
94.47
Freebase
Mic-F1
62.76
63.92
60.44
66.40
Mac-F1
55.66
56.15
54.27
58.72
TABLE V: Ablation results of MVMAE on five datasets.
Datasets
Metrics
Learnable Aggregator
Expressive Decoder
MVMAE
ACM
Mic-F1
92.54
92.42
92.46
Mac-F1
92.21
92.38
92.36
DBLP
Mic-F1
94.92
94.75
94.96
Mac-F1
94.35
94.41
94.47
Freebase
Mic-F1
66.25
66.31
66.40
Mac-F1
58.68
58.84
58.72
TABLE VI: Further Ablation results of MVMAE on five datasets.
Method
ACM (N: 11,246)
AMiner (N: 439,448)
OAG (N: 764,914)
# Params
Memory
Time
# Params
Memory
Time
# Params
Memory
Time
HAN
394,499
15,526M
0.30s
OOM
OOM
OOM
OOM
OOM
OOM
HGT
462,231
806M
0.27s
728,114
4,568M
2.3s
1,679,676
23,940M
5.7s
HGMAE
285,404
7,624M
0.32s
OOM
OOM
OOM
OOM
OOM
OOM
RMR
85,216
762M
0.23s
92,352
3,052M
0.2s
174,208
15,996M
0.5s
MV-HetGNN
103,636
976M
0.28s
114,538
4,372M
0.8s
242,446
22,784M
3.8s
TABLE VII: Efficiency comparison of different methods regarding parameter count (# Params), memory footprint, and training time. Memory consumption is reported as the peak GPU memory usage in megabytes (MBs). Training time denotes the average duration per epoch in seconds (s), calculated over ten training epochs. All measurements are conducted on an NVIDIA GeForce RTX 3090 GPU (24-GB memory) and 128-GB RAM.
Fig. 3: Sensitivity analysis of different meta-paths’ contributions to the model performance on the AMiner dataset.
Fig. 4: Sensitivity analysis of the masking ratio on the AMiner dataset.
Fig. 5: Sensitivity analysis of the hop of meta-paths on the ACM and Freebase datasets.
Dataset
ACM
DBLP
Freebase
AMiner
OAG
RCDD
Masking ratio ( ρ )
0.2
0.3
0.5
0.5
0.3
0.5
Embedding dimension ( d )
256
256
256
256
128
128
Encoder depth ( Le )
3
1
2
3
2
2
Decoder depth ( Ld )
1
1
1
1
1
1
Learning rate ( η )
1e-3
1e-3
1e-3
1e-3
1e-3
1e-3
Dropout rate
0.4
0.5
0.4
0.2
0.2
0.2
TABLE VIII: Hyperparameter settings of MVMAE on various datasets.