More Domains, No Gains? FLO: Fast-Slow Reciprocal Learning for Multi-Domain Graph Pre-training

Yupeng Tong; Lianze Shan; Jitao Zhao; Di Jin; Dongxiao He

cs.LGOct 10, 2026

More Domains, No Gains? FLO: Fast-Slow Reciprocal Learning for Multi-Domain Graph Pre-training

Authors: Yupeng Tong, Lianze Shan, Jitao Zhao, Di Jin, Dongxiao He

Organizations: School of Computer Science and Technology, Tianjin University, Tianjin, China

Abstract

Inspired by the great success of foundation models in the development of artificial general intelligence, Graph Foundation Models (GFMs) have recently attracted significant attention and have the potential to advance graph intelligence. Exhibiting such intelligence requires effectively learning knowledge from large-scale, multi-domain graph data during pre-training. Increasing domain diversity during pre-training is expected to yield richer and more generalizable knowledge. Most existing methods pursue this goal by designing domain alignment strategies that help a single graph model learn from multiple domains. However, we observe that existing methods struggle to benefit from increasing domain diversity. Through empirical and theoretical analysis, we reveal two key limitations. Firstly, pre-training in most GFMs is confined to a single message-passing paradigm. Since propagation flattens relations among neighboring nodes, the graph model can hardly perceive the knowledge beyond propagation, namely the attribute-predictable component of these relations. Secondly, domain alignment strategies often rely on randomly initialized parameters that are jointly updated with the graph model using immediate loss feedback, leaving early knowledge acquisition vulnerable to cross-domain interference. To address these issues, we propose FLO, a Fast--sLow reciprOcal approach for multi-domain graph pre-training. Specifically, a propagation-free attribute branch predicts the graph model's relations without message passing and returns these predictions to refine the model. We further stabilize knowledge acquisition by pairing a fast process that adapts to the current domain with a slow process that integrates knowledge across domains. On 12 datasets, FLO benefits from increasing domain diversity and achieves the best average performance in cross-domain and cross-task transfer.

Figures & tables

Figure 1: Motivation experiments with Cora as the target graph. (a) Accuracy change as source domains are added. (b)–(c) Pre-training loss of two baselines.

Figure 2: Overview of FLO. (a) Multi-domain pre-training: source graphs are visited in turn, and the graph branch

f_{G}

and the attribute branch

f_{M}

each reconstruct node features. In Fast–Slow Knowledge Acquisition (FSKA), each branch keeps a slow copy updated by an Exponential Moving Average (EMA) after each step. In Propagation–Attribute Relation Refinement (PARR), each branch learns the relation distribution of the other branch’s slow copy over the same candidates. (b) Downstream tasks: only the frozen graph encoder

f_{G}

is applied to graphs from unseen domains.

Method	Cora	CiteSeer	PubMed	CS	Photo	Computers	Avg.
DGI	75.47 ±0.78	65.76 ±1.19	82.83 ±0.24	92.82 ±0.18	91.48 ±0.26	87.24 ±0.21	82.60
GRACE	72.00 ±0.36	56.29 ±0.71	76.80 ±0.37	90.29 ±0.09	90.76 ±0.15	86.82 ±0.15	78.83
BGRL	79.43 ±1.24	64.46 ±1.14	83.91 ±0.26	92.63 ±0.21	92.76 ±0.21	88.98 ±0.16	83.69
FUG	82.79 ±0.72	70.52 ±0.15	84.42 ±0.23	92.41 ±0.02	92.31 ±0.16	87.30 ±0.16	84.96
TIG	84.39 ±1.08	73.08 ±0.52	86.45 ±0.13	93.42 ±0.05	93.27 ±0.22	89.51 ±0.07	86.68
GFT	72.99 ±0.94	64.79 ±0.99	83.59 ±0.12	93.06 ±0.27	90.25 ±0.18	84.18 ±0.19	81.48

Table 1: Accuracy (%) of cross-domain node classification with standard deviations. Each column is an unseen target graph. The best and second-best results are in bold and underlined.

Figure 3: Few-shot node classification. Left: accuracy (%) on Cora, CS, and CiteSeer for FLO and the five baselines with the highest average accuracy. Right: accuracy gain (%) of FLO over each baseline, averaged over the 1-, 3-, and 5-shot settings; Avg. is the mean over the six target graphs.

	Link prediction: Hits@50 (%)				Graph classification: accuracy (%)
Method	Physics	WikiCS	DBLP	Avg.	COIL-RAG	ENZYMES	PROTEINS	Avg.
FUG	55.99 ±3.38	66.23 ±1.22	64.18 ±5.29	62.13	84.97 ±1.50	33.41 ±4.31	71.64 ±1.66	63.34
TIG	44.46 ±2.88	69.93 ±0.59	61.85 ±3.43	58.75	77.16 ±1.53	33.33 ±3.36	71.56 ±2.36	60.68
GFT	42.93 ±2.90	58.78 ±2.13	43.41 ±5.49	48.37	83.42 ±1.62	35.88 ±5.91	72.37 ±2.92	63.89
MDGPT	47.79 ±1.52	68.03 ±1.26	57.25 ±1.56	57.69	80.14 ±2.19	31.24 ±4.22	71.59 ±3.22	60.99
SAMGPT	49.82 ±1.86	70.12 ±0.73	57.60 ±1.75	59.18	81.91 ±1.99	29.57 ±4.13	70.65 ±2.71	60.71

Table 2: Hits@50 (%) of link prediction and accuracy (%) of graph classification with standard deviations. The best and second-best results are in bold and underlined.

Figure 4: Domain diversity and training stability. (a) Accuracy change (%) as source domains are added in the order on the horizontal axis. (b) Min–max normalized log loss in the first 40 epochs.

Variant	Cora	CiteSeer	PubMed	CS	Photo	Computers
w/o $\mathcal{L}_{\rm kd}$	83.26 ±0.84	66.52 ±1.31	86.63 ±0.23	93.13 ±0.09	93.34 ±0.07	90.04 ±0.12
w/o slow attribute branch	83.37 ±0.89	66.98 ±1.11	86.67 ±0.26	93.03 ±0.14	93.47 ±0.15	90.04 ±0.11
w/o slow graph branch	64.65 ±0.95	42.01 ±1.61	80.57 ±0.32	82.58 ±0.23	89.62 ±0.36	84.83 ±0.20
w/o $\mathcal{L}_{\rm rec}$	30.33 ±0.19	20.66 ±0.63	82.48 ±0.31	76.14 ±0.92	59.72 ±1.94	52.40 ±1.16
w/ GCN attribute branch	79.74 ±0.10	55.56 ±0.22	86.60 ±0.26	91.52 ±0.15	92.02 ±0.55	89.48 ±0.17
FLO	85.83 ±0.65	72.50 ±0.91	86.74 ±0.35	93.62 ±0.05	93.62 ±0.23	90.13 ±0.07

Table 3: Ablation study on node classification: accuracy (%) with standard deviations. w/o slow attribute branch removes

\mathcal{L}^{G}_{\rm kd}

, the relation targets returned to the graph branch; w/o slow graph branch removes

\mathcal{L}^{M}_{\rm kd}

, the relation targets that train the attribute branch. Removing one doubles the weight of the other, so the total relation weight is unchanged. The best results are in bold.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Result	What it shows	Main tools	Proof
Eq. ( 9 )	The relational information of the graph branch splits into an attribute-predicted part and the propagation-induced part $R$	Chain rule, tower property	A.2
Lemma A.1	Close embeddings give close relation distributions	Lipschitz bounds of cosine and softmax	A.3
Theorem 3.2	Propagation drives the relations among nearby nodes toward uniform	Edge form of the Dirichlet energy, spectral decomposition	A.4
Theorem 3.3	Targets returned by the attribute branch remove $R$	Conditional mean, convexity of $\operatorname{KL}$ , Pinsker’s inequality	A.5
Proposition A.2	Reconstruction rules out uniform relations	Bounds of the decoder on directions	A.6
Proposition 3.4	Slow copies damp period- $S$ oscillations by $\rho$	Frequency response, Parseval’s identity	A.7

Appendix

Table 4: Results proved in this appendix.

	Cora	CiteSeer	PubMed	CS	Photo	Computers
Relation change, $K=1$	0.70	0.77	0.41	0.44	0.53	0.53
Relation change, $K=2$	1.10	1.09	0.83	0.67	0.61	0.61
Drop without $\mathcal{L}_{\rm kd}$	2.57	5.98	0.11	0.49	0.28	0.09

Appendix

Table 5: Relation change caused by propagation,

\mathbb{E}\,\operatorname{KL}(P\|Q_{B})

, and the accuracy drop (%) when relation learning is removed.

Figure 5: Relation change caused by propagation and the accuracy drop (%) when relation learning is removed, for each target graph. The values are those of Table 5 . The vertical axis is logarithmic, and the dashed lines are least-squares fits of the logarithm of the drop.

Component	Parameter	Value
Input	Common dimension $d$	1,024
Both branches	Hidden width; output width	512; 1,024
Graph branch	Architecture	GCN–PReLU–GCN
Attribute branch	Architecture	Linear–GELU–Linear
Decoders	Architecture	Separate linear maps
Mask	Elementwise keep probability	0.40

Appendix

Table 6: Hyperparameters of FLO. The pre-training hyperparameters are shared across all target graphs; the last two rows apply only to few-shot node classification.

Method	Cora	CiteSeer	PubMed	CS	Photo	Computers	Memory (GB)	Params (M)
Pre-training time per epoch (s)
DGI	0.189	0.189	0.145	0.148	0.177	0.159	19.6	0.39
GRACE	0.747	1.063	0.532	0.560	0.813	0.739	12.9	0.39
BGRL	0.102	0.102	0.099	0.097	0.076	0.064	3.5	0.53
FUG	0.167	0.165	0.152	0.133	0.138	0.110	10.6	10.49
TIG	0.231	0.230	0.209	0.201	0.183	0.152	14.7	3.81

Appendix

Table 7: Computational cost. Top: pre-training time per epoch (s) with each graph held out as the target, peak GPU memory during pre-training, and trainable parameters. Bottom: forward time (ms) of the frozen encoder on each target graph and parameters at inference. All methods are measured on the same NVIDIA RTX 5090 GPU.

Dataset	Graphs	Nodes	Edges	Features	Classes	Domain
Node classification
Cora	1	2,708	5,278	1,433	7	Citation
CiteSeer	1	3,327	4,552	3,703	6	Citation
PubMed	1	19,717	44,324	500	3	Citation
CS	1	18,333	81,894	6,805	15	Co-authorship
Photo	1	7,650	119,081	745	8	Co-purchase

Appendix

Table 8: Statistics of the datasets. Edges are counted once per undirected pair; for graph classification, the numbers of nodes and edges are averaged over the graphs of each dataset.

Method	Code	Method	Code
DGI	github.com/PetarV-/DGI	GFT	github.com/Zehong-Wang/GFT
GRACE	github.com/CRIPAC-DIG/GRACE	MDGPT	anonymous.4open.science/r/MDGPT
BGRL	github.com/nerdslab/bgrl	SAMGPT	github.com/blue-soda/SAMGPT
FUG	github.com/hedongxiao-tju/FUG	MDGFM	github.com/wbkzwqtzw/MDGFM
TIG	github.com/hedongxiao-tju/TIG	BRIDGE	github.com/RingBDStack/BRIDGE

Appendix

Table 9: Code of the baselines.

Component	Specification
Operating system	Ubuntu 22.04.5 LTS, Linux kernel 6.8.0
CPU	Intel Xeon Silver 4410Y, 48 threads
Memory	128 GB
GPU	NVIDIA GeForce RTX 5090, 32 GB
Software	Python 3.10.12, PyTorch 2.11.0 with CUDA 12.8, PyTorch Geometric 2.8.0
	NumPy 2.2.6, scikit-learn 1.7.2

Appendix

Table 10: Experimental environment.

Method	Cora	CiteSeer	PubMed	CS	Photo	Computers	Avg.
DGI	33.09 ±5.51	33.79 ±5.60	47.10 ±6.45	71.82 ±6.36	56.42 ±8.15	42.48 ±9.90	47.45
GRACE	28.13 ±4.46	24.20 ±3.88	39.81 ±5.61	47.23 ±5.77	42.35 ±7.61	38.77 ±7.31	36.75
BGRL	34.19 ±4.42	25.04 ±3.75	43.37 ±5.66	57.58 ±5.84	60.32 ±10.14	50.95 ±11.17	45.24
FUG	46.40 ±8.37	36.07 ±6.81	52.23 ±9.47	78.02 ±5.88	63.62 ±9.77	51.27 ±12.82	54.60
TIG	48.88 ±8.80	44.71 ±9.02	50.94 ±10.00	75.28 ±6.22	63.52 ±10.47	49.47 ±13.08	55.47
GFT	27.25 ±4.78	26.78 ±4.31	45.36 ±6.00	63.49 ±6.42	20.01 ±4.37	14.62 ±5.12	32.92

Appendix

Table 11: Accuracy (%) of 1-shot node classification with standard deviations over the 500 tasks shared by all methods. The best and second-best results are in bold and underlined.

Method	Cora	CiteSeer	PubMed	CS	Photo	Computers	Avg.
DGI	45.56 ±5.08	44.96 ±4.42	55.58 ±5.12	85.26 ±2.08	70.17 ±5.23	56.98 ±7.27	59.75
GRACE	38.59 ±4.00	30.63 ±3.88	45.68 ±4.85	65.49 ±3.51	57.22 ±6.52	50.64 ±6.08	48.04
BGRL	46.74 ±4.09	31.66 ±3.58	51.06 ±5.51	75.12 ±3.29	72.63 ±7.23	62.05 ±8.43	56.54
FUG	61.63 ±5.42	47.62 ±4.94	60.82 ±6.32	86.55 ±1.70	65.91 ±8.90	58.85 ±10.17	63.56
TIG	58.53 ±6.17	56.39 ±5.03	56.48 ±7.81	83.63 ±3.04	68.24 ±10.30	56.31 ±11.03	63.26
GFT	37.84 ±4.25	35.49 ±3.97	51.90 ±5.65	78.95 ±2.74	22.25 ±4.74	17.49 ±6.30	40.65

Appendix

Table 12: Accuracy (%) of 3-shot node classification with standard deviations over the 500 tasks shared by all methods. The best and second-best results are in bold and underlined.

Method	Cora	CiteSeer	PubMed	CS	Photo	Computers	Avg.
DGI	53.07 ±3.79	50.55 ±3.54	60.34 ±4.47	87.77 ±1.20	74.10 ±4.33	63.16 ±4.49	64.83
GRACE	44.50 ±3.25	34.72 ±3.42	49.95 ±4.23	71.96 ±2.75	64.24 ±5.42	56.71 ±5.15	53.68
BGRL	53.28 ±3.37	35.74 ±3.11	56.07 ±4.68	81.12 ±2.22	77.83 ±5.18	67.13 ±6.08	61.86
FUG	67.31 ±3.23	53.28 ±3.86	64.51 ±4.64	88.13 ±1.13	66.29 ±7.50	62.83 ±8.18	67.06
TIG	63.70 ±4.59	60.66 ±3.45	59.64 ±6.58	86.62 ±1.98	69.20 ±8.48	60.80 ±8.49	66.77
GFT	44.13 ±3.73	40.85 ±3.38	56.09 ±4.75	82.69 ±1.62	24.92 ±4.83	19.76 ±6.37	44.74

Appendix

Table 13: Accuracy (%) of 5-shot node classification with standard deviations over the 500 tasks shared by all methods. The best and second-best results are in bold and underlined.

Figure 6: Accuracy (%) of node classification on each target graph at pre-training budgets of 300, 500, 1,000, 1,500, 2,000, 2,500, and 3,000 epochs. Shaded bands show the standard deviation over the three random splits of node classification, all panels share the same vertical span, and the circled point is the budget used in all other experiments.

Figure 7: Accuracy (%) of node classification after 2,000 pre-training epochs when the temperature

\tau

, the relation weight

\lambda

, the momentum

\mu

, the learning rate, or the hidden width of both branches is varied. Each row is a target graph, and the circled point is the default value.

Graph Foundation Models

Graph Foundation Models are pretrained graph-learning systems designed to transfer across datasets, tasks, and domains with different structures and features.

See all 96 papers

MDGMIX: Boundary-Aware Subgraph Mixing for Multi-Domain Graph Pre-Training
May 25, 2026Ziyu Zheng, Yaming Yang, Ziyu Guan +2Cross-Domain Generalization Cross-Domain Transfer Learning
School of Computer Science and Technology, Xidian University, Xi’an, China · School of Artificial Intelligence, Xidian University, Xi’an, China
Multi-Domain Riemannian Graph Gluing for Building Graph Foundation Models
Feb 28, 2026Li Sun, Zhenhao Huang, Silei Chen +4Graph Foundation Models Multi-Source Domain Adaptation
Beijing University of Posts and Telecommunications, Beijing 100876, China · North China Electric Power University, Beijing 102206, China · University of Illinois Chicago, IL, USA
Decoupled and Divergence-Conditioned Prompt for Multi-domain Dynamic Graph Foundation Models
May 13, 2026Haonan Yuan, Qingyun Sun, Junhua Shi +3Domain-Adaptive Pretraining Prompt Tuning
School of Computer Science and Engineering, Beihang University, Beijing 100191, China · Key Lab of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University, Guilin 541004, China · Department of Computer Science, University of Illinois at Chicago, Chicago 60607, USA

May 25, 2026J/K move · Enter open · S save

MDGMIX: Boundary-Aware Subgraph Mixing for Multi-Domain Graph Pre-Training

Ziyu Zheng, Yaming Yang, Ziyu Guan, Wei Zhao+1

School of Computer Science and Technology, Xidian University, Xi’an, China · School of Artificial Intelligence, Xidian University, Xi’an, China

Cross-Domain Generalization Cross-Domain Transfer Learning Graph Foundation Models

Open paper AskPDF arXiv

Multi-domain graph pre-training is a crucial step in constructing foundational graph models with cross-domain generalization capabilities. However, existing methods predominantly rely on jointly training all source domain graphs, resulting in high computational costs. Furthermore, it remains unclear whether all source domain graph data contribute equally to effective transfer. This paper empirically reveals significant data redundancy in multi-domain graph pre-training. Based on this finding, we propose the Multi-domain Graph Pre-training Framework, MDGMIX, which combines boundary-aware subgraph mixing with hierarchical discrimination. By selecting boundary nodes to construct challenging mixed-domain subgraphs, MDGMIX employs coarse-grained domain discrimination and fine-grained domain decomposition losses to decouple shared patterns from domain-specific patterns. During adaptation, MDGMIX employs a lightweight prompt weighting mechanism to transfer source domain knowledge. Extensive experiments demonstrate that MDGMIX consistently outperforms strong baselines in few-shot classification tasks while exhibiting superior time and memory efficiency. The code is available at: https://github.com/zhengziyu77/MDGMIX.

Abstract

Graph Foundation Models

Explore similar work

MDGMIX: Boundary-Aware Subgraph Mixing for Multi-Domain Graph Pre-Training

Multi-Domain Riemannian Graph Gluing for Building Graph Foundation Models

Decoupled and Divergence-Conditioned Prompt for Multi-domain Dynamic Graph Foundation Models