Pre-training with Graph Transformers

Jiaming Wang; Thomas Laurent; Xavier Bresson

cs.LGSep 12, 2026

Pre-training with Graph Transformers

Authors: Jiaming Wang, Thomas Laurent, Xavier Bresson

Organizations: National University of Singapore Singapore · Loyola Marymount University Los Angeles, USA

Abstract

This article investigates pre-training strategies for graph transformers in the biochemistry domain. By conducting comprehensive experiments, the study reveals that supervised pre-training using computed properties as labels provides the highest performance gain on downstream tasks. The results also highlight the importance of constraining model capacity to mitigate overfitting in graph transformers.

Figures & tables

	PRE-TRAINING METHODS			DOWNSTREAM TASKS
ID	GNN	Method	Dataset	CLINTOX	MUV	HIV	SIDER	BACE	TOX21	BBBP	TOXCAST	AVERAGE
1	GTL - LapPE	-	-	54.2	51.7	53.3	51.0	59.8	58.4	69.9	55.5	56.7
2	GTM - LapPE	-	-	55.8	52.2	56.3	56.2	69.8	61.2	72.6	57.1	60.2
3	GTL - RWPE	-	-	66.3	56.1	61.9	55.0	50.9	58.1	72.5	56.5	59.7
4	GTM - RWPE	-	-	65.1	51.8	59.6	54.1	67.0	63.6	72.3	61.1	61.8
5	GTL - NoPE	-	-	64.7	52.1	61.2	51.1	63.8	52.8	68.8	59.3	59.2

Table 1. Test ROC-AUC (%) performance on molecular prediction benchmarks using different pre-training strategies with GT and GIN. The rightmost column averages the mean of test performance across the 8 datasets. The best results for GT and GIN are bolded. The experiments are categorized as follows: experiments 1-8 involve direct training and evaluation of GNN models without any pre-training; experiments 9-13 utilize self-supervised pre-training; experiments 14-31 employ supervised pre-training; and experiments 32-41 utilize combined pre-training. For detailed explanations of the acronyms used, please refer to Section 2.3 .

Graph Transformer

Graph Transformers adapt self-attention to represent relationships, topology, and long-range dependencies in graph-structured data.

See all 101 papers

Toward a universal foundation model for graph-structured data
Apr 7, 2026Sakib Mostafa, Lei Xing, Md. Tauhidul IslamGraph Transfer Learning Graph Representation Learning
Department of Radiation Oncology, Stanford University, Stanford, CA, USA · Institute of Computational and Mathematical Engineering, Stanford University, Stanford, CA, USA · Department of Electrical Engineering, Stanford University, Stanford, CA, USA
Probabilistic Contrastive Pretraining for Multi-task ADME Property Prediction
Jun 9, 2026Yifan Xue, Srimukh Prasad Veccham, Saee Paliwal +2Contrastive Learning Self-Supervised Pre-Training
NVIDIA Santa Clara, CA 95051
Gaussian Process Limit Reveals Structural Benefits of Graph Transformers
Mar 18, 2026Nil Ayday, Lingchu Yang, Debarghya GhoshdastidarGNN Oversmoothing Graph Neural Networks
School of Computation, Information and Technology, Technical University of Munich · Graduate School of Information Science and Technology, The University of Tokyo

Apr 7, 2026J/K move · Enter open · S save

Toward a universal foundation model for graph-structured data

Sakib Mostafa, Lei Xing, Md. Tauhidul Islam

Department of Radiation Oncology, Stanford University, Stanford, CA, USA · Institute of Computational and Mathematical Engineering, Stanford University, Stanford, CA, USA · Department of Electrical Engineering, Stanford University, Stanford, CA, USA

Graph Transfer Learning Graph Representation Learning Graph Foundation Models Transfer Learning Prompt Engineering

Open paper AskPDF arXiv

Graphs are a central representation in biomedical research, capturing molecular interaction networks, gene regulatory circuits, cell--cell communication maps, and knowledge graphs. Despite their importance, currently there is not a broadly reusable foundation model available for graph analysis comparable to those that have transformed language and vision. Existing graph neural networks are typically trained on a single dataset and learn representations specific only to that graph's node features, topology, and label space, limiting their ability to transfer across domains. This lack of generalization is particularly problematic in biology and medicine, where networks vary substantially across cohorts, assays, and institutions. Here we introduce a graph foundation model designed to learn transferable structural representations that are not specific to specific node identities or feature schemes. Our approach leverages feature-agnostic graph properties, including degree statistics, centrality measures, community structure indicators, and diffusion-based signatures, and encodes them as structural prompts. These prompts are integrated with a message-passing backbone to embed diverse graphs into a shared representation space. The model is pretrained once on heterogeneous graphs and subsequently reused on unseen datasets with minimal adaptation. Across multiple benchmarks, our pretrained model matches or exceeds strong supervised baselines while demonstrating superior zero-shot and few-shot generalization on held-out graphs. On the SagePPI benchmark, supervised fine-tuning of the pretrained backbone achieves a mean ROC-AUC of 95.5%, a gain of 21.8% over the best supervised message-passing baseline. The proposed technique thus provides a unique approach toward reusable, foundation-scale models for graph-structured data in biomedical and network science applications.

Abstract

Graph Transformer

Explore similar work

Toward a universal foundation model for graph-structured data

Probabilistic Contrastive Pretraining for Multi-task ADME Property Prediction

Gaussian Process Limit Reveals Structural Benefits of Graph Transformers