BiScale-GTR: Fragment-Aware Graph Transformers for Multi-Scale Molecular Representation Learning

Yi Yang; Ovidiu Daescu

cs.LGApr 7, 2026

BiScale-GTR: Fragment-Aware Graph Transformers for Multi-Scale Molecular Representation Learning

Authors: Yi Yang, Ovidiu Daescu

Organizations: Department of Computer Science, The University of Texas at Dallas

Abstract

Fragment-level representations provide a natural way to capture recurring molecular substructures and reuse their learned representations across molecules. However, a shared fragment identity alone may not fully describe how a fragment is instantiated in a particular molecule, since the same fragment can exhibit different chemical behavior depending on its surrounding atomic environment. Effective fragment-based molecular learning therefore requires representations that are both reusable across molecules and sensitive to local atomic context. We introduce BiScale-GTR, a self-supervised molecular representation framework built around context-grounded shared fragment tokens. BiScale-GTR constructs a reusable graph Byte Pair Encoding (graph-BPE) vocabulary using Weisfeiler-Lehman (WL)-based fragment identity, chemical validity filtering, and recursive out-of-vocabulary (OOV) decomposition. Each shared fragment token is then grounded with atom-level GNN representations through atom-to-fragment pooling and gated fusion, allowing the same fragment identity to acquire context-dependent representations in different molecular environments. A structure-aware fragment Transformer performs global reasoning over these atom-grounded tokens, capturing reusable substructure identity, local chemical context, and long-range molecular dependencies. Experiments on MoleculeNet, PharmaBench, and the Long Range Graph Benchmark demonstrate strong performance across classification and regression tasks. Attribution analysis further shows that BiScale-GTR highlights chemically meaningful recurring motifs, providing interpretable links between molecular structure and predicted properties.

Figures & tables

Figure 1: BiScale-GTR combines atom-level GNN encoding and fragment-level token representations. Multi-scale fusion integrates atom and fragment embeddings, which are modeled by a structure-aware fragment Transformer for molecular property prediction.

Figure 2: Overview of Graph BPE vocabulary construction and tokenization.

Model	Pretrain Data Size	BBBP	Tox21	MUV	BACE	ToxCast	SIDER	HIV
GraphMVP ( Liu et al., 2021 )	50k	70.8 $\pm$ 0.5	74.9 $\pm$ 0.8	77.7 $\pm$ 0.6	79.3 $\pm$ 1.5	63.1 $\pm$ 0.2	60.2 $\pm$ 1.1	76.0 $\pm$ 0.1
GraphMAE ( Hou et al., 2022 )	2M	71.2 $\pm$ 1.0	75.2 $\pm$ 0.9	76.4 $\pm$ 2.0	78.2 $\pm$ 1.5	63.6 $\pm$ 0.3	60.5 $\pm$ 1.2	76.8 $\pm$ 0.6
Mole-BERT ( Xia et al., 2023 )	2M	71.9 $\pm$ 0.8	76.8 $\pm$ 0.5	78.9 $\pm$ 1.8	80.8 $\pm$ 1.4	64.3 $\pm$ 0.2	62.8 $\pm$ 1.1	78.2 $\pm$ 0.8
SimSGT ( Liu et al., 2023 )	2M	72.2 $\pm$ 0.9	76.8 $\pm$ 0.9	81.4 $\pm$ 1.4	84.3 $\pm$ 0.6	65.9 $\pm$ 0.8	61.7 $\pm$ 0.8	78.0 $\pm$ 1.9
GraphFP ( Luong and Singh, 2023 )	430k	72.0 $\pm$ 1.7	74.0 $\pm$ 0.7	75.4 $\pm$ 1.9	80.5 $\pm$ 1.8	63.9 $\pm$ 0.9	63.6 $\pm$ 1.2	78.0 $\pm$ 1.5
MORE ( Son et al., 2025 )	2M	71.9 $\pm$ 0.9	75.6 $\pm$ 0.5	–	82.8 $\pm$ 1.3	64.6 $\pm$ 0.6	60.9 $\pm$ 0.6	77.0 $\pm$ 0.7

Table 1: ROC-AUC (%) comparison on MoleculeNet biological classification tasks. Results are reported as mean

\pm

standard deviation when available. Baseline results are taken directly from the corresponding original papers using the same data split protocol. Bold indicates the best result and underlined values denote the second-best performance.

Model	CYP2C9	CYP2D6	CYP3A4	HLMC	MLMC	RLMC	LogD	PPB	Sol
RF ( Rigatti, 2017 )	18.471	18.041	16.540	0.813	0.987	0.958	1.249	0.204	0.918
XGBoost ( Chen and Guestrin, 2016 )	17.582	17.819	16.123	0.647	0.844	0.819	1.071	0.186	0.832
CMPNN ( Swanson, 2019 )	18.377	19.156	16.701	0.921	1.130	0.939	0.807	0.236	0.858
FPGNN ( Cai et al., 2022 )	16.933	17.611	15.606	0.604	0.774	0.716	0.838	0.179	0.747
DHTNN ( Song et al., 2023 )	17.449	17.890	16.156	0.729	0.926	0.915	0.912	0.235	0.828
KANO ( Li et al., 2023 )	17.350	17.622	15.307	0.554	0.767	0.762	0.766	0.185	0.772

Table 2: Performance comparison on PharmaBench regression tasks. Results are reported as RMSE (

\downarrow

). Baseline results are taken directly from PharmaBench paper ( Niu et al., 2024 )

Model	Peptides-func (AP $\uparrow$ )	Peptides-struct (MAE $\downarrow$ )
GCN ( Kipf and Welling, 2016 )	0.5930 $\pm$ 0.0023	0.3496 $\pm$ 0.0013
GCNII ( Chen et al., 2020 )	0.5543 $\pm$ 0.0078	0.3471 $\pm$ 0.0010
GINE ( Hu et al., 2019 )	0.5498 $\pm$ 0.0079	0.3547 $\pm$ 0.0045
GatedGCN ( Bresson and Laurent, 2017 )	0.5864 $\pm$ 0.0077	0.3420 $\pm$ 0.0013
GatedGCN+RWSE ( Dwivedi et al., 2022 )	0.6069 $\pm$ 0.0035	0.3357 $\pm$ 0.0006
Transformer ( Vaswani et al., 2017 ) + LapPE ( Dwivedi et al., 2023 )	0.6326 $\pm$ 0.0126	0.2529 $\pm$ 0.0016

Table 3: Performance comparison on peptide benchmarks. Average Precision (AP

\uparrow

) is reported for Peptides-func and Mean Absolute Error (MAE

\downarrow

) for Peptides-struct. Results are shown as mean

\pm

standard deviation. Baseline results are taken directly from the LRGB benchmark

Variant	BBBP	Tox21	MUV	BACE	ToxCast	SIDER	HIV
Full model (GNN + Transformer)	68.4 $\pm$ 0.8	76.1 $\pm$ 0.4	81.6 $\pm$ 0.9	85.0 $\pm$ 1.1	62.2 $\pm$ 0.2	64.2 $\pm$ 0.9	79.2 $\pm$ 0.6
Transformer-only (w/o GNN)	62.8 $\pm$ 0.8	73.2 $\pm$ 0.5	71.6 $\pm$ 0.8	65.0 $\pm$ 0.9	61.1 $\pm$ 0.7	59.8 $\pm$ 0.5	77.1 $\pm$ 0.4
GNN-only (w/o Transformer)	58.7 $\pm$ 0.3	51.9 $\pm$ 0.5	51.8 $\pm$ 0.3	59.2 $\pm$ 1.1	50.2 $\pm$ 0.3	53.3 $\pm$ 0.4	51.0 $\pm$ 0.5
Fixed sum fusion	67.4 $\pm$ 0.8	75.1 $\pm$ 0.6	78.1 $\pm$ 0.5	77.0 $\pm$ 1.0	60.8 $\pm$ 0.3	62.2 $\pm$ 1.1	78.4 $\pm$ 0.5

Table 4: Architecture and fusion ablation on MoleculeNet datasets (ROC-AUC %). The fixed-sum variant replaces the adaptive gate with

z_{k}=e_{k}+\tilde{h}_{k}

Tokenizer	BBBP	Tox21	MUV	BACE	ToxCast	SIDER	HIV	Peptide-func
Atom	52.6 $\pm$ 10.2	74.5 $\pm$ 0.3	76.7 $\pm$ 1.8	71.5 $\pm$ 13.6	61.9 $\pm$ 0.2	55.1 $\pm$ 1.1	75.4 $\pm$ 0.2	42.2 $\pm$ 0.03
BRICS	68.0 $\pm$ 1.4	75.0 $\pm$ 0.3	73.0 $\pm$ 1.7	79.1 $\pm$ 2.2	62.0 $\pm$ 0.1	62.7 $\pm$ 0.4	75.1 $\pm$ 0.6	62.5 $\pm$ 0.01
Graph-BPE (SMILES)	63.4 $\pm$ 0.7	73.4 $\pm$ 0.1	76.1 $\pm$ 0.5	79.3 $\pm$ 2.5	60.9 $\pm$ 0.6	60.1 $\pm$ 0.7	76.4 $\pm$ 1.1	58.8 $\pm$ 0.03
Graph-BPE + WL (Ours)	68.4 $\pm$ 0.8	76.1 $\pm$ 0.4	81.6 $\pm$ 0.9	85.0 $\pm$ 1.1	62.2 $\pm$ 0.2	64.2 $\pm$ 0.9	79.2 $\pm$ 0.6	67.2 $\pm$ 0.01

Table 5: Tokenizer ablation on MoleculeNet and Peptide-func benchmarks. All models use the same BiScale-GTR architecture, pretraining corpus, and optimization settings; only the tokenizer differs. Results are reported as ROC-AUC (%) for MoleculeNet datasets and AP (%) for Peptide-func, averaged over three random seeds.

Variant	BBBP	Tox21	MUV	BACE	ToxCast	SIDER	HIV
w/o pretraining	59.8 $\pm$ 0.3	70.5 $\pm$ 0.8	59.9 $\pm$ 0.6	66.2 $\pm$ 1.3	59.8 $\pm$ 0.3	59.6 $\pm$ 0.7	74.6 $\pm$ 0.4
mask ratio = 0.1	69.3 $\pm$ 0.5	74.3 $\pm$ 0.7	77.9 $\pm$ 0.5	83.2 $\pm$ 0.8	61.3 $\pm$ 0.4	62.1 $\pm$ 0.6	77.9 $\pm$ 0.3
mask ratio = 0.2 (Full model)	68.4 $\pm$ 0.8	76.1 $\pm$ 0.4	81.6 $\pm$ 0.9	85.0 $\pm$ 1.1	62.2 $\pm$ 0.2	64.2 $\pm$ 0.9	79.2 $\pm$ 0.6
mask ratio = 0.3	66.9 $\pm$ 0.4	75.2 $\pm$ 0.6	76.8 $\pm$ 0.6	81.0 $\pm$ 1.4	61.9 $\pm$ 0.6	62.3 $\pm$ 0.4	75.7 $\pm$ 0.4

Table 6: Effect of pretraining and masking ratio (ROC-AUC %).

Dataset	Metric	$\Delta_{\text{top}}$	$\Delta_{\text{bottom}}$	Gap ( $\Delta_{\text{top}}-\Delta_{\text{bottom}}$ )
HIV	ROC-AUC $\downarrow$	28.9	10.2	18.7
Tox21	ROC-AUC $\downarrow$	11.1	4.1	7.0
BACE	ROC-AUC $\downarrow$	29.8	11.2	18.6
BBBP	ROC-AUC $\downarrow$	37.9	10	27.9

Table 7: Faithfulness evaluation via fragment masking.

\Delta_{\text{top}}

measures the ROC-AUC drop (%) after removing top-

3

most important fragments, while

\Delta_{\text{bottom}}

measures drop (%) after removing bottom-

3

fragments.

Panel of Figure 3 — Figure 3: Fragment-level attention attribution on representative molecules from the HIV and Tox21 datasets. Fragment colors show rank-normalized importance within each molecule based on CLS attention rollout scores. Circled substructures correspond to the highest-attribution learned fragment tokens selected by CLS attention rollout. Additional details are provided in Appendix A.13

Panel of Figure 4 — Figure 4: t-SNE visualization of fragment embeddings learned by the Transformer without (left) and with (right) the GNN encoder. Each point represents a fragment, and colors denote clusters derived from ECFP fingerprints. Fragments with similar embeddings are positioned closer together in the 2D t-SNE space. NMI measures the agreement between the embedding distribution and the ECFP-based clustering.

Panel of Figure 5 — Figure 5: Top-3 relative ROC-AUC drop across datasets, computed using Eq. 5

Model	Within-token spread ( $\downarrow$ )	Centroid separation ( $\uparrow$ )
BiScale-GTR (Molecule)	0.3627	0.4052
BiScale-GTR (Fragment)	0.2595	0.5549

Table 8: Token-space analysis on the BBBP test set. We report the average within-token spread and average centroid separation of the final contextualized token representations. Lower within-token spread and higher centroid separation indicate a sharper and more discriminative token space.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Size	Task Type	Description
CYP2C9	999	Regression	Binding affinity to CYP2C9
CYP2D6	1,214	Regression	Binding affinity to CYP2D6
CYP3A4	1,980	Regression	Binding affinity to CYP3A4
HLMC	2,286	Regression	Human liver microsomal clearance
MLMC	1,403	Regression	Mouse liver microsomal clearance
RLMC	1,129	Regression	Rat liver microsomal clearance

Appendix

Table 9: Dataset profiles for PharmaBench benchmarks.

Dataset	Size	Tasks	Pos : Neg Ratio	Description
BBBP	2,039	1	$\sim$ 0.8 : 1	Blood-brain barrier permeability
Tox21	7,831	12	varies 1–5 : 1	Toxicity on 12 biological targets
ToxCast	8,575	617	widely variable	High-throughput toxicity screening
SIDER	1,427	27	1–15 : 1	Adverse drug reactions
MUV	93,087	17	$>$ 500 : 1	Virtual screening validation
HIV	41,127	1	$\approx$ 11 : 1	HIV replication inhibition

Appendix

Table 10: Dataset profiles for MoleculeNet classification benchmarks. Pos:Neg ratios indicate approximate class imbalance in each dataset.

	Peptides-func	Peptides-struct
Number of Graphs	15,535	15,535
Number of Tasks	1	5
Number of Classes	10	–
Task Type	Multi-label classification	Multi-label regression
Description	Peptides-func predicts peptide biological activities such as antibacterial and antiviral functions. Peptides-struct predicts global structural properties derived from peptide 3D conformations, including descriptors such as length and sphericity.

Appendix

Table 11: The classification and regression tasks in the LRGB benchmark.

Benchmark	Batch Size	Weight Decay	Dropout	Head LR	Backbone LR	Scheduler (factor, patience)
MoleculeNet	64	$5\times 10^{-5}$	0.10, 0.20	$2\times 10^{-4}$	$5\times 10^{-5}$	None
PharmaBench	256	$1\times 10^{-5}$	0.10	$1\times 10^{-3}$	$1\times 10^{-3}$	Plateau (0.8, 6)
Long-range Peptides	128	$1\times 10^{-5}$	0.05	$3\times 10^{-4}$	$2\times 10^{-4}$	Plateau (0.5, 20)

Appendix

Table 12: Fine-tuning hyperparameters for different benchmark groups. LR denotes learning rate.

Stage	Cost	Notes
Vocabulary construction, 800 fragments	22 min wall-clock	One-time offline preprocessing
Pretraining	8 GPU-hours	503k optimization steps
Fine-tuning	0.33–2.5 GPU-hours per dataset	Varies with dataset size and early stopping
Inference	$\sim$ 15–25 ms/molecule	Tokenization + forward pass

Appendix

Table 13: Empirical computational cost of BiScale-GTR. GPU stages are measured on a single NVIDIA RTX 4090.

MoleculeNet
Dataset	Fallback Rate	UNK Rate
BACE	0.1248	0.0000
BBBP	0.0525	0.0022
Tox21	0.0742	0.0033
ToxCast	0.0867	0.0089
SIDER	0.1153	0.0067

Appendix

Table 14: Tokenizer statistics across downstream datasets.

Figure 6: Cumulative coverage of fragment occurrences when fragments are sorted by corpus frequency. A small number of fragments accounts for the majority of occurrences, indicating that the learned vocabulary captures common structural motifs.

Figure 7: Frequency-weighted distribution of fragment sizes measured by the number of atoms per fragment. Curves correspond to vocabularies learned from different corpus sizes of the same dataset (10k, 50k, 100k molecules), while orig denotes the full ChEMBL corpus used for vocabulary construction (430k molecules). Across corpus sizes, most fragments remain small substructures, while larger fragments correspond to recurring chemical motifs captured by the vocabulary.

Vocabulary Size	Valid Fragments	Functional Group Categories
400	331	25
800	677	27
1600	1374	28

Appendix

Table 15: Chemical diversity statistics for different vocabulary sizes.

Vocabulary Size	Tox21 ROC-AUC	HIV ROC-AUC	Peptides-func AP
400	74.9 $\pm$ 0.3	78.0. $\pm$ 0.8	0.6571 $\pm$ 0.0139
800	76.1 $\pm$ 0.4	79.2 $\pm$ 0.6	0.6717 $\pm$ 0.0107
1600	76.3 $\pm$ 0.3	78.9 $\pm$ 0.4	0.6721 $\pm$ 0.0201

Appendix

Table 16: Downstream performance for different vocabulary sizes.

Method	Peptides-func (AP)	Peptides-struct (MAE)
Graph BPE (without validity/OOV handling)	0.6321	0.2929
Graph BPE (with validity/OOV handling)	0.6717	0.2621

Appendix

Table 17: Effect of tokenization refinements on long-range molecular datasets. Removing the validity filtering and OOV fallback mechanisms leads to degraded performance.

Variant	BBBP	Tox21	MUV	BACE	ToxCast	SIDER	HIV
Full model	68.4 $\pm$ 0.8	76.1 $\pm$ 0.4	81.6 $\pm$ 0.9	85.0 $\pm$ 1.1	62.2 $\pm$ 0.9	64.2 $\pm$ 0.9	79.2 $\pm$ 0.6
w/o $B_{adj}$	67.4 $\pm$ 0.8	75.2 $\pm$ 0.4	79.8 $\pm$ 0.7	83.3 $\pm$ 0.9	61.5 $\pm$ 0.3	63.1 $\pm$ 0.8	78.3 $\pm$ 0.4
w/o $B_{dist}$	67.0 $\pm$ 0.7	74.1 $\pm$ 0.6	79.1 $\pm$ 0.7	82.1 $\pm$ 0.9	61.3 $\pm$ 0.5	62.2 $\pm$ 0.7	77.1 $\pm$ 0.5
w/o $B_{bond}$	68.3 $\pm$ 0.9	75.9 $\pm$ 0.5	80.9 $\pm$ 0.8	84.1 $\pm$ 1.2	62.1 $\pm$ 0.8	63.4 $\pm$ 0.8	78.4 $\pm$ 0.4
w/o all biases	65.8 $\pm$ 0.8	73.8 $\pm$ 0.8	77.9 $\pm$ 0.8	80.6 $\pm$ 0.8	60.5 $\pm$ 0.4	61.4 $\pm$ 0.8	77.6 $\pm$ 0.6

Appendix

Table 18: Ablation of structure-aware attention biases.

Variant	BBBP	Tox21	MUV	BACE	ToxCast	SIDER	HIV
3-layer GNN (Full model)	68.4 $\pm$ 0.8	76.1 $\pm$ 0.4	81.6 $\pm$ 0.9	85.0 $\pm$ 1.1	62.2 $\pm$ 0.2	64.2 $\pm$ 0.9	79.2 $\pm$ 0.6
6-layer GNN	62.9 $\pm$ 0.9	76.2 $\pm$ 0.3	78.2 $\pm$ 0.7	81.7 $\pm$ 0.8	61.3 $\pm$ 0.3	59.8 $\pm$ 0.6	79.1 $\pm$ 0.4
8-layer GNN	60.2 $\pm$ 0.7	75.3 $\pm$ 0.6	77.8 $\pm$ 0.8	79.1 $\pm$ 0.9	61.1 $\pm$ 0.5	59.6 $\pm$ 0.4	78.3 $\pm$ 0.7

Appendix

Table 19: Effect of GNN depth (ROC-AUC %).

Mask Ratio	MFP Accuracy (%) $\uparrow$	MFP Loss $\downarrow$
10%	76.8	0.8740
20%	70.2	1.037
30%	70.9	0.9991

Appendix

Table 20: Effect of masking ratio on the masked fragment prediction (MFP) pretraining objective.

Variant	BBBP	Tox21	MUV	BACE	ToxCast	SIDER	HIV
Frequency-aware masking (Full)	68.4	76.1	81.6	85.0	62.2	64.2	79.2
Uniform masking	67.2	75.3	80.1	83.1	61.9	63.7	77.4

Appendix

Table 21: Effect of masking strategy during pretraining. The full model uses the proposed frequency-aware masking strategy, while Uniform replaces it with standard random masking. Results for the Uniform baseline are averaged over three independent runs.

Feature	Description
Max valence	Maximum valence of the atom as determined by RDKit
Bond order sum	Sum of bond orders of all bonds connected to the atom
Remaining valence	Difference between maximum valence and bond order sum
Aromatic indicator	Binary flag indicating whether the atom is aromatic

Appendix

Table 22: Atom-level constraint features used by the GNN encoder.

Explore similar work

Hierarchical Molecular Representation Learning via Fragment-Based Self-Supervised Embedding Prediction
Feb 23, 2026Jiele Wu, Haozhe Ma, Zhihan Guo +2Molecular Representation Learning Graph Representation Learning
School of Computing, National University of Singapore, Singapore · Computer Science and Engineering, The Chinese University of Hong Kong, China
Multi-View Molecular Representation Learning with Hierarchical Graphs and Contextualized Fingerprints
Sep 14, 2026Gwang-Hyeon Yun, Jong-Hoon Park, Bing Hu +3Graph SSL Hierarchical Representation Learning
Yonsei University · Wonju-si, Republic of Korea · University of Waterloo +1
Multi-level Self-supervised Pretraining on Compositional Hierarchical Graph for Molecular Property Prediction
May 15, 2026Xiayu Liu, Zhengyi Lu, Hou-biao LiSelf-Supervised Pre-Training Hierarchical Representation Learning
School of Mathematical Sciences, University of Electronic Science and Technology of China, No.2006, Xiyuan Avenue, West Hi-tech Zone,2026 611731, Chengdu, China · Department of Computer Science and Engineering, Oakland University, 201 Meadow Brook RD, Rochester Hills, 48309, MI, USA

Feb 23, 2026J/K move · Enter open · S save

Hierarchical Molecular Representation Learning via Fragment-Based Self-Supervised Embedding Prediction

Jiele Wu, Haozhe Ma, Zhihan Guo, Thanh Vinh Vo+1

School of Computing, National University of Singapore, Singapore · Computer Science and Engineering, The Chinese University of Hong Kong, China

Molecular Representation Learning Graph Representation Learning Graph SSL Hierarchical Representation Learning

Open paper AskPDF arXiv

Graph self-supervised learning (GSSL) has demonstrated strong potential for generating expressive graph embeddings without the need for human annotations, making it particularly valuable in domains with high labeling costs such as molecular graph analysis. However, existing GSSL methods mostly focus on node- or edge-level information, often ignoring chemically relevant substructures which strongly influence molecular properties. In this work, we propose Graph Semantic Predictive Network (GraSPNet), a hierarchical self-supervised framework that explicitly models both atomic-level and fragment-level semantics. GraSPNet decomposes molecular graphs into chemically meaningful fragments without predefined vocabularies and learns node- and fragment-level representations through multi-level message passing with masked semantic prediction at both levels. This hierarchical semantic supervision enables GraSPNet to learn multi-resolution structural information that is both expressive and transferable. Extensive experiments on multiple molecular property prediction benchmarks demonstrate that GraSPNet learns chemically meaningful representations and consistently outperforms state-of-the-art GSSL methods in transfer learning settings.

Abstract

Explore similar work

Hierarchical Molecular Representation Learning via Fragment-Based Self-Supervised Embedding Prediction

Multi-View Molecular Representation Learning with Hierarchical Graphs and Contextualized Fingerprints

Multi-level Self-supervised Pretraining on Compositional Hierarchical Graph for Molecular Property Prediction