Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learning

Mateusz Praski; Jakub Adamczyk; Wojciech Czech

cs.LGAug 8, 2025

Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learning

Authors: Mateusz Praski, Jakub Adamczyk, Wojciech Czech

Organizations: Faculty of Computer Science, AGH University of Krakow, al. Mickiewicza 30, 30-059, Krakow, Poland

Abstract

Pretrained molecular embedding models spanning graph neural networks, text transformers, and hybrid architectures have been widely used in drug discovery, yet no large-scale, rigorous comparison of their representations existed. We evaluated 28 models across 25 classification datasets under a controlled framework using hierarchical Bayesian Bradley-Terry statistical testing. All neural models except CheMeleon failed to outperform the classical ECFP count fingerprint; CheMeleon was the only model rated as better. However, further testing on external validation datasets, based on scaffold cross-validation and a Bayesian hierarchical correlated t-test, revealed no definite advantage of either embedding method. As a practical takeaway, we recommend evaluating both ECFP fingerprints and CheMeleon embeddings in novel QSAR/QSPR studies. Overall, in the context of all neural networks evaluated, these results also challenge the perceived progress in molecular representation learning and highlight the need for rigorous evaluation standards in the field.

Figures & tables

Dataset	Reference	Endpoint
NICEATM Acute Oral Toxicity	[ 49 , 33 ]	Acute oral rat toxicity: non-toxic (0), intermediate (1), very toxic (2)
Rat carcinogenicity	[ 64 ]	In vivo classification as carcinogen or non-carcinogen
DILIrank 2.0	[ 52 ]	Drug-induced liver injury: none (0), low (1), high (2)
B3DB	[ 44 ]	Blood-brain barrier penetration, $\log BB$ threshold $-1$
BSEP	[ 43 ]	Bile salt export pump inhibition, $\mathrm{pIC}_{50}=4$ threshold
Mitochondrial toxicity	[ 23 ]	Toxicity to mitochondria

Table 1: Selected external datasets used for evaluation.

Model	Mean rank $\downarrow$	Mean AUROC $\uparrow$
CheMeleon	5.24	81.17%
R-MAT	6.52	80.91%
MolBERT	8.16	80.51%
ChemBERTa	8.68	80.01%
ECFP	8.80	79.95%
Mordred	8.92	80.22%

Table 2: Average model results, with ECFP marked.

BBT decision	Models
Better	CheMeleon
Practically equivalent	R-MAT, MolBERT, ChemBERTa, Mordred, CDDD, MAT, Atom Pair, MolE, MoLFormer
Worse	Mol2vec, CLAMP, TT, …, (all other models)

Table 3: Bayesian Bradley-Terry (BBT) model decisions for all models compared to the ECFP baseline, with ROPE

[0.35,0.65]

Figure 2: Critical difference diagram, models ordered by BBT posterior mean $\beta_{i}$ . Horizontal bars join groups of pairwise practically equivalent models.

Figure 3: Pairwise win ratio between all evaluated models. Green marks wins of the model on the Y axis against the model on the X axis, red means losses (i.e. the model from X axis won more).

Model	Number of datasets
CheMeleon	12
ECFP	8
MolBERT	8
Mordred	8
R-MAT	8

Table 4: Number of datasets on which a model achieved the highest AUROC or was within 0.01 AUROC of it, for the five models with the most such datasets.

Figure 4: Percentage of models outperforming ECFP (AUROC higher by more than 0.01) on each dataset.

Dataset	CheMeleon AUROC [%]	ECFP AUROC [%]
NICEATM Acute Oral Toxicity	80.69 $\pm$ 2.48	80.09 $\pm$ 2.37
Rat carcinogenicity	68.43 $\pm$ 3.83	67.35 $\pm$ 4.49
DILIrank 2.0	70.62 $\pm$ 3.11	70.83 $\pm$ 3.38
B3DB	91.41 $\pm$ 2.46	92.26 $\pm$ 2.38
BSEP	86.30 $\pm$ 2.93	84.92 $\pm$ 4.85
Mitochondrial toxicity	87.00 $\pm$ 2.08	83.78 $\pm$ 4.53

Table 5: Results of the post-hoc comparison on external validation datasets. Mean

\pm

standard deviation is reported across 5-fold scaffold cross-validation. For each dataset and model, the classification head with the best mean AUROC is used.

Figure 5: Per-dataset posterior of the mean AUROC difference (CheMeleon $-$ ECFP) from the hierarchical t-test, $\rho=0$ . Filled dots and bars show the posterior mean and 89% HDI after shrinkage, empty dots the raw mean difference over folds. The shaded area is the ROPE $[-0.01,0.01]$ .

$\rho$	$\mathbb{E}[\mu_{0}]$	89% HDI	P(ECFP)	P(equivalent)	P(CheMeleon)	Decision
0	+0.0073	[-0.0004, +0.0155]	0.005	0.662	0.333	Inconclusive
0.001	+0.0074	[-0.0008, +0.0151]	0.005	0.662	0.333	Inconclusive
0.01	+0.0073	[-0.0007, +0.0154]	0.005	0.666	0.329	Inconclusive
0.05	+0.0071	[-0.0012, +0.0156]	0.006	0.681	0.313	Inconclusive
0.1	+0.0069	[-0.0023, +0.0156]	0.007	0.679	0.314	Inconclusive
0.2	+0.0065	[-0.0033, +0.0162]	0.013	0.676	0.311	Inconclusive

Table 6: Results of the Bayesian hierarchical correlated t-test, CheMeleon against ECFP, for different values of the fold correlation

\rho

Panel of Figure 6 — Figure 6: Posterior predictive distribution of the CheMeleon $-$ ECFP AUROC difference on a new dataset, from the hierarchical correlated $t$ -test on the scaffold-split folds (ROPE $=[-0.01,0.01]$ ).

Figure 7: Embedding time of each evaluated model, using its representative variant: mean time to embed 10,000 molecules from MUV, with error bars indicating the standard deviation over batches. Orange marks models run on GPU.

Model	Device	Mean time [s] $\downarrow$	Std time [s]
CheMeleon	GPU	9.88	0.51
R-MAT	GPU	246.92	22.24
MolBERT	CPU	268.15	11.60
ChemBERTa	GPU	15.96	0.29
ECFP	CPU	0.77	0.08

Table 7: Embedding time of the 5 best models by mean rank (Table 2 ), using each model’s representative variant. MUV is embedded in batches of 10,000 molecules.

Molecular Representation Learning

Molecular representation learning encodes chemical structures and context for prediction, generation, and discovery across chemistry and biology.

See all 216 papers

On the limitations of molecular representations
Nov 5, 2026Acer Blake, Fergus Imrie, Charlotte M. DeaneMolecular Representation Learning
Department of Statistics, University of Oxford, Oxford, UK
Multi-View Molecular Representation Learning with Hierarchical Graphs and Contextualized Fingerprints
Sep 14, 2026Gwang-Hyeon Yun, Jong-Hoon Park, Bing Hu +3Graph SSL Hierarchical Representation Learning
Yonsei University · Wonju-si, Republic of Korea · University of Waterloo +1
Self-Supervised Pretraining of Molecular Graph Encoders with LeJEPA
Sep 2, 2026Michał Kulczykowski, Rafał ŁabędzkiSelf-Supervised Pre-Training Molecular Representation Learning
SGH Warsaw School of Economics

Nov 5, 2026J/K move · Enter open · S save

On the limitations of molecular representations

Acer Blake, Fergus Imrie, Charlotte M. Deane

Department of Statistics, University of Oxford, Oxford, UK

Molecular Representation Learning

Open paper AskPDF arXiv

Molecular representations such as fingerprints and learned embeddings play a central role in cheminformatics and small-molecule machine learning. They serve as the primary input for a diverse array of tasks, including clustering, dataset splitting, and virtual screening, yet their evaluation is often limited to downstream model performance on narrow benchmark tasks. We carried out a systematic investigation of the quality of molecular representations, finding that none of the state-of-the-art molecular embedding models capture graph structure, 3D shape, or ADMET endpoint similarity better than fingerprints or physicochemical descriptors. However, classical fingerprints also revealed their own material limitations. For example, we show that depending on the representation, between 225M and

\sim

4B molecules have non-unique representations in the 10.8B Enamine REAL database, with implications for virtual screening. Our findings provide an insight into the current limitations of molecular representations, outlining future research directions for the development of more robust encodings of small molecules.

Abstract

Molecular Representation Learning

Explore similar work

On the limitations of molecular representations

Multi-View Molecular Representation Learning with Hierarchical Graphs and Contextualized Fingerprints

Self-Supervised Pretraining of Molecular Graph Encoders with LeJEPA