BERTology of Molecular Property Prediction

Mohammad Mostafanejad; Paul Saxe; T. Daniel Crawford

cs.LGMar 13, 2026

BERTology of Molecular Property Prediction

Authors: Mohammad Mostafanejad, Paul Saxe, T. Daniel Crawford

Organizations: Department of Chemistry, Virginia Tech, Blacksburg, Virginia 24061, USA · Molecular Sciences Software Institute, Blacksburg, Virginia 24060, USA

Abstract

Chemical language models (CLMs) have emerged as promising competitors to popular classical machine learning models for molecular property prediction (MPP) tasks. However, an increasing number of studies have reported inconsistent and contradictory results for the performance of CLMs across various MPP benchmark tasks. In this study, we conduct and analyze hundreds of meticulously controlled experiments to systematically investigate the effects of various factors, such as dataset size, model size, and standardization, on the pre-training and fine-tuning performance of CLMs for MPP. In the absence of well-established scaling laws for encoder-only masked language models, our aim is to provide comprehensive numerical evidence and a deeper understanding of the underlying mechanisms affecting the performance of CLMs for MPP tasks, some of which appear to be entirely overlooked in the literature.

Figures & tables

BERT	T-Loss $\downarrow$ c	V-Loss $\downarrow$ c	V-Acc $\uparrow$ c	V-wF1 $\uparrow$ c	V-PPPL $\downarrow$ c
Variant	T-Loss $\downarrow$ c	V-Loss $\downarrow$ c	V-Acc $\uparrow$ c	V-wF1 $\uparrow$ c	V-PPPL $\downarrow$ c
Tiny	0.6511 $\pm$ 0.0090	0.4696 $\pm$ 0.0053	0.8823 $\pm$ 0.0077	0.8686 $\pm$ 0.0085	1.5993 $\pm$ 0.0085
Small	0.2353 $\pm$ 0.0009	0.1836 $\pm$ 0.0008	0.9453 $\pm$ 0.0109	0.9412 $\pm$ 0.0116	1.2016 $\pm$ 0.0010
Base b	0.1779 $\pm$ 0.0006	0.1359 $\pm$ 0.0011	0.9595 $\pm$ 0.0046	0.9564 $\pm$ 0.0041	1.1455 $\pm$ 0.0013

Table 1: Variations of the averaged pre-training performance metrics with 95% confidence intervals a for masked language modeling with respect to the model weight initialization and data sampling seeds

Panel of Figure 1 — Figure 1: The effect of standardization noise on the pre-training validation weighted-F1 score (a,c,e) and pseudo-perplexity (b,d,f) of BERT for masked language modeling. Both metrics are averaged over three independent runs with different data sampling and model initialization seeds. $\tau$ and $\nu$ control the standardization noise in the pre-training and validation splits, respectively. The error bars are based on 95% confidence interval which are not defined (NaN) for samples of $N=1$ runs, if the other two have diverged and removed from calculating the statistical summaries.

Figure 2: Variations of BERT’s pre-training performance metrics: (a) validation loss (V-Loss), (b) accuracy (V-Acc), (c) weighted-F1 score (V-wF1), and (d) pseudo-perplexity (V-PPPL) versus the dataset size bin index,

k

(the corresponding data percentages are shown on the top axes). Results pertinent to the Tiny-, Small- and Base-BERT are shown in blue circles, red squares and green triangles, respectively. Shaded bands are used to indicate the 95% confidence intervals across three independent runs with different model initialization and data sampling random seeds.

Figure 3: Variations of the fine-tuned BERT’s testing performance metrics, (a) Pearson

R

, (b)

R^{2}

, (c) RMSE, (d) MAE, versus the pre-training dataset size bin index,

k

, for the HLM endpoint (the corresponding data percentages are shown on the top axes)

Figure 4: Variations of the fine-tuned BERT’s testing performance metrics, (a) Pearson

R

, (b)

R^{2}

, (c) RMSE, (d) MAE, versus the pre-training dataset size bin index,

k

, for the hPPB endpoint (the corresponding data percentages are shown on the top axes)

Figure 5: Variations of the fine-tuned BERT’s testing performance metrics, (a) Pearson

R

, (b)

R^{2}

, (c) RMSE, (d) MAE, versus the pre-training dataset size bin index,

k

, for the solubility endpoint (the corresponding data percentages are shown on the top axes)

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Variations of BERT’s 3-fold cross-validation fine-tuning metrics, (a) Pearson

R

, (b)

R^{2}

, (c) RMSE, (d) MAE, versus the pre-training dataset size bin index,

k

, for the HLM endpoint (the corresponding data percentages are shown on the top axes). A hyperparameter search of 50 models is performed on each fold for each model variant and dataset size bin index and the best model is selected based on the validation

R^{2}

, following Ref. 12

Figure 3: Variations of BERT’s 3-fold cross-validation fine-tuning metrics, (a) Pearson

R

, (b)

R^{2}

, (c) RMSE, (d) MAE, versus the pre-training dataset size bin index,

k

, for the hPPB endpoint (the corresponding data percentages are shown on the top axes). A hyperparameter search of 50 models is performed on each fold for each model variant and dataset size bin index and the best model is selected based on the validation

R^{2}

, following Ref. 12

Figure 4: Variations of BERT’s 3-fold cross-validation fine-tuning metrics, (a) Pearson

R

, (b)

R^{2}

, (c) RMSE, (d) MAE, versus the pre-training dataset size bin index,

k

, for the solubility endpoint (the corresponding data percentages are shown on the top axes). A hyperparameter search of 50 models is performed on each fold for each model variant and dataset size bin index and the best model is selected based on the validation

R^{2}

, following Ref. 12

Figure 1: ChEMBL standardization protocol

Figure 2: PubChem standardization protocol

PubChem	ChEMBL
CC(=O)[O–]	CC ( – – O ) O
O=As([O–])O	O – – [ As ] ( O ) ( O ) O
[NH4+]	N
[CH3–]	C
CS ( C ) – – O	CS+[O–]

Appendix

Table 1: The SMILES representation of five different compounds standardized by PubChem and ChEMBL protocols

Figure 3: Corrupting the PubChem dataset with ChEMBL standardized SMILES

Models	MAE $\downarrow$ c	RMSE $\downarrow$ c	Pearson R $\uparrow$ c	R 2 $\uparrow$ c
Lasso Regression	0.403 $\pm$ 0.006 (0.407)	0.498 $\pm$ 0.007 (0.518)	0.602 $\pm$ 0.010 (0.567)	0.360 $\pm$ 0.012 (0.318)
Support Vector Machine	0.371 $\pm$ 0.004 (0.356)	0.466 $\pm$ 0.004 (0.452)	0.667 $\pm$ 0.010 (0.696)	0.441 $\pm$ 0.013 (0.479)
Random Forest	0.399 $\pm$ 0.006 (0.385)	0.492 $\pm$ 0.006 (0.483)	0.621 $\pm$ 0.011 (0.641)	0.378 $\pm$ 0.010 (0.406)
XGBoost	0.380 $\pm$ 0.009 (0.377)	0.482 $\pm$ 0.005 (0.489)	0.640 $\pm$ 0.013 (0.640)	0.402 $\pm$ 0.018 (0.392)
LightGBM	0.368 $\pm$ 0.008 (0.366)	0.466 $\pm$ 0.004 (0.472)	0.665 $\pm$ 0.024 (0.661)	0.440 $\pm$ 0.031 (0.434)
BERT-Tiny	0.441 $\pm$ 0.017 (0.450)	0.545 $\pm$ 0.009 (0.556)	0.495 $\pm$ 0.033 (0.470)	0.224 $\pm$ 0.042 (0.198)

Appendix

Table 0: Supervised fine-tuning performance metrics obtained via 3-fold cross-validation hyperparameter optimization on Biogen’s public ADME dataset with experimental HLM data as labels a,b

Models	MAE $\downarrow$ c	RMSE $\downarrow$ c	Pearson R $\uparrow$ c	R 2 $\uparrow$ c
Lasso Regression	0.442 $\pm$ 0.010 (0.440)	0.558 $\pm$ 0.011 (0.563)	0.676 $\pm$ 0.017 (0.680)	0.445 $\pm$ 0.021 (0.452)
Support Vector Machine	0.414 $\pm$ 0.004 (0.416)	0.523 $\pm$ 0.008 (0.535)	0.718 $\pm$ 0.011 (0.714)	0.512 $\pm$ 0.016 (0.505)
Random Forest	0.453 $\pm$ 0.008 (0.454)	0.562 $\pm$ 0.008 (0.568)	0.667 $\pm$ 0.020 (0.669)	0.437 $\pm$ 0.024 (0.442)
XGBoost	0.436 $\pm$ 0.005 (0.431)	0.548 $\pm$ 0.011 (0.553)	0.687 $\pm$ 0.016 (0.693)	0.464 $\pm$ 0.025 (0.471)
LightGBM	0.417 $\pm$ 0.006 (0.409)	0.527 $\pm$ 0.009 (0.532)	0.710 $\pm$ 0.015 (0.715)	0.504 $\pm$ 0.022 (0.511)
BERT-Tiny	0.498 $\pm$ 0.007 (0.499)	0.626 $\pm$ 0.011 (0.634)	0.560 $\pm$ 0.023 (0.590)	0.272 $\pm$ 0.037 (0.298)

Appendix

Table 1: Supervised fine-tuning performance metrics obtained via 3-fold cross-validation hyperparameter optimization on Biogen’s public ADME dataset with experimental RLM data as labels a,b

Models	MAE $\downarrow$ c	RMSE $\downarrow$ c	Pearson R $\uparrow$ c	R 2 $\uparrow$ c
Lasso Regression	0.387 $\pm$ 0.010 (0.410)	0.502 $\pm$ 0.013 (0.542)	0.721 $\pm$ 0.010 (0.686)	0.519 $\pm$ 0.015 (0.469)
Support Vector Machine	0.363 $\pm$ 0.014 (0.388)	0.479 $\pm$ 0.018 (0.536)	0.752 $\pm$ 0.010 (0.701)	0.562 $\pm$ 0.016 (0.481)
Random Forest	0.408 $\pm$ 0.015 (0.422)	0.523 $\pm$ 0.018 (0.548)	0.696 $\pm$ 0.019 (0.679)	0.477 $\pm$ 0.024 (0.458)
XGBoost	0.390 $\pm$ 0.005 (0.421)	0.509 $\pm$ 0.010 (0.547)	0.713 $\pm$ 0.006 (0.681)	0.505 $\pm$ 0.008 (0.459)
LightGBM	0.374 $\pm$ 0.010 ( 0.408 )	0.491 $\pm$ 0.015 ( 0.529 )	0.735 $\pm$ 0.008 ( 0.703 )	0.539 $\pm$ 0.011 ( 0.494 )
BERT-Tiny	0.464 $\pm$ 0.012 (0.494)	0.595 $\pm$ 0.017 (0.612)	0.573 $\pm$ 0.021 (0.564)	0.312 $\pm$ 0.015 (0.312)

Appendix

Table 2: Supervised fine-tuning performance metrics obtained via 3-fold cross-validation hyperparameter optimization on Biogen’s public ADME dataset with experimental hPPB data as labels a,b

Models	MAE $\downarrow$ c	RMSE $\downarrow$ c	Pearson R $\uparrow$ c	R 2 $\uparrow$ c
Lasso Regression	0.430 $\pm$ 0.021 (0.416)	0.565 $\pm$ 0.032 (0.534)	0.681 $\pm$ 0.033 (0.697)	0.453 $\pm$ 0.044 (0.474)
Support Vector Machine	0.411 $\pm$ 0.014 ( 0.381 )	0.550 $\pm$ 0.031 ( 0.512 )	0.696 $\pm$ 0.029 ( 0.720 )	0.481 $\pm$ 0.042 ( 0.516 )
Random Forest	0.450 $\pm$ 0.017 (0.420)	0.576 $\pm$ 0.030 (0.529)	0.667 $\pm$ 0.042 (0.702)	0.430 $\pm$ 0.044 (0.484)
XGBoost	0.430 $\pm$ 0.014 (0.425)	0.570 $\pm$ 0.028 (0.543)	0.668 $\pm$ 0.026 (0.681)	0.443 $\pm$ 0.036 (0.457)
LightGBM	0.427 $\pm$ 0.019 (0.417)	0.558 $\pm$ 0.026 (0.548)	0.684 $\pm$ 0.024 (0.671)	0.466 $\pm$ 0.033 (0.447)
BERT-Tiny	0.525 $\pm$ 0.026 (0.514)	0.649 $\pm$ 0.040 (0.620)	0.531 $\pm$ 0.054 (0.560)	0.259 $\pm$ 0.062 (0.287)

Appendix

Table 3: Supervised fine-tuning performance metrics obtained via 3-fold cross-validation hyperparameter optimization on Biogen’s public ADME dataset with experimental rPPB data as labels a,b

Models	MAE $\downarrow$ c	RMSE $\downarrow$ c	Pearson R $\uparrow$ c	R 2 $\uparrow$ c
Lasso Regression	0.429 $\pm$ 0.015 (0.416)	0.574 $\pm$ 0.019 (0.569)	0.551 $\pm$ 0.002 (0.535)	0.290 $\pm$ 0.008 (0.282)
Support Vector Machine	0.414 $\pm$ 0.020 (0.390)	0.565 $\pm$ 0.025 (0.551)	0.569 $\pm$ 0.010 (0.572)	0.313 $\pm$ 0.014 (0.326)
Random Forest	0.416 $\pm$ 0.010 (0.388)	0.575 $\pm$ 0.021 (0.558)	0.553 $\pm$ 0.011 (0.565)	0.289 $\pm$ 0.006 (0.310)
XGBoost	0.413 $\pm$ 0.005 (0.404)	0.584 $\pm$ 0.010 (0.593)	0.532 $\pm$ 0.020 (0.497)	0.264 $\pm$ 0.033 (0.221)
LightGBM	0.394 $\pm$ 0.008 (0.371)	0.563 $\pm$ 0.014 (0.549)	0.571 $\pm$ 0.026 (0.577)	0.316 $\pm$ 0.031 (0.331)
BERT-Tiny	0.424 $\pm$ 0.020 (0.403)	0.592 $\pm$ 0.029 (0.594)	0.490 $\pm$ 0.022 (0.467)	0.231 $\pm$ 0.019 (0.209)

Appendix

Table 4: Supervised fine-tuning performance metrics obtained via 3-fold cross-validation hyperparameter optimization on Biogen’s public ADME dataset with experimental solubility data as labels a,b

Models	MAE $\downarrow$ c	RMSE $\downarrow$ c	Pearson R $\uparrow$ c	R 2 $\uparrow$ c
Lasso Regression	0.373 $\pm$ 0.007 (0.383)	0.485 $\pm$ 0.008 (0.516)	0.708 $\pm$ 0.010 (0.677)	0.499 $\pm$ 0.014 (0.458)
Support Vector Machine	0.341 $\pm$ 0.008 (0.338)	0.457 $\pm$ 0.011 (0.476)	0.747 $\pm$ 0.013 (0.736)	0.555 $\pm$ 0.018 (0.539)
Random Forest	0.390 $\pm$ 0.007 (0.386)	0.502 $\pm$ 0.004 (0.502)	0.686 $\pm$ 0.008 (0.709)	0.462 $\pm$ 0.011 (0.488)
XGBoost	0.359 $\pm$ 0.005 (0.351)	0.480 $\pm$ 0.006 (0.482)	0.716 $\pm$ 0.010 (0.728)	0.510 $\pm$ 0.013 (0.527)
LightGBM	0.353 $\pm$ 0.005 (0.340)	0.469 $\pm$ 0.006 (0.471)	0.730 $\pm$ 0.007 (0.742)	0.531 $\pm$ 0.009 (0.549)
BERT-Tiny	0.408 $\pm$ 0.008 (0.402)	0.539 $\pm$ 0.016 (0.541)	0.622 $\pm$ 0.025 (0.640)	0.376 $\pm$ 0.035 (0.393)

Appendix

Table 5: Supervised fine-tuning performance metrics obtained via 3-fold cross-validation hyperparameter optimization on Biogen’s public ADME dataset with experimental MDR1-ER data as labels a,b

Model	Parameter	Allowed Values
BERT	stack_depth	$\{$ 0, 1, 2, 3, 4, 5 $\}$
	p	$\{$ 0.0, 0.1, 0.2, 0.3, 0.4, 0.5 $\}$
	batch_size	$\{$ 8, 16, 32, 64, 128, 256, 512 $\}$
	learning_rate	$\{$ 1E-06, 1E-05, 5E-05, 1E-04, 5E-04, 1E-03 $\}$
	weight_decay	$\{$ 0.0, 1E-04, 5E-04, 1E-03, 5E-03, 1E-02, 5E-02, 1E-01 $\}$
	freeze_base_model	$\{$ True, False $\}$

Appendix

Table 6: Pre-defined search space for grid-search hyperparameter optimization of classical machine learning models performed over ADME dataset a

Parameter	Target Property
Parameter	HLM	RLM	hPPB	rPPB	SOL	MDR1-ER
LASSO__alpha	0.005	0.001	0.005	0.005	0.005	0.005

Appendix

Table 7: Optimized parameters for the LASSO model obtained from grid-search hyperparameter optimization with 3-fold cross-validation using ADME dataset

Parameter	Target Property
Parameter	HLM	RLM	hPPB	rPPB	SOL	MDR1-ER
SVM__C	5	5	10	5	5	5
SVM__epsilon	0.1	0.1	0.1	0.1	0.3	0.1
SVM__gamma	“scale”	“auto”	“auto”	“auto”	“auto”	“auto”

Appendix

Table 8: Optimized hyperparameters for the support vector machine model obtained from grid-search hyperparameter optimization with 3-fold cross-validation using ADME dataset

Parameter	Target Property
Parameter	HLM	RLM	hPPB	rPPB	SOL	MDR1-ER
max_depth	15	40	25	25	None	None
max_features	0.67	None	0.67	0.67	0.33	0.67
n_estimators	750	500	1000	1000	500	1000

Appendix

Table 9: Optimized hyperparameters for the random forest model obtained from grid-search hyperparameter optimization with 3-fold cross-validation using ADME dataset

Parameter	Target Property
Parameter	HLM	RLM	hPPB	rPPB	SOL	MDR1-ER
n_estimators (Round 1)	750	750	100	100	750	1000
max_depth (Round 2)	3	3	3	3	5	4
min_child_weight (Round 2)	1	2	2	2	2	1
gamma (Round 3)	0.0	0.0	0.0	0.0	0.0	0.1
colsample_bytree (Round 4)	0.8	0.9	0.7	0.9	0.6	0.6

Appendix

Table 10: Optimized hyperparameters for the XGBoost model obtained from grid-search hyperparameter optimization with 3-fold cross-validation using ADME dataset a,b

Parameter	Target Property
Parameter	HLM	RLM	hPPB	rPPB	SOL	MDR1-ER
n_estimators (Round 1)	1000	750	1000	100	500	100
min_child_samples (Round 2)	40	10	20	10	10	20
num_leaves (Round 2)	75	15	15	15	15	15
colsample_bytree (Round 3)	0.9	1.0	0.6	0.8	1.0	0.8
subsample (Round 3)	0.8	0.9	0.9	0.8	0.9	1.0

Appendix

Table 11: Optimized hyperparameters for the LightGBM model obtained from grid-search hyperparameter optimization with 3-fold cross-validation using ADME dataset a,b

Model	Parameter	Target Property
Model	Parameter	HLM	RLM	hPPB	rPPB	SOL	MDR1-ER
Tiny-BERT	stack_depth	5	1	5	5	1	2
	p	0.3	0.4	0.0	0.4	0.4	0.5
	batch_size	128	256	128	64	64	128
	learning_rate	5E-04	1E-04	5E-05	1E-04	5E-05	5E-05
	weight_decay	0.05	0.00	0.01	0.1	0.0001	0.01

Appendix

Table 12: Optimized hyperparameters for different variants of BERT model obtained from conducting Bayesian search hyperparameter optimization with 3-fold cross-validation over ADME dataset

Instance	vCPU	GPU	On-demand	GPU	Instance	Instance
Name	Count	Types	Price ($/h) a	Memory (GB)	Memory (GB)	Storage (GB)
p5en.48xlarge	192	8 $\times$ H200	63.30	8 $\times$ 141 (HBM3e)	2048	8 $\times$ 3840 NVMe SSD
p4de.24xlarge	96	8 $\times$ A100	27.45	8 $\times$ 80 (HBM2e)	1152	8 $\times$ 1000 NVMe SSD
g6e.12xlarge	48	4 $\times$ L40S	10.49	4 $\times$ 48	384	2 $\times$ 1900 NVMe SSD

Appendix

Table 13: Estimated costs required for pre-training BERT models on the entire PubChem (version 4-18-2025) dataset

Model Details	BERT Variant
Model Details	Tiny	Small	Base
Hidden Size	128	512	768
Hidden Activation	GELU	GELU	GELU
Vocabulary Size	30522	30522	30522
Max Position Embeddings	512	512	512
Intermediate Size	512	2048	3072

Appendix

Table 14: Architectural details of different BERT model variants

Chemical Language Models

See all 8 papers

Does Pretraining Teach Chemical Language Models Chemistry?
Oct 13, 2026Leif Sieben, Yoel ZimmermannChemical Language Models Language Model Pretraining
Harvard University, Cambridge, MA, USA
Probing Chemical Language Models: Effects of Pre-training and Fine-tuning
Jul 2, 2026Anna Karnysheva, Dietrich Klakow, Ji-Ung LeeLanguage Model Pretraining Chemical Language Models
1RTG Neuroexplicit Models · 2Spoken Language Systems · Saarland University +1
Can Decision Trees Teach Large Language Models? Distilling Verbalized Knowledge for Molecular Property Prediction
Mar 12, 2026Khiem Le, Sreejata Dey, Marcos Martínez Galindo +4Molecular Property Prediction Knowledge Distillation
University of Notre Dame, IN, USA · IBM Research

Oct 13, 2026J/K move · Enter open · S save

Does Pretraining Teach Chemical Language Models Chemistry?

Leif Sieben, Yoel Zimmermann

Harvard University, Cambridge, MA, USA

Chemical Language Models Language Model Pretraining Molecular Representation Learning Language Modeling Molecular Property Prediction

Open paper AskPDF arXiv

Chemical language models (CLMs) are pretrained on large corpora of molecules with the aim of learning representations that transfer to diverse downstream tasks. Here, we systematically study the effects of pretraining using a series of such CLMs across 64 molecular property prediction datasets. We find that the downstream gains of pretraining in CLMs do not scale and performance saturates below gradient-boosting baselines. Across the regimes evaluated, only supervised pretraining on traditional physico-chemical descriptors consistently improves downstream performance for the CLMs tested whereas some forms of pretraining result in models worse than a randomly initialized encoder. While models can resolve molecular structure well, they do so no better than a randomly initialized model, suggesting that, at best, CLMs learn descriptor-like properties in addition to low-level token statistics during pretraining, but surprisingly little chemistry beyond what classical molecular features already capture.

Abstract

Chemical Language Models

Explore similar work

Does Pretraining Teach Chemical Language Models Chemistry?

Probing Chemical Language Models: Effects of Pre-training and Fine-tuning

Can Decision Trees Teach Large Language Models? Distilling Verbalized Knowledge for Molecular Property Prediction