Authors: Mohammad Mostafanejad, Paul Saxe, T. Daniel Crawford
Organizations: Department of Chemistry, Virginia Tech, Blacksburg, Virginia 24061, USA · Molecular Sciences Software Institute, Blacksburg, Virginia 24060, USA
Chemical language models (CLMs) have emerged as promising competitors to popular classical machine learning models for molecular property prediction (MPP) tasks. However, an increasing number of studies have reported inconsistent and contradictory results for the performance of CLMs across various MPP benchmark tasks. In this study, we conduct and analyze hundreds of meticulously controlled experiments to systematically investigate the effects of various factors, such as dataset size, model size, and standardization, on the pre-training and fine-tuning performance of CLMs for MPP. In the absence of well-established scaling laws for encoder-only masked language models, our aim is to provide comprehensive numerical evidence and a deeper understanding of the underlying mechanisms affecting the performance of CLMs for MPP tasks, some of which appear to be entirely overlooked in the literature.
Figures & tables
BERT
T-Loss ↓ c
V-Loss ↓ c
V-Acc ↑ c
V-wF1 ↑ c
V-PPPL ↓ c
Variant
Tiny
0.6511 ± 0.0090
0.4696 ± 0.0053
0.8823 ± 0.0077
0.8686 ± 0.0085
1.5993 ± 0.0085
Small
0.2353 ± 0.0009
0.1836 ± 0.0008
0.9453 ± 0.0109
0.9412 ± 0.0116
1.2016 ± 0.0010
Base b
0.1779 ± 0.0006
0.1359 ± 0.0011
0.9595 ± 0.0046
0.9564 ± 0.0041
1.1455 ± 0.0013
Table 1: Variations of the averaged pre-training performance metrics with 95% confidence intervals a for masked language modeling with respect to the model weight initialization and data sampling seeds
Figure 1: The effect of standardization noise on the pre-training validation weighted-F1 score (a,c,e) and pseudo-perplexity (b,d,f) of BERT for masked language modeling. Both metrics are averaged over three independent runs with different data sampling and model initialization seeds. τ and ν control the standardization noise in the pre-training and validation splits, respectively. The error bars are based on 95% confidence interval which are not defined (NaN) for samples of N=1 runs, if the other two have diverged and removed from calculating the statistical summaries.
Figure 2: Variations of BERT’s pre-training performance metrics: (a) validation loss (V-Loss), (b) accuracy (V-Acc), (c) weighted-F1 score (V-wF1), and (d) pseudo-perplexity (V-PPPL) versus the dataset size bin index, k (the corresponding data percentages are shown on the top axes). Results pertinent to the Tiny-, Small- and Base-BERT are shown in blue circles, red squares and green triangles, respectively. Shaded bands are used to indicate the 95% confidence intervals across three independent runs with different model initialization and data sampling random seeds.
Figure 3: Variations of the fine-tuned BERT’s testing performance metrics, (a) Pearson R , (b) R2 , (c) RMSE, (d) MAE, versus the pre-training dataset size bin index, k , for the HLM endpoint (the corresponding data percentages are shown on the top axes)
Figure 4: Variations of the fine-tuned BERT’s testing performance metrics, (a) Pearson R , (b) R2 , (c) RMSE, (d) MAE, versus the pre-training dataset size bin index, k , for the hPPB endpoint (the corresponding data percentages are shown on the top axes)
Figure 5: Variations of the fine-tuned BERT’s testing performance metrics, (a) Pearson R , (b) R2 , (c) RMSE, (d) MAE, versus the pre-training dataset size bin index, k , for the solubility endpoint (the corresponding data percentages are shown on the top axes)
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: The effect of standardization noise on the pre-training validation loss (a,c,e) and accuracy (b,d,f) of BERT for masked language modeling. Both metrics are averaged over three independent runs with different data sampling and model initialization seeds. The error bars are based on 95% confidence interval which are not defined (NaN) for samples of N=1 runs, if the other two have diverged and removed from the calculating the statistical summaries. See Supporting Information for more details.
Figure 2: Variations of BERT’s 3-fold cross-validation fine-tuning metrics, (a) Pearson R , (b) R2 , (c) RMSE, (d) MAE, versus the pre-training dataset size bin index, k , for the HLM endpoint (the corresponding data percentages are shown on the top axes). A hyperparameter search of 50 models is performed on each fold for each model variant and dataset size bin index and the best model is selected based on the validation R2 , following Ref. 12
Figure 3: Variations of BERT’s 3-fold cross-validation fine-tuning metrics, (a) Pearson R , (b) R2 , (c) RMSE, (d) MAE, versus the pre-training dataset size bin index, k , for the hPPB endpoint (the corresponding data percentages are shown on the top axes). A hyperparameter search of 50 models is performed on each fold for each model variant and dataset size bin index and the best model is selected based on the validation R2 , following Ref. 12
Figure 4: Variations of BERT’s 3-fold cross-validation fine-tuning metrics, (a) Pearson R , (b) R2 , (c) RMSE, (d) MAE, versus the pre-training dataset size bin index, k , for the solubility endpoint (the corresponding data percentages are shown on the top axes). A hyperparameter search of 50 models is performed on each fold for each model variant and dataset size bin index and the best model is selected based on the validation R2 , following Ref. 12
Figure 1: ChEMBL standardization protocol
Figure 2: PubChem standardization protocol
PubChem
ChEMBL
CC(=O)[O–]
CC ( – – O ) O
O=As([O–])O
O – – [ As ] ( O ) ( O ) O
[NH4+]
N
[CH3–]
C
CS ( C ) – – O
CS+[O–]
Appendix
Table 1: The SMILES representation of five different compounds standardized by PubChem and ChEMBL protocols
Figure 3: Corrupting the PubChem dataset with ChEMBL standardized SMILES
Models
MAE ↓ c
RMSE ↓ c
Pearson R ↑ c
R 2 ↑ c
Lasso Regression
0.403 ± 0.006 (0.407)
0.498 ± 0.007 (0.518)
0.602 ± 0.010 (0.567)
0.360 ± 0.012 (0.318)
Support Vector Machine
0.371 ± 0.004 (0.356)
0.466 ± 0.004 (0.452)
0.667 ± 0.010 (0.696)
0.441 ± 0.013 (0.479)
Random Forest
0.399 ± 0.006 (0.385)
0.492 ± 0.006 (0.483)
0.621 ± 0.011 (0.641)
0.378 ± 0.010 (0.406)
XGBoost
0.380 ± 0.009 (0.377)
0.482 ± 0.005 (0.489)
0.640 ± 0.013 (0.640)
0.402 ± 0.018 (0.392)
LightGBM
0.368 ± 0.008 (0.366)
0.466 ± 0.004 (0.472)
0.665 ± 0.024 (0.661)
0.440 ± 0.031 (0.434)
BERT-Tiny
0.441 ± 0.017 (0.450)
0.545 ± 0.009 (0.556)
0.495 ± 0.033 (0.470)
0.224 ± 0.042 (0.198)
Appendix
Table 0: Supervised fine-tuning performance metrics obtained via 3-fold cross-validation hyperparameter optimization on Biogen’s public ADME dataset with experimental HLM data as labels a,b
Models
MAE ↓ c
RMSE ↓ c
Pearson R ↑ c
R 2 ↑ c
Lasso Regression
0.442 ± 0.010 (0.440)
0.558 ± 0.011 (0.563)
0.676 ± 0.017 (0.680)
0.445 ± 0.021 (0.452)
Support Vector Machine
0.414 ± 0.004 (0.416)
0.523 ± 0.008 (0.535)
0.718 ± 0.011 (0.714)
0.512 ± 0.016 (0.505)
Random Forest
0.453 ± 0.008 (0.454)
0.562 ± 0.008 (0.568)
0.667 ± 0.020 (0.669)
0.437 ± 0.024 (0.442)
XGBoost
0.436 ± 0.005 (0.431)
0.548 ± 0.011 (0.553)
0.687 ± 0.016 (0.693)
0.464 ± 0.025 (0.471)
LightGBM
0.417 ± 0.006 (0.409)
0.527 ± 0.009 (0.532)
0.710 ± 0.015 (0.715)
0.504 ± 0.022 (0.511)
BERT-Tiny
0.498 ± 0.007 (0.499)
0.626 ± 0.011 (0.634)
0.560 ± 0.023 (0.590)
0.272 ± 0.037 (0.298)
Appendix
Table 1: Supervised fine-tuning performance metrics obtained via 3-fold cross-validation hyperparameter optimization on Biogen’s public ADME dataset with experimental RLM data as labels a,b
Models
MAE ↓ c
RMSE ↓ c
Pearson R ↑ c
R 2 ↑ c
Lasso Regression
0.387 ± 0.010 (0.410)
0.502 ± 0.013 (0.542)
0.721 ± 0.010 (0.686)
0.519 ± 0.015 (0.469)
Support Vector Machine
0.363 ± 0.014 (0.388)
0.479 ± 0.018 (0.536)
0.752 ± 0.010 (0.701)
0.562 ± 0.016 (0.481)
Random Forest
0.408 ± 0.015 (0.422)
0.523 ± 0.018 (0.548)
0.696 ± 0.019 (0.679)
0.477 ± 0.024 (0.458)
XGBoost
0.390 ± 0.005 (0.421)
0.509 ± 0.010 (0.547)
0.713 ± 0.006 (0.681)
0.505 ± 0.008 (0.459)
LightGBM
0.374 ± 0.010 ( 0.408 )
0.491 ± 0.015 ( 0.529 )
0.735 ± 0.008 ( 0.703 )
0.539 ± 0.011 ( 0.494 )
BERT-Tiny
0.464 ± 0.012 (0.494)
0.595 ± 0.017 (0.612)
0.573 ± 0.021 (0.564)
0.312 ± 0.015 (0.312)
Appendix
Table 2: Supervised fine-tuning performance metrics obtained via 3-fold cross-validation hyperparameter optimization on Biogen’s public ADME dataset with experimental hPPB data as labels a,b
Models
MAE ↓ c
RMSE ↓ c
Pearson R ↑ c
R 2 ↑ c
Lasso Regression
0.430 ± 0.021 (0.416)
0.565 ± 0.032 (0.534)
0.681 ± 0.033 (0.697)
0.453 ± 0.044 (0.474)
Support Vector Machine
0.411 ± 0.014 ( 0.381 )
0.550 ± 0.031 ( 0.512 )
0.696 ± 0.029 ( 0.720 )
0.481 ± 0.042 ( 0.516 )
Random Forest
0.450 ± 0.017 (0.420)
0.576 ± 0.030 (0.529)
0.667 ± 0.042 (0.702)
0.430 ± 0.044 (0.484)
XGBoost
0.430 ± 0.014 (0.425)
0.570 ± 0.028 (0.543)
0.668 ± 0.026 (0.681)
0.443 ± 0.036 (0.457)
LightGBM
0.427 ± 0.019 (0.417)
0.558 ± 0.026 (0.548)
0.684 ± 0.024 (0.671)
0.466 ± 0.033 (0.447)
BERT-Tiny
0.525 ± 0.026 (0.514)
0.649 ± 0.040 (0.620)
0.531 ± 0.054 (0.560)
0.259 ± 0.062 (0.287)
Appendix
Table 3: Supervised fine-tuning performance metrics obtained via 3-fold cross-validation hyperparameter optimization on Biogen’s public ADME dataset with experimental rPPB data as labels a,b
Models
MAE ↓ c
RMSE ↓ c
Pearson R ↑ c
R 2 ↑ c
Lasso Regression
0.429 ± 0.015 (0.416)
0.574 ± 0.019 (0.569)
0.551 ± 0.002 (0.535)
0.290 ± 0.008 (0.282)
Support Vector Machine
0.414 ± 0.020 (0.390)
0.565 ± 0.025 (0.551)
0.569 ± 0.010 (0.572)
0.313 ± 0.014 (0.326)
Random Forest
0.416 ± 0.010 (0.388)
0.575 ± 0.021 (0.558)
0.553 ± 0.011 (0.565)
0.289 ± 0.006 (0.310)
XGBoost
0.413 ± 0.005 (0.404)
0.584 ± 0.010 (0.593)
0.532 ± 0.020 (0.497)
0.264 ± 0.033 (0.221)
LightGBM
0.394 ± 0.008 (0.371)
0.563 ± 0.014 (0.549)
0.571 ± 0.026 (0.577)
0.316 ± 0.031 (0.331)
BERT-Tiny
0.424 ± 0.020 (0.403)
0.592 ± 0.029 (0.594)
0.490 ± 0.022 (0.467)
0.231 ± 0.019 (0.209)
Appendix
Table 4: Supervised fine-tuning performance metrics obtained via 3-fold cross-validation hyperparameter optimization on Biogen’s public ADME dataset with experimental solubility data as labels a,b
Models
MAE ↓ c
RMSE ↓ c
Pearson R ↑ c
R 2 ↑ c
Lasso Regression
0.373 ± 0.007 (0.383)
0.485 ± 0.008 (0.516)
0.708 ± 0.010 (0.677)
0.499 ± 0.014 (0.458)
Support Vector Machine
0.341 ± 0.008 (0.338)
0.457 ± 0.011 (0.476)
0.747 ± 0.013 (0.736)
0.555 ± 0.018 (0.539)
Random Forest
0.390 ± 0.007 (0.386)
0.502 ± 0.004 (0.502)
0.686 ± 0.008 (0.709)
0.462 ± 0.011 (0.488)
XGBoost
0.359 ± 0.005 (0.351)
0.480 ± 0.006 (0.482)
0.716 ± 0.010 (0.728)
0.510 ± 0.013 (0.527)
LightGBM
0.353 ± 0.005 (0.340)
0.469 ± 0.006 (0.471)
0.730 ± 0.007 (0.742)
0.531 ± 0.009 (0.549)
BERT-Tiny
0.408 ± 0.008 (0.402)
0.539 ± 0.016 (0.541)
0.622 ± 0.025 (0.640)
0.376 ± 0.035 (0.393)
Appendix
Table 5: Supervised fine-tuning performance metrics obtained via 3-fold cross-validation hyperparameter optimization on Biogen’s public ADME dataset with experimental MDR1-ER data as labels a,b
Table 6: Pre-defined search space for grid-search hyperparameter optimization of classical machine learning models performed over ADME dataset a
Parameter
Target Property
HLM
RLM
hPPB
rPPB
SOL
MDR1-ER
LASSO__alpha
0.005
0.001
0.005
0.005
0.005
0.005
Appendix
Table 7: Optimized parameters for the LASSO model obtained from grid-search hyperparameter optimization with 3-fold cross-validation using ADME dataset
Parameter
Target Property
HLM
RLM
hPPB
rPPB
SOL
MDR1-ER
SVM__C
5
5
10
5
5
5
SVM__epsilon
0.1
0.1
0.1
0.1
0.3
0.1
SVM__gamma
“scale”
“auto”
“auto”
“auto”
“auto”
“auto”
Appendix
Table 8: Optimized hyperparameters for the support vector machine model obtained from grid-search hyperparameter optimization with 3-fold cross-validation using ADME dataset
Parameter
Target Property
HLM
RLM
hPPB
rPPB
SOL
MDR1-ER
max_depth
15
40
25
25
None
None
max_features
0.67
None
0.67
0.67
0.33
0.67
n_estimators
750
500
1000
1000
500
1000
Appendix
Table 9: Optimized hyperparameters for the random forest model obtained from grid-search hyperparameter optimization with 3-fold cross-validation using ADME dataset
Parameter
Target Property
HLM
RLM
hPPB
rPPB
SOL
MDR1-ER
n_estimators (Round 1)
750
750
100
100
750
1000
max_depth (Round 2)
3
3
3
3
5
4
min_child_weight (Round 2)
1
2
2
2
2
1
gamma (Round 3)
0.0
0.0
0.0
0.0
0.0
0.1
colsample_bytree (Round 4)
0.8
0.9
0.7
0.9
0.6
0.6
Appendix
Table 10: Optimized hyperparameters for the XGBoost model obtained from grid-search hyperparameter optimization with 3-fold cross-validation using ADME dataset a,b
Parameter
Target Property
HLM
RLM
hPPB
rPPB
SOL
MDR1-ER
n_estimators (Round 1)
1000
750
1000
100
500
100
min_child_samples (Round 2)
40
10
20
10
10
20
num_leaves (Round 2)
75
15
15
15
15
15
colsample_bytree (Round 3)
0.9
1.0
0.6
0.8
1.0
0.8
subsample (Round 3)
0.8
0.9
0.9
0.8
0.9
1.0
Appendix
Table 11: Optimized hyperparameters for the LightGBM model obtained from grid-search hyperparameter optimization with 3-fold cross-validation using ADME dataset a,b
Model
Parameter
Target Property
HLM
RLM
hPPB
rPPB
SOL
MDR1-ER
Tiny-BERT
stack_depth
5
1
5
5
1
2
p
0.3
0.4
0.0
0.4
0.4
0.5
batch_size
128
256
128
64
64
128
learning_rate
5E-04
1E-04
5E-05
1E-04
5E-05
5E-05
weight_decay
0.05
0.00
0.01
0.1
0.0001
0.01
Appendix
Table 12: Optimized hyperparameters for different variants of BERT model obtained from conducting Bayesian search hyperparameter optimization with 3-fold cross-validation over ADME dataset
Instance
vCPU
GPU
On-demand
GPU
Instance
Instance
Name
Count
Types
Price ($/h) a
Memory (GB)
Memory (GB)
Storage (GB)
p5en.48xlarge
192
8 × H200
63.30
8 × 141 (HBM3e)
2048
8 × 3840 NVMe SSD
p4de.24xlarge
96
8 × A100
27.45
8 × 80 (HBM2e)
1152
8 × 1000 NVMe SSD
g6e.12xlarge
48
4 × L40S
10.49
4 × 48
384
2 × 1900 NVMe SSD
Appendix
Table 13: Estimated costs required for pre-training BERT models on the entire PubChem (version 4-18-2025) dataset
Model Details
BERT Variant
Tiny
Small
Base
Hidden Size
128
512
768
Hidden Activation
GELU
GELU
GELU
Vocabulary Size
30522
30522
30522
Max Position Embeddings
512
512
512
Intermediate Size
512
2048
3072
Appendix
Table 14: Architectural details of different BERT model variants
Chemical language models (CLMs) are pretrained on large corpora of molecules with the aim of learning representations that transfer to diverse downstream tasks. Here, we systematically study the effects of pretraining using a series of such CLMs across 64 molecular property prediction datasets. We find that the downstream gains of pretraining in CLMs do not scale and performance saturates below gradient-boosting baselines. Across the regimes evaluated, only supervised pretraining on traditional physico-chemical descriptors consistently improves downstream performance for the CLMs tested whereas some forms of pretraining result in models worse than a randomly initialized encoder. While models can resolve molecular structure well, they do so no better than a randomly initialized model, suggesting that, at best, CLMs learn descriptor-like properties in addition to low-level token statistics during pretraining, but surprisingly little chemistry beyond what classical molecular features already capture.
Chemical language models (CLMs) are trained with linearized representations such as SMILES, yet it remains unclear which chemically meaningful substructures they encode. To foster a better understanding of CLMs, we conduct a systematic study and probe for 78 molecular substructures across eight pre-trained and six randomly initialized models. We furthermore study how fine-tuning on chemical downstream tasks affects the learned representations of molecular substructures. Our results show that pre-training generally improves molecular structure awareness of CLMs, particularly in the upper layers. Moreover, randomly initialized models already encode ring structures well in the first layer. Our analysis on two chemical downstream tasks further reveals that, interestingly, fine-tuning affects task-relevant molecular substructures more than others, indicating that the changes in the representations follow chemical theory.
Anna Karnysheva, Dietrich Klakow, Ji-Ung Lee
1RTG Neuroexplicit Models · 2Spoken Language Systems · Saarland University +1
Molecular Property Prediction (MPP) is a fundamental problem in drug discovery that has recently attracted growing attention. Large Language Models (LLMs), known for their impressive proficiency across domains, show promise as generalist models for MPP. However, their current performance remains below the threshold needed for practical adoption. To bridge this gap, we propose TreeKD for distilling the knowledge of tree-based specialist models into LLMs to complement the internal knowledge of LLMs and improve their predictive accuracy. For each property, we train a specialist decision tree using features derived from 40K functional groups in the input molecules. Then, the predictive rule learned by the decision tree, which encodes its knowledge, is verbalized and incorporated into the prompts for training LLMs. In addition, by replacing a single decision tree with a Random Forest, we introduce a test-time scaling technique called rule-consistency, which aggregates predictions generated from different prompts constructed with different rules. An extensive evaluation with two LLMs, Gemma-2-2B and Granite-3.3-2B, on the TDC benchmark with 22 prediction tasks shows that our method substantially enhances the performance of LLMs, advancing the development of generalist models for MPP.
Khiem Le, Sreejata Dey, Marcos Martínez Galindo +4