cs.LGOct 13, 2026
SaveDoes Pretraining Teach Chemical Language Models Chemistry?
Organizations: Harvard University, Cambridge, MA, USA
Abstract
Chemical language models (CLMs) are pretrained on large corpora of molecules with the aim of learning representations that transfer to diverse downstream tasks. Here, we systematically study the effects of pretraining using a series of such CLMs across 64 molecular property prediction datasets. We find that the downstream gains of pretraining in CLMs do not scale and performance saturates below gradient-boosting baselines. Across the regimes evaluated, only supervised pretraining on traditional physico-chemical descriptors consistently improves downstream performance for the CLMs tested whereas some forms of pretraining result in models worse than a randomly initialized encoder. While models can resolve molecular structure well, they do so no better than a randomly initialized model, suggesting that, at best, CLMs learn descriptor-like properties in addition to low-level token statistics during pretraining, but surprisingly little chemistry beyond what classical molecular features already capture.
Figures & tables
Figure 1: Overall benchmarking performance. Mean rank across 64 datasets grouped into four task families. ( ) All CLIMB models use three seeds, except the 100M models at one seed.
Figure 2: Pretraining scaling. Downstream performance as a function of pretraining tokens processed. Pretraining is enlarged by sampling more unique molecules from the corpus with all other settings staying constant.
Figure 3: Performance against pretraining and downstream similarity. (a) Performance gain over a random encoder (“Lift”) for molecular structures that appear in the pretraining corpus (corpus-identical), as well as the top 25% most similar and bottom 25% least similar molecules. (b) Lift over a random encoder relative to the average nearest-neighbor distances across pretraining and downstream molecules. (c) Lift over random for different pairs of pretraining datasets and downstream tasks, see SI Sections A.3 and A for further details on datasets.
Figure 4: Corrupted pretraining objectives. Downstream lift defined as performance gains over a random encoder after corrupting supervised molecule-descriptor mappings (a) or progressively removing chemical structure from the unsupervised corpus (b).
Figure 5: What information do CLIMB embeddings carry? CLIMB embeddings were appended to either fingerprints, descriptors, or both and combinations were used as input to an XGBoost model to isolate how much additional information our embeddings carry. We report the improvements in prediction performance, downstream lift, relative to the respective feature set named on the x-axis.
Figure 6: Can representations resolve chemical edits? We tested ten chemical edits (1,000 molecule pairs each), where each parent is labeled 0 and every edited partner 1, giving 2,000 labeled vectors. An XGBoost model is tasked with predicting the labels. We report the area under the curve (AUC) of the receiver operating characteristic, where 0.5 would correspond to random predictions. Notation-only edits (SMILES enumeration, Kekulization) should score .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Setting |
|---|---|
| Backbone | ModernBERT (RoPE, pre-norm, GeGLU, global/local attention) |
| Layers / hidden size / heads | 12 / 512 / 8 |
| Feed-forward width (GeGLU) | 1,536 |
| Global attention | every 3 layers; local window 128 |
| Position encoding | rotary; max position 256 |
| Dropout (attn / mlp / embed) | 0 / 0 / 0 |
Table A.1: Encoder architecture and optimizer settings.
| Source | Objective | Labels | Molecules | Label distribution |
| Unsupervised corpora | ||||
| PubChem, filtered | MLM, MTR | — | 12,000,000 | — |
| PubChem, full | MLM, MTR | — | 123,400,000 | — |
| Computed targets | ||||
| RDKit descriptors | MTR | 217 | all of the above | continuous, -scored, clipped |
| Assay label families | ||||
Table A.2: Pretraining data.
| Dataset | Task type | Metric | Molecules | Actives |
|---|---|---|---|---|
| MoleculeACE | potency regression, 30 targets | macro RMSE | 48,714 | — |
| HIV | rare-active screen | NEF1% | 41,120 | 1,443 (3.51%) |
| BACE | binary classification | ROC-AUC | 1,513 | 691 (45.7%) |
| Tox21 | binary classification, 12 assays | mean ROC-AUC | 7,823 | 5,858 (7.52%) |
| QM7 | quantum regression | RMSE (kcal/mol) | 6,838 | — |
| Ames | binary classification | ROC-AUC | 7,278 | 53.3% a |
Table A.3: Panel of downstream datasets.
Figure C.1: Absolute errors on benchmarking suite. To show the absolute performance of various models rather than just relative rankings, we also report the average error per dataset type.
| topology and shape | substructure and electronic | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Representation | Chi | Kappa | BCUT | Ipc/SPS | EState | VSA | fr_* | Bulk | ALL |
| ECFP4 (2,048-bit) | 0.73 | 0.69 | 0.66 | 0.69 | 0.65 | 0.69 | 0.61 | 0.72 | 0.69 |
| CLIMB unsup | 0.86 | 0.79 | 0.73 | 0.75 | 0.56 | 0.62 | 0.33 | 0.70 | 0.59 |
| CLIMB sup, desc | 0.97 | 0.95 | 0.86 | 0.89 | 0.87 | 0.77 | 0.74 | 0.91 | 0.82 |
| CLIMB random | 0.42 | 0.34 | 0.53 | 0.42 | 0.31 | 0.34 | 0.16 | 0.41 | 0.34 |
Table D.1: Descriptor prediction from embeddings. Median of a ridge head fit from the frozen representation to each of 217 RDKit descriptors, on 1,940 held-out molecules. Targets are -scored with the pretraining corpus statistics and clipped at , matching the descriptor objective.
Figure E.1: Is end-to-end fine-tuning necessary? The same pretrained encoder used two ways at full downstream data: frozen with a trained probe, versus the whole network fine-tuned. Intervals are 95% resampling intervals. Dotted line: XGBoost on ECFP4+descriptors.
Figure F.1: The probe head is not neutral. The same frozen representations scored through an MLP head and through XGBoost, everything else fixed. The preference is representation-dependent and consistent across panels: the CLIMB encoders do better with the MLP head, while fingerprints and descriptors do better with XGBoost. In the main text, each representation is reported at its own best head in Fig. 1.
Figure G.1: Tokenizer family and vocabulary size. We compare two tokenizer types: byte-level BPE and Unigram-LM. The actual vocabulary sizes achieved on the pretraining corpus are: BPE ; Unigram . No discernible effect on downstream performance is found.
Figure H.1: Label efficiency. Absolute performance against the number of labeled training molecules (log scale), with an identical hold-out split, label fractions and seed grid across models.
Figure I.1: Canonical versus enumerated SMILES in pretraining. Canonical and enumerated pretraining are matched at five corpus fractions 3 pretraining seeds.
| measured lift | weight conditioning | attributable to | |||||
|---|---|---|---|---|---|---|---|
| Panel | real | wiki | real | wiki | generic | chemistry | Chemistry share |
| QM7 | 2.74 | 3.12 | — | ||||
| MoleculeACE | 0.83 | 1.44 | 19% | ||||
| HIV | 5.42 | 2.75 | 25% | ||||
| BACE | 0.78 | 0.97 | 83% | ||||
| Tox21 | 0.26 | 81% | |||||
Table J.1: Controlling for weight conditioning in unsupervised pretraining. We report the gain in performance over a randomly initialized encoder (lift). Weight conditioning refers to the weight-matched control models (mean of 3 seeds). Generic is gain attributable to the Wikipedia arm; chemistry is the gain due to using real SMILES; both with the effect of conditioning subtracted.