A Large Scale Investigation of Scaling Limits in Chemical Language Models

Roshan Balaji; Kamran Chitsaz; Quentin Fournier; Nirav Pravinbhai Bhatt; Sarath Chandar

cs.LGAug 19, 2025

A Large Scale Investigation of Scaling Limits in Chemical Language Models

Authors: Roshan Balaji, Kamran Chitsaz, Quentin Fournier, Nirav Pravinbhai Bhatt, Sarath Chandar

Organizations: BioSystems Engineering and Control Lab · Wadhwani School of Data Science and AI, IIT Madras · The Centre for Integrative Biology and Systems medicinE (IBSE) · Chandar Research Lab · Mila – Quebec AI Institute · IIT Madras Zanzibar · Polytechnique Montréal · Canada CIFAR AI Chair

Abstract

Chemical Language Models (CLMs) are increasingly used in de novo drug design, driven by recent growth in model scale, compute, and dataset size. However, the relationship between design choices, training dynamics, and downstream generation quality remains poorly understood. We present a compute-controlled scaling study of CLMs comprising more than 30,000 experiments across molecular representations (SMILES, SELFIES, SAFE), tokenizations (atom-level and byte-pair encoding), model scales (0.5M-1B parameters), leakage-controlled datasets (MOSES, ChEMBL, PubChem, ZINC-22), and architectures (decoder-only and encoder-decoder). By fitting IsoFLOP profiles, we establish clear scaling trends in pretraining loss, but find that these improvements do not translate into comparable gains in goal-directed molecular design. Layer-wise probing and sparse autoencoder analysis reveal continued development of chemical representations: chemical syntax saturates early, while semantic properties emerge more slowly and become increasingly accessible with further training and model scale. These representational gains coexist with diminishing improvements in goal-directed generation under the evaluated protocols and oracle budgets. Our resulting suite of models, NovoMolGen, achieves state-of-the-art results, outperforming prior CLMs and specialized generative models in goal-directed molecular generation across drug discovery tasks. These findings expose a disconnect between chemical representation learning and downstream molecular design, motivating the development of pretraining paradigms that more directly learn chemical semantics.

Figures & tables

Figure 1: Pretraining and internal representations improve with compute, while goal-directed generation shows diminishing returns. Metrics against pretraining compute

C=6ND

for models from 0.5M to 1B parameters. (a) Next-token accuracy rises with scale. (b, e) After RL fine-tuning, the total top 5% docking score over five targets (

\downarrow

) and the total PMO score over 22 tasks show only small gains beyond

\sim

5M. (c) Middle-layer linear-probe

R^{2}

, averaged over 36 property targets, keeps rising. (d) Validation set bits per molecule decrease steadily as compute increases. (f) SAE monosemanticity, the mean over features of the largest absolute property correlation, keeps rising.

Figure 2: Chemical sequence modelling follows compute-optimal scaling. Validation set bits per molecule (

\downarrow

) of the PubChem–SMILES–atomwise decoder-only models. (a) IsoFLOP profiles at the five compute budgets of our IsoFLOP grid, evaluated at each budget’s bracketing model sizes; curves and points come from a parametric loss surface fit to all training trajectories, and stars mark the loss-minimising size. (b) Compute-optimal parameters and training tokens follow power laws in compute, with shaded bootstrap intervals. (c) Compute-optimal loss on the random (in-distribution) and maximum-dissimilarity (out-of-distribution) splits, with colour and marker denoting model size.

Figure 3: Representation and corpus change the loss far more than tokenisation or architecture. Validation set bits per molecule (

\downarrow

) at

D=20N

, varying one design axis at a time around the PubChem–SMILES–atomwise decoder-only reference. (a) SMILES is lowest at every scale. (b) Byte-pair tokenisation is slightly worse than atom-level tokenisation at small scale and lower from 32M onwards. (c) T5Gemma matches the decoder-only model at 32M and trails it at larger scales. (d) Corpora are scored on their own validation splits, so their ordering reflects corpus diversity.

Figure 4: Out-Of-Distribution (OOD) generalisation improves with scale but depends on representation and corpus. The OOD gap is the relative increase in bits per molecule from the random to the maximum-dissimilarity split (

\downarrow

). (a) Validation set loss of the reference configuration on each split. (b) OOD gap for each design axis. (c) The OOD gap for each corpus, measured on its own splits, narrows with scale on PubChem and ZINC-22 but widens on MOSES and ChEMBL. (d) FCD between 30,000 molecules sampled from each 32M model and each split.

Figure 5: Chemical syntax is learned rapidly, while semantic properties emerge later and more slowly. Linear probes and SAEs on the active set activations. Validity is the accuracy of separating valid molecules from corrupted sequences, and the other properties report probe

R^{2}

. (a) Middle-layer probe score from 0.5M to 1B parameters. (b) Probe gain over the randomly initialised network as a fraction of the gain at the final checkpoint (mean over layers), against training tokens expressed for the 32M model. (c) Probe score by depth at 32M. (d) Probe score at 32M across design axes. (e) SAE monosemanticity of the 32M model during training by depth. (f) Validation set F1 of linear probes on SAE latents and on residual-stream neurons for ten common functional groups at 32M.

Figure 6: NovoMolGen outperforms strong baselines on goal-directed generation from 5M parameters, with little gain from further scale. Rows compare baselines (grey) with NovoMolGen-SMILES models from 0.5M to 1B parameters. (a) PMO AUC top-10 per task with 10,000 oracle calls ( $\uparrow$ ). (b) Total PMO score over the 22 tasks. (c) Top-5% docking score of novel molecules per target with 3,000 oracle calls, in kcal/mol. (d) Total top 5% docking score over the five targets in kcal/mol ( $\downarrow$ ). Baseline results are taken from Lee et al. (2024a) .

Appendix figures & tables40 assets

Supplementary material from the paper’s appendix.

Appendix

Corpus	Molecules	Median heavy atoms	Median SMILES tokens	Character
MOSES	1.9M	21	28	Narrow, drug-like, restricted atoms
ChEMBL	1.9M	27	48	Bioactive, literature-derived
PubChem	100M	24	47	Broad drug-like (reference corpus)
ZINC-22	1.5B (sampled)	29	52	Make-on-demand, largest coverage

Appendix

Table B1: Pretraining corpora. Token counts use the atom-level SMILES tokeniser. Sizes are reported after standardisation and leakage filtering.

Axis	Variants	Model sizes
Reference	PubChem, SMILES, atom-level, Qwen3	0.5M, 5M, 32M, 157M, 300M, 1B
Representation	SELFIES, SAFE	0.5M, 5M, 32M
Tokenisation	Byte-pair encoding	0.5M, 5M, 32M, 157M, 300M, 1B
Architecture	T5Gemma (encoder–decoder)	0.5M, 5M, 32M, 157M, 300M, 1B
Pretraining corpus	MOSES, ChEMBL, ZINC-22	0.5M, 5M, 32M, 157M, 300M, 1B

Appendix

Table B2: Experimental design grid. Each axis is varied around the PubChem–SMILES–atomwise Qwen3 reference at matched model size and training tokens.

Figure B1: Diversity of the data across batches. The left plot shows the distribution of molecular lengths, tokenized using an atomwise tokenizer, indicating variation in molecule length within each batch. The right plot shows Tanimoto similarity between molecule pairs, demonstrating structural diversity in each batch.

Figure B2: Molecular-property distributions of the four pretraining corpora. Each corpus is represented by 6,000 molecules sampled uniformly across all of its shards, with properties computed by RDKit on the largest fragment. (a–d) Kernel density estimates of QED, logP, molecular weight and TPSA. (e, f) Fraction of molecules by ring count and number of rotatable bonds, with the last bin including all larger values.

Panel of Figure 11 — Figure B3: Molecular-property distributions for the random and scaffold reference splits (175,000 molecules each) and a generated set from NovoMolGen-SMILES-BPE-32M (100,000 molecules). The properties include QED, SA, logP, MW, TPSA, Bertz complexity, number of rotatable bonds, and number of rings.

Components	NovoMolGen-32M	NovoMolGen-157M	NovoMolGen-300M
Attention Heads	8	10	12
Hidden Layers	12	24	32
Hidden Size	512	640	768
Intermediate Size	1024	2560	3072

Appendix

Table C1: NovoMolGen Configurations

Figure D1: Training curves for loss and fraction of variance explained for NovoMolGen-SMILES-BPE-32M at checkpoint 75000 and layer 6.

Hyperparameter	Value	Description
Dictionary Multiplier	$16\times$	Expansion factor ( $16\times 512=8192$ features)
Sparsity ( $k$ )	$\approx 2\%$	Top- $k$ active features allowed
Optimizer	Adam	Standard configuration
Learning Rate	$3\times 10^{-4}$	Linear warmup over first 2,500 steps
Batch Size (SAE)	4,096	Activation vectors per optimization step
Training Steps	50,000	Total updates ( $\approx 2\times 10^{8}$ tokens)

Appendix

Table D1: Hyperparameters for Sparse Autoencoder Training.

Functional Group	SAE latents	Neurons	$\Delta$
Carbonyl	0.999	0.982	$+0.017$
Carbonyl (non-acid)	0.997	0.965	$+0.032$
Bicyclic	0.997	0.984	$+0.013$
Ether	0.995	0.968	$+0.027$
Secondary amine	0.993	0.963	$+0.030$
Amide	0.993	0.960	$+0.033$

Appendix

Table D2: Validation set

F_{1}

for detecting the ten most common functional groups from a linear probe on SAE latents against a probe on residual-stream neurons (middle layer of the 32M PubChem–SMILES–atomwise reference at the final checkpoint, averaged over folds). SAE latents give the higher

F_{1}

for seven of the ten groups, with gains of up to about 0.04. Where neurons win, the margin is within 0.003. Bold indicates the higher score.

Figure E1: Parallel Coordinates Plot for the SMILES (AtomWise) model (32M), showing the importance and effects of hyperparameters on the Aggregated Score.

Figure E2: Parallel Coordinates Plot for the SMILES (BPE) model (32M).

Figure E3: Parallel Coordinates Plot for the SAFE (AtomWise) model.

Figure E4: Parallel Coordinates Plot for the SAFE (BPE) model.

Figure E5: Parallel Coordinates Plot for the SELFIES (AtomWise) model.

Figure E6: Parallel Coordinates Plot for the SELFIES (BPE) model.

Figure E7: Parallel Coordinates Plot for the SMILES (AtomWise) model (157M).

Figure E8: Parallel Coordinates Plot for the SMILES (BPE) model (157M).

Figure E9: Parallel Coordinates Plot for the SMILES (AtomWise) model (300M).

Figure E10: Parallel Coordinates Plot for the SMILES (BPE) model (300M).

Budget $C$ (FLOPs)	Evaluated sizes	$N_{\mathrm{opt}}$	$D_{\mathrm{opt}}$ (tokens)
$3\times 10^{15}$	0.5M, 5M, 32M	2.5M	$2.0\times 10^{8}$
$3\times 10^{16}$	5M, 32M, 157M	12M	$4.2\times 10^{8}$
$3\times 10^{17}$	5M, 32M, 157M	55M	$9.1\times 10^{8}$
$1\times 10^{18}$	32M, 157M, 300M	130M	$1.3\times 10^{9}$
$1\times 10^{19}$	157M, 300M, 1B	700M	$2.4\times 10^{9}$

Appendix

Table F1: IsoFLOP grid. For each compute budget we evaluate the model sizes that bracket the optimum and read the compute-optimal parameter count

N_{\mathrm{opt}}

and token count

D_{\mathrm{opt}}

from the fitted loss surface.

Axis	Setting	Bits / molecule ( $\downarrow$ )	Bits / token	Tokens / molecule
Representation	SMILES (atomwise)	48.1	1.02	47
	SELFIES (atomwise)	50.9	0.33	155
	SAFE (atomwise)	50.2	0.95	53
Tokenisation	SMILES (atom-level)	48.1	1.02	47
Tokenisation	SMILES (byte-pair)	46.4	1.86	25
Corpus (SMILES, atomwise)	MOSES	25.6	0.91	28

Appendix

Table G1: Validation set loss at 32M decomposed into bits per token and tokens per molecule on the random split. Bits per molecule equals bits per token times tokens per molecule.

Model size	0.5M	5M	32M	157M	300M	1B
Bits / molecule ( $\downarrow$ )	71.5	54.1	48.1	46.0	45.5	45.0

Appendix

Table G2: Validation set bits per molecule of the PubChem–SMILES–atomwise reference on the random split, trained at

D=20N

Figure G1: Substructures present in BPE tokenization

Split	0.5M	5M	32M	157M	300M	1B
Random	71.5	54.1	48.1	46.0	45.5	45.0
Scaffold	72.9	55.2	49.1	46.9	46.4	45.9
Molecular weight	422	243	168	143	132	126
Maximum dissimilarity	172	92	68	62	60	59
Max-dissim. / random	2.4	1.7	1.4	1.35	1.32	1.3

Appendix

Table H1: Validation set bits per molecule of the PubChem–SMILES–atomwise reference on all four evaluation splits (

\downarrow

). The final row is the maximum-dissimilarity loss as a multiple of the random-split loss.

Axis	Setting	OOD gap (%)
Representation	SMILES (atomwise)	41
	SELFIES (atomwise)	57
	SAFE (atomwise)	50
Tokenisation	Atom-level	41
Tokenisation	Byte-pair	32
Architecture	Qwen3 (decoder)	41

Appendix

Table H2: Out-of-distribution gap at 32M, the relative increase in bits per molecule from the random to the maximum-dissimilarity split (

\downarrow

Axis	Setting	Size	Valid ( $\uparrow$ )	IntDiv ( $\uparrow$ )	Novelty ( $\uparrow$ )	FCD ( $\downarrow$ )	SNN ( $\uparrow$ )	Frag ( $\uparrow$ )	Scaf ( $\uparrow$ )
Ref. size	PubChem, SMILES	0.5M	0.310	0.892	0.989	3.92	0.386	0.948	0.469
		5M	0.901	0.894	0.970	2.84	0.402	0.892	0.722
		32M	0.961	0.884	0.979	5.12	0.395	0.901	0.314
		157M	0.968	0.883	0.980	5.05	0.398	0.905	0.330
		300M	0.972	0.882	0.980	4.95	0.399	0.908	0.335
		1B	0.978	0.882	0.981	4.82	0.401	0.910	0.340

Appendix

Table H3: Unconstrained-generation metrics on the random split for NovoMolGen across design axes. Values are averaged over pretraining seeds where available. Each corpus is scored on its own random split, so the FCD scale differs across corpora. Arrows show the preferred direction.

Representation (32M)	Random	Scaffold	Molecular weight	Maximum dissimilarity
SMILES (atomwise)	5.12	13.9	23.4	29.0
SELFIES (atomwise)	2.43	12.6	22.9	27.4
SAFE (atomwise)	2.93	12.5	22.8	26.2

Appendix

Table H4: Fréchet ChemNet Distance (

\downarrow

) between 30,000 molecules sampled from each 32M PubChem model and each evaluation split. Samples stay anchored to the training distribution, so the FCD grows sharply on the shifted splits.

Representation	Size	FCD ( $\downarrow$ )		SNN ( $\uparrow$ )		Scaf ( $\uparrow$ )
Representation	Size	Test	TestSF	Test	TestSF	Test	TestSF
SMILES (atomwise)	0.5M	3.92	15.03	0.386	0.333	0.469	0.064
	5M	2.84	13.17	0.402	0.347	0.722	0.063
	32M	5.12	13.91	0.395	0.345	0.314	0.028
SELFIES (atomwise)	32M	2.43	12.61	0.400	0.345	0.662	0.041
SAFE (atomwise)	32M	2.93	12.53	0.383	0.336	0.595	0.092

Appendix

Table H5: Random split (Test) versus scaffold split (TestSF) for NovoMolGen. FCD is lower-is-better, SNN and scaffold similarity are higher-is-better. The scaffold split degrades sample-space metrics even though its validation loss stays close to the random split.

Figure I1: Lower pretraining loss does not buy better goal-directed design. Downstream performance after RL fine-tuning of decoder-only PubChem models across design axes (colour) and model sizes from 0.5M to 1B (marker). (a) Total PMO score over 22 tasks (higher is better) against validation set bits per molecule, with the loss axis reversed. (b) Every axis plateaus in total PMO score beyond

\sim

5M (shaded). (c, d) The same comparisons for the total top-5% novel docking score over five targets (lower is better). Dashed lines mark

f

-RAG and REINVENT ( Lee et al., 2024a ) .

Figure I2: PMO benchmark results for NovoMolGen-SMILES across model sizes (32M, 157M, 300M) and tokenization strategies. The heatmap (left) shows normalized scores per task, while the bar chart (right) presents total scores (higher is better). Baselines are REINVENT and $f$ -RAG .

Figure I3: PMO benchmark results for NovoMolGen-SMILES-32M (AtomWise) across intermediate training checkpoints. The heatmap (left) shows normalized scores per task, while the bar chart (right) presents total scores (higher is better). Baselines are REINVENT and

f

-RAG .

Figure I4: PMO benchmark results for all molecule types NovoMolGen-32M . The heatmap (left) shows normalized scores per task, while the bar chart (right) presents total scores (higher is better). Baselines are REINVENT and $f$ -RAG .

Oracle	$f$ -RAG	REINVENT	NovoMolGen‑32M (AtomWise)	NovoMolGen‑157M (AtomWise)	NovoMolGen‑300M (AtomWise)
albuterol_similarity	0.977 (0.002)	$0.882_{(0.006)}$	$0.958_{(0.003)}$	0.968 (0.002)	$0.954_{(0.002)}$
amlodipine_mpo	0.749 (0.019)	$0.635_{(0.035)}$	0.732 (0.024)	$0.730_{(0.037)}$	$0.730_{(0.007)}$
celecoxib_rediscovery	0.778 (0.007)	$0.713_{(0.067)}$	$0.732_{(0.009)}$	$0.777_{(0.132)}$	0.904 (0.022)
drd2	$0.936_{(0.011)}$	$0.666_{(0.044)}$	0.973 (0.001)	0.983 (0.002)	$0.972_{(0.001)}$
deco_hop	0.992 (0.000)	0.945 (0.007)	$0.902_{(0.007)}$	$0.853_{(0.115)}$	$0.785_{(0.130)}$
fexofenadine_mpo	0.856 (0.016)	$0.784_{(0.006)}$	$0.802_{(0.008)}$	$0.790_{(0.028)}$	0.803 (0.018)

Appendix

Table I1: PMO AUC Top-10 results. Results are reported as mean (std) over three independent model initializations for SMILES molecule type. Blue denotes the best performing model, while Pink represents the second-best model by mean score.

Oracle	$f$ -RAG	REINVENT	NovoMolGen-32M (BPE)	NovoMolGen-157M (BPE)	NovoMolGen-300M (BPE)
albuterol_similarity	0.977 (0.002)	$0.882_{(0.006)}$	0.981 (0.001)	$0.951_{(0.000)}$	$0.949_{(0.001)}$
amlodipine_mpo	0.749 (0.019)	$0.635_{(0.035)}$	0.739 (0.021)	$0.739_{(0.026)}$	$0.713_{(0.010)}$
celecoxib_rediscovery	$0.778_{(0.007)}$	$0.713_{(0.067)}$	$0.751_{(0.190)}$	0.872 (0.061)	0.890 (0.018)
drd2	$0.936_{(0.011)}$	$0.666_{(0.044)}$	0.987 (0.001)	$0.967_{(0.001)}$	0.967 (0.000)
deco_hop	0.992 (0.000)	0.945 (0.007)	$0.844_{(0.121)}$	$0.937_{(0.008)}$	$0.914_{(0.051)}$
fexofenadine_mpo	0.856 (0.016)	$0.784_{(0.006)}$	0.837 (0.022)	$0.810_{(0.012)}$	$0.804_{(0.002)}$

Appendix

Table I2: PMO AUC Top-10 results. Results are reported as mean (std) over three independent model initializations for SMILES molecule type. Blue denotes the best performing model, while Pink represents the second-best model by mean score.

Figure I5: Sample efficiency of NovoMolGen-SMILES-BPE-32M on goal-directed generation tasks from the PMO benchmark. The plot tracks the top-10 reward as a function of oracle calls, showing how quickly the model discovers high-reward molecules.

Figure I6: Examples of the generated top-5 molecules from a single run of NovoMolGen-SMILES-BPE-32M . The scores are provided at the bottom of each generated molecule.

Method	parp1	fa7	5ht1b	braf	jak2	Sum
REINVENT	$-8.7$	$-7.2$	$-8.8$	$-8.4$	$-8.2$	$-41.2$
MOOD	$-10.9$	$-8.2$	$-11.2$	$-11.1$	$-10.2$	$-51.4$
GEAM	$-12.9$	$-9.9$	$-12.4$	$-12.3$	$-11.8$	$-59.3$
$f$ -RAG	$-12.9$	$-9.9$	$-12.7$	$-12.4$	$-11.8$	$-59.8$
NovoMolGen-0.5M	$-13.6$	$-10.4$	$-13.4$	$-13.1$	$-12.5$	$-63.0$
NovoMolGen-5M	$-14.2$	$-10.9$	$-14.0$	$-13.6$	$-13.0$	$-65.8$

Appendix

Table J1: Top-5% novel docking score per target (kcal/mol,

\downarrow

). Baselines are taken from Lee et al. (2024a) . Blue denotes the best score per target. Totals are the reported sums, so rounded per-target entries may not add up exactly.

Method	parp1	fa7	5ht1b	braf	jak2
REINVENT	$9.894_{(2.178)}$	$10.731_{(1.516)}$	$11.605_{(3.688)}$	$8.715_{(2.712)}$	$11.456_{(1.793)}$
MORLD	$\mathbf{98.433_{(1.189)}}$	$\mathbf{97.967_{(1.764)}}$	$\mathbf{98.787_{(0.743)}}$	$\mathbf{96.993_{(2.787)}}$	$\mathbf{97.720_{(0.995)}}$
HierVAE	$60.453_{(17.165)}$	$24.853_{(15.416)}$	$48.107_{(1.988)}$	$59.747_{(16.403)}$	$85.200_{(14.262)}$
FREED	$74.640_{(2.953)}$	$78.787_{(2.132)}$	$75.027_{(5.194)}$	$73.653_{(4.312)}$	$75.907_{(5.916)}$
MOOD	$84.180_{(2.123)}$	$83.180_{(1.519)}$	$84.613_{(0.822)}$	$87.413_{(0.830)}$	$83.273_{(1.455)}$
GEAM	$\mathbf{88.611_{(3.107)}}$	$\mathbf{89.378_{(2.619)}}$	$\mathbf{84.222_{(2.968)}}$	$\mathbf{90.322_{(3.467)}}$	$\mathbf{89.222_{(1.824)}}$

Appendix

Table J2: Novelty comparison on protein-ligand docking tasks. We report Novelty (%), calculated with respect to the ZINC-250k dataset. Results show the mean and standard deviation from 3 independent runs. Baseline data is sourced from Lee et al. (2024b) . Blue denotes the best model by mean score, while Pink represents the second-best model by mean score.

Figure J1: Sample efficiency of NovoMolGen-SMILES-BPE-32M on Protein-ligand docking goal-directed generation tasks. The plot tracks the top 5th percentile of the reward distribution as a function of oracle calls, demonstrating our model’s ability to rapidly discover high-reward molecules.

Figure J2: Examples of the generated top-10 molecules from a single run of NovoMolGen-SMILES-BPE-32M , shown for the stricter showcase filter QED $>0.5$ and SA $<4$ rather than the SA $<5$ used in the novel-hit definition. The docking scores are given at the bottom of each molecule.

Explore similar work

Probing Chemical Language Models: Effects of Pre-training and Fine-tuning
Jul 2, 2026Anna Karnysheva, Dietrich Klakow, Ji-Ung LeeMoleculenet Molecules
1RTG Neuroexplicit Models · 2Spoken Language Systems · Saarland University +1
SMolLM: Small Language Models Learn Small Molecular Grammar
May 7, 2026Akhil Jindal, Harang JuMolecular Generation Chemistry
Moku
What Does a Chemical Language Model Know About Molecules?
Jun 22, 2026Christian Kenneth, Etowah Adams, Liam Bai +1Molecules Language Modeling
Independent · Department of Systems Biology, Columbia University, New York · Generate:Biomedicines, Massachusetts +1

Jul 2, 2026, cs.LGJ/K move · Enter open · S save

Probing Chemical Language Models: Effects of Pre-training and Fine-tuning

Anna Karnysheva, Dietrich Klakow, Ji-Ung Lee

1RTG Neuroexplicit Models · 2Spoken Language Systems · Saarland University+1

Moleculenet Molecules Pretraining Model Fine-Tuning Passage Encoding

Open paper AskPDF arXiv

Chemical language models (CLMs) are trained with linearized representations such as SMILES, yet it remains unclear which chemically meaningful substructures they encode. To foster a better understanding of CLMs, we conduct a systematic study and probe for 78 molecular substructures across eight pre-trained and six randomly initialized models. We furthermore study how fine-tuning on chemical downstream tasks affects the learned representations of molecular substructures. Our results show that pre-training generally improves molecular structure awareness of CLMs, particularly in the upper layers. Moreover, randomly initialized models already encode ring structures well in the first layer. Our analysis on two chemical downstream tasks further reveals that, interestingly, fine-tuning affects task-relevant molecular substructures more than others, indicating that the changes in the representations follow chemical theory.

Abstract

Explore similar work

Probing Chemical Language Models: Effects of Pre-training and Fine-tuning

SMolLM: Small Language Models Learn Small Molecular Grammar

What Does a Chemical Language Model Know About Molecules?