Training Large Language Models for Small-Molecule Design with Synthetic Task Scaling

Frank Hu; Shriram Chennakesavalu; Zichen Wang; Patricia Suriana; Bodhi Vani; Kirill Shmilovich; Kangway Chuang; Colin Grambow

cs.LGSep 4, 2026

Training Large Language Models for Small-Molecule Design with Synthetic Task Scaling

Authors: Frank Hu, Shriram Chennakesavalu, Zichen Wang, Patricia Suriana, Bodhi Vani, Kirill Shmilovich, Kangway Chuang, Colin Grambow

Organizations: Prescient Design, Genentech, South San Francisco, CA, USA

Abstract

Designing viable drug candidates requires searching a combinatorially large and rugged chemical space for molecules that satisfy multiple, often competing, objectives. Large language models (LLMs) provide a useful generative prior for this problem because of their representational capacity, reasoning ability, and flexibility when incorporating information from the external environment. While reinforcement learning from verifiable rewards (RLVR) can be used to improve the capabilities of LLMs, many chemically relevant scoring functions require hours or even days per evaluation, making them prohibitively expensive to use directly during online training. Here, we investigate whether LLMs can learn molecular design strategies from cheaper synthetic tasks that generalize to expensive molecular lead optimization settings. We find that curriculum-based training recipes that gradually incorporate more challenging synthetic design tasks enable strong performance that surpasses that of much larger frontier models on structure-based lead optimization. Our results suggest that scaling post-training using synthetic tasks is an effective strategy for adapting LLMs to high-cost experimental scenarios that are too expensive to directly train on.

Figures & tables

Figure 1: Training on synthetic Tier-1 and Tier-2 tasks using low- and medium-cost chemistry oracles enables strong performance on expensive evaluation tasks. All tasks involve lead optimization, where the model must iteratively design new molecules with desired properties conditioned on a seed molecule. (Top) Tier-1 tasks involve simple and cheap RDKit ( RDKit, 2026 ) and OpenEye ( OpenEye, Cadence Molecular Sciences, 2025 ) oracles. (Middle) Tier-2 tasks involve designing strong small-molecule binders to a protein target as measured by the medium-cost Chemgauss4 ( McGann, 2011 ; McGann, 2012 ) scoring function. (Bottom) Evaluation tasks involve designing strong small-molecule binders to a protein target as measured by the expensive SQM2.20 ( Pecina et al., 2024 ) scoring function. For Tier-2 and evaluation tasks, model proposals are docked into the protein pocket using POSIT docking ( Kelley et al., 2015 ) before scoring.

Condition	Tier-1 steps	Second-stage steps	Second-stage batch (Tier-1 : Tier-2)
T1(500)	500	0	—
T2(500)	0	500	0% : 100%
T1(250)–T2(250)	250	250	0% : 100%
T1(300)–T2(200)	300	200	0% : 100%
T1(350)–T2(150)	350	150	0% : 100%
T1(400)–T2(100)	400	100	0% : 100%

Table 1: The training curricula where every run is trained for a total of 500 steps. Runs are named by the tier of tasks used for training—“T1” for Tier-1 tasks and “T2” for Tier-2 tasks—and the number of steps trained using those tasks, in parentheses. For example, T1(

a

)–T2(

b

) trains for

a

steps on Tier-1 tasks followed by

b

steps on only Tier-2 tasks. We vary this condition in T1(

a

)–Mix(

b

) where we train for

a

steps on Tier-1 tasks followed by

b

steps on a 25% : 75% mixture of Tier-1 to Tier-2 tasks.

Figure 2: Progression of the average reward for the Tier-1 environment tasks (left) and the Tier-2 environment tasks (right) as a function of the number of training steps. T1(

a

)–T2(

b

) are not shown on the left plot as they do not contain any Tier-1 tasks in the second curriculum learning stage. Plotted rewards are smoothed using an exponential moving average with

\alpha=0.15

Figure 3: (Left): Relative binding affinity improvement for all trained models and frontier models using the T1(500) model as the baseline, where higher is better. All results are computed from

n=200

rollouts with five rollouts for each of 40 test set systems, where invalid rollouts are assigned an improvement of 0, with trained models having increased reliability with fewer invalid rollouts (see Table A3 ). All error bars are

\pm

1 SE for the average relative binding affinity improvement over the entire test set of 40 systems. (Right): The relative binding affinity improvement is plotted against the total training wall time in hours, computed as the sum of the Tier-1 only training time followed by the curriculum hot-start training time. Error bars are omitted here for clarity.

Model	Reward (bal.) $\uparrow$	Rel. BA impr. $\uparrow$	Sys. impr. $\uparrow$	Turns	SQM turns	Constr. % $\uparrow$
T1(400)–T2(100)	0.623 $\pm$ 0.034	+3.18 $\pm$ 1.35	37	7.7	5.7	93.3%
T1(300)–T2(200)	0.622 $\pm$ 0.034	+3.88 $\pm$ 1.98	34	7.3	6.0	93.7%
T1(250)–Mix(250)	0.611 $\pm$ 0.034	+3.65 $\pm$ 1.81	37	7.2	5.8	91.9%
T1(350)–Mix(150)	0.611 $\pm$ 0.031	+2.17 $\pm$ 0.78	39	11.1	5.8	93.1%
T1(450)–Mix(50)	0.608 $\pm$ 0.036	+2.31 $\pm$ 0.93	35	9.4	5.3	92.4%
T1(450)–T2(50)	0.607 $\pm$ 0.032	+2.07 $\pm$ 0.65	36	11.6	5.6	93.9%

Table 2: Lead-opt SQM evaluation with

n=200

rollouts per model across 40 systems. Reward (bal.) is the balanced reward

0.5\cdot r_{\mathrm{obj}}+0.5\cdot r_{\mathrm{constraint}}+\text{penalties}

\pm

1 system-clustered SE. Rel. BA impr. is the relative binding affinity improvement against T1(500) (the Tier-1 only base path; higher

=

tighter binding),

\pm

1 paired SE over the 40 shared systems, the same as in Figure 3 . Sys. impr. is the number of systems (out of 40) for which at least one rollout improves on the seed. Turns is the mean number of interaction turns. SQM turns is the mean number of turns in which score_sqm is called. Constr. % is the mean graded constraint reward averaged over all

n

rollouts – rollouts that produced no valid scored molecule count as 0. Models are ranked by balanced reward.

Figure 4: Average cumulative maximum relative binding affinity throughout trajectories averaged across rollouts for 10 systems and four models, where higher is better. The x-axis is the molecule number scored by SQM, which helps standardize the comparison because all models have at most five SQM calls in their budget; models can make additional calls, but further scoring is refused. In cases where the model does not make five SQM scoring calls, the best design is carried forward up to five SQM calls.

Figure 5: Examples of the best molecules out of five rollouts generated by GPT-5.6 Sol, Claude Opus 4.8, T1(500), and T1(400)–T2(100) for the 6pyy and 4ise systems. The required BRICS scaffold is highlighted in green for the seed and all generated molecules, and the $\Delta$ SQM is shown relative to the seed molecule where higher is better.

Figure 6: Change in the number of heavy atoms versus change in the number of halogens per edit averaged over all SQM-scored candidates for each model. The change in each quantity is computed relative to the seed structure. T1(500) is shown as a black diamond.

Figure 7: The difference between the percentage of rollouts that added a functional group and the percentage of rollouts that removed a functional group for each model. Percentages are computed only over rollouts that terminated in a final valid and scorable design. Numbers are only shown in the heatmap for differences of magnitude $\geq 10\%$ .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Tier-1	Tier-2
Objective scaling factor $k$	$10.0$	$10.0$
Objective normalization $\tau$	$\sigma_{y}$	$\lvert y_{\mathrm{ref}}\rvert$
Soft-constraint scaling factor $s$	$0.1$	$0.1$
Objective weight $w_{\mathrm{obj}}$	$0.5$	$0.75$
Constraint weight $w_{\mathrm{constraint}}$	$0.5$	$0.25$
Tool call penalty $r_{\mathrm{tool}}$	$-0.001$	$-0.001$

Appendix

Table A1: Reward hyperparameters used for Tier-1 and Tier-2 tasks. Here,

\sigma_{y}

is the standard deviation of the Tier-1 target property computed over all molecules in ZINC, and

y_{\mathrm{ref}}

is the seed binding-affinity score for Tier-2 tasks.

Hyperparameter	Value
Policy optimization loss	IPO ( ipo_loss_fn ) ( Prime Intellect, 2025 )
Group size (completions/prompt)	16
KL coefficient ( $\beta$ )	$1{\times}10^{-3}$
Off-policy staleness ( $\mathrm{max\_off\_policy\_steps}$ )	2 (async)
Optimizer	AdamW
Learning rate	$1{\times}10^{-6}$

Appendix

Table A2: Training hyperparameters used across all runs.

Figure A1: T2(500) training, one panel per reward weighting. Each panel shows the total reward (

R_{\mathrm{total}}

) together with the unweighted binding and constraint subrewards (

r_{\mathrm{obj}}

and

r_{\mathrm{constraint}}

). The total is the weighted combination of these subrewards plus the tool/format penalties, so the

r_{\mathrm{obj}}

and

r_{\mathrm{constraint}}

curves do not sum directly to

R_{\mathrm{total}}

. In both settings the binding subreward stays near zero while the constraint subreward saturates, i.e., training collapses to a constraint-only policy. Rewards are smoothed with an exponential moving average (

\alpha=0.15

Figure A2: T1(400)–T2(100) reward decomposition for the curriculum-trained arm hot-started at step 400 (docking-only second stage), under three objective/constraint reward weightings. Each panel shows the total reward (

R_{\mathrm{total}}

) together with the unweighted binding and constraint subrewards (

r_{\mathrm{obj}}

and

r_{\mathrm{constraint}}

); the total is the weighted combination of these subrewards plus the tool/format penalties, so the

r_{\mathrm{obj}}

and

r_{\mathrm{constraint}}

curves do not sum directly to

R_{\mathrm{total}}

. Curves span the docking-stage steps 401–500.

Figure A3: Curriculum-trained models with

w_{\mathrm{obj}}=w_{\mathrm{constraint}}=0.5

. The data presentation here is consistent with Figure 3 , and higher relative binding affinity improvement is better.

Figure A4: Curriculum-trained models with

w_{\mathrm{obj}}=0.25

and

w_{\mathrm{constraint}}=0.75

. The data presentation here is consistent with Figure 3 , and higher relative binding affinity improvement is better.

Figure A5: Percentage of scored rollouts for each model broken down into five categories by the final submitted molecule: the best (or a worse) molecule the model scored during the rollout (Best/Suboptimal (scored)), a new molecule submitted unscored that ends up better (or worse) than its best scored (Best/Suboptimal (unscored)), or no valid submission (Failure). A scored molecule is one the model runs SQM scoring on during the rollout; an unscored molecule is submitted without scoring.

Model	Successful	Invalid formatting	Scoring failed
T1(250)–T2(250)	200	0	0
T1(250)–Mix(250)	200	0	0
T1(300)–T2(200)	200	0	0
T1(350)–T2(150)	200	0	0
T1(350)–Mix(150)	200	0	0
T1(400)–Mix(100)	200	0	0

Appendix

Table A3: Rollout outcomes over all

n=200

rollouts per model (

40

systems

\times\,5

). A rollout is Successful if it produced a valid final molecule (RDKit-parsable SMILES in its <|answer_start|> block) that received a real SQM score. The two failure modes are Invalid formatting —no or unparsable final answer block—and Scoring failed —a parsable molecule that did not score via SQM (e.g., POSIT/docking failure).

Figure A6: The average number of characters each model emitted across all

n=200

rollouts, split into reasoning (chain of thought) and non-reasoning output (visible messages). For closed-source frontier models the reasoning text is unavailable, shown as a tilde (

\sim

) rather than zero, so only non-reasoning output is counted.

Figure A7: For each model, and the aggregated Trained and Frontier rows, the dots give the average turn of each of the five SQM calls and the shaded bar spans the average rollout length; models can make additional calls, but further scoring is refused. Average turn for the

n^{\mathrm{th}}

call is averaged only over those rollouts that made an

n^{\mathrm{th}}

call to the SQM scorer.

Figure A8: Relative binding affinity improvement for all trained models and frontier models using the T1(500) model as the baseline, where higher is better. The left panel is over the

n=14

anionic starting ligands and the right is over the

n=26

neutral starting ligands. All error bars are

\pm

1 SE in the average relative binding affinity improvement with the T1(500) baseline over the respective subset of the test set.

Figure A9: For each model, the percentage of its scored candidates that remove an anionic group, over the 14 test set cases whose starting ligand was anionic.

Figure A10: The fraction of unique molecules averaged for each model over the test set systems where that model generated five valid final molecules; the number of such systems is annotated above each bar.

Group	Property	Description
Size / physico.	molecular_weight	Average molecular weight in Daltons (Da)
	exact_molecular_weight	Monoisotopic (exact) molecular weight in Da
	heavy_atom_molecular_weight	Average molecular weight ignoring hydrogens
	heavy_atom_count	Number of heavy (non-hydrogen) atoms
	num_heteroatoms	Number of heteroatoms (non-carbon atoms)
	tpsa	Topological polar surface area in Å 2

Appendix

Table A4: The 116 molecular properties presented to the model in the Tier-1 molecular-optimization environment (as optimization targets and constraints), grouped by family: 101 computed with RDKit and 15 with OpenEye. Property names are the identifiers shown in the prompt. Most are scalar targets; scaffold is a BRICS-substructure constraint (a string) and oe_is_macrocycle is a boolean flag.

Molecule Design

See all 44 papers

Apr 17, 2026

Evaluating the Progression of Large Language Model Capabilities for Small-Molecule Drug Design

Large Language Models (LLMs) have the potential to accelerate small molecule drug design due to their ability to reason about information from diverse sources and formats. However, their practical utility remains unclear due to the lack of benchmarks that reflect real-world scenarios. In this work, we introduce a suite of chemically-grounded tasks spanning molecular property prediction, molecular representation transformations, and molecular design. Importantly, we formulate these tasks as reinforcement learning (RL) environments, enabling a unified approach for evaluation and post-training. Across three model families, we find that frontier models are increasingly proficient at chemical tasks, but that there is significant room for improvement, especially in experimental settings with low data. Critically, we show that RL-based post-training can substantially improve performance. A smaller model post-trained on our environments becomes competitive with state-of-the-art frontier models, despite a significantly weaker base model. This suggests a practical route toward employing LLMs in drug discovery; by combining carefully-designed evaluation tasks with targeted post-training, we can both elucidate and close critical capability gaps.

Shriram Chennakesavalu, Kirill Shmilovich, Hayley Weir +5

Prescient Design, Genentech, South San Francisco, CA, USA

LLM Evaluation Molecular Property Prediction LLM Post-Training Drug Discovery

Ask PDF

Jun 8, 2026

My Chemical Harness: Evolutionary Molecular Design over Synthetic Pathways with Large Language Model Agents

Designing molecules with target properties is most useful when candidate structures are accompanied by feasible synthetic routes. We introduce My Chemical Harness, a route-native evolutionary framework for goal-directed molecular design in which the search population consists of executable synthetic pathways rather than isolated molecular graphs. Each route is built from purchasable building blocks and reaction templates, executed by deterministic chemistry tools, and scored through task-specific molecular oracles. Large language models (LLMs) are used only as strategy controllers that select high-level preferences over route length, move type, reaction families, motifs, and exploration pressure, while local code performs route construction, validation, deduplication, scoring, selection, and memory updates. This separation lets the LLM guide exploration without allowing it to introduce hallucinated products or unsupported reaction steps. On a soluble epoxide hydrolase proxy task, our LLM agent improves over single pass LLM and deterministic controllers, reaching state-of-the-art performance across the sEH score, synthetic accessibility score, and AiZynthFinder success rate metrics. These results suggest that constrained LLM agents can play a significant role in molecular discovery without requiring training, fine-tuning, or dedicated generative models.

César Ojeda, Darius A. Faroughy, Maryam Karimi +3

Institute of Mathematics, Faculty of Science, University of Potsdam, Campus Golm, Building 9, Karl-Liebknecht-Str. 24–25, 14476 Potsdam-Golm, Germany · NHETC, Department of Physics and Astronomy, Rutgers University, Piscataway, NJ 08854, USA · Potsdam Transfer, University of Potsdam, Karl-Liebknecht-Str. 24-25, 14476 Potsdam. +1

Molecular Optimization LLM Agents Molecule Design

Ask PDF

Jul 26, 2024

Small Molecule Optimization with Large Language Models

Molecular optimization, the process of designing molecules with desirable properties, represents a critical challenge in drug discovery. The recent advancements in large language models (LLMs) have opened new opportunities for their integration with traditional molecular optimization algorithms to improve performance. In this work, we propose Molecular Language Model powered Evolutionary Algorithm (Mol-E), an evolutionary algorithm that relies on the generative capabilities of LLMs trained on molecules and molecular properties. Scientific Contribution. Mol-E obtains the highest aggregate Top-10 AUC among the comparable full-23-task results considered here, scoring 17.500 in the task-agnostic regime, in which the oracle is treated strictly as a black box, and 20.551 in the task-informed regime, in which the optimizer receives a fixed semantic description of the objective. Mol-E also improves over the evaluated baselines on multi-property optimization with docking against DRD2, MK2, and AChE.

Philipp Guevorguian, Menua Bedrosian, Tigran Fahradyan +3

YerevaNN research lab, Yerevan, Armenia.

Molecular Optimization Evolutionary Optimization Large Language Model-Guided Optimization AI-Assisted Scientific Research Molecular Generation Drug Discovery

Ask PDF

Abstract

Molecule Design

Explore similar work

Evaluating the Progression of Large Language Model Capabilities for Small-Molecule Drug Design

My Chemical Harness: Evolutionary Molecular Design over Synthetic Pathways with Large Language Model Agents

Small Molecule Optimization with Large Language Models