Improving molecular properties, such as drug-likeness or binding affinity, is a recurring task in early-stage drug discovery. However, molecules optimized in an unconstrained chemical space have limited practical value if they cannot be synthesized. Policy Gradient for Forward Synthesis (PGFS) is a synthesis-aware reinforcement learning method for molecular improvement, but its use of reactant embedding prediction makes reactant selection indirect, which, as we show, limits learning effectiveness. We first develop PGFS+, in which reaction templates and second reactants are represented by trainable embedding lookup tables. Combined with a more effective scoring function and RL algorithm, PGFS+ significantly improves the desired property. However, it exposes a reward-hacking failure mode: a powerful reactant search can map diverse input molecules to the same high-reward magnet molecule, improving the reward while collapsing the output diversity. We therefore introduce PGFS++, a synthesis-aware reinforcement learning framework for input-specific molecular improvement. Given an input molecule, PGFS++ treats it as the start of a forward-synthesis trajectory, applies learned reaction templates with compatible in-stock building blocks, and produces a molecule with improved target properties, an explicit synthesis route, and structural similarity to the input. Experiments on molecular improvement tasks show that PGFS++ improves target properties while preserving high output diversity.
Figures & tables
Figure 1: Reward hacking phenomenon in PGFS+. A , PGFS+ maps hundreds of input molecules to the same output molecule, causing high QED but low output diversity. This is referred to as the magnet output phenomenon. Highlighted atoms indicate the portions of the output molecule that are traced back to the corresponding input molecules. B , Trade-off between mean QED and mean diversity for PGFS, its variants, and PGFS+. Because of the magnet output, PGFS+ achieves high mean QED but has output diversity close to zero.
Figure 2: Workflow comparison between PGFS, PGFS+, and PGFS++. At step t , all methods use an f -network conditioned on the current first reactant R1′(t) to predict the reaction template T . PGFS uses the fingerprint representation F(R1′(t)) directly, whereas PGFS+ and PGFS++ first transform the R1 representation through an additional MLP. For second-reactant selection, PGFS represents R2 candidates using fixed fingerprints and retrieves candidates by k -nearest neighbours. In contrast, PGFS+ and PGFS++ represent template-compatible R2 candidates with a trainable embedding lookup table and score them globally using an attention-like mechanism with query q=concat(ϕ1(R1′),ϕT(T)) . Once R2′(t) is selected, the reaction runner applies the chosen reaction and returns the product, which becomes the next state R1′(t+1) .
Figure 3: Performance comparison on the QED improving task among ours (PGFS++) and random search, PGFS, PGFS TAC, PGFS MB, REINVENT4 (mol2mol), SynFlowNet. The comparison with SynFlowNet used the reversible reaction templates provided in the SynFlowNet repository to avoid potential issues arising from irreversible templates. Comparison with other baselines uses a general template set derived from Button et al. (2019) , containing both reversible and irreversible templates.
Figure 4: Performance comparison on the SEH improving task among ours (PGFS++) and random search, PGFS, PGFS TAC, PGFS MB, REINVENT4 (mol2mol), SynFlowNet. The comparison with SynFlowNet used the reversible reaction templates provided in the SynFlowNet repository to avoid potential issues arising from irreversible templates. Comparison with other baselines uses a general template set derived from Button et al. (2019) , containing both reversible and irreversible templates.
Figure 5: Ablation of input–output similarity bonus. We compare PGFS+ without an input–output similarity bonus against PGFS++ with additive and multiplicative similarity bonuses. Bold entries indicate the best value for each metric, and underlined entries indicate the second-best value. Without input–output similarity bonus, PGFS+ suffers from the magnet output problem, as reflected by high mean QED, but very low output diversity, input–output similarity, and RAS.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Value
Symbol
Value
α
3×10−4
λ
0.95
nsteps
2048
ϵ
0.2
B
64
ce
0.05
K
10
cv
0.5
γ
0.99
gmax
0.5
KLtarg
0.02
Appendix
Table S1: Additional PPO hyperparameters used in PGFS++ experiments (notations as in Equation ( S1 )).
Method
General templates
Reversible templates
SynFlowNet
0.766
0.856
Ours
0.895
0.867
Appendix
Table S2: Median QED on the test set under general vs. reversible templates. Ours denotes the matched PGFS++ QED models from the main paper.
Tem.
Obj.
Seed/stat.
Prop. ↑
Div. ↑
1−SA↑
I/O sim. ↑
RAS ↑
Gen.
QED
123456
0.895
0.873
0.749
0.389
0.705
654321
0.897
0.867
0.698
0.360
0.267
246135
0.897
0.877
0.722
0.371
0.685
Mean
0.896
0.872
0.723
0.373
0.552
s.d.
0.001
0.005
0.026
0.015
0.247
Gen.
SEH
123456
0.812
0.772
0.748
0.312
0.545
Appendix
Table S3: Seed-wise median performance of PGFS++. Gen. and Rev. denote general and reversible reaction templates, respectively. The first three rows of each block report the median over generated molecules for each random seed. The final two rows report the mean and sample standard deviation, respectively, across the three seed-wise medians.
Figure S2: Property–novelty trade-offs on the test set. Top: general templates (PGFS++ versus REINVENT4) for QED (left) and SEH (right). Bottom: reversible templates (PGFS++ versus SynFlowNet) for QED (left) and SEH (right). Each point is one episode’s generated molecule. Colored stars mark each method’s median (property, novelty). The black star marks an ideal high-property / high-novelty corner. Numerical values of medians are presented in Table S1 .
Molecular optimization computationally designs or edits chemical structures to improve desired properties while respecting chemical and practical constraints.
Synthesizable molecular optimization seeks to improve target properties while ensuring that molecular modifications follow feasible synthetic pathways. Existing synthesis-aware methods typically rely on exploring a large space of candidate transformations defined by reaction templates and purchasable building blocks. This search becomes even more challenging when property improvement requires multiple reaction steps, as the space expands further along the pathway. To address this challenge, we introduce MolReAct, which reformulates molecular optimization as search over compact reaction spaces proposed by a tool-augmented large language model (LLM). At each step, the LLM combines its prior chemical knowledge with cheminformatics tools to identify a molecule-specific set of compatible reactions, preserving synthesizability while making multi-step optimization feasible. Given this compact action space, we further leverage Group Relative Policy Optimization (GRPO) with the terminal oracle reward to improve long-term decision-making over multiple reaction steps. Across diverse molecular optimization tasks, MolReAct achieves the highest Top-10 score on 11 of 14 tasks and the best sample efficiency on 12 of 14 tasks, outperforming existing baselines under limited oracle budgets. Beyond these gains, MolReAct also provides each optimized molecule with a template-grounded synthetic pathway.
Tao Li, Kaiyuan Hou, Tuan Vinh +4
Department of Computer Science, Emory University · School of Computer Science, Carnegie Mellon University · MRC Brain Network Dynamics Unit, University of Oxford +3
Achieving precise control over multiple properties without sacrificing chemical validity remains a central challenge in molecular inverse design. Existing reinforcement learning (RL) methods fine-tune graph diffusion models by treating reverse sampling as a sequential policy, using a single terminal reward to optimize hundreds of coupled decisions. They often suffer from instability, validity collapse, and limited property gains. We introduce GraphFDM (Graph Forward Distribution Matching), a new online RL paradigm for graph diffusion that performs optimization through the forward process. GraphFDM uses valid generations to define a reward-tilted target distribution jointly optimized over graph size and molecular structure for each property condition, incorporating reinforcement signals into supervised learning without storing reverse trajectories. We derive the unique optimal target, prove a condition-wise improvement guarantee, and show that the fixed graph-size prior of standard graph diffusion leaves an irreducible matching gap. In multi-conditional polymer and small-molecule generation, GraphFDM achieves the lowest MAE on every target property, with reductions of up to 53.0% relative to the strongest baselines and chemical validity above 0.99. It further generalizes to out-of-distribution property combinations.
Machine learning can accelerate molecular discovery by designing molecules and planning experiments. However, many scientific challenges demand molecules with very rare properties, and in this sparse setting, existing algorithms offer little gain over random guessing. We propose a method to efficiently search large regions of molecular space using algorithmically controlled stochastic synthesis. Rather than design, make and test individual molecules, we design and make complex mixtures, test them as a pool, then deconvolute the molecule-activity map. We optimize synthesis to encode maximal information. Theoretically, this approach can reduce the number of experiments required to find the optimal molecule among d candidates from O(d) to O(logd) or O(1). In simulation, on estimated protein fitness landscapes, it finds active molecules with an order of magnitude fewer experiments than existing Bayesian optimization methods.
Kasper K. Jakobsen, Eli N. Weinstein
Department of Chemistry, Technical University of Denmark, Kgs. Lyngby, DK.