Sample Efficient Generative Molecular Optimization with Joint Self-Improvement
Authors: Serra Korkmaz, Adam Izdebski, Jonathan Pirnay, Rasmus Møller-Larsen, Michal Kmicikiewicz, Pankhil Gawade, Maksim Pavlov, Dominik G. Grimm, Ewa Szczurek
Organizations: Institute of AI for Health, Helmholtz Zentrum Munchen · Technical University of Munich, TUM School of Computation, Information and Technology · Technical University of Munich, Campus Straubing for Biotechnology and Sustainability · University of Applied Sciences Weihenstephan Triesdorf · Faculty of Mathematics, Informatics and Mechanics, University of Warsaw
Generative molecular optimization aims to design molecules with properties surpassing those of existing compounds. However, evaluating candidate molecules is expensive, so sample efficiency, i.e., generating optimized molecules within a fixed evaluation budget, is essential. Moreover, in the offline setting, where a surrogate model approximates candidate evaluation, sample inefficiency compounds with surrogate error, further limiting performance. To address these challenges, we introduce Joint Self-Improvement (JSI), a framework for sample-efficient online and offline molecular optimization. JSI combines (i) a joint generative-predictive model trained via a joint likelihood-based loss, which aligns molecule generation and surrogate modeling, and (ii) inference-time self-improvement, which iteratively biases the generative component of the joint model toward higher-scoring molecules, using direct objective function evaluations in the online setting and the predictive component in the offline setting. Experiments across docking score optimization in both online and offline settings, as well as antimicrobial peptide optimization, demonstrate that JSI outperforms state-of-the-art methods under limited evaluation budgets. Code is available at https://github.com/szczurek-lab/jsi.
Figures & tables
Figure 1: Overview of JSI . In the offline setting, a joint generative-predictive model, with a language modeling (LM) head for next-token probability prediction and a predictive head for property prediction, is fine-tuned on an offline dataset of molecules and their property scores, and the model self-improves by iteratively biasing next-token probabilities (blue) using predicted property scores (green), steering generation towards high-scoring molecules at inference time. In the online setting, ground-truth oracle evaluations are used in place of the predictive head.
Figure 2: Example logit update in Eq. ( 10 ) with K=2 , σ=1 and V={C,O} . JSI uses SBS to autoregressively generate K molecules, inducing a tree where each node is a partial sequence and edges carry next-token probabilities. The predictive head scores each completed sequence, and the scores are aggregated into a global advantage that biases next-token probabilities. Panel (a) shows the initial probabilities, while panel (b) shows probabilities biased towards high-scoring branches.
Target Protein, Hit Ratio ( ↑ )
Method
parp1
fa7
5ht1b
braf
jak2
ZINC250K
3.993 (0.355)
1.097 (0.192)
24.26 (0.622)
1.020 (0.193)
6.183 (0.344)
REINVENT
4.693 (1.776)
1.967 (0.661)
26.047 (2.497)
2.207 (0.800)
5.667 (1.067)
JT-VAE
3.200 (0.348)
0.933 (0.152)
18.044 (0.747)
0.644 (0.157)
5.856 (0.204)
GraphAF
0.822 (0.113)
0.011 (0.016)
6.978 (0.952)
1.422 (0.556)
1.233 (0.284)
MORLD
0.047 (0.050)
0.007 (0.013)
0.893 (0.758)
0.047 (0.040)
0.227 (0.118)
Table 1: Molecular optimization performance on docking score optimization in the online setting. We mark the best method bold .
Figure 3: Hit ratio versus oracle budget across five protein targets. Curves show the mean across 10 seeds for JSI (purple) and SATURN (gray); shaded regions indicate ±1 standard deviation over seeds.
Target Protein
Method
parp1
fa7
5ht1b
braf
jak2
SATURN
OB(1) ( ↓ )
35 (28)
201 (172)
4 (3)
130 (144)
30 (29)
OB(10) ( ↓ )
282 (116)
715 (187)
45 (13)
517 (167)
227 (93)
OB(100) ( ↓ )
721 (137)
1408 (304)
347 (49)
1047 (301)
720 (131)
IntDiv1 ( ↑ )
0.627 (0.023)
0.597 (0.056)
0.665 (0.032)
0.572 (0.063)
0.641 (0.056)
JSI (ours)
OB(1) ( ↓ )
25 (21)
106 (77)
3 (2)
68 (72)
13 (9)
Table 2: Oracle burden in the online setting. We mark the best method bold .
Target Protein, Hit Ratio ( ↑ )
Method
parp1
fa7
5ht1b
braf
jak2
GraphGA
29.631 (20.478)
0.913 (0.942)
73.875 (4.945)
3.340 (4.056)
28.912 (14.910)
REINVENT
2.686 (0.612)
0.813 (0.252)
19.653 (0.937)
0.693 (0.204)
4.020 (0.616)
SATURN
6.319 (0.845)
0.733 (0.270)
32.287 (1.687)
1.927 (0.353)
7.667 (1.279)
GeneticGFN
8.893 (1.532)
0.767 (0.252)
56.910 (2.730)
2.760 (0.907)
10.693 (1.927)
RaM
38.373 (5.196)
5.712 (5.382)
62.932 (9.472)
10.732 (4.771)
33.189 (7.850)
Table 3: Molecular optimization performance on docking score optimization in the offline setting. We mark the best method bold .
Figure 4: Binding poses of best molecules generated by JSI in the offline optimization setting.
Figure 8
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
[N]
Set of integers 1,…,N
a
Vector (column-vector)
a(i)
Vector indexed for some purpose
ai:j
Vector with values from the i -th to j -th (inclusive) entry of vector a
ai
The i -th entry of vector a
a
Scalar
Appendix
Table 7: Mathematical notation used throughout the paper.
Target Protein, Hit Ratio ( ↑ )
Normalization
Aggregation
braf
Z-scoring
Sum
58.547 (11.456)
Min-max
Sum
23.223 (6.426)
Min-max
Product
38.897 (7.024)
Appendix
Table 8: Hit Ratio for different reward normalization and aggregation methods in the online docking score optimization setting. We mark the best configuration bold . Values are mean (standard deviation) across 10 random seeds.
Target Protein, Hit Ratio ( ↑ )
wDS / wSA / wQED
parp1
fa7
5ht1b
braf
jak2
0.25 / 0.375 / 0.375
65.163 (7.858)
34.350 (11.492)
87.827 (3.689)
49.883 (10.589)
68.553 (12.792)
0.33 / 0.33 / 0.33
67.280 (7.961)
36.587 (10.270)
91.357 (3.204)
58.547 (11.456)
71.863 (6.127)
0.5 / 0.25 / 0.25
75.470 (6.973)
43.530 (11.954)
84.413 (6.580)
64.997 (7.474)
80.230 (4.915)
0.75 / 0.125 / 0.125
68.913 (9.576)
41.317 (10.804)
78.410 (4.293)
59.570 (9.114)
67.867 (5.982)
Appendix
Table 9: Hit Ratio for different weightings w of the reward Rw (Eq. ( 24 )) in the online docking score optimization setting. We mark the best weighting per target bold . Values are mean (standard deviation) across 10 random seeds.
Target Protein, Pearson Correlation ( ↑ )
Sequences
parp1
fa7
5ht1b
braf
jak2
Test
0.879 (0.005)
0.762 (0.012)
0.867 (0.007)
0.802 (0.010)
0.832 (0.006)
Generated
0.224 (0.190)
0.284 (0.150)
0.305 (0.194)
0.304 (0.141)
0.384 (0.122)
Appendix
Table 10: Surrogate predictive performance on held-out (Test) and generated (Generated) sequences in the offline docking score optimization setting, measured by the Pearson correlation with the ground-truth oracle docking score. Values are mean (standard deviation) across 10 random seeds.
Target Protein, Hit Ratio ( ↑ )
Step Size σ
braf
0.25
33.829 (14.137)
0.5
29.907 (10.995)
0.75
27.402 (7.025)
1
27.418 (9.888)
Appendix
Table 11: Sensitivity analysis for the step size σ in the offline docking score optimization setting. Values are mean (standard deviation) across 3 random seeds.
Target Protein, Hit Ratio ( ↑ )
Beam Width K
braf
16
18.414 (3.963)
32
21.275 (9.278)
64
21.857 (7.136)
128
27.418 (9.888)
Appendix
Table 12: Sensitivity analysis for the beam width K in the offline docking score optimization setting. Values are mean (standard deviation) across 3 random seeds.
Target Protein, Hit Ratio ( ↑ )
Joint-Loss Weight λ
braf
0.1
24.410 (8.217)
0.25
27.418 (9.888)
0.4
18.867 (2.780)
0.5
16.564 (2.297)
Appendix
Table 13: Sensitivity analysis for the joint-loss weight λ in the offline docking score optimization setting. Values are mean (standard deviation) across 3 random seeds.
Method
Uniqueness ( ↑ )
Novelty ( ↑ )
3-gram Jaccard Similarity
Diversity ( ↑ )
AMP-Diffusion
0.960
1.000
0.003
0.820
Hyformer
1.000
0.999
0.002
0.740
JSI (ours)
1.000
0.916
0.004
0.780
Appendix
Table 14: Uniqueness, novelty and diversity of the AMP sequences generated in the offline AMP optimization setting. Values are computed on the 1,000 sequences per method evaluated in Table 4 .
Method
#Candidates
Length
Time(s)
Unconditional
1
39.4
0.004 (0.001)
Best-of- K
128
39.6
0.552 (0.014)
JSI
Nrounds=1
128
39.7
0.662 (0.045)
Nrounds=2
256
40.2
1.321 (0.068)
Nrounds=3
384
41.0
2.041 (0.195)
Appendix
Table 18: Sampling cost of the inference-time methods on the BRAF protein target in the offline docking score optimization setting. Values are mean (standard deviation) across 3 random seeds.
Figure 5: Offline and online GMO as compared to the JSI framework. (a) In the offline setting, objective f is not available, and a surrogate f^ is used to predict its values. (b) In the online setting, objective f is available and is used for generative model updates. (c) Joint Self-Improvement can perform in both settings: either using the offline dataset to train the joint model and sample new molecules, or updating based on newly generated objective evaluations.
Hyperparameter
Pretraining
Finetuning
Batch Size
512
32
Learning Rate
6e-4
1e-4
Weight Decay
1e-1
1e-1
(β1,β2)
(0.9, 0.95)
(0.9, 0.95)
Gradient Clip
1.0
1.0
Minimum Learning Rate
1e-6
1e-6
Appendix
Table 19: Hyperparameter settings used for pre-training and fine-tuning of Hyformer for docking optimization task.
Target Protein
Hyperparameter
parp1
fa7
5ht1b
braf
jak2
Beam width K
16
16
32
16
16
Number of rounds Nrounds
10
14
10
12
10
Step size σ
1.5
1.5
0.75
1.75
1.5
Appendix
Table 20: Best hyperparameters for Joint Self-Improvement in the online optimization experiment.
Target Protein
Hyperparameter
parp1
fa7
5ht1b
braf
jak2
Beam width K
128
128
16
128
128
Number of rounds Nrounds
10
10
8
8
8
Step size σ
0.25
0.5
1.0
1.0
0.5
Appendix
Table 21: Best hyperparameters for Joint Self-Improvement in the offline optimization experiment.
Hyperparameter
Pretraining
Finetuning
Batch Size
512
64
Learning Rate
4e-4
1e-4
Weight Decay
1e-1
1e-1
(β1,β2)
(0.9, 0.95)
(0.9, 0.95)
Gradient Clip
1.0
1.0
Minimum Learning Rate
1e-6
1e-6
Appendix
Table 22: Hyperparameter settings used for pre-training and fine-tuning of Hyformer for AMP optimization task.
Target Protein, MAE ( ↓ )
Surrogate
parp1
fa7
5ht1b
braf
jak2
Chemprop
0.517 (0.012)
0.383 (0.005)
0.528 (0.007)
0.464 (0.006)
0.439 (0.013)
JSI
0.446 (0.013)
0.400 (0.012)
0.426 (0.015)
0.466 (0.012)
0.408 (0.007)
Appendix
Table 23: Predictive performance of Chemprop and JSI on a held-out test set of 500 molecules sampled from ZINC250K in docking score prediction, as measured by mean absolute error (MAE).
Molecular optimization computationally designs or edits chemical structures to improve desired properties while respecting chemical and practical constraints.
Molecular optimization in drug discovery, materials design, and catalysis requires searching vast chemical spaces under tight evaluation budgets, since high-fidelity oracles and experimental measurements are costly. The practical impact of an optimization method therefore hinges on its sample efficiency: how few evaluations it needs to find strong candidates. We introduce Sample Efficient Generative Optimization (SEGO), a framework for Bayesian optimization on adaptively generated molecules. In SEGO, a probabilistic surrogate model forms a hypothesis about where hits lie in chemical space, a generative model is steered to propose candidates in that region, the most promising candidate is selected via an acquisition function, and the resulting oracle call is used both to sharpen the surrogate and to anchor the generator in real reward. SEGO attains state-of-the-art performance on the practical molecular optimization (PMO) benchmark using only one tenth of the oracle calls consumed by other methods, and on a multiparameter docking task it reaches ten hits in roughly half the oracle calls of existing approaches. These gains move molecular optimization closer to campaigns driven by direct experimental feedback.
Sarina Kopf, Cristina Nevado, Philippe Schwaller
LIAC, ISIC, EPFL, Lausanne, Switzerland · Department of Chemistry, University of Zurich, Zurich, Switzerland · Centre of Competence in Research (NCCR) Catalysis, Zurich, Switzerland
Molecular discovery is increasingly framed as an inverse design problem: identifying molecular structures that satisfy desired property profiles under feasibility constraints. While recent generative models provide continuous latent representations of chemical space, targeted optimization within these representations often leads to degraded validity, loss of structural fidelity, or unstable behavior. We introduce MoltenFlow, a modular framework that combines property-organized latent representations with flow-matching generative priors and gradient-based guidance. This formulation supports both conditioned generation and local optimization within a single latent-space framework. We show that guided latent flows enable efficient multi-objective molecular optimization under fixed oracle budgets with controllable trade-offs, while a learned flow prior improves unconditional generation quality.
Generating novel molecules with desired properties is a central challenge in drug discovery. Although foundational work focused on de novo generation, practical use cases require a more complex setting: conditioning on an initial seed molecule, optimizing multiple properties, and operating in data-sparse regimes. We introduce PairIT, a flexible training framework for autoregressive transformers over SMILES sequences that enables seeded generation by learning from pairs of structurally similar molecules. By leveraging matched pairs, the model learns to generate local, chemically meaningful edits that improve a property of the seed molecule. Prompt tokens extend this framework by specifying the target property, desired direction, and approximate magnitude of change. Multiple prompt tokens can be concatenated to enable multi-property optimization at inference time. For properties where pair supervision alone is insufficient, reinforcement learning post-training can further improve optimization, providing a tunable trade-off between proximity to the seed and the optimization objective. PairIT naturally supports pretraining on a large dataset and fine-tuning on a smaller dataset, which is critical for performance in data-sparse regimes. Surprisingly, we observe diminishing returns to the scale of the pretraining dataset. We demonstrate empirically on MoleculeNet and Therapeutics Data Commons datasets that, with appropriate paired-data curation, a standard transformer model of modest size (68M parameters) can achieve robust seeded molecular optimization in low-data regimes.
Hannah Lawrence, Bodhi P. Vani, Ji Won Park +5
Genentech, South San Francisco, CA, USA · Massachusetts Institute of Technology, Cambridge, MA, USA · Prescient Design, Genentech, Basel, Switzerland +1