Amortized Molecular Optimization via Group Relative Policy Optimization
Authors: Muhammad bin Javaid, Hasham Hussain, Ashima Khanna-Reiter, Berke Kişin, Jonathan Pirnay, Alexander Mitsos, Dominik G. Grimm, Martin Grohe
Organizations: RWTH Aachen University, Department of Computer Science, Aachen, Germany · Alfred E. Tiefenbacher (GmbH & Co. KG), Hamburg, Germany · Technical University of Munich, TUM Campus Straubing for Biotechnology and Sustainability · University of Applied Sciences Weihenstephan-Triesdorf, Bioinformatics · RWTH Aachen University, Process Systems Engineering (AVT.SVT), Aachen, Germany
In structurally constrained molecular optimization, state-of-the-art methods treat every new input structure as a separate instance, restarting an expensive oracle-driven search from scratch and scaling poorly to settings with many starting structures or expensive oracles. While amortized approaches that learn a transferable policy could in principle remove this bottleneck, existing methods struggle to generalize to diverse structural constraints at inference time. We present AMORTIX, a Graph Transformer model that natively supports such constraints, optimizing molecular structures in a single rollout with zero inference-time oracle calls. A central challenge for amortized training in this domain is that optimization difficulty varies drastically across starting structures. We address this by centering rewards within groups of completions that share the same starting structure. We evaluate on goal-directed scaffold decoration, including single- and multi-target kinase inhibitor design, and on a few-shot prodrug case study. AMORTIX outperforms both amortized baselines and search-based instance optimizers on kinase scaffold decoration, and ablations across 23 objectives show that group-relative centering improves over REINFORCE and PPO and is substantially more consistent across scaffold splits. The prodrug case study further demonstrates transfer of learned modification rules to unseen drug structures.
Figures & tables
Figure 1: Comparison of optimization paradigms. (A) Instance Optimization: Requires an expensive iterative search with hundreds to thousands of oracle calls for every new input structure Si , resulting in high cost that scales linearly with library size. (B) Amortized Optimization: Front-loads computation into offline training. The learned policy πθ generates optimized molecules for new, unseen inputs Si in a single rollout without inference-time oracle calls, enabling scalable, high-throughput design.
Figure 2: Overview of AMORTIX. (A) The policy decodes a composite action in three levels (operation, target atom, bond order) under a validity mask. (B) For each starting scaffold, G completions are sampled independently from the policy. (C) Rewards are centered by the group mean, and the resulting advantages weight the policy-gradient update.
Method
GSK3 β
JNK3
Kinase MPO
Amortized Policies
LibINVENT
0.414±0.122
0.261±0.065
0.453±0.007
GraphXForm
0.110±0.001
0.051±0.002
0.409±0.003
DrugEx v3
0.194±0.004
0.356±0.003
0.354±0.003
AMORTIX-DeNovo
0.572±0.015
0.381±0.029
0.397±0.005
AMORTIX-REINFORCE
0.757±0.084
0.754±0.092
0.475±0.120
Table 1: Optimization scores across three tasks (mean ± SD over 3 scaffold splits). AMORTIX denotes the default GRPO variant. Amortized methods generate one molecule per test scaffold; instance optimizers report the best molecule found per scaffold.
Figure 3: Cost vs. performance on Kinase MPO. Cumulative oracle calls per method as test scaffolds are processed; numbers next to each curve are mean test-set objective scores. Amortized methods front-load cost into a one-time training budget and use no oracle calls at inference. Instance optimizers pay per scaffold; we charge them the number of oracle calls until their mean best-so-far score over all test scaffolds converges (within 0.01 of its final value): ≈ 500 for Mol GA and ≈ 330 for GenMol (Table 4 ). Open circles mark where Mol GA ( ≈ 100 scaffolds) and GenMol ( ≈ 150 scaffolds) match AMORTIX’s training budget.
Method
Mean score
Best on
Split SD
GRPO higher on
GRPO
0.570
18/23
0.004
—
PPO
0.527
5/23
0.013
18/23
REINFORCE
0.512
0/23
0.072
22/23
LibINVENT
0.320
0/23
0.021
23/23
Table 2: Summary of the 23-objective ablation. Mean score is averaged over objectives. Split SD is the standard deviation across 3 scaffold splits, averaged over objectives. Further details are in Appendix H.1 .
Figure 4: Training signal on three scaffold-decoration tasks. Top: mean training reward. Bottom: fraction of the reward variance removed by the baseline (defined in the text); the dashed line is the global baseline. Thin lines: per epoch; thick lines: Gaussian-smoothed ( σ=3 epochs); values below −1 are clipped. One run per method on a single scaffold split.
Figure 5: Pure prodrugs generated for unseen parent drugs. Four of the 16 pure prodrugs (every parent–addon bond cleavable, parent skeleton unmodified), each shown next to its parent, covering three cleavable groups found (alkyl ester, alkyl carbamate, benzyl ester). Green: addon atoms released on hydrolysis; red: cleavable bond. ΔPBBB : predicted BBB-penetration lift over the parent. The other 12 are in Appendix I .
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Taxonomy of generative molecular optimization methods, organized by amortization (rows) and structural constraint (columns). The amortized + structurally-constrained quadrant (highlighted) is comparatively underserved: existing methods in this regime either rely on expensive search-based imitation (GraphXForm), struggle to scale across diverse scaffolds (LibINVENT), or use global rewards that do not handle heterogeneous scaffold difficulty (DrugEx v3). AMORTIX targets this regime. Mol GA can be applied to constrained tasks via a hard scaffold-preserving penalty; it is not natively scaffold-aware.
Figure 7: Structurally-constrained molecular design tasks. Each task corresponds to a different choice of initial subgraph s0 (outer molecules), to which atoms and bonds are added to construct the target molecule (center).
Figure 8: AMORTIX policy architecture. A partial molecular graph st and a virtual super-node are embedded and processed by an L -layer Graph Transformer with bond-order bias in the attention scores. A composite action is decoded in three sequential steps (Level 0: operation; Level 1: target atom; Level 2: bond order), each with its own encoder pass: after every decision, the action-state embeddings w(0),w(1) and the action-level embedding rℓ of the super-node are updated and the graph is re-encoded. A validity mask M(st) removes valence-violating actions at every level, so every generated molecule is chemically valid by construction.
Figure 9: Schematic comparison of advantage estimation strategies (illustrative, not measured data). (A) Global baseline (REINFORCE): a single mean over all starting structures mixes scaffold difficulty with policy quality, which inflates the variance of the advantages. (B) Group-relative baseline (GRPO): per-structure means ( μA,μB ) make each advantage reflect only the choices made on that structure, stabilizing the learning signal across easy and hard structures. The measured counterpart on three tasks is shown in Figure 4 .
Figure 10: Structural Generalization Split. t-SNE visualization of the chemical space (Morgan Fingerprints) for Training, Validation, and Test scaffolds. The cluster-based splitting strategy ensures that Test scaffolds (purple) occupy distinct regions of chemical space compared to the Training set (blue) and Validation set (orange), enforcing a test of out-of-distribution generalization.
Task
AMORTIX
f-RAG
Genetic GFN
REINVENT
GenMol
SynGBO
GP BO
Mol GA
(Ours)
Albuterol Similarity
0.972 ± 0.004
0.977 ± 0.002
0.949 ± 0.010
0.881 ± 0.016
0.937 ± 0.010
0.947 ± 0.024
0.964 ± 0.050
0.928 ± 0.015
Amlodipine MPO
0.666 ± 0.021
0.749 ± 0.019
0.761 ± 0.019
0.644 ± 0.019
0.810 ± 0.012
0.670 ± 0.088
0.720 ± 0.061
0.740 ± 0.055
Celecoxib Rediscovery
0.860 ± 0.005
0.778 ± 0.007
0.802 ± 0.029
0.717 ± 0.027
0.826 ± 0.018
0.856 ± 0.013
0.860 ± 0.002
0.629 ± 0.062
Deco Hop
0.944 ± 0.026
0.936 ± 0.011
0.733 ± 0.109
0.662 ± 0.044
0.960 ± 0.010
0.831 ± 0.039
0.672 ± 0.118
0.656 ± 0.013
DRD2
0.991 ± 0.001
0.992 ± 0.000
0.974 ± 0.006
0.957 ± 0.007
0.995 ± 0.000
0.981 ± 0.010
0.902 ± 0.117
0.950 ± 0.004
Appendix
Table 3: PMO benchmark (top-10 AUC, 22 tasks). AMORTIX is run as an online optimizer with REINFORCE and a fixed entropy coefficient β=0 on every task (mean ± SD over 3 runs). Baseline results are taken from their respective publications (3 to 5 runs), as compiled in [ 33 ] (Mol GA and REINVENT from [ 25 ] ), except GenMol, which is taken from [ 30 ] . All methods optimize online within a 10,000-call budget. The top three results per task are colored blue (1st), orange (2nd) and green (3rd); tied scores share a color.
Type
Method
Training Cost
Inference Cost
(Fixed Total)
(Per Instance)
Amortized
AMORTIX
≈ 50,000
0
LibINVENT
≈ 50,000
0
GraphXForm
≈ 335,000
0
DrugEx v3
≈ 640,000
0
Instance
Mol GA
N/A
≈ 500
Appendix
Table 4: Oracle consumption on the Kinase MPO task. For amortized models, values denote the cumulative calls allowed while training the policy. For instance optimizers, values denote the number of oracle calls until the mean best-so-far score over test scaffolds converges (Appendix F ). AMORTIX becomes cheaper in cumulative cost than Mol GA after ≈100 test scaffolds and GenMol after ≈150 test scaffolds; see Figure 3 . Within a run, only oracle calls for previously unseen molecules are counted.
Figure 11: Mol GA convergence across objectives under hard scaffold constraints. Mean best score found so far as a function of cumulative oracle calls, averaged over all test scaffolds (500 per split, 3 splits), for GSK3 β , JNK3, and Kinase MPO. Mol GA was run for a fixed budget of 10,000 oracle calls per scaffold with no early stopping; only scaffold-preserving, unique molecules count as oracle calls. Dashed lines show AMORTIX’s final scores from Table 1 . Curves reach within ≈ 0.01 of their final value after roughly 500 oracle calls for Kinase MPO, 5,000 for JNK3, and 7,000 for GSK3 β , which continues to climb slowly until the end of the budget.
Figure 12: GenMol convergence across objectives under hard scaffold constraints. Mean best score found so far as a function of cumulative oracle calls, averaged over all test scaffolds (500 per split, 3 splits), for GSK3 β , JNK3, and Kinase MPO. GenMol was run for 32 generations of 32 samples ( ≈ 1,000 calls per scaffold). Dashed lines show AMORTIX’s final scores from Table 1 . Curves reach within ≈ 0.01 of their final value after roughly 330 oracle calls for Kinase MPO, and after roughly 800 (GSK3 β ) and 900 (JNK3) calls.
Method
Used in
Pretraining data
Training size
AMORTIX
S, P
ChEMBL 35 [ 12 ]
≈ 1.5M molecules
GraphXForm [ 42 ]
S
ChEMBL 35 (same as AMORTIX)
≈ 1.5M molecules
LibINVENT [ 14 , 34 ]
S
ChEMBL 27, sliced into scaffold–decoration pairs
23.1M pairs
DrugEx v3 [ 31 ]
S
ChEMBL 27
≈ 1.7M molecules
GenMol [ 30 ]
S (V2), P (V1)
SAFE dataset [ 39 ] (ZINC + UniChem)
945M molecules
f-RAG [ 29 ]
P
SAFE-GPT [ 39 ] (ZINC + UniChem); ZINC-250k for the injection module
1.1B molecules (SAFE-GPT)
Appendix
Table 5: Pretraining data of each compared method. S: goal-directed scaffold decoration (Section 4.1 ); P: PMO benchmark (Appendix E ).
Parameter
Value
Network Architecture
Latent Dimension
512
Number of Transformer Blocks
10
Number of Attention Heads
16
Optimization (ADAM)
Learning Rate
1×10−4
Appendix
Table 6: Global Network Architecture and Optimization Hyperparameters.
AMORTIX-DeNovo
AMORTIX-REINFORCE
AMORTIX-GRPO
AMORTIX-PPO
GraphXForm
Initialization
Single Carbon
Scaffolds + C *
Scaffolds + C *
Scaffolds + C *
Single Carbon
Method
Dr. GRPO
REINFORCE †
Dr. GRPO
PPO (value critic)
TASAR (SIL)
Grouping
False
False
True
False
False
Starting Structures per Epoch
1
10
10
10
1
Group Size ( G ) ‡
160
16
16
16
160
Total Batch Size
≈160
≈160 ( 16×10 )
≈160 ( 16×10 )
≈160 ( 16×10 )
N/A §
Appendix
Table 7: Experiment-specific hyperparameters for goal-directed scaffold decoration tasks. The AMORTIX variants train until either their maximum budget of 50,000 oracle calls is exhausted or 500 epochs are completed (whichever comes first); GraphXForm runs until the maximum number of epochs is completed.
Parameter
Value
General GA Settings
Starting Population
1 (the starting scaffold)
Offspring Size
1,000
Max Generations
100
Selection Strategy
Greedy (Top-N)
Parent Sampling Strategy
Uniform Quantile Sampling (25 quantiles)
Appendix
Table 8: Hyperparameters and configuration details for Mol GA.
Objective
GRPO
PPO
REINFORCE
LibINVENT
Albuterol Similarity
0.584 ± 0.002
0.579 ± 0.004
0.557 ± 0.018
0.259 ± 0.019
Amlodipine MPO
0.462 ± 0.003
0.454 ± 0.013
0.441 ± 0.093
0.313 ± 0.031
Celecoxib Rediscovery
0.397 ± 0.002
0.345 ± 0.004
0.394 ± 0.024
0.232 ± 0.021
Deco Hop
0.848 ± 0.005
0.850 ± 0.015
0.842 ± 0.092
0.535 ± 0.005
DRD2
0.900 ± 0.002
0.600 ± 0.003
0.540 ± 0.028
0.681 ± 0.040
Fexofenadine MPO
0.802 ± 0.004
0.785 ± 0.013
0.797 ± 0.100
0.609 ± 0.008
Appendix
Table 9: Scaffold decoration across 23 objectives. Mean ± SD over 3 scaffold splits (one training run per split), where each score is the mean objective value over the 500 held-out test scaffolds of a split. The best mean per objective is in bold. Objectives marked with † are the three tasks of Table 1 .
Figure 13: Advantage SD during training. Standard deviation of the advantage over the scaffold-started trajectories of each batch, before any normalization. Thin lines: per epoch; thick lines: Gaussian-smoothed ( σ=3 epochs). One run per method on a single scaffold split. After the initial phase of fast reward improvement, GRPO gives the smallest spread on all three tasks.
Figure 14: Within- and between-scaffold reward variance for the runs of Figure 4 . Within: variance among samples of the same scaffold (the squared advantage SD); between: variance of the scaffold means. Thin lines: per epoch; thick lines: Gaussian-smoothed ( σ=3 epochs). Shows one run per task.
Task
Method
IntDiv1 (W)
IntDiv1 (A)
Uniq.
Murcko
#Circles
GSK3 β
GRPO
0.491
0.781
0.762
0.682
87
PPO
0.505
0.761
0.751
0.680
113
REINFORCE
0.550
0.784
0.940
0.627
111
JNK3
GRPO
0.459
0.764
0.687
0.598
91
PPO
0.528
0.765
0.741
0.673
118
REINFORCE
0.540
0.778
0.879
0.590
114
Appendix
Table 10: Diversity of the generated molecules. The last block averages 23 objectives. IntDiv1 (W): within the completions of a scaffold; (A): across scaffolds.
Method
Obj. Score
GSK3 β ( ↑ )
JNK3 ( ↑ )
QED ( ↑ )
SA ( ↓ )
Amortized Policies
LibINVENT
0.453±0.007
0.280±0.012
0.164±0.019
0.596±0.016
3.052±0.045
GraphXForm
0.409±0.003
0.064±0.007
0.024±0.005
0.793±0.008
3.205±0.023
DrugEx v3
0.354±0.003
0.036±0.009
0.018±0.003
0.613±0.004
3.266±0.056
AMORTIX-DeNovo
0.397±0.005
0.071±0.012
0.030±0.004
0.710±0.014
3.007±0.059
AMORTIX-REINFORCE
0.475±0.120
0.352±0.293
0.285±0.205
0.505±0.223
3.178±2.068
Appendix
Table 11: Detailed Kinase MPO sub-task breakdown. Values represent the global mean ± standard deviation across 3 scaffold splits. AMORTIX denotes the default GRPO variant. While instance optimizers default to the easier heuristic metric QED, AMORTIX successfully prioritizes the complex biological targets under hard topological constraints while keeping synthetic accessibility in a comparable range.
Figure 15: Top six partial prodrugs by ΔPBBB : each has a cleavable methyl ester (green addon, red bond) and other modifications that are not cleavable (orange addons, blue bonds).
Figure 16: Pure prodrugs (continued from Figure 5 ). The remaining 12 of the 16 pure prodrugs, each shown next to its parent. Eleven are methyl esters of a carboxylic acid. Cysteine carries a carbamate and an ester, and its thiol is left unmodified.
Molecular optimization computationally designs or edits chemical structures to improve desired properties while respecting chemical and practical constraints.