MolRGen: A Training and Evaluation Setting for De Novo Molecular Generation with Reasonning Models
Authors: Philippe Formont, Maxime Darrin, Ismail Ben Ayed, Pablo Piantanida
Organizations: Universit´e Paris-Saclay · ´ETS Montr´eal · ILLS – International Laboratory on Learning Systems · Mila – Quebec AI Institute · Mistral AI · LIVIA · CNRS, CentraleSup´elec
Recent reasoning-based large language models have shown strong performance on tasks with verifiable outcomes, but their use in de novo molecular generation remains limited by the lack of training environments where rewards can be computed without reference molecules. We introduce MolRGen, a benchmark and molecular verifier for training and evaluating reasoning LLMs on de novo molecular generation. MolRGen contains approximately 4,500 protein-pocket targets, resulting in 50k multi-objective optimization prompts combining docking scores with molecular properties such as QED, synthetic accessibility, logP, and physicochemical descriptors. Unlike caption-based generation or molecule-editing benchmarks, MolRGen evaluates molecules proposed from scratch by computing rewards at generation time. We benchmark general-purpose and chemistry-specialized open-source LLMs and introduce a diversity-aware top-k metric to measure whether models can generate a diverse set of high-scoring molecules. Finally, we use the verifier to fine-tune a 128B LLM with GRPO, showing improved performance, at the cost of a diversity-exploitation trade-off. MolRGen provides a scalable testbed for studying verifier-based reasoning and reinforcement learning in molecular design.
Figures & tables
Figure 1
Objective νi
Property γi
Reference xi
Margin σi
Answer o^
Reward r(q,o^)
De Novo Generation (per-property reward)
49k
maximize
property
n/a
threshold
gen. SMILES
γi(o^)
minimize
property
n/a
threshold
gen. SMILES
1−γi(o^)
below
property
n/a
threshold
gen. SMILES
\mathds1γi(o^)≤σi
above
property
n/a
threshold
gen. SMILES
\mathds1γi(o^)≥σi
Property Prediction
55k
Table 1: Reward and prompt definition. We propose different reward functions r(q,o^) depending on the objective νi of the task, that are designed to reflect how close the extracted answer o^ is to the desired outcome. All properties (and rewards) are normalized in [0,1] .
nr= 16
nr= 24
nr= 32
nr= 48
Model
Size
top1
top16
top1
top16
top1
top16
top1
top16
General LLMs
gemma-4
31B
0.65 ± 0.05
0.37 ± 0.05
0.67 ± 0.04
0.48 ± 0.03
0.68 ± 0.03
0.52 ± 0.02
0.7 ± 0.01
0.56 ± 0.01
Mistral-Small-4
128B
0.6 ± 0.06
0.32 ± 0.04
0.62 ± 0.05
0.44 ± 0.03
0.64 ± 0.04
0.48 ± 0.02
0.66 ± 0.02
0.52 ± 0.01
MiniMax-M2
229B
0.61 ± 0.06
0.31 ± 0.05
0.64 ± 0.05
0.43 ± 0.03
0.66 ± 0.04
0.48 ± 0.02
0.68 ± 0.02
0.52 ± 0.01
Qwen3-Next
80B
0.58 ± 0.05
0.3 ± 0.04
0.6 ± 0.04
0.41 ± 0.03
0.61 ± 0.03
0.45 ± 0.02
0.63 ± 0.02
0.49 ± 0.01
Table 2: De novo generation results. Evaluation of various LLMs on the molecular generation task. We report the average top-k scores among nr generations per prompt and standard deviations, averaged over the unique prompts. Models are ordered by their release date.
Figure 2: Diversity-aware top-k score. Evaluation of the diversity-aware top-k score (y-axis) against varying similarity thresholds (x-axis) between candidate clusters. As the similarity threshold increases, the diversity constraint relaxes, and the average score of the selected molecule pool increases.
Figure 3: GRPO Training Effect. (Top) topk,nr scores across varying sampling scales nr/k . (Bottom) Distribution of pairwise Tanimoto similarities for molecules generated for the same prompt.
Figure 4: Property prediction performances. ROC-AUC scores of the LLMs on classification tasks (top), and normalized Pearson correlation on regression tasks (bottom).
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Property
Allowed Objectives
Relative Frequency
Objectives
Proba.
SA
minimize
0.5
3.0
below
0.5
QED
maximize
0.7
7.0
above
0.3
ExactMolWt
above
0.3
0.8
Appendix
Table 3: Allowed objectives for molecular properties, with their associated selection probability. We also include the relative frequency, defining the probability of selecting a given property (higher values, such as QED or logP, are sampled more often).
Figure 5: Overview of the target proteins. (a) Function of the proteins extracted from the PDB, our dataset comprises 21 molecular functions with at least 10 targets, the majority of which are kinases (30%). (b) Annotation score of the proteins on UniProt (from 1 to 5). The vast majority of the target proteins are high-quality proteins with strong evidence of their existence.
Figure 6: Assessment of our pocket detection strategy compared to fpocket predictions. Histogram of the Intersection over Union (IoU) between the pockets we extracted and the pockets predicted by fpocket, for 1000 targets. We show the predicted druggability score of the pockets with fpocket, with more than half of the pockets having a druggability score above 0.5 (considered druggable by fpocket).
Figure 7: Task sizes in the molecular property prediction objectives. The vast majority of tasks consist of regression tasks, and the largest benchmark used is the TDC benchmark.
Figure 8: Scaffold occurrence in the various benchmarks. Occurrences of the most frequent Murcko scaffolds (of at least 6 atoms) in each benchmark, illustrating the chemical diversity across tasks.
Figure 9: Overview of the molecular reaction dataset generation pipeline. Iterative stochastic process of synthesis generation: initialization with seed reactions, relaxed filtering for early steps, property filtering for later steps, probabilistic product selection, and chain extension up to 5 reaction steps.
Property
Min
Max
QED (Drug-likeness)
0.30
1.00
Molecular Weight (Da)
0
600
TPSA (Å 2 )
0
160
H-Bond Acceptors
0
10
H-Bond Donors
0
10
Rotatable Bonds
1
10
Appendix
Table 4: Molecular property constraints applied during synthesis generation.
Figure 14
Figure 11: Frequency and chemical diversity of reaction templates.
Figure 12: Training curves of RL-Molstral. Evolution of the average reward (left) and average completion length (right) during training.
Figure 13: Validity of the generated completions. Description of the validity of the generated completions. Generations can be invalid due to no answer being generated in the expected format, no SMILES being parsed in the answer, or no valid SMILES or multiple SMILES being proposed.
Figure 14: Uniqueness and diversity evolution with the number of rollouts. We display the uniqueness (left) and diversity (right) of the generated molecules with respect to the number of rollouts.
Figure 15: Diversity-aware top-k score for different fingerprints. We display the diversity-aware metric when the similarity between molecules is based on various families of fingerprints.
nr= 16
nr= 24
nr= 32
nr= 48
Model
Size
top1
top16
top1
top16
top1
top16
top1
top16
reinvent
0.2B
0.78 ± 0.03
0.53 ± 0.04
0.78 ± 0.01
0.63 ± 0.03
0.79 ± 0.02
0.66 ± 0.02
0.8 ± 0.01
0.7 ± 0.01
gemma-4
31B
0.64 ± 0.08
0.33 ± 0.05
0.68 ± 0.05
0.45 ± 0.03
0.68 ± 0.05
0.5 ± 0.02
0.71 ± 0.02
0.54 ± 0.01
Mistral-Small-4
128B
0.57 ± 0.06
0.29 ± 0.04
0.59 ± 0.05
0.4 ± 0.03
0.62 ± 0.04
0.43 ± 0.02
0.63 ± 0.02
0.47 ± 0.01
MiniMax-M2
229B
0.59 ± 0.09
0.3 ± 0.05
0.62 ± 0.06
0.41 ± 0.04
0.63 ± 0.07
0.46 ± 0.02
0.68 ± 0.01
0.49 ± 0.01
ChemDFM-R
14B
0.58 ± 0.08
0.3 ± 0.04
0.62 ± 0.05
0.39 ± 0.03
0.63 ± 0.05
0.43 ± 0.02
0.65 ± 0.02
0.47 ± 0.02
Appendix
Table 5: De novo generation results. Evaluation of various LLMs on the molecular generation task. We report the average top-k scores among nr generations per prompt and standard deviations, averaged over the unique prompts. Models are ordered by their release date.
Figure 16: Diversity-Aware scores with Reinvent. We compare the performance of RL-Molstral-g16 to a traditional generation algorithm baseline, which is a genetic algorithm optimizing the same reward as the one used for the LLMs.
Figure 17: Extraction results for the molecular property prediction. Proportion of prompts for which none of the 5 generated answers could be processed into a numerical prediction. We highlight with the colors of the barplots the proportion of prompts where at least one of the completion had a number in the extracted answer, but was rejected (most often due to ambiguous answer).
ames
bbb
cyp2c9
cyp2d6
cyp3a4
dili
herg
pgp
pkis2-drewry
ROC-AUC
ChemDFM-R
0.51
0.73
0.48
0.45
0.54
0.48
0.53
0.50
0.49
MiniMax-M2
0.60
0.54
0.45
0.50
0.41
0.52
0.57
0.45
0.39
Mistral-Small-4
0.70
0.73
0.55
0.64
0.54
0.56
0.75
0.59
0.50
Qwen3
0.71
0.62
0.60
0.65
0.56
0.68
0.74
0.66
0.55
Qwen3-Next
0.74
0.73
0.64
0.65
0.56
0.79
0.73
0.72
0.55
Appendix
Table 6: Comprehensive Classification Results.
Figure 18: Effect of reinforcement learning on the molecular property prediction performance. We display the difference Δ in performance between RL-Mistral and Mistral-Small-4 on all classification and regression tasks.
Figure 19: Docking poses for sample 772296 generated by different models trying to minimize the docking score to this target.
Figure 20: Docking poses for sample 803976 generated by different models trying to minimize the docking score to this target.
Figure 21: Docking poses for sample 941067 generated by different models trying to minimize the docking score to this target along with minimizing the SA score.
Figure 30: Ether0 refusal. Representative examples where Ether0 refuses to generate a molecule, interpreting property-optimization, or prediction instructions as requests to produce harmful substances.
Molecular generation develops computational methods to create chemical structures and conformations for discovery, optimization, and structure elucidation.
Leveraging large language models (LLMs) for molecular generation has shown remarkable potential in chemical and drug design. Current methods primarily rely on supervised training or fine-tuning with limited datasets, which are insufficient to capture complex molecular design objectives. While some approaches attempt to guide generation toward specific goals, they often lack direct optimization mechanisms, making it difficult to align generated molecules with desired properties. To tackle these challenges, we propose LLMol, a principled reinforcement learning framework that directly incorporates verifiable rewards for targeted molecule generation. The key insight is to formulate molecular design as a goal-conditioned sequence prediction task, where verifiable rewards serve as explicit supervision to drive generation toward desired objectives. LLMol follows a two-stage training paradigm combining supervised learning and reinforcement learning. In the first stage, large language models are supervised fine-tuned to capture chemical syntax and molecular distributions. In the second stage, we introduce Reinforcement Learning with Verifiable Rewards (RLVR), which directly integrates property-based reward signals to guide molecular generation toward task-specific objectives. To address the high variance and instability common in discrete sequence optimization, we adopt Group Relative Policy Optimization (GRPO), a stable on-policy algorithm that smooths reward signals and improves training robustness. This framework enables LLMol to effectively handle a range of molecular design tasks, including single-property targeting (e.g., penalized logP, QED) and structure-constrained optimization. Experimental results demonstrate that LLMol consistently outperforms existing methods, achieving higher success rates and improved efficiency across diverse molecular benchmarks.
Real-world molecular design remains challenging for large language model (LLM)-based agents. It requires them to interpret design contexts, satisfy multiple constraints, identify infeasible specifications, and reason over multi-step tool outputs. Existing benchmarks do not capture this complexity, focusing instead on explicit and narrow constraints, only feasible problems, and single-path solutions. To address this gap, we propose MolDesignBench, a scenario-grounded benchmark that more closely reflects real-world molecular design for evaluating tool-augmented LLM agents. MolDesignBench comprises 2K generation and optimization instances that combine implicit requirements embedded in design narratives with explicit property and functional-group constraints, including infeasible cases, and require the effective use of 17 specialized chemistry tools. Experiments across diverse frontier LLMs reveal low success rates--with the best achieving only ∼43%--and frequent failures in implicit-constraint reasoning, infeasibility detection, and tool reasoning. The corresponding fine-grained failure-mode analysis identifies implicit constraint interpretation and infeasibility detection as the primary bottlenecks, establishing MolDesignBench as a rigorous testbed to guide future research on chemical agents. The benchmark, tool interface, and evaluation code are publicly available.
Yongjun Jeong, Hanbum Ko, Ye Rin Kim +6
Department of Artificial Intelligence, Korea University · LG AI Research · Department of Statistics, Korea University
Language models for molecular design have scaled to hundreds of millions of parameters, yet how they learn chemical grammar is poorly understood. We train SMolLM, a 53K-parameter weight-shared transformer, to generate novel SMILES with 95% validity on the ZINC-250K drug-like-molecule benchmark, outperforming a standard GPT with 10 times more parameters. Mechanistically, the same block resolves SMILES constraints across passes in a fixed hierarchy: brackets first, rings second, and valence last, as shown by error classification and linear probing, with ablation isolating the bracket-matching head. Together, these results yield a compact, mechanistically interpretable molecular generator and a testbed for studying iterative computation in formal-language domains.