MolRGen: A Training and Evaluation Setting for De Novo Molecular Generation with Reasonning Models

Philippe Formont; Maxime Darrin; Ismail Ben Ayed; Pablo Piantanida

cs.LGMar 18, 2026

MolRGen: A Training and Evaluation Setting for De Novo Molecular Generation with Reasonning Models

Authors: Philippe Formont, Maxime Darrin, Ismail Ben Ayed, Pablo Piantanida

Organizations: Universit´e Paris-Saclay · ´ETS Montr´eal · ILLS – International Laboratory on Learning Systems · Mila – Quebec AI Institute ·

Abstract

Recent reasoning-based large language models have shown strong performance on tasks with verifiable outcomes, but their use in de novo molecular generation remains limited by the lack of training environments where rewards can be computed without reference molecules. We introduce MolRGen, a benchmark and molecular verifier for training and evaluating reasoning LLMs on de novo molecular generation. MolRGen contains approximately 4,500 protein-pocket targets, resulting in 50k multi-objective optimization prompts combining docking scores with molecular properties such as QED, synthetic accessibility, logP, and physicochemical descriptors. Unlike caption-based generation or molecule-editing benchmarks, MolRGen evaluates molecules proposed from scratch by computing rewards at generation time. We benchmark general-purpose and chemistry-specialized open-source LLMs and introduce a diversity-aware top-k metric to measure whether models can generate a diverse set of high-scoring molecules. Finally, we use the verifier to fine-tune a 128B LLM with GRPO, showing improved performance, at the cost of a diversity-exploitation trade-off. MolRGen provides a scalable testbed for studying verifier-based reasoning and reinforcement learning in molecular design.

Figures & tables

Objective ${\nu_{i}}$	Property ${\gamma_{i}}$	Reference ${x_{i}}$	Margin ${\sigma_{i}}$	Answer $\hat{o}$	Reward $r(q,\hat{o})$
De Novo Generation (per-property reward)					49k
maximize	property	n/a	threshold	gen. SMILES	${\gamma_{i}}(\hat{o})$
minimize	property	n/a	threshold	gen. SMILES	$1-{\gamma_{i}}(\hat{o})$
below	property	n/a	threshold	gen. SMILES	$\mathds{1}_{{\gamma_{i}}(\hat{o})\leq{\sigma_{i}}}$
above	property	n/a	threshold	gen. SMILES	$\mathds{1}_{{\gamma_{i}}(\hat{o})\geq{\sigma_{i}}}$
Property Prediction					55k

Table 1: Reward and prompt definition. We propose different reward functions

r(q,\hat{o})

depending on the objective

{\nu_{i}}

of the task, that are designed to reflect how close the extracted answer

\hat{o}

is to the desired outcome. All properties (and rewards) are normalized in

[0,1]

		$n_{r}=$ 16		$n_{r}=$ 24		$n_{r}=$ 32		$n_{r}=$ 48
Model	Size	$\text{top}_{1}$	$\text{top}_{16}$	$\text{top}_{1}$	$\text{top}_{16}$	$\text{top}_{1}$	$\text{top}_{16}$	$\text{top}_{1}$	$\text{top}_{16}$
General LLMs
gemma-4	31B	0.65 $\pm$ 0.05	0.37 $\pm$ 0.05	0.67 $\pm$ 0.04	0.48 $\pm$ 0.03	0.68 $\pm$ 0.03	0.52 $\pm$ 0.02	0.7 $\pm$ 0.01	0.56 $\pm$ 0.01
Mistral-Small-4	128B	0.6 $\pm$ 0.06	0.32 $\pm$ 0.04	0.62 $\pm$ 0.05	0.44 $\pm$ 0.03	0.64 $\pm$ 0.04	0.48 $\pm$ 0.02	0.66 $\pm$ 0.02	0.52 $\pm$ 0.01
MiniMax-M2	229B	0.61 $\pm$ 0.06	0.31 $\pm$ 0.05	0.64 $\pm$ 0.05	0.43 $\pm$ 0.03	0.66 $\pm$ 0.04	0.48 $\pm$ 0.02	0.68 $\pm$ 0.02	0.52 $\pm$ 0.01
Qwen3-Next	80B	0.58 $\pm$ 0.05	0.3 $\pm$ 0.04	0.6 $\pm$ 0.04	0.41 $\pm$ 0.03	0.61 $\pm$ 0.03	0.45 $\pm$ 0.02	0.63 $\pm$ 0.02	0.49 $\pm$ 0.01

Table 2: De novo generation results. Evaluation of various LLMs on the molecular generation task. We report the average top-k scores among

n_{r}

generations per prompt and standard deviations, averaged over the unique prompts. Models are ordered by their release date.

Figure 2: Diversity-aware top-k score. Evaluation of the diversity-aware top-k score (y-axis) against varying similarity thresholds (x-axis) between candidate clusters. As the similarity threshold increases, the diversity constraint relaxes, and the average score of the selected molecule pool increases.

Figure 3: GRPO Training Effect. (Top)

\text{top}_{k,n_{r}}

scores across varying sampling scales

n_{r}/k

. (Bottom) Distribution of pairwise Tanimoto similarities for molecules generated for the same prompt.

Figure 4: Property prediction performances. ROC-AUC scores of the LLMs on classification tasks (top), and normalized Pearson correlation on regression tasks (bottom).

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Property	Allowed Objectives		Relative Frequency
Property	Objectives	Proba.	Relative Frequency
SA	minimize	0.5	3.0
	below	0.5	3.0
QED	maximize	0.7	7.0
	above	0.3	7.0
ExactMolWt	above	0.3	0.8

Appendix

Table 3: Allowed objectives for molecular properties, with their associated selection probability. We also include the relative frequency, defining the probability of selecting a given property (higher values, such as QED or logP, are sampled more often).

Panel of Figure 5 — Figure 5: Overview of the target proteins. (a) Function of the proteins extracted from the PDB, our dataset comprises 21 molecular functions with at least 10 targets, the majority of which are kinases (30%). (b) Annotation score of the proteins on UniProt (from 1 to 5). The vast majority of the target proteins are high-quality proteins with strong evidence of their existence.

Figure 6: Assessment of our pocket detection strategy compared to fpocket predictions. Histogram of the Intersection over Union (IoU) between the pockets we extracted and the pockets predicted by fpocket, for 1000 targets. We show the predicted druggability score of the pockets with fpocket, with more than half of the pockets having a druggability score above 0.5 (considered druggable by fpocket).

Figure 7: Task sizes in the molecular property prediction objectives. The vast majority of tasks consist of regression tasks, and the largest benchmark used is the TDC benchmark.

Figure 8: Scaffold occurrence in the various benchmarks. Occurrences of the most frequent Murcko scaffolds (of at least 6 atoms) in each benchmark, illustrating the chemical diversity across tasks.

Figure 9: Overview of the molecular reaction dataset generation pipeline. Iterative stochastic process of synthesis generation: initialization with seed reactions, relaxed filtering for early steps, property filtering for later steps, probabilistic product selection, and chain extension up to 5 reaction steps.

Property	Min	Max
QED (Drug-likeness)	0.30	1.00
Molecular Weight (Da)	0	600
TPSA (Å 2 )	0	160
H-Bond Acceptors	0	10
H-Bond Donors	0	10
Rotatable Bonds	1	10

Appendix

Table 4: Molecular property constraints applied during synthesis generation.

Figure 11: Frequency and chemical diversity of reaction templates.

Figure 12: Training curves of RL-Molstral. Evolution of the average reward (left) and average completion length (right) during training.

Figure 13: Validity of the generated completions. Description of the validity of the generated completions. Generations can be invalid due to no answer being generated in the expected format, no SMILES being parsed in the answer, or no valid SMILES or multiple SMILES being proposed.

Figure 14: Uniqueness and diversity evolution with the number of rollouts. We display the uniqueness (left) and diversity (right) of the generated molecules with respect to the number of rollouts.

Panel of Figure 15 — Figure 15: Diversity-aware top-k score for different fingerprints. We display the diversity-aware metric when the similarity between molecules is based on various families of fingerprints.

		$n_{r}=$ 16		$n_{r}=$ 24		$n_{r}=$ 32		$n_{r}=$ 48
Model	Size	$\text{top}_{1}$	$\text{top}_{16}$	$\text{top}_{1}$	$\text{top}_{16}$	$\text{top}_{1}$	$\text{top}_{16}$	$\text{top}_{1}$	$\text{top}_{16}$
reinvent	0.2B	0.78 $\pm$ 0.03	0.53 $\pm$ 0.04	0.78 $\pm$ 0.01	0.63 $\pm$ 0.03	0.79 $\pm$ 0.02	0.66 $\pm$ 0.02	0.8 $\pm$ 0.01	0.7 $\pm$ 0.01
gemma-4	31B	0.64 $\pm$ 0.08	0.33 $\pm$ 0.05	0.68 $\pm$ 0.05	0.45 $\pm$ 0.03	0.68 $\pm$ 0.05	0.5 $\pm$ 0.02	0.71 $\pm$ 0.02	0.54 $\pm$ 0.01
Mistral-Small-4	128B	0.57 $\pm$ 0.06	0.29 $\pm$ 0.04	0.59 $\pm$ 0.05	0.4 $\pm$ 0.03	0.62 $\pm$ 0.04	0.43 $\pm$ 0.02	0.63 $\pm$ 0.02	0.47 $\pm$ 0.01
MiniMax-M2	229B	0.59 $\pm$ 0.09	0.3 $\pm$ 0.05	0.62 $\pm$ 0.06	0.41 $\pm$ 0.04	0.63 $\pm$ 0.07	0.46 $\pm$ 0.02	0.68 $\pm$ 0.01	0.49 $\pm$ 0.01
ChemDFM-R	14B	0.58 $\pm$ 0.08	0.3 $\pm$ 0.04	0.62 $\pm$ 0.05	0.39 $\pm$ 0.03	0.63 $\pm$ 0.05	0.43 $\pm$ 0.02	0.65 $\pm$ 0.02	0.47 $\pm$ 0.02

Appendix

Table 5: De novo generation results. Evaluation of various LLMs on the molecular generation task. We report the average top-k scores among

n_{r}

generations per prompt and standard deviations, averaged over the unique prompts. Models are ordered by their release date.

Figure 16: Diversity-Aware scores with Reinvent. We compare the performance of RL-Molstral-g16 to a traditional generation algorithm baseline, which is a genetic algorithm optimizing the same reward as the one used for the LLMs.

Figure 17: Extraction results for the molecular property prediction. Proportion of prompts for which none of the 5 generated answers could be processed into a numerical prediction. We highlight with the colors of the barplots the proportion of prompts where at least one of the completion had a number in the extracted answer, but was rejected (most often due to ambiguous answer).

	ames	bbb	cyp2c9	cyp2d6	cyp3a4	dili	herg	pgp	pkis2-drewry
	ROC-AUC
ChemDFM-R	0.51	0.73	0.48	0.45	0.54	0.48	0.53	0.50	0.49
MiniMax-M2	0.60	0.54	0.45	0.50	0.41	0.52	0.57	0.45	0.39
Mistral-Small-4	0.70	0.73	0.55	0.64	0.54	0.56	0.75	0.59	0.50
Qwen3	0.71	0.62	0.60	0.65	0.56	0.68	0.74	0.66	0.55
Qwen3-Next	0.74	0.73	0.64	0.65	0.56	0.79	0.73	0.72	0.55

Appendix

Table 6: Comprehensive Classification Results.

Figure 18: Effect of reinforcement learning on the molecular property prediction performance. We display the difference

\Delta

in performance between RL-Mistral and Mistral-Small-4 on all classification and regression tasks.

Panel of Figure 19 — Figure 19: Docking poses for sample 772296 generated by different models trying to minimize the docking score to this target.

Panel of Figure 20 — Figure 20: Docking poses for sample 803976 generated by different models trying to minimize the docking score to this target.

Panel of Figure 21 — Figure 21: Docking poses for sample 941067 generated by different models trying to minimize the docking score to this target along with minimizing the SA score.

Figure 30: Ether0 refusal. Representative examples where Ether0 refuses to generate a molecule, interpreting property-optimization, or prediction instructions as requests to produce harmful substances.

Molecular Generation

Molecular generation develops computational methods to create chemical structures and conformations for discovery, optimization, and structure elucidation.

See all 231 papers

Adopting Reinforcement Learning with Verifiable Rewards for Molecular Generation
Jul 21, 2026Mingxuan Ouyang, Hao Lan, Wanyu LinMolecular Optimization Reinforcement Learning with Verifiable Rewards
MolDesignBench: Evaluating LLM-based Agent for Scenario-grounded Molecular Design
Sep 23, 2026Yongjun Jeong, Hanbum Ko, Ye Rin Kim +6Molecular Optimization LLM Agent Evaluation
Department of Artificial Intelligence, Korea University · LG AI Research · Department of Statistics, Korea University
SMolLM: Small Language Models Learn Small Molecular Grammar
May 7, 2026Akhil Jindal, Harang JuSmall Language Models Mechanistic Interpretability
Moku

Jul 21, 2026J/K move · Enter open · S save

Adopting Reinforcement Learning with Verifiable Rewards for Molecular Generation

Mingxuan Ouyang, Hao Lan, Wanyu Lin

Molecular Optimization Reinforcement Learning with Verifiable Rewards Molecular Generation

Open paper AskPDF arXiv

Leveraging large language models (LLMs) for molecular generation has shown remarkable potential in chemical and drug design. Current methods primarily rely on supervised training or fine-tuning with limited datasets, which are insufficient to capture complex molecular design objectives. While some approaches attempt to guide generation toward specific goals, they often lack direct optimization mechanisms, making it difficult to align generated molecules with desired properties. To tackle these challenges, we propose LLMol, a principled reinforcement learning framework that directly incorporates verifiable rewards for targeted molecule generation. The key insight is to formulate molecular design as a goal-conditioned sequence prediction task, where verifiable rewards serve as explicit supervision to drive generation toward desired objectives. LLMol follows a two-stage training paradigm combining supervised learning and reinforcement learning. In the first stage, large language models are supervised fine-tuned to capture chemical syntax and molecular distributions. In the second stage, we introduce Reinforcement Learning with Verifiable Rewards (RLVR), which directly integrates property-based reward signals to guide molecular generation toward task-specific objectives. To address the high variance and instability common in discrete sequence optimization, we adopt Group Relative Policy Optimization (GRPO), a stable on-policy algorithm that smooths reward signals and improves training robustness. This framework enables LLMol to effectively handle a range of molecular design tasks, including single-property targeting (e.g., penalized logP, QED) and structure-constrained optimization. Experimental results demonstrate that LLMol consistently outperforms existing methods, achieving higher success rates and improved efficiency across diverse molecular benchmarks.

Abstract

Molecular Generation

Explore similar work

Adopting Reinforcement Learning with Verifiable Rewards for Molecular Generation

MolDesignBench: Evaluating LLM-based Agent for Scenario-grounded Molecular Design

SMolLM: Small Language Models Learn Small Molecular Grammar