MolHop: Scaffold-Hopping Molecular Design with Agentic Reinforcement Learning

Yanyi Su; Chenyan Wang; Yulei Gong; Yeyun Chen; Xiaohui Yang; Jun Cheng

cs.LGOct 14, 2026

MolHop: Scaffold-Hopping Molecular Design with Agentic Reinforcement Learning

Authors: Yanyi Su, Chenyan Wang, Yulei Gong, Yeyun Chen, Xiaohui Yang, Jun Cheng

Organizations: State Key Laboratory of Physical Chemistry of Solid Surface, College of Chemistry and Chemical Engineering, Xiamen University, Xiamen, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Institute of Artificial Intelligence, Xiamen University, Xiamen, China · Laboratory of AI for Electrochemistry (AI4EC), Tan Kah Kee Innovation Laboratory (IKKEM), Xiamen, China

Abstract

Molecular design is often formulated as an optimization problem over existing molecular spaces. However, molecular optimization and the discovery of structurally distinct replacement molecules represent two fundamentally different search problems. LLM-based molecular design agents typically perform the former, conducting local refinement within the neighborhood of existing structures to improve target properties through structural modifications. In contrast, the latter requires exploration of a broader chemical space to discover molecules with entirely different scaffolds while preserving the functional characteristics of reference molecules. We propose MolHop, an agentic framework for cross-scaffold molecular discovery that enables models to navigate and edit within a pre-organized chemical space. MolHop organizes approximately 1.6 million molecular scaffolds from commercial molecules into a coarse-to-fine hierarchical action space. The agent selects new exploration regions through scaffold navigation actions and adjusts candidate properties through molecular editing. The framework maintains historical trajectory memory to support long-horizon search processes. On drug molecule and commercial compound tasks, connecting Grok-4.6 and DeepSeek-Flash-V4.1 to the MolHop framework substantially improves their cross-scaffold joint success rates. We develop MolHop-9B through agentic reinforcement learning, and MolHop-9B outperforms generative baselines including GPT-5.6-sol, Claude-5-Opus, and Gemini-3.1-Pro, while generating candidate molecules with larger scaffold differences from the reference molecules. These results demonstrate that effective molecular discovery beyond the traditional optimization paradigm requires structured chemical and action spaces, enabling agents to explore new structural regions rather than merely refine existing molecules.

Figures & tables

Figure 1: Overview of MolHop, a hierarchical scaffold navigation framework with memory-guided molecular editing. (a) Illustration of scaffold hopping across molecular space. (b) Hierarchical scaffold space and coarse-to-fine scaffold query. (c) Agent interaction with MolHop tools for molecular inspection and editing. (d) Joint success criteria for candidate evaluation. (e) Trajectory memory for long-horizon scaffold search.

Dataset	Model	Method	Joint@H1 $\uparrow$	Tanimoto median $\downarrow$	MCS median $\downarrow$
Drug	Grok	Baseline	41.0%	0.292	0.913
		With harness	64.0%	0.141	0.684
	DeepSeek	Baseline	15.5%	0.381	0.966
		With harness	52.5%	0.133	0.700
Commercial	Grok	Baseline	36.5%	0.317	0.833
		With harness	69.0%	0.145	0.600

Table 1: Cross-Scaffold Molecular Discovery Performance Comparison

Table 2: Cross-scaffold molecular discovery results. Joint denotes joint@H1, Tan. denotes scaffold Tanimoto similarity, and MCS denotes scaffold MCS ratio. Higher Joint values indicate better scaffold-constrained molecular discovery performance.

Figure 2: (a) Joint@H1 versus median scaffold MCS on the ChEMBL test set. (b) Joint@H1 versus median scaffold MCS on the Enamine test set. Points toward the upper-left region indicate higher joint success with larger scaffold transition.

Model	VER $\uparrow$	BOR $\downarrow$	EFER $\uparrow$	Eff. $\downarrow$	ITI $\downarrow$
2B Base	21.80%	71.80%	0.20%	1490.9	29.50%
2B SFT	91.10%	8.10%	2.20%	441.8	0.60%
4B Base	18.10%	81.70%	3.70%	218.2	3.30%
4B SFT	97.80%	2.20%	19.40%	125.1	0.40%
9B Base	4.60%	95.40%	13.70%	174.2	0.90%
9B SFT	97.80%	2.00%	19.70%	118.9	0.20%

Table 3: Effect of SFT on Tool-Use Capability

Figure 3: Tool-use trajectory distributions of MolHop agents and frozen LLM agents. (a) MolHop-2B. (b) MolHop-4B. (c) MolHop-9B. (d) Grok-4.6. (e) DeepSeek-Flash-V4.1. The stacked areas show the fraction of trajectories assigned to the four action categories Navigation, Scaffold Selection, Edit, and Finish over normalized trajectory progress.

Figure 4: Ablation study of MolHop components. Three variants are retrained by removing molecular editing, trajectory memory, or hierarchical scaffold library from MolHop-4B. (a) Training reward. (b) Validation joint@H1. (c) Final joint@H1 on ChEMBL. (d) Final joint@H1 on Enamine.

Figure 5: OOD targets, restricted-library samples, and DeepSeek outputs in the property PCA space. PCA projection based on five molecular properties, logP, TPSA, MW, Labute ASA, and MolMR. Red open circles denote OOD targets, blue points denote library samples, and purple points denote MolHop harness predictions, with filled and open purple markers indicating joint@H1 met and joint@H1 not met outputs, respectively.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Level	Retained structural information	Abstraction rule	Number of nodes
H1	Ring-count category	Record the number of ring nodes generated by the approximate smallest-ring representation	14
H2	Connectivity topology between rings	Represent each ring as a node, retain connections between rings, and remove connection types	498
H3	Ring topology with connection types	Distinguish connection types between rings on the basis of H2	1,451
H4	Minimal scaffold topology	Generalize atoms and bonds and reduce degree-2 nodes while retaining minimal ring structures	14,378
H5	Ring sizes and reduced connecting chains	Retain topology within rings and compress non-ring connecting chains of degree-2 nodes	153,699
H6	Complete connectivity graph of a generalized scaffold	Remove atom and bond types while retaining connectivity and connecting-chain lengths	412,853

Appendix

Table A1: Hierarchical Scaffold Space Statistics

Figure A1: Construction and query process of the hierarchical scaffold space. (a) Coarse-to-fine scaffold abstraction from H1 ring-level categories to H7 typed scaffolds. (b) Example scaffold query process, including scope selection, exact scaffold filtering, regrouping, and scaffold selection.

Action group	Action	Description
Scaffold navigation	nav_where	Query the position of the current molecule in H1-H7 space and the scope available for exploration
Scaffold navigation	nav_list	List categories, subregions, and specific scaffolds at a specified level or within a candidate set
Scaffold navigation	nav_filter	Apply combined structural constraints to H7 leaf nodes and return a candidate set
Scaffold navigation	nav_pick	Select a scaffold from previously returned candidates and load it as a new starting point for molecular editing
Molecular inspection	mol_show	Enumerate editable sites for a specified action category, with all mode returning only the number of sites in each category
Molecular inspection	mol_site	Obtain permitted actions based on site identifiers returned by mol_show , and obtain the action and fragment identifiers allowed at that site

Appendix

Table A2: MolHop action space.

Object	Action	Description
Fragment	add_fragment	Add a fragment at a permitted attachment site
Fragment	delete_fragment	Delete a fragment whose removal is permitted
Fragment	replace_fragment	Replace the current fragment with another fragment
Fragment	fragment_swap	Replace a degree-2 atom whose two adjacent bonds are both non-ring bonds with a structural fragment having two attachment points to alter the connecting chain
Atoms and bonds	mutate_atom	Modify the atom type at a permitted site
Atoms and bonds	change_bond	Modify a permitted bond type or bond order

Appendix

Table A3: Molecular editing actions implemented by mol_edit.

Content	Recorded information	Role in search
Reference target	Starting molecule, five target properties, and initial SA score	Keep the optimization target fixed across multi-turn interactions
Current property feedback	Signed differences for each property, absolute relative deviations, match distance, normalized SA score, total cost, and the property with the largest deviation	Indicate which properties should be prioritized for adjustment in the next step and compare different candidates
Evaluation status	Current scaffold load identifier, molecular revision identifier, evaluated structure, and gap_is_current	Distinguish feedback on the current candidate from feedback on historical structures
Best historical record	Lowest cost observed so far, corresponding SMILES, step identifier, and revision identifier	Retain a search reference for review, which is not equivalent to the best candidate already satisfying joint
Scaffold visitation trajectory	Visited scaffold categories, abstraction levels, entry sources, and corresponding scaffolds	Record search transitions between different structural regions
Within-category editing trajectory	Revision identifiers, action summaries, property gaps, undo flags, and restoration targets	Associate local structural changes with property feedback

Appendix

Table A4: Trajectory Memory State Structure

Figure A2: Construction and use of trajectory memory in MolHop. (a) Search history within an illustrative trajectory, including scaffold transitions, molecular revisions, and undo operations. (b) Memory state at the current revision, summarizing target status, retrieval trail, revision ledger, and current property gaps. (c) The agent reads memory and selects the next action.

Data use	ChEMBL	Enamine	Total size	Data split
Drug test set	200	0	200	Scaffold deduplication, ring-count stratification, and diversity selection
Commercial test set	0	200	200	Scaffold deduplication, ring-count stratification, and diversity selection
SFT starting points	500	1,500	2,000	Stratified sampling, 1,800 train / 200 validation
RL starting points	750	4,250	5,000	Supplement with the remaining eligible samples, 4,872 train / 128 validation

Appendix

Table B1: Experimental Dataset Split

Setting	Configuration
Number of training and validation trajectories	1,800 and 200
Fine-tuning method	Full-parameter fine-tuning, bf16, FSDP
Training epochs	3 epochs, 675 optimization steps in total
Global batch size	8
Maximum sequence length	16,384 tokens
Optimizer	AdamW

Appendix

Table B2: SFT training configuration

Setting	Configuration
Number of training and validation starting points	4,872 and 128
Starting points per batch	16
Rollouts per starting point	4
Maximum assistant turns per episode	50
Training sampling temperature and top-p	1.0 and 1.0
Validation sampling temperature	0

Appendix

Table B3: GRPO training configuration

	joint	Valid	md<1	SA $\uparrow$	H1	Valid $\land$ md $\land$ SA
Grok	82	170	148	144	103	123
Grok + harness	128	200	181	146	190	134
DeepSeek	31	187	107	142	87	70
DeepSeek + harness	105	200	183	123	192	111

Appendix

Table C1: Drug dataset

	joint	Valid	md<1	SA $\uparrow$	H1	Valid $\land$ md $\land$ SA
Grok	73	166	145	137	86	119
Grok + harness	138	200	186	149	195	140
DeepSeek	35	186	128	134	82	85
DeepSeek + harness	121	200	190	136	188	129

Appendix

Table C2: Commercial dataset

Figure C1: Agentic RL training and cross-scaffold discovery performance of MolHop agents. (a) GRPO training reward curves for MolHop-2B, MolHop-4B, and MolHop-9B. (b) Validation joint@H1 performance during GRPO training.

Figure C2: Joint success versus median scaffold MCS under different scaffold transition criteria from H1 to H7. Higher joint success and lower MCS correspond to the upper-left region. LLM-based baselines improve mainly under relaxed scaffold criteria, while MolHop and GenMol remain stable.

	joint	Valid	md<1	SA $\uparrow$	H1	Valid $\land$ md $\land$ SA
MolHop-2B	57	200	88	134	199	57
MolHop-4B	96	200	137	139	197	97
MolHop-9B	111	200	142	157	195	111
Gemini-3.1-Pro	87	199	186	154	109	144
GPT-5.6-sol	99	196	166	168	132	141
Claude-5-Opus	107	192	162	173	132	150

Appendix

Table C3: Drug dataset

	joint	Valid	md<1	SA $\uparrow$	H1	Valid $\land$ md $\land$ SA
MolHop-2B	67	200	100	136	200	67
MolHop-4B	105	200	144	137	197	105
MolHop-9B	147	200	165	170	200	147
Gemini-3.1-Pro	94	197	190	165	105	159
GPT-5.6-sol	123	199	180	182	143	163
Claude-5-Opus	108	195	159	189	132	154

Appendix

Table C4: Commercial dataset

	Drug molecules			Commercial molecules
Model	TESR a	Efficiency $\downarrow$	Avg Turns	TESR a	Efficiency $\downarrow$	Avg Turns
MolHop-9B	86.2%	28.0	15.6	87.8%	20.0	14.7
MolHop-4B	74.2%	58.2	27.9	72.8%	55.9	27.9
MolHop-2B	79.7%	25.7	7.3	80.8%	21.1	7.1
DeepSeek-Flash-V4.1	94.0%	90.1	45.9	94.3%	79.5	46.6
Grok-4.6	78.4%	76.0	48.6	78.3%	70.0	48.4

Appendix

Table C5: MolHop Tool-Use Efficiency Comparison

Property	p25	p75
logP	1.50	3.26
TPSA	52.83	84.50
MW	274.32	361.43
LabuteASA	116.73	153.55
MolMR	75.13	100.20

Appendix

Table D1: Property Range Statistics for Restricted Scaffold Library

Method	joint@H1	md<1	Mean md $\downarrow$
Scaffold library upper bound	0.0% (0/83)	0.0% (0/83)	2.376
DeepSeek + harness	43.4% (36/83)	90.4% (75/83)	0.542

Appendix

Table D2: OOD Property Optimization Results

Figure E1: Recovery of a candidate retained in memory. In this successful MolHop-4B trajectory, an N-to-C edit improves TPSA matching but worsens logP matching and is undone. After exploring other scaffolds, the agent reloads candidate A at turn 21 and submits it at turn 23, achieving joint success.

Figure E2: Comparison from the same starting molecule. Under identical 50-turn budgets, Grok-4.6 attains a smaller property distance but ends with a higher SA than the reference and no submission. DeepSeek-Flash-V4.1 submits a candidate with a lower SA and achieves joint success.

Figure E3: MolHop-4B failure despite close property matching. The agent submits its recorded best candidate at turn 28. All five relative property gaps are at most 3.04%, but SA increases from 2.427 to 3.289. The submitted candidate therefore fails the SA condition and the joint objective.

Figure E4: Additional baseline trajectories from different starting molecules. Grok-4.6 explores alternative scaffolds and local edits, restores a previous candidate, and submits successfully at turn 50. DeepSeek-Flash-V4.1 exhausts its budget after an undo operation without submitting. All of its evaluated non-reference states have SA scores above the reference.

Molecule Design

See all 43 papers

My Chemical Harness: Evolutionary Molecular Design over Synthetic Pathways with Large Language Model Agents
Jun 8, 2026César Ojeda, Darius A. Faroughy, Maryam Karimi +3Molecular Optimization LLM Agents
Institute of Mathematics, Faculty of Science, University of Potsdam, Campus Golm, Building 9, Karl-Liebknecht-Str. 24–25, 14476 Potsdam-Golm, Germany · NHETC, Department of Physics and Astronomy, Rutgers University, Piscataway, NJ 08854, USA · Potsdam Transfer, University of Potsdam, Karl-Liebknecht-Str. 24-25, 14476 Potsdam. +1
Mol-Debate: Multi-Agent Debate Improves Structural Reasoning in Molecular Design
Apr 22, 2026Wengyu Zhang, Xiao-Yong Wei, Qing LiAI-Assisted Scientific Research Multi-Agent Debate
♠The Hong Kong Polytechnic University · ♡Sichuan University
One Sequence, Many Decodings: CAGenMol-2 Recasts Drug Design as Masked Molecular Inference
Sep 28, 2026Yanting Li, Enyan Dai, Lei Wang +2Molecular Optimization Masked Diffusion Models
The Hong Kong University of Science and Technology (Guangzhou) · Jinan University, Guangzhou

Jun 8, 2026J/K move · Enter open · S save

My Chemical Harness: Evolutionary Molecular Design over Synthetic Pathways with Large Language Model Agents

César Ojeda, Darius A. Faroughy, Maryam Karimi, Payam Zarrintaj+2

Institute of Mathematics, Faculty of Science, University of Potsdam, Campus Golm, Building 9, Karl-Liebknecht-Str. 24–25, 14476 Potsdam-Golm, Germany · NHETC, Department of Physics and Astronomy, Rutgers University, Piscataway, NJ 08854, USA · Potsdam Transfer, University of Potsdam, Karl-Liebknecht-Str. 24-25, 14476 Potsdam.+1

Molecular Optimization LLM Agents Molecule Design

Open paper AskPDF arXiv

Designing molecules with target properties is most useful when candidate structures are accompanied by feasible synthetic routes. We introduce My Chemical Harness, a route-native evolutionary framework for goal-directed molecular design in which the search population consists of executable synthetic pathways rather than isolated molecular graphs. Each route is built from purchasable building blocks and reaction templates, executed by deterministic chemistry tools, and scored through task-specific molecular oracles. Large language models (LLMs) are used only as strategy controllers that select high-level preferences over route length, move type, reaction families, motifs, and exploration pressure, while local code performs route construction, validation, deduplication, scoring, selection, and memory updates. This separation lets the LLM guide exploration without allowing it to introduce hallucinated products or unsupported reaction steps. On a soluble epoxide hydrolase proxy task, our LLM agent improves over single pass LLM and deterministic controllers, reaching state-of-the-art performance across the sEH score, synthetic accessibility score, and AiZynthFinder success rate metrics. These results suggest that constrained LLM agents can play a significant role in molecular discovery without requiring training, fine-tuning, or dedicated generative models.

Abstract

Molecule Design

Explore similar work

My Chemical Harness: Evolutionary Molecular Design over Synthetic Pathways with Large Language Model Agents

Mol-Debate: Multi-Agent Debate Improves Structural Reasoning in Molecular Design

One Sequence, Many Decodings: CAGenMol-2 Recasts Drug Design as Masked Molecular Inference