MARCO: Multi-Round Agentic Reinforcement for Conditional Molecular Optimization

Shicheng Fang; Yuxin Wang; Zhuo Yang; Xiaohu Xu; Jiahao Lu; Chuanyuan Tan; Tong Zhu; Yining Zheng

cs.LGSep 29, 2026

MARCO: Multi-Round Agentic Reinforcement for Conditional Molecular Optimization

Authors: Shicheng Fang, Yuxin Wang, Zhuo Yang, Xiaohu Xu, Jiahao Lu, Chuanyuan Tan, Tong Zhu, Yining Zheng,

Organizations: Fudan University · Shanghai Innovation Institute · Southeast University · East China Normal University ·

Abstract

Molecular optimization is inherently iterative: a candidate is proposed, evaluated against several objectives, and revised while preserving a relationship to the source molecule. Most instruction-following models instead emit one edited molecule, forcing validity, property improvement, and similarity control into a single response. We introduce MARCO, an evaluator-grounded reinforcement-learning framework that trains molecular editors on bounded proposal--feedback--revision trajectories. MARCO aggregates shaped turn rewards into an undiscounted trajectory return for group-relative policy optimization. We evaluate two consequences of this training: Same-1 tests the trained policy under a one-response budget, while Same-5 tests whether the same policy can use verifier feedback when up to five responses are available. Across the three-objective MuMOInstruct benchmark, three Qwen backbones, and seen/unseen instruction splits, SFT-initialized MARCO obtains the highest product of property success rate and similarity in every reported primary setting. Same-5 further improves the observed score under the tested budget, while four-objective and public-checkpoint experiments test transfer across constraint sets and initialization regimes.

Figures & tables

Figure 1: Overview of MARCO trajectory learning. The policy samples bounded trajectories in which candidate molecules alternate with verifier feedback on validity, directional property progress, and source similarity. After rollout, the shaped turn rewards are aggregated into one trajectory return; group-relative comparison of these returns updates the policy.

Task	Requested directions
BDP	BBBP $\uparrow$ + DRD2 $\uparrow$ + pLogP $\uparrow$
BDQ	BBBP $\uparrow$ + DRD2 $\uparrow$ + QED $\uparrow$
BPQ	BBBP $\uparrow$ + pLogP $\uparrow$ + QED $\uparrow$
HMPQ	HIA $\uparrow$ + mutagenicity $\downarrow$ + pLogP $\uparrow$ + QED $\uparrow$
BDPQ	BBBP $\uparrow$ + DRD2 $\uparrow$ + pLogP $\uparrow$ + QED $\uparrow$

Table 1: Molecular optimization tasks and requested directions. BDP, BDQ, and BPQ form the primary three-objective benchmark; HMPQ and BDPQ are four-objective extensions. pLogP denotes penalized logP.

Figure 2: Five-turn BPQ optimization trajectory. Each panel shows the edited local fragment, with $R$ denoting the omitted structure; the Turn-1 label describes the change from the source and later labels describe changes from the preceding candidate. The targets are BBBP $\uparrow$ , pLogP $\uparrow$ , and QED $\uparrow$ , and values are signed changes from the source.

Train horizon	Eval. budget	BDP Seen	BDP Unseen	BDQ Seen	BDQ Unseen	BPQ Seen	BPQ Unseen
1 turn	Same-1	0.184	0.165	0.158	0.142	0.225	0.267
5 turns	Same-1	0.203	0.175	0.166	0.156	0.232	0.268
1 turn	Same-5	0.222	0.200	0.180	0.170	0.290	0.312
5 turns	Same-5	0.232	0.211	0.258	0.244	0.358	0.369

Table 5: Training-horizon control on Qwen2.5-3B-Instruct MARCO-from-SFT (SR

\times

Sim).

Task	Split	Released checkpoint			After feedback RL
		SR	Sim	SR $\times$ Sim	SR	Sim	SR $\times$ Sim
BDP	Seen	0.554	0.598	0.332	0.584	0.614	0.358
BDP	Unseen	0.518	0.616	0.319	0.536	0.630	0.338
BDQ	Seen	0.556	0.613	0.341	0.578	0.605	0.350
BDQ	Unseen	0.462	0.650	0.300	0.500	0.637	0.319
BPQ	Seen	0.700	0.581	0.407	0.732	0.594	0.435

Table 6: Public-checkpoint adaptation of

\mathrm{GeLLM}^{3}\mathrm{O}\text{-}\mathrm{P}(6)_{\mathrm{Mistral}}

under the direct-SMILES five-turn protocol.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Evaluation and SFT		Feedback RL and reward
Maximum new tokens	1024	Initialization	Per-subtask SFT policy
Inference batch size	8	Rollout horizon	5 turns
Decoding	Deterministic	Training batch / group size $K$	32 / 8
SFT sequence length	4096	Response length	1024
SFT epochs	2	Actor learning rate	$1\times 10^{-6}$
SFT learning rate	$5\times 10^{-6}$	Temperature / top- $p$	1.0 / 0.8

Appendix

Table 9: Canonical MARCO training, evaluation, and reward settings. Reward coefficients correspond to the turn-reward equations in the main paper; the similarity bounds remain

\delta_{\mathrm{low}}

and

\delta_{\mathrm{high}}

Metric	Method	Seen instruction			Unseen instruction
		BDP	BDQ	BPQ	BDP	BDQ	BPQ
SR	Base	0.052	0.034	0.052	0.052	0.042	0.050
	SFT	0.398	0.319	0.471	0.310	0.342	0.419
	GRPO	0.156	0.082	0.212	0.148	0.078	0.186
	GRPO *	0.090	0.128	0.160	0.072	0.138	0.186
	RePO	0.206	0.160	0.274	0.198	0.170	0.242

Appendix

Table 11: Complete SFT-initialization ablation on Qwen2.5-3B-Instruct under Same-1.

Metric	Method	Same-1				Same-5
		HMPQ		BDPQ		HMPQ		BDPQ
		Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen
SR	Base	0.042	0.062	0.024	0.040	0.146	0.146	0.054	0.082
	SFT	0.385	0.469	0.202	0.186	0.469	0.510	0.252	0.232
	GRPO	0.042	0.021	0.068	0.040	0.167	0.104	0.096	0.058
	RePO	0.135	0.156	0.090	0.068	0.156	0.198	0.138	0.084

Appendix

Table 12: Complete four-objective Same-1 and Same-5 results on Qwen2.5-7B-Instruct.

Figure 4: Complete view of the illustrative Qwen2.5-3B-Instruct SFT-initialized MARCO BPQ trajectory summarized by Figure 2 in the main paper. Panel (a) shows directional BBBP, pLogP, and QED changes with source similarity; panel (b) shows the five complete RDKit-rendered candidates. The green frame marks the first property-success candidate at Turn 5.

Molecular Optimization

Molecular optimization computationally designs or edits chemical structures to improve desired properties while respecting chemical and practical constraints.

See all 103 papers

From Single-Step Edit Response to Multi-Step Molecular Optimization
May 11, 2026Haojie Rao, Kun Li, Yida Xiong +5Molecular Optimization Constrained Optimization
School of Computer Science, Wuhan University, Wuhan, China · School of Computer Science, Wuhan University Wuhan, China · Department of Data Science and Artificial Intelligence, Monash University, Victoria, Australia +5
Reference-guided Policy Optimization for Molecular Optimization via LLM Reasoning
Mar 6, 2026Xuan Li, Zhanke Zhou, Zongze Li +4Molecular Optimization Large Language Model-Guided Optimization
Hong Kong Baptist University · CMIC, Shanghai Jiao Tong University · DAMO Academy, Alibaba Group +1
C-MORAL: Controllable Multi-Objective Molecular Optimization with Reinforcement Alignment for LLMs
Apr 24, 2026Rui Gao, Youngseung Jeon, Swastik Roy +2RL for Language Models Molecular Optimization
University of California, Los Angeles · Amazon

May 11, 2026J/K move · Enter open · S save

From Single-Step Edit Response to Multi-Step Molecular Optimization

Haojie Rao, Kun Li, Yida Xiong, Jiameng Chen+4

School of Computer Science, Wuhan University, Wuhan, China · School of Computer Science, Wuhan University Wuhan, China · Department of Data Science and Artificial Intelligence, Monash University, Victoria, Australia+5

Molecular Optimization Constrained Optimization

Open paper AskPDF arXiv

Conditional molecular optimization aims to edit a molecule to realize a specified property shift. In practice, structurally similar molecule data is scarce, while decisions are inherently action-level: at each step, the system must select one local structural edit from a candidate set that is strictly filtered by chemical feasibility rules. This level mismatch between supervision and decision makes oracle-in-the-loop search unstable in molecular optimization. Regressing on property differences between molecule pairs improves data efficiency but relies on oracle-in-the-loop search, entangling transformation effects with global context and providing limited guidance for selecting the next feasible edit, often resorting to oracle-in-the-loop search. For this reason, we propose a response-oriented discrete edit optimization approach comprising two tightly coupled components: a single-step molecular edit response predictor (SMER) and a multi-step planner that composes local predictions into optimization trajectories via guided tree search (SMER-Opt). The approach learns a directional evaluation model over edit actions to support constraint-aware planning. It mines weakly related molecule pairs and decomposes their structural differences into minimal edit units, turning endpoint property annotations into process-level supervision and yielding reusable, transferable action primitives. A directional edit evaluator then scores feasible candidate edits by their likelihood of moving the molecule toward the desired property change, substantially reducing dependence on external evaluator queries at decision time. Code is available at https://anonymous.4open.science/r/SMER.

Abstract

Molecular Optimization

Explore similar work

From Single-Step Edit Response to Multi-Step Molecular Optimization

Reference-guided Policy Optimization for Molecular Optimization via LLM Reasoning

C-MORAL: Controllable Multi-Objective Molecular Optimization with Reinforcement Alignment for LLMs