Agents on a Tree: Pathwise Coordination for Multi-Objective Molecular Optimization

Jia Zhang; Tengfei Ma; Tianle Li; Daojian Zeng; Xieping Gao; Xiangxiang Zeng

cs.AIMar 27, 2026

Agents on a Tree: Pathwise Coordination for Multi-Objective Molecular Optimization

Authors: Jia Zhang, Tengfei Ma, Tianle Li, Daojian Zeng, Xieping Gao, Xiangxiang Zeng

Organizations: College of Computer Science and Electronic Engineering, Hunan University · College of Information and Science, Hunan Normal University.

Abstract

Multi-objective molecular optimization requires searching vast chemical spaces under conflicting objectives, where early design decisions strongly constrain downstream outcomes. Existing methods typically rely on a single policy or fixed scalarization, which limits their ability to represent diverse trade-offs and to explore multiple promising design trajectories. We propose ATOM, a multi-agent framework that formulates molecular optimization as a tree-structured search. Each node corresponds to an atomic operation and hosts an agent specialized for a particular objective or decision context. Agents coordinate along different paths of the tree rather than enforcing a global consensus, enabling the method to maintain and compare alternative molecular evolution trajectories. A global memory of past optimization behaviors further supports balanced exploration and exploitation across objectives. This tree-structured interaction enables reasoning over long-horizon dependencies inherent in molecular design. Experiments on challenging multi-objective benchmarks involving activity, synthesizability, and ADMET-related properties show that ATOM consistently achieves improved Pareto coverage and hypervolume over strong baselines. These results demonstrate the effectiveness of pathwise multi-agent coordination for molecular optimization. Code is available at https://anonymous.4open.science/r/ATOM-41CE.

Figures & tables

Figure 1: (a) Correlations between representative molecular properties are weak or conflicting, illustrating the intrinsic difficulty of balancing multiple objectives. (b) Our tree-structured framework coordinates specialized agents along different search paths, enabling the exploration of alternative molecular evolution trajectories without enforcing a single global policy.

Figure 2: Agents-on-a-Tree framework for multi-objective molecular optimization, combining pathwise MCTS planning with knowledge-mediated coordination among specialized agents to improve Pareto coverage under conflicting objectives.

Method	QED+SA	JNK3+GSK3 $\beta$	JNK3+QED+SA	GSK3 $\beta$ +QED+SA	QED+JNK3+GSK3 $\beta$	QED+JNK3+GSK3 $\beta$ +SA	Sum(↑)
SMILES LSTM	0.925±0.004	0.592±0.035	0.512±0.025	0.706±0.037	0.445±0.061	0.438±0.031	3.618
SMILES GA	0.885±0.003	0.706±0.014	0.510±0.007	0.655±0.070	0.609±0.021	0.424±0.034	3.788
GRAPH GA	0.899±0.002	0.725±0.059	0.582±0.072	0.704±0.035	0.610±0.071	0.449±0.096	3.969
STONED	0.876±0.001	0.689±0.023	0.497±0.035	0.623±0.027	0.603±0.037	0.411±0.041	3.698
GP BO	0.843±0.003	0.783±0.043	0.505±0.045	0.629±0.030	0.640±0.103	0.422±0.101	3.822
Drugassist	0.874±0.002	0.645±0.011	0.581±0.029	0.692±0.047	0.501±0.021	0.484±0.061	3.776

Table 1: Performance comparison on multi-objective molecular optimization tasks. Higher values indicate better performance.

Figure 3: Normalized distributions of the generated molecules, QED and SA GSK3 $\beta$ and JNK3.

Figure 4: Non-dominated solutions of various methods on GSK3

\beta

+JNK3, QED+SA, GSK3

\beta

+QED and JNK3+QED objectives.

Figure 5: Examples of molecules generated by ATOM on the JNK3-GSK3 $\beta$ target pair.

Figure 6: Molecular Optimization Trajectory with ATOM.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Prompt design template for the GSK3 $\beta$ expert agent.

Panel of Figure 8 — Figure 8: Prompt design template for the SA expert agent.

Method	Top10 GSK3 $\beta$	Top10 JNK3	Top50 GSK3 $\beta$	Top50 JNK3	Avg GSK3 $\beta$	Avg JNK3	Avg Rank
SMILES LSTM	0.840	0.608	0.678	0.428	0.090	0.040	9
SMILES GA	0.906	0.763	0.896	0.729	0.572	0.388	3
GRAPH GA	0.927	0.772	0.902	0.753	0.501	0.354	4
STONED	0.910	0.738	0.897	0.726	0.342	0.207	5
GP BO	0.932	0.787	0.914	0.776	0.764	0.541	2
Drugassist	0.860	0.688	0.810	0.625	0.330	0.211	6

Appendix

Table 2: Comparison of GSK3

\beta

and JNK3 performance across different methods.

Method	Top10 Avg QED	Top10 Avg SA	Top50 Avg QED	Top50 Avg SA	Avg QED	Avg SA	Avg Rank
SMILES LSTM	0.940	0.953	0.925	0.926	0.572	0.746	4
SMILES GA	0.814	0.997	0.762	0.880	0.437	0.369	5
GRAPH GA	0.933	0.958	0.890	0.879	0.346	0.418	8
STONED	0.855	0.910	0.794	0.786	0.379	0.401	9
GPBO	0.924	0.806	0.883	0.724	0.551	0.524	6
Drugassist	0.941	0.930	0.941	0.901	0.339	0.703	7

Appendix

Table 3: Comparison of QED and SA performance across different methods.

Method	QED+SA		GSK3 $\beta$ +JNK3		QED+JNK3+GSK3 $\beta$		GSK3 $\beta$ +QED+SA		JNK3+QED+SA		QED+JNK3+GSK3 $\beta$ +SA
	Div	Nov	Div	Nov	Div	Nov	Div	Nov	Div	Nov	Div	Nov
SMILES LSTM	0.8833	0.9996	0.8799	0.9994	0.8780	0.9994	0.8780	0.9994	0.8800	0.9994	0.8800	0.9994
SMILES GA	0.8555	0.9994	0.7465	0.9994	0.7444	0.9996	0.8253	0.9997	0.8689	0.9997	0.7776	0.9997
GRAPH GA	0.8690	0.9997	0.7717	0.9995	0.8162	0.9997	0.8518	0.9997	0.8899	0.9997	0.8485	0.9997
STONED	0.7700	0.9996	0.8313	0.9997	0.8389	0.9997	0.7828	0.9997	0.9063	0.9991	0.7878	0.9996
GB PO	0.5944	0.9967	0.6721	0.9967	0.6560	0.9967	0.6253	0.9967	0.6000	0.9967	0.5475	0.9967

Appendix

Table 4: Diversity (Div) and Novelty (Nov) of generated molecules under different multi-objective settings. All values are rounded to 4 decimal places.

Molecular Optimization

Molecular optimization computationally designs or edits chemical structures to improve desired properties while respecting chemical and practical constraints.

See all 103 papers

M3OS: A Monte Carlo Graph Search-Orchestrated Multi-Agent LLM System for Evidence-Traced Molecular Optimization
Sep 28, 2026Junjie Wang, Yaowei Jin, Ruohui Tang +7Molecular Optimization Multi-Agent LLM Systems
Lingang Laboratory, Shanghai, 200031, China. · School of Information Science and Technology, ShanghaiTech University, Shanghai, 201210, China. · Global Institute of Future Technology, Shanghai Jiao Tong University, Shanghai, 200240, China.
Molecular Lead Optimization via Agentic Tool Planning
May 21, 2026Lingxiao Li, Haobo Zhang, Ruohao Fan +2Molecular Optimization Large Language Model-Guided Optimization
University of Michigan Ann Arbor, Michigan, USA · Department of Biochemistry and Molecular Medicine University of California, Davis Davis, California, USA · Department of Pharmacology and Toxicology Michigan State University East Lansing, Michigan, USA
MolMem: Memory-Augmented Agentic Reinforcement Learning for Sample-Efficient Molecular Optimization
Apr 14, 2026Ziqing Wang, Yibo Wen, Abhishek Pandy +2Molecular Optimization Sample-Efficient RL
Northwestern University · AbbVie

Sep 28, 2026J/K move · Enter open · S save

M3OS: A Monte Carlo Graph Search-Orchestrated Multi-Agent LLM System for Evidence-Traced Molecular Optimization

Junjie Wang, Yaowei Jin, Ruohui Tang, Guonan Cui+6

Lingang Laboratory, Shanghai, 200031, China. · School of Information Science and Technology, ShanghaiTech University, Shanghai, 201210, China. · Global Institute of Future Technology, Shanghai Jiao Tong University, Shanghai, 200240, China.

Molecular Optimization Multi-Agent LLM Systems Large Language Model-Guided Optimization Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Small-molecule optimization integrates medicinal-chemistry reasoning and computational evidence through iterative, multi-objective decisions. When large language models (LLMs) reason over optimization histories stored primarily in conversational context, they must recover candidate identities, prior evaluations, and task constraints to guide subsequent decisions. We present M3OS, a multi-agent LLM system that decouples molecular-design reasoning from optimization-state management through Monte Carlo graph search. A persistent graph links evaluated candidates, parent-child transformations and evaluation evidence, while rewards and visit statistics guide LLM-assisted parent selection. Two branches combine tool-driven candidate generation with knowledge- and case-guided medicinal-chemistry editing. An execution harness controls graph updates through structured output extraction, molecular validation and task-bound evaluation. Agents receive role-specific contexts, while the graph preserves optimization trajectories beyond their active contexts. Across three molecular optimization benchmarks, M3OS achieves higher success rates than baselines, supporting the integration of persistent search state, specialized agents and controlled execution for multi-constraint optimization.

Abstract

Molecular Optimization

Explore similar work

M3OS: A Monte Carlo Graph Search-Orchestrated Multi-Agent LLM System for Evidence-Traced Molecular Optimization

Molecular Lead Optimization via Agentic Tool Planning

MolMem: Memory-Augmented Agentic Reinforcement Learning for Sample-Efficient Molecular Optimization