AgentFold: Closed-Loop Agentic Search for Protein Folding Model Design

Mingquan Liu; Jiangyu Chen; Hanqun Cao; Xujun Zhang; Pengsen Ma; Xiangru Tang; Shuting Jin; Zhuo Yang

cs.AIAug 27, 2026

AgentFold: Closed-Loop Agentic Search for Protein Folding Model Design

Authors: Mingquan Liu, Jiangyu Chen, Hanqun Cao, Xujun Zhang, Pengsen Ma, Xiangru Tang, Shuting Jin, Zhuo Yang, +4 more

Organizations: State Key Lab. of Chemo & Biosensing, Coll. of Comp. Sci. & Electron. Eng., Hunan University · State Key Laboratory for Novel Software Technology, Sch. of Comput. Sci., Nanjing University · The Chinese University of Hong Kong · Zhejiang University · Yale University · Wuhan University of Science and Technology · South China Normal University Affiliated High School · Southeast University · Stanford University

Abstract

Scientific LLM agents have shown promise in literature reasoning, tool use, and experiment planning, but it remains unclear whether they can autonomously improve large, tightly coupled scientific machine-learning systems through executable code changes and computationally expensive validation. We study this question in protein folding, where progress requires coordinated architectural modifications, multi-objective evaluation, and domain-aware interpretation. We present AgentFold, a multi-agent framework that formulates folding-model development as a closed-loop search over executable code variants. Starting from ESMFold, AgentFold proposes hypotheses, implements and debugs code-level modifications, evaluates model variants, analyzes experimental outcomes, and stores both successful and failed interventions in structured memory. An MCTS-style policy allocates computational resources across high-scoring search branches. On an engineering-scale protein-folding codebase comprising more than 2,000 lines of code, AgentFold explores approximately 80 model variants using approximately 5,000 GPU-hours and 170 million LLM tokens. Under a matched computational budget, AgentFold improves the best lDDT by 7.5% over independent Codex proposals and outperforms a random-search control. Beyond model improvement, the resulting intervention traces reveal recurring empirical design patterns: stable gains tend to arise from early, soft, learnable priors and gated refinement, whereas direct geometric perturbations and geometry-conditioned feedback often destabilize training. The code and experimental resources are publicly available at https://github.com/lmqfly/AgentFold.

Figures & tables

Figure 1: AgentFold system overview. We cast model improvement as MCTS-style search over a code-variant tree, coupling an inner loop (sample $\rightarrow$ evolve $\rightarrow$ run $\rightarrow$ analyze) with a database-backed memory, and an outer periodic update that re-scores candidates to refine the search policy.

Figure 2: MCTS-style tree evolution. Each node is a sampled variant scored by average lDDT ( lddt_mean ). Color encodes performance (darker indicates higher lddt_mean ); gray marks low-scoring variants with $\texttt{lddt\_mean}<0.1$ .

Method	Best lDDT	Top-5 lDDT	Best NWRS	Top-5 NWRS
AgentFold	0.285	0.267	0.526	0.516
Codex proposals	0.265	0.257	0.512	0.509
Random controller	0.260	0.242	0.510	0.506

Table 1: Matched comparison at 36 evaluations; Top-5 by NWRS.

Variant	NWRS $\uparrow$	bb_lddt $\uparrow$	lddt $\uparrow$	oligo_gdtts $\uparrow$	rmsd $\downarrow$	tm_score $\uparrow$
esmfold	0.500	0.644/0.651	0.232/0.220	0.564/0.570	7.380/5.358	0.648/0.693
esmfold_struct_enhanced_v4	+0.026	+0.009/+0.010	+0.053 / +0.059	+0.005/-0.003	+0.082/-0.038	+0.004/-0.012
esmfold_struct_local_context_v1	+0.020	+0.002/+0.006	+0.049 /+0.044	-0.001/-0.005	+0.176/ -0.129	+0.001/-0.012
esmfold_struct_dist_aware_v1	+0.018	+0.011 / +0.014	+0.027/+0.024	+0.011 / +0.020	-0.088/ -0.134	+0.011 /-0.009
esmfold_struct_enhanced_multiscale_v2	+0.017	+0.007/ +0.014	+0.045/ +0.049	+0.004/+0.006	+0.261/+0.338	+0.001/-0.019
esmfold_net_conformal_geometric_attention	+0.017	-0.006/-0.008	+0.043/+0.046	-0.006/ +0.011	-0.063/+0.240	-0.005/-0.001

Table 2: CAMEO2022 development-benchmark performance for representative variants. We show the top NWRS variants and variants used in later targeted analyses. The ESMFold row reports absolute mean/median values; other rows report deltas relative to ESMFold. Bold and underline mark the largest and second-largest favorable changes among displayed variants.

Variant	Aspect	Loop		Physical	Contact
		loop lDDT $\uparrow$	loop bb-lDDT $\uparrow$	MolProbity $\downarrow$	Prec ${}_{12\text{-}24}$ $\uparrow$	F1 ${}_{12\text{-}24}$ $\uparrow$
esmfold	Base	0.162	0.613	3.773	0.599	0.606
esmfold_struct_enhanced_v4	L/P/C	+0.060	+0.008	-0.157	+0.019	+0.010
esmfold_struct_local_context_v1	L	+0.063	+0.002	–	–	–
esmfold_struct_enhanced_v1_dup2	L/C	+0.031	+0.007	–	+0.020	+0.013
esmfold_struct_attn_frame_v1	L/P	+0.025	+0.001	-0.049	–	–

Table 3: Targeted evaluation summary by motivation aspect. The ESMFold row reports absolute means; other rows report changes relative to ESMFold. L/P/C denote loop-quality, physical-plausibility, and contact-modeling motivations. Bold indicates the largest improvement, and underline indicates the second largest.

Figure 3: Selected variant subtree used in the evolutionary analysis. Colors distinguish high-lDDT variants, collapse cases, and ancestor/context nodes; each node reports mean lDDT.

Figure 4: Robustness under repeated runs and deeper Folding Trunks. Matched repeated-run settings are used; RMSD is omitted due to its different scale. Error bars denote standard deviations.

Figure 5: Component ablation of esmfold_struct_enhanced_v4 . Each panel shows one metric; RMSD is omitted due to its different scale. Error bars denote standard deviations.

Figure 6: Loop-region case studies on four CAMEO targets. Superpositions of ground truth (green), ESMFold (magenta), and esmfold_struct_enhanced_v4 (cyan) are shown for 7wj0_A, 7vd7_B, 7ycv_A, and 8d08_D. Text in each panel reports loop lDDT, loop backbone lDDT, and loop RMSD for ESMFold and esmfold_struct_enhanced_v4 . esmfold_struct_enhanced_v4 improves loop placement and backbone alignment in most cases, while 7vd7_B illustrates a residual metric trade-off.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Main Goal	Specific Objective	Count
Global Improvement	pLDDT/LDDT metric	41
Global Improvement	RMSD reduction	21
Loop Quality	Loop region prediction	42
Physical Plausibility	Regularization constraints	31
Physical Plausibility	Torsion angle constraints	22
Long-range Contact	Long-range contact modeling	28

Appendix

Table 4: Motivation taxonomy of proposed variants. We group each variant’s stated goal into five high-level categories (global improvement, loop quality, physical plausibility, long-range contact, and long-sequence quality), and further refine each category by its specific objective. Categories are not mutually exclusive; counts indicate how many variant motivations fall into each objective.

Pattern	Representative variants	Evidence from stored reports	Outcome signal
P1: Bias before geometry	#28 dist_aware_v1 ; #47 local_context_v1	Reports describe these variants as injecting information into the initial pair representation or IPA logits before coordinates are produced. The reports attribute gains to making residue-pair relations more learnable without directly changing rigid-frame updates.	#28 improves backbone/global metrics; #47 achieves the largest loop-lDDT gain in Table 3 .
P1 failure contrast	#11 frame_reg_v4 ; #18 enhanced_frame_pred_v1	Reports note that fixed frame/torsion biases are added directly to BackboneUpdate or combined with attention biases after structural refinement is already coupled. The stored analyses describe collapse or destructive interaction despite plausible motivations.	Both variants have near-zero lDDT and high RMSD in the search logs, indicating failed folding behavior.
P2: Multiplicative refinement	#36 enhanced_v4 ; #48 adaptive_backbone_v1	Reports describe sigmoid or confidence-related scaling of backbone updates. These mechanisms modulate update magnitude rather than adding a fixed displacement, allowing uncertain regions to be dampened.	#36 is the highest-NWRS variant; #48 improves backbone lDDT and RMSD relative to ESMFold in the stored report.
P3: Avoid geometry-to-attention feedback	#18 enhanced_frame_pred ; #57 geom_alg_phys	Reports describe variants that feed structure/geometric features into attention or replace IPA with geometry-heavy attention modules. The reports emphasize that such feedback can introduce conflicting optimization signals when geometric features are immature or noisy.	#18 collapses; #57 is much larger but does not dominate compact variants and shows weaker all-atom/local performance than #36.
Hard geometric perturbation failure	#60 differential_geometry	The report states that curvature/torsion features are computed from sequence features and then fed back into the standard IPA path, while the intended geometric mechanism is not realized. The stored evaluation records a catastrophic failure.	bb-lDDT $=0.015$ , lDDT $=0.000$ , TM-score $=0.091$ in the stored test record.

Appendix

Table 5: Representative agent-report evidence used to derive the empirical pattern set

\mathcal{P}

in Section 4.3.1 . The evidence is summarized from the stored intervention reports in my_tree_data_dup.json ; it is descriptive rather than causal proof.

Variant	Aspect	loop bb-lDDT $\uparrow$	loop lDDT $\uparrow$	loop RMSD $\downarrow$
esmfold	Base	0.613	0.162	5.433
esmfold_struct_enhanced_v4	L/P/C	+0.008	+0.060	+0.041
esmfold_struct_local_context_v1	L	+0.002	+0.063	+0.046
esmfold_struct_enhanced_v1_dup2	L/C	+0.007	+0.031	+0.022
esmfold_struct_attn_frame_v1	L/P	+0.001	+0.025	-0.007
esmfold_struct_enhanced_multiscale_v2	L/P/C	+0.002	+0.056	+0.129

Appendix

Table 6: Complete loop-region metrics for the motivation-aspect summary. The ESMFold row reports absolute means; other rows report changes relative to ESMFold.

Variant	Aspect	Ram. out. $\downarrow$	Ram. fav. $\uparrow$	Rot. out. $\downarrow$	C $\beta$ dev. $\downarrow$	Clashscore $\downarrow$	RMS bonds $\downarrow$	RMS angles $\downarrow$	MolProbity $\downarrow$
esmfold	Base	5.238	86.597	4.678	0.000	186.953	0.091	6.099	3.773
esmfold_struct_enhanced_v4	L/P/C	-1.435	+3.240	-0.518	+0.000	-18.169	-0.014	-1.023	-0.157
esmfold_struct_local_context_v1	L	–	–	–	–	–	–	–	–
esmfold_struct_enhanced_v1_dup2	L/C	–	–	–	–	–	–	–	–
esmfold_struct_attn_frame_v1	L/P	-0.752	+1.808	-0.109	+0.000	-3.748	-0.007	-0.488	-0.049
esmfold_struct_enhanced_multiscale_v2	L/P/C	-1.034	+2.562	+0.469	+0.000	-8.692	-0.012	-0.852	-0.043

Appendix

Table 7: Complete physical-plausibility metrics for the motivation-aspect summary. The ESMFold row reports absolute means; other rows report changes relative to ESMFold.

Variant	Aspect	Prec ${}_{0\text{-}6}$ $\uparrow$	Prec ${}_{6\text{-}12}$ $\uparrow$	Prec ${}_{12\text{-}24}$ $\uparrow$	Prec ≥24 $\uparrow$	F1 ${}_{0\text{-}6}$ $\uparrow$	F1 ${}_{6\text{-}12}$ $\uparrow$	F1 ${}_{12\text{-}24}$ $\uparrow$	F1 ≥24 $\uparrow$
esmfold	Base	0.944	0.621	0.599	0.537	0.952	0.617	0.606	0.521
esmfold_struct_enhanced_v4	L/P/C	+0.003	+0.032	+0.019	-0.010	+0.001	+0.024	+0.010	-0.007
esmfold_struct_local_context_v1	L	–	–	–	–	–	–	–	–
esmfold_struct_enhanced_v1_dup2	L/C	+0.000	+0.025	+0.020	+0.003	+0.000	+0.023	+0.013	+0.001
esmfold_struct_attn_frame_v1	L/P	–	–	–	–	–	–	–	–
esmfold_struct_enhanced_multiscale_v2	L/P/C	-0.000	+0.009	+0.009	-0.017	-0.002	+0.008	+0.007	-0.021

Appendix

Table 8: Complete contact metrics for the motivation-aspect summary. The ESMFold row reports absolute means; other rows report changes relative to ESMFold.

Index	Variant Name	Parameters (M)
1	esmfold	22.606659
2	esmfold_struct_enhanced_v1	22.606659
3	esmfold_struct_dynamic_head_weights	22.608207
4	esmfold_struct_sequence_distance_bias_v2	22.607595
5	esmfold_struct_attention_bias_v1	22.606660
6	esmfold_struct_residue_type_bias	22.606659

Appendix

Table 9: List of variants with their corresponding indices and parameter counts.

Figure 7: Structure module comparison. Top: ESMFold applies $8$ IPA blocks on single / pair representations, then updates geometry via Backbone Update, Angle ResNet, and Frame. Bottom: Our variant adds a residue-index-conditioned bias MLP before IPA; the remaining geometric heads are unchanged.

Explore similar work

Protein Design with Agent Rosetta: A Case Study for Specialized Scientific Agents
Mar 16, 2026Jacopo Teneggi, S. M. Bargeen A. Turzo, Tanya Marwah +4Tool-Augmented Language Model Agents AI Agents for Scientific Discovery
Polymathic AI · Johns Hopkins University · Center for Computational Biology, Flatiron Institute +3
AgentPLM: Agentic Protein Language Models with Reasoning-Augmented Decoding for Protein Sequence Design
Jun 1, 2026Sahil Rahman, Maxx Richard RahmanRL for Language Models Direct Preference Optimization
Bedford College, London, United Kingdom · Saarland University, Saarbrücken, Germany
Agentic BAIM-LLM Evaluation (ABLE): Benchmarking LLM Use of Protein Design Tools
Sep 5, 2026Bryce Cai, Geetha Jeyapragasan, Samira Nedungadi +2LLM Tool Use LLM Agent Evaluation
SecureBio, Cambridge, MA, USA.

Mar 16, 2026J/K move · Enter open · S save

Protein Design with Agent Rosetta: A Case Study for Specialized Scientific Agents

Jacopo Teneggi, S. M. Bargeen A. Turzo, Tanya Marwah, Alberto Bietti+3

Polymathic AI · Johns Hopkins University · Center for Computational Biology, Flatiron Institute+3

Tool-Augmented Language Model Agents AI Agents for Scientific Discovery Protein Design LLM Agent Harnesses

Open paper AskPDF arXiv

Large language models (LLMs) are capable of emulating reasoning and using tools, creating opportunities for autonomous agents that execute complex scientific tasks. Protein design provides a natural testbed: although machine learning (ML) methods achieve strong results, these are largely restricted to canonical amino acids and narrow objectives, leaving unfilled need for a generalist tool for broad design pipelines. We introduce Agent Rosetta, an LLM agent paired with a structured environment for operating Rosetta, the leading physics-based heteropolymer design software, capable of modeling non-canonical building blocks and geometries. Agent Rosetta iteratively refines designs to achieve user-defined objectives, combining LLM reasoning with Rosetta's generality. We evaluate Agent Rosetta on design with canonical amino acids, matching specialized models and expert baselines, and with non-canonical residues -- where ML approaches fail -- achieving comparable performance. Critically, prompt engineering alone often fails to generate Rosetta actions, demonstrating that environment design is essential for integrating LLM agents with specialized software. Our results show that properly designed environments enable LLM agents to make scientific software accessible while matching specialized tools and human experts.

Abstract

Explore similar work

Protein Design with Agent Rosetta: A Case Study for Specialized Scientific Agents

AgentPLM: Agentic Protein Language Models with Reasoning-Augmented Decoding for Protein Sequence Design

Agentic BAIM-LLM Evaluation (ABLE): Benchmarking LLM Use of Protein Design Tools