Evaluator-in-the-Loop Monte Carlo Tree Search via LLM Agents for Motif Scaffolding in Protein Design

Haotian Hu; Oguzhan Gungordo; Siheng Xiong; Faramarz Fekri

cs.MAOct 2, 2026

Evaluator-in-the-Loop Monte Carlo Tree Search via LLM Agents for Motif Scaffolding in Protein Design

Authors: Haotian Hu, Oguzhan Gungordo, Siheng Xiong, Faramarz Fekri

Organizations: Georgia Institute of Technology

Abstract

Motif-scaffolding systems commonly follow a generate-then-filter paradigm, in which candidate proteins are generated independently and structural evaluation is used primarily for terminal screening or ranking. This paradigm underuses evaluation: failed predictions contain state-specific evidence about whether a design requires repair of motif geometry, global foldability, or other structural constraints. We introduce ELMS (Evidence-based LLM-guided Monte Carlo Search), an evaluator-in-the-loop search framework for motif scaffolding that turns such evaluator feedback into targeted design actions. Effective reuse of structural feedback is nontrivial because different scaffold states exhibit different failure modes, and repeatedly refining a single trajectory can prematurely commit computation to an unproductive region of sequence space. ELMS therefore retains evaluated scaffolds as persistent search states: a Critic Agent diagnoses state-local structural failures, a Policy Agent selects targeted operators with execution parameters, motif-locked operators realize legal sequence modifications, and MCTS determines which historical states should receive further design effort. Under the standard GeomMotif protocol (100 candidates per task), ELMS achieves Successful rates of 86.41% on single-motif tasks and 84.57% on paired-motif tasks, exceeding the strongest prior baseline by 19.3 and 21.9 percentage points, respectively. On MotifBench, under a matched 100-candidate search budget, it solves 26.7 of 30 tasks on average (88.89% Task Success), compared with 16.0 tasks (53.33%) for the strongest baseline. These results establish ELMS as an effective approach for converting structural evaluation from a terminal filter into actionable guidance for iterative motif scaffolding.

Figures & tables

Figure 1: Overview of ELMS. The search tree selects an evaluated state for expansion; the Critic Agent analyzes the state-local evidence to identify which structural or task-specific constraints are limiting the current candidate, the Policy Agent translates this critic into a concrete edit plan over mutable sequence regions, and a motif-locked operator applies the edit while preserving the required motif positions to generate a new candidate sequence. Structural evaluation returns the resulting child to the search tree, where value and visit information is backpropagated to guide subsequent expansions. After search, evaluated states are ranked deterministically for benchmark reporting.

Role	Implementation	LLM?	Output
MCTS search loop	Deterministic UCB/back propagate functions	No	Selected tree node, visit counts, back propagate values
Critic Agent	gpt-5-nano with ontology-constrained JSON	Yes	Candidate-local critic labels and rationales
Policy Agent	gpt-5-nano with action-schema-constrained JSON	Yes	Legal operator choices plus execution parameters
Motif-locked operator	Motif-lock template plus sequence-model backend	No	Apply selected actions to legal scaffold regions while preserving motif residues
Structure evaluator	ESMFold plus benchmark-specific adapters	No	Fold confidence, motif RMSD, hard-validity signals, online proxy status
Benchmark reporter	Deterministic ranking procedure	No	Ranked evaluated candidate set

Table 1: Role map for ELMS. The table identifies which roles are dedicated LLM agents and which ones are non-LLM modules or fixed protein-model calls.

Model / system	Successful (%)		Novel (%)
Model / system	Single	Paired	Single	Paired
Genie2	$60.1\pm 1.0$	$32.9\pm 0.4$	$60.1\pm 1.0$	$26.6\pm 0.5$
La-Proteina	$67.1\pm 0.7$	$62.7\pm 0.5$	$67.1\pm 0.6$	$35.2\pm 0.9$
RFdiffusion	$65.1\pm 0.4$	$43.7\pm 1.0$	$65.1\pm 0.4$	$25.0\pm 0.5$
Protpardelle-1C	$56.2\pm 0.7$	$44.6\pm 0.5$	$56.2\pm 0.7$	$25.2\pm 0.4$
FrameFlow	$30.6\pm 0.4$	$25.1\pm 0.4$	$30.6\pm 0.4$	$19.7\pm 0.3$

Table 2: GeomMotif success and novelty comparison. All rows use the standard GeomMotif protocol (100 candidates per task) and benchmark Successful definition. Values are reported as mean

\pm

standard deviation. Post hoc Unique and additional diversity statistics are reported in Appendix.

Method	Solutions	Success Rate(%)
RFdiffusion3	15.7	$52.22\pm 4.84$
La-Proteina	16.0	$53.33\pm 1.92$
Protpardelle-1c (cc91-large)	14.7	$48.89\pm 1.11$
Genie3	9.0	$30.00\pm 1.92$
Baseline LLM(GPT-5-nano)	8.3	27.78 $\pm$ 2.22
ELMS (GPT-5-nano)	26.7	$\mathbf{88.89\pm 1.11}$

Table 3: MotifBench fixed-search-budget comparison. Results are averaged over three independent runs; each run evaluates 100 candidates per task. Solutions denotes the mean number of tasks for which at least one evaluated candidate satisfies the MotifBench success criterion.

Variant	Evaluator feedback	Action routing	Tree selection / value backpropagation	Single Success (%)	Paired Success (%)
Random Operator Search	No	Random legal routing	No	0.0	0.0
Generate-then-filter	No	Independent direct generation	No	6.1	10.4
Greedy Operator Search	Yes	Evidence-guided greedy routing	No; single trajectory	48.0	53.7
ELMS	Yes	Evidence-guided action selection	UCB selection + value backpropagation	88.6	86.4

Table 4: Matched-budget GeomMotif search ablation with identical LLM, inputs, evaluator, success gate, and action space, using

B_{\mathrm{search}}=100

evaluated candidates per task. All rows report Task Success: the percentage of tasks with at least one successful candidate.

Panel of Figure 2 — Figure 2: GeomMotif Task Success@1 as a function of internal search budget with $K_{\mathrm{sub}}=1$ . Greedy feedback improves at small budgets but largely plateaus, whereas ELMS continues to benefit from additional search.

Explore similar work

RosettaSearch: Multi-Objective Inference-Time Search for Protein Sequence Design
Apr 19, 2026Meghana Kshirsagar, Allen Nie, Ching-An Cheng +5Inference-Time Search Large Language Model-Guided Optimization
AI for Good, Microsoft Redmond · Google DeepMind · Google Research +3
AgentFold: Closed-Loop Agentic Search for Protein Folding Model Design
Aug 27, 2026Mingquan Liu, Jiangyu Chen, Hanqun Cao +9AI Agents for Scientific Discovery Protein Structure Prediction
State Key Lab. of Chemo & Biosensing, Coll. of Comp. Sci. & Electron. Eng., Hunan University · State Key Laboratory for Novel Software Technology, Sch. of Comput. Sci., Nanjing University · The Chinese University of Hong Kong +6
Agentic BAIM-LLM Evaluation (ABLE): Benchmarking LLM Use of Protein Design Tools
Sep 5, 2026Bryce Cai, Geetha Jeyapragasan, Samira Nedungadi +2LLM Tool Use LLM Agent Evaluation
SecureBio, Cambridge, MA, USA.

Apr 19, 2026J/K move · Enter open · S save

RosettaSearch: Multi-Objective Inference-Time Search for Protein Sequence Design

Meghana Kshirsagar, Allen Nie, Ching-An Cheng, Fanglei Xue+4

AI for Good, Microsoft Redmond · Google DeepMind · Google Research+3

Inference-Time Search Large Language Model-Guided Optimization Molecular Generation Protein Design Multi-Objective Optimization

Open paper AskPDF arXiv

We introduce RosettaSearch, an inference-time multi-objective optimization approach for backbone conditioned protein sequence design. We use large language models (LLMs) as a generative optimizer within a search algorithm capable of controlled exploration and exploitation, using rewards computed from RosettaFold3, a structure prediction model, under a strict computational budget. In a large-scale evaluation, we apply RosettaSearch to 400 suboptimal sequences generated by LigandMPNN (a state-of-the-art model trained for protein sequence design), recovering high-fidelity designs that LigandMPNN's single-pass decoding fails to produce. RosettaSearch's designs show improvements in structural fidelity metrics ranging between 18% to 68%, translating to a 2.5x improvement in design success rate. We observe that these gains in success rate are robust when RosettaSearch-designed sequences are evaluated with an independent structure prediction oracle (Chai-1) and generalize across two distinct LLM families (o4-mini and Gemini-3), with performance scaling consistently with reasoning capability. We further demonstrate that RosettaSearch improves the sequence fidelity of ProteinMPNN designs for de novo backbones from the Dayhoff atlas, showing that the approach generalizes beyond native protein structures to computationally generated backbones. We also demonstrate a multi-modal extension of RosettaSearch with vision-language models, where images of predicted protein structures are used as feedback to incorporate structural context to guide protein sequence generation. To our knowledge, this is the first large-scale demonstration that LLMs can serve as effective generative optimizers for backbone-conditioned protein sequence design, yielding systematic gains without any model retraining.

Abstract

Explore similar work

RosettaSearch: Multi-Objective Inference-Time Search for Protein Sequence Design

AgentFold: Closed-Loop Agentic Search for Protein Folding Model Design

Agentic BAIM-LLM Evaluation (ABLE): Benchmarking LLM Use of Protein Design Tools