Static Recovery Is Not Dynamic Stability: Dynamics-Aware Benchmarking of Protein Motif Scaffolding
Authors: Emile de Bruyn, Moritz Schäffler, Jannik Schneider, Sebastian R. Schmidt, Chanwoong Hwang, Aleena Siji, Utkarsh Upadhyay, Stefan Bauer, Alexander Schug, Stefan Kesselheim, Artur Yakimovich, Alina Bazarova
Organizations: Jülich Supercomputing Centre, Forschungszentrum Jülich · Center for Advanced Systems Understanding (CASUS), Görlitz, Germany · Helmholtz Munich · Scientific Computing Center, Karlsruhe Institute of Technology · Department of Physics, University of Cologne · Helmholtz-Zentrum Dresden-Rossendorf e. V. (HZDR), Dresden, Germany · Institute of Computer Science, University of Wrocław, Wrocław, Poland · Cluster of Excellence Physics of Life, TU Dresden, Dresden, Germany
Protein motif scaffolding benchmarks assess whether generative models can build scaffolds around functionally important residues. Their conclusions depend on which motifs are tested and which criteria define success. Existing benchmarks typically use small hand-curated motif sets and static refolding criteria, but protein function often depends on conformational ensembles and transitions. Here, we address both benchmark scope and success criteria. We introduce BILBY-PRO, a systematically derived benchmark of structurally conserved functional motifs from PROSITE-linked experimental structures, comprising 220 cases from 174 PROSITE motif-pattern entries. Evaluating six recent scaffold-generation models reveals distinct patterns in target coverage and unique-solution yield: RFDiffusion and Proteina solve more targets, whereas ESM3 produces more unique successful scaffolds on the targets it solves. We further introduce BILBY-MD, a reference-based all-atom molecular dynamics evaluation framework. Applied to statically solved designs from the original MotifBench benchmark, it compares motif drift, scaffold drift, motif-contact preservation, and motif-dihedral free-energy landscapes against reference simulations. Of 55 statically solved model-problem pairs, 31 satisfy our MD-supported criterion and only 8 pass all checks. Compact, self-contained motifs are often MD-supported, whereas interface-like or context-dependent motifs fail more frequently. Static refolding success therefore provides useful evidence of local motif recovery but does not establish dynamical fidelity of the full scaffold. BILBY highlights three complementary axes of motif-scaffolding evaluation: target coverage, unique-solution yield, and MD-supported dynamical fidelity.
Figures & tables
Statistic
Count
Benchmark cases
220
Unique PROSITE motif-pattern entries
174
Core assemblies
179
Ribosomal assemblies
30
Huge assemblies
11
Median motif length
19 residues
Table 1: Composition of BILBY-PRO. The benchmark contains compact, structurally conserved within-chain motifs derived from PROSITE-linked experimental structures.
Figure 1: Static motif-scaffolding performance across benchmark sets and broad functional classes. (a) Coverage is the fraction of cases with at least one successful scaffold among 100 generation attempts. (b) MotifBench score rewards unique successful solutions per case with diminishing return, so additional successes contribute less once a case is already well solved. Reporting both metrics separates target-level reliability from unique-solution yield: RFDiffusion and Proteina tend to solve more targets, while ESM3 often achieves higher MotifBench scores on the cases it solves. Numbers in parentheses indicate the number of cases in each subset.
Figure 2: Model-level RMSD behavior for statically solved base-motif cases. Values are averaged over solved pairs for each model; labels report the number of contributing solved pairs. The left panel shows motif RMSD mean difference relative to the reference, with dashed lines at ±1\text{,}\mathrm{\SIUnitSymbolAngstrom} . The right panel shows scaffold RMSD mean difference relative to the reference, with dashed lines at $\pm$2\text{\,}\mathrm{\SIUnitSymbolAngstrom} . Takeaway: motif RMSD remains near threshold for all models, while scaffold RMSD exposes substantial global drift for several methods.
Figure 3: Model-level motif-contact preservation for statically solved cases. The left panel shows contact Pearson similarity; the right panel shows weighted contact-map absolute difference. Dashed lines mark the heuristic MD-support thresholds used in the analysis: Pearson similarity above 0.8 and weighted absolute difference below 1.0 . Takeaway: contact preservation is generally stronger than scaffold stability, especially for the smaller FrameFlow and DPLM2 solved sets.
Figure 4: Raw MD metric profiles for representative MotifBench problems discussed in the text. Rows are benchmark problems, with the number of contributing models solved by MotifBench in parentheses. Columns show motif RMSD mean difference, scaffold RMSD mean difference, contact Pearson similarity, weighted contact-map absolute difference, PCA FES Pearson similarity, and TICA FES Pearson similarity. Blue/red cells pass/fail the metric cutoffs as discussed in appendix C . Takeaway: compact local motifs pass most checks, while interface-like or context-dependent cases fail in contacts, FES, or scaffold stability.
Group
Count
Group 1
113
Group 2
9
Group 3
2
Group 4 (multi-conformational)
96
Table A.1: MotifBench-style group composition of BILBY-PRO. Groups 1–3 follow the MotifBench definitions; Group 4 denotes multi-conformational motifs identified by structural clustering.
Place
Model
MotifBench Score
1
Proteina
37.16
2
ESM3
34.92
3
RFDiffusion
27.72
4
Genie2
17.39
5
FrameFlow
12.19
6
DPLM2
6.98
Table A.2: MotifBench leaderboard for the 30 MotifBench cases.
Place
Model
MotifBench Score
1
RFDiffusion
62.20
2
Proteina
60.30
3
FrameFlow
45.94
4
Genie2
32.35
5
ESM3
24.98
6
DPLM2
5.77
Table A.3: Static evaluation leaderboard for BILBY-PRO (220 cases), using the MotifBench score.
Testcase
Group
RFDiffusion
FrameFlow
Genie-2
Proteina
DPLM-2
ESM3
01_1LDB
1
4
0
0
0
0
0
02_1ITU
1
4
0
1
0
0
72
03_2CGA
1
0
0
0
0
0
0
04_5WN9
1
0
0
8
17
0
0
05_5ZE9
1
31
54
32
33
0
39
06_6E6R
1
56
60
54
92
84
19
Table A.4: Static evaluation results for original MotifBench Testcases
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Check cutoff
Strict
Baseline
Lenient
Motif RMSD
≤0.5
\SIUnitSymbolAngstrom
≤1.0
\SIUnitSymbolAngstrom
≤1.5
\SIUnitSymbolAngstrom
Scaffold RMSD
≤1.0
\SIUnitSymbolAngstrom
≤2.0
\SIUnitSymbolAngstrom
≤3.0
\SIUnitSymbolAngstrom
Contact Pearson
≥0.9
≥0.8
≥0.7
Contact absolute difference
≤0.75
≤1.0
≤1.25
PCA/TICA FES Pearson
≥0.5
≥0.3
≥0.2
Appendix
Table A.5: Cutoff settings used for the MD-support sensitivity analysis. Motif RMSD and scaffold RMSD are absolute mean differences from the reference trajectory. The contact check requires both the Pearson and absolute-difference criteria to pass. The FES check requires both PCA and TICA FES Pearson correlations to pass the listed threshold.
Figure A.1: Sensitivity of individual MD-support checks to threshold choices. Lines show the fraction of statically solved model/problem pairs passing each MD check under strict, baseline, and lenient cutoff settings, including isolated changes to RMSD, contact, and FES thresholds. Motif RMSD is consistently at least as permissive as scaffold RMSD and remains above it under the baseline and all-lenient settings, supporting the interpretation that static success is more predictive of motif-local geometry than of whole-scaffold dynamical stability.
MD-support result
Negative-case model/problem pairs
Motif-RMSD check passes
42/79
( 53.2% )
Scaffold-RMSD check passes
53/79
( 67.1% )
Combined contact check passes
9/79
( 11.4% ; 9/75 evaluable)
Combined PCA/TICA landscape check passes
6/79
( 7.6% ; 6/55 evaluable)
At least 2/4 checks pass
37/79
( 46.8% )
At least 3/4 checks pass
6/79
( 7.6% )
Appendix
Table A.6: BILBY-MD support outcomes for 79 statically unsuccessful model–problem pairs spanning 28 original MotifBench problems.
Figure A.2: Model-level RMSD behavior for statically unsolved base-motif cases. Values are averaged over unsolved pairs for each model; labels report the number of contributing unsolved pairs. The left panel shows motif RMSD mean difference relative to the reference, with dashed lines at ±1 Å. The right panel shows scaffold RMSD mean difference relative to the reference, with dashed lines at ±2 Å. Takeaway: similarily to the statically solved case evaluation the motif RMSD mean difference remains more conserved than the scaffold RMSD mean difference.
Figure A.3: Model-level motif-contact preservation for statically unsolved cases. The left panel shows contact Pearson similarity; the right panel shows weighted contact-map absolute difference. Dashed lines mark the heuristic MD-support thresholds used in the analysis: Pearson similarity above 0.8 and weighted absolute difference below 1.0. Takeaway: Especially when it comes to the contact Pearson score the models perform quite similarly, however none manage to breach the 0.8 threshold.
Figure A.4: Model-level free-energy-surface similarity for statically unsolved base-motif cases. Bars show model means and error bars show standard errors over unsolved pairs. Panels show PCA Pearson correlation, PCA absolute difference, TICA Pearson correlation, and TICA absolute difference between designed and reference motif-dihedral landscapes. Dashed lines in the Pearson panels mark the 0.3 threshold used in the MD sanity check. Takeaway: FES agreement is considerably worse than RMSD or scaffold, pointing to a clear dynamical failure of the statically unsolved simulations.
Figure A.5: Raw MD metric profiles for representative MotifBench problems discussed in the text. Rows are benchmark problems, with the number of contributing models not-solved by MotifBench in parentheses. Columns show motif RMSD mean difference, scaffold RMSD mean difference, contact Pearson similarity, weighted contact-map absolute difference, PCA FES Pearson similarity, and TICA FES Pearson similarity. Blue/red cells pass/fail the metric cutoffs as discussed in appendix C . Takeaway: Much lower pass-rate than the statically solved cases, with no clear distinction between models.
Figure A.6: Model-level free-energy-surface similarity for solved base-motif cases. Bars show model means and error bars show standard errors over solved pairs. Panels show PCA Pearson correlation, PCA absolute difference, TICA Pearson correlation, and TICA absolute difference between designed and reference motif-dihedral landscapes. Dashed lines in the Pearson panels mark the 0.3 threshold used in the MD sanity check. Takeaway: FES agreement separates local-contact preservation from ensemble-level agreement; many high-scoring models do not reproduce the ensemble dynamics even when they score high on the static benchmark.
Motif-scaffolding systems commonly follow a generate-then-filter paradigm, in which candidate proteins are generated independently and structural evaluation is used primarily for terminal screening or ranking. This paradigm underuses evaluation: failed predictions contain state-specific evidence about whether a design requires repair of motif geometry, global foldability, or other structural constraints. We introduce ELMS (Evidence-based LLM-guided Monte Carlo Search), an evaluator-in-the-loop search framework for motif scaffolding that turns such evaluator feedback into targeted design actions. Effective reuse of structural feedback is nontrivial because different scaffold states exhibit different failure modes, and repeatedly refining a single trajectory can prematurely commit computation to an unproductive region of sequence space. ELMS therefore retains evaluated scaffolds as persistent search states: a Critic Agent diagnoses state-local structural failures, a Policy Agent selects targeted operators with execution parameters, motif-locked operators realize legal sequence modifications, and MCTS determines which historical states should receive further design effort. Under the standard GeomMotif protocol (100 candidates per task), ELMS achieves Successful rates of 86.41% on single-motif tasks and 84.57% on paired-motif tasks, exceeding the strongest prior baseline by 19.3 and 21.9 percentage points, respectively. On MotifBench, under a matched 100-candidate search budget, it solves 26.7 of 30 tasks on average (88.89% Task Success), compared with 16.0 tasks (53.33%) for the strongest baseline. These results establish ELMS as an effective approach for converting structural evaluation from a terminal filter into actionable guidance for iterative motif scaffolding.
Recent advances in de novo protein binder design have enabled increasing experimental validation, yet reported in silico metrics remain difficult to interpret or compare across studies due to non-standardized evaluation protocols. We introduce ProtDBench, a standardized and throughput-aware evaluation framework for protein binder design. ProtDBench defines unified benchmark tasks, evaluation protocols, and success criteria, enabling systematic analysis of how evaluation design influences observed performance. Using a large wet-lab annotated dataset, we analyze commonly used structure prediction models as evaluation verifiers, revealing substantial verifier-dependent bias and limited agreement under identical filtering protocols. We then benchmark representative open-source generative binder design methods across ten diverse protein targets under a fixed evaluation protocol. Beyond per-sequence success rates, ProtDBench incorporates throughput-aware metrics based on a fixed 24-hour budget, as well as cluster-level success criteria to account for structural diversity. Together, these results expose systematic differences induced by filtering rules, success definitions, and throughput-aware evaluation between computational efficiency, success rate, and structural diversity. Overall, ProtDBench provides a fair and reproducible evaluation pipeline that supports systematic and controlled comparison of protein binder design methods under realistic evaluation settings.
Cong Liu, Milong Ren, Jiaqi Guan +4
AMLab, AI4Science Lab, University of Amsterdam, Amsterdam, The Netherlands · Work done at ByteDance Seed · ByteDance Seed
Designing proteins with specific biological functions is of considerable interest for both the development of new therapeutics and the engineering of enzymes, yet achieving the intended activity remains challenging. Generative artificial intelligence (AI) has emerged as a powerful tool to explore the vast space of possible designs by learning from protein sequence and structural data. Here, we review 59 generative approaches across six protein design tasks, including sequence generation, backbone generation, structure-conditioned sequence design, joint sequence-structure design, motif scaffolding, and binder and antibody design. We outline common generative approaches and protein representations and discuss how structural constraints and functional information are utilized during generation. This is followed by a discussion of the computational and experimental approaches used to evaluate designed proteins and what such assessments reveal about their properties. Finally, we discuss current limitations in designing proteins beyond well-represented families and incorporating experimental feedback into the design process. We expect that rapid advances in generative modeling will enable more reliable design of a wider range of protein structures and functions over the coming years.
Senura Hansaja Wanasekara, Manoj M Wagle, Minh-Duong Nguyen +4
Biomedical Engineering, The University of Sydney, Darlington, NSW 2006, Australia. · College of Engineering and Computer Science, VinUniversity, Vietnam. · School of Computer Science and School , The University of Sydney, Darlington, NSW 2006, Australia.