Static Recovery Is Not Dynamic Stability: Dynamics-Aware Benchmarking of Protein Motif Scaffolding
Authors: Emile de Bruyn, Moritz Schäffler, Jannik Schneider, Sebastian R. Schmidt, Chanwoong Hwang, Aleena Siji, Utkarsh Upadhyay, Stefan Bauer, Alexander Schug, Stefan Kesselheim, Artur Yakimovich, Alina Bazarova
Organizations: Jülich Supercomputing Centre, Forschungszentrum Jülich · Center for Advanced Systems Understanding (CASUS), Görlitz, Germany · Helmholtz Munich · Scientific Computing Center, Karlsruhe Institute of Technology · Department of Physics, University of Cologne · Helmholtz-Zentrum Dresden-Rossendorf e. V. (HZDR), Dresden, Germany · Institute of Computer Science, University of Wrocław, Wrocław, Poland · Cluster of Excellence Physics of Life, TU Dresden, Dresden, Germany
Protein motif scaffolding benchmarks assess whether generative models can build scaffolds around functionally important residues. Their conclusions depend on which motifs are tested and which criteria define success. Existing benchmarks typically use small hand-curated motif sets and static refolding criteria, but protein function often depends on conformational ensembles and transitions. Here, we address both benchmark scope and success criteria. We introduce BILBY-PRO, a systematically derived benchmark of structurally conserved functional motifs from PROSITE-linked experimental structures, comprising 220 cases from 174 PROSITE motif-pattern entries. Evaluating six recent scaffold-generation models reveals distinct patterns in target coverage and unique-solution yield: RFDiffusion and Proteina solve more targets, whereas ESM3 produces more unique successful scaffolds on the targets it solves. We further introduce BILBY-MD, a reference-based all-atom molecular dynamics evaluation framework. Applied to statically solved designs from the original MotifBench benchmark, it compares motif drift, scaffold drift, motif-contact preservation, and motif-dihedral free-energy landscapes against reference simulations. Of 55 statically solved model-problem pairs, 31 satisfy our MD-supported criterion and only 8 pass all checks. Compact, self-contained motifs are often MD-supported, whereas interface-like or context-dependent motifs fail more frequently. Static refolding success therefore provides useful evidence of local motif recovery but does not establish dynamical fidelity of the full scaffold. BILBY highlights three complementary axes of motif-scaffolding evaluation: target coverage, unique-solution yield, and MD-supported dynamical fidelity.
Figures & tables
Statistic
Count
Benchmark cases
220
Unique PROSITE motif-pattern entries
174
Core assemblies
179
Ribosomal assemblies
30
Huge assemblies
11
Median motif length
19 residues
Table 1: Composition of BILBY-PRO. The benchmark contains compact, structurally conserved within-chain motifs derived from PROSITE-linked experimental structures.
Figure 1: Static motif-scaffolding performance across benchmark sets and broad functional classes. (a) Coverage is the fraction of cases with at least one successful scaffold among 100 generation attempts. (b) MotifBench score rewards unique successful solutions per case with diminishing return, so additional successes contribute less once a case is already well solved. Reporting both metrics separates target-level reliability from unique-solution yield: RFDiffusion and Proteina tend to solve more targets, while ESM3 often achieves higher MotifBench scores on the cases it solves. Numbers in parentheses indicate the number of cases in each subset.
Figure 2: Model-level RMSD behavior for statically solved base-motif cases. Values are averaged over solved pairs for each model; labels report the number of contributing solved pairs. The left panel shows motif RMSD mean difference relative to the reference, with dashed lines at ±1\text{,}\mathrm{\SIUnitSymbolAngstrom} . The right panel shows scaffold RMSD mean difference relative to the reference, with dashed lines at $\pm$2\text{\,}\mathrm{\SIUnitSymbolAngstrom} . Takeaway: motif RMSD remains near threshold for all models, while scaffold RMSD exposes substantial global drift for several methods.
Figure 3: Model-level motif-contact preservation for statically solved cases. The left panel shows contact Pearson similarity; the right panel shows weighted contact-map absolute difference. Dashed lines mark the heuristic MD-support thresholds used in the analysis: Pearson similarity above 0.8 and weighted absolute difference below 1.0 . Takeaway: contact preservation is generally stronger than scaffold stability, especially for the smaller FrameFlow and DPLM2 solved sets.
Figure 4: Raw MD metric profiles for representative MotifBench problems discussed in the text. Rows are benchmark problems, with the number of contributing models solved by MotifBench in parentheses. Columns show motif RMSD mean difference, scaffold RMSD mean difference, contact Pearson similarity, weighted contact-map absolute difference, PCA FES Pearson similarity, and TICA FES Pearson similarity. Blue/red cells pass/fail the metric cutoffs as discussed in appendix C . Takeaway: compact local motifs pass most checks, while interface-like or context-dependent cases fail in contacts, FES, or scaffold stability.
Group
Count
Group 1
113
Group 2
9
Group 3
2
Group 4 (multi-conformational)
96
Table A.1: MotifBench-style group composition of BILBY-PRO. Groups 1–3 follow the MotifBench definitions; Group 4 denotes multi-conformational motifs identified by structural clustering.
Place
Model
MotifBench Score
1
Proteina
37.16
2
ESM3
34.92
3
RFDiffusion
27.72
4
Genie2
17.39
5
FrameFlow
12.19
6
DPLM2
6.98
Table A.2: MotifBench leaderboard for the 30 MotifBench cases.
Place
Model
MotifBench Score
1
RFDiffusion
62.20
2
Proteina
60.30
3
FrameFlow
45.94
4
Genie2
32.35
5
ESM3
24.98
6
DPLM2
5.77
Table A.3: Static evaluation leaderboard for BILBY-PRO (220 cases), using the MotifBench score.
Testcase
Group
RFDiffusion
FrameFlow
Genie-2
Proteina
DPLM-2
ESM3
01_1LDB
1
4
0
0
0
0
0
02_1ITU
1
4
0
1
0
0
72
03_2CGA
1
0
0
0
0
0
0
04_5WN9
1
0
0
8
17
0
0
05_5ZE9
1
31
54
32
33
0
39
06_6E6R
1
56
60
54
92
84
19
Table A.4: Static evaluation results for original MotifBench Testcases
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Check cutoff
Strict
Baseline
Lenient
Motif RMSD
≤0.5
\SIUnitSymbolAngstrom
≤1.0
\SIUnitSymbolAngstrom
≤1.5
\SIUnitSymbolAngstrom
Scaffold RMSD
≤1.0
\SIUnitSymbolAngstrom
≤2.0
\SIUnitSymbolAngstrom
≤3.0
\SIUnitSymbolAngstrom
Contact Pearson
≥0.9
≥0.8
≥0.7
Contact absolute difference
≤0.75
≤1.0
≤1.25
PCA/TICA FES Pearson
≥0.5
≥0.3
≥0.2
Appendix
Table A.5: Cutoff settings used for the MD-support sensitivity analysis. Motif RMSD and scaffold RMSD are absolute mean differences from the reference trajectory. The contact check requires both the Pearson and absolute-difference criteria to pass. The FES check requires both PCA and TICA FES Pearson correlations to pass the listed threshold.
Figure A.1: Sensitivity of individual MD-support checks to threshold choices. Lines show the fraction of statically solved model/problem pairs passing each MD check under strict, baseline, and lenient cutoff settings, including isolated changes to RMSD, contact, and FES thresholds. Motif RMSD is consistently at least as permissive as scaffold RMSD and remains above it under the baseline and all-lenient settings, supporting the interpretation that static success is more predictive of motif-local geometry than of whole-scaffold dynamical stability.
MD-support result
Negative-case model/problem pairs
Motif-RMSD check passes
42/79
( 53.2% )
Scaffold-RMSD check passes
53/79
( 67.1% )
Combined contact check passes
9/79
( 11.4% ; 9/75 evaluable)
Combined PCA/TICA landscape check passes
6/79
( 7.6% ; 6/55 evaluable)
At least 2/4 checks pass
37/79
( 46.8% )
At least 3/4 checks pass
6/79
( 7.6% )
Appendix
Table A.6: BILBY-MD support outcomes for 79 statically unsuccessful model–problem pairs spanning 28 original MotifBench problems.
Figure A.2: Model-level RMSD behavior for statically unsolved base-motif cases. Values are averaged over unsolved pairs for each model; labels report the number of contributing unsolved pairs. The left panel shows motif RMSD mean difference relative to the reference, with dashed lines at ±1 Å. The right panel shows scaffold RMSD mean difference relative to the reference, with dashed lines at ±2 Å. Takeaway: similarily to the statically solved case evaluation the motif RMSD mean difference remains more conserved than the scaffold RMSD mean difference.
Figure A.3: Model-level motif-contact preservation for statically unsolved cases. The left panel shows contact Pearson similarity; the right panel shows weighted contact-map absolute difference. Dashed lines mark the heuristic MD-support thresholds used in the analysis: Pearson similarity above 0.8 and weighted absolute difference below 1.0. Takeaway: Especially when it comes to the contact Pearson score the models perform quite similarly, however none manage to breach the 0.8 threshold.
Figure A.4: Model-level free-energy-surface similarity for statically unsolved base-motif cases. Bars show model means and error bars show standard errors over unsolved pairs. Panels show PCA Pearson correlation, PCA absolute difference, TICA Pearson correlation, and TICA absolute difference between designed and reference motif-dihedral landscapes. Dashed lines in the Pearson panels mark the 0.3 threshold used in the MD sanity check. Takeaway: FES agreement is considerably worse than RMSD or scaffold, pointing to a clear dynamical failure of the statically unsolved simulations.
Figure A.5: Raw MD metric profiles for representative MotifBench problems discussed in the text. Rows are benchmark problems, with the number of contributing models not-solved by MotifBench in parentheses. Columns show motif RMSD mean difference, scaffold RMSD mean difference, contact Pearson similarity, weighted contact-map absolute difference, PCA FES Pearson similarity, and TICA FES Pearson similarity. Blue/red cells pass/fail the metric cutoffs as discussed in appendix C . Takeaway: Much lower pass-rate than the statically solved cases, with no clear distinction between models.
Figure A.6: Model-level free-energy-surface similarity for solved base-motif cases. Bars show model means and error bars show standard errors over solved pairs. Panels show PCA Pearson correlation, PCA absolute difference, TICA Pearson correlation, and TICA absolute difference between designed and reference motif-dihedral landscapes. Dashed lines in the Pearson panels mark the 0.3 threshold used in the MD sanity check. Takeaway: FES agreement separates local-contact preservation from ensemble-level agreement; many high-scoring models do not reproduce the ensemble dynamics even when they score high on the static benchmark.
Biomedical Engineering, The University of Sydney, Darlington, NSW 2006, Australia. · College of Engineering and Computer Science, VinUniversity, Vietnam. · School of Computer Science and School , The University of Sydney, Darlington, NSW 2006, Australia.