cs.AISep 24, 2026
SavePFArena: Benchmarking Language Models for Protein Modification
Organizations: Shanghai Artificial Intelligence Laboratory · Generative Symbolic Intelligence Lab (GenSI), Tsinghua University · Institute for AI Industry Research (AIR), Tsinghua University · School of Pharmaceutical Sciences, Tsinghua University
Abstract
Protein modification requires navigating an immense sequence space, yet wet-lab validation remains low-throughput and costly. Although computational paradigms including protein language models (PLMs), large language models (LLMs), and LLM-based agents have shown promise in protein modification, their relative efficacy across realistic experimental decision-making settings remains unclear. To bridge this gap, we introduce PFArena, a benchmark comprising four controlled task interfaces that cover single-mutant generation and multi-mutant ranking. By providing varying levels of mutation fitness data, PFArena reflects four representative research scenarios characterized by differing degrees of prior experimental context. We assess six PLMs, six LLMs, and five LLM-based agents using complementary metrics to measure both peak and overall protein modification performance. Our evaluation reveals that model performance shifts systematically with the availability of target-specific experimental evidence: PLMs demonstrate proficiency in open-ended single-mutant generation by leveraging protein-specific priors, whereas LLMs and agents perform strongly in multi-mutant ranking, particularly when target-specific fitness data are available. Nevertheless, all model families face fundamental challenges with increasing search-space size and mutation depth. We release our code and benchmark suite to facilitate reproducible research in model-assisted protein modification.
Figures & tables
Figure 1 : Benchmark performance across the four protein modification tasks in PFArena. Method classes are distinguished by cell background color: protein language models, large language models, and agents. For agents, Claude denotes Claude Opus 5 and GPT denotes GPT-6 Astra. Methods are ranked by Recall@ for task T1 and by Spearman correlation for tasks T2 – T4 . Results for the remaining metrics are detailed in the experiments section.
Figure 2 : PFArena evaluates protein modification from the perspective of experimental scientists. The benchmark is organized around four representative research scenarios that correspond to the questions scientists ask when deciding which mutants to study next. T1: Single-mutant generation generates high-performing single mutations without candidate input. T2: Measurement-free multi-mutant ranking ranks multi-mutant candidates without prior fitness measurements. T3: Anchor-informed multi-mutant ranking ranks multi-mutant successors conditioned on a tested anchor mutant. T4: Single-mutant-informed multi-mutant ranking ranks multi-mutant candidates using measured single-mutant fitness information. These scenarios span single- and multi-mutant discovery under progressively richer experimental evidence, enabling systematic comparison of PLMs, LLMs, and agents in realistic protein-engineering workflows. The protein structure depicted here corresponds to PDB entry 1RQC [ 10 ] .
| Task component | Specification |
| Definition | Budgeted generation of legal single substitutions without target-specific mutation measurements. |
| Input | Wild-type sequence , assay context and proposed budget . |
| Output | Proposed mutation set of at most single substitutions. |
| Instance | Input: Wild-type sequence: MSSSLGKE...AMV Assay context: - Primary task class: activity function. - Fitness type: organismal or cellular fitness. - Readout subclass: growth or selection proxy. Proposed budget : 40 Output: Proposed mutation set: P557L , K369Y , N434D , ... . |
| Statistics | Assays: 123; measured single mutants: 594,695. |
Table 1: Task specification for T1: Single-mutant generation .
| Task component | Specification |
| Definition | Ranking multi-mutant candidates without target-specific mutation measurements. |
| Input | Wild-type sequence , assay context , and a score-blind candidate pool. |
| Output | Complete ranking of the supplied candidate pool. |
| Instance | Input: Wild-type sequence: MLEGKVKW...KEA Assay context: - Primary task class: stability. - Fitness type: stability. - Readout subclass: folding free energy stability readout. Candidate pool: { V28C+V63L , V28Q+V63P , V28M+V63F , ... }. Output: Ranked candidate set: V28C+V63L , V28M+V63F , V28Q+V63P , ... . |
| Statistics | Assays: 74; candidate number: 6,288. |
Table 2: Task specification for T2: Measurement-free multi-mutant ranking .
| Task component | Specification |
| Definition | Ranking strict successors of a measured anchor mutant. |
| Input | Wild-type sequence , assay context , a measured anchor mutant with score , and a score-blind successor pool . |
| Output | Complete ranking of the supplied candidate pool. |
| Instance | Input: Wild-type sequence: QVQLVQSG...VSS Assay context: - Primary task class: binding. - Fitness type: binding. - Readout subclass: binding. Anchor: T28P with score -1.586. Successor pool: { T28P+S30R+N59K+T76A , T28P+S30R+N59K+Q62P+S75F , T28P+S30R+L104V , ... }. Output: Ranked candidate set: T28P+S30R+N59K+Q62P+S75F , T28P+S30R+N59K+T76A , T28P+S30R+L104V , ... . |
| Statistics | Assays: 67; candidate number: 3,648. |
Table 3: Task specification for T3: Anchor-informed multi-mutant ranking .
| Task component | Specification |
| Definition | Multi-mutant ranking with measured context for every component substitution. |
| Input | Wild-type sequence , assay context , a measured single-mutant context map , and a score-blind multi-mutant candidate pool . |
| Output | Complete ranking of the supplied candidate pool. |
| Instance | Input: Wild-type sequence: EVKLDETG...EIK Assay context: - Primary task class: stability. - Fitness type: abundance or expression. - Readout subclass: cellular abundance stability proxy. Single-mutant context: { W108E : -0.354, G109P : -1.062, M34Q : 0.605, N35S : 1.154, ... }. Candidate pool: { W108E+G109P , Y102P+M105K , M34Q+N35S , ... }. Output: Ranked candidate set: M34Q+N35S , Y102P+M105K , W108E+G109P , ... . |
| Statistics | Assays: 29; candidate number: 2,638; visible single-mutant context rows: 28,609. |
Table 4: Task specification for T4: Single-mutant-informed multi-mutant ranking .
| Objective | Metric | T1 | T2 | T3 | T4 |
| Global ranking | Spearman Correlation | ||||
| Top-weighted ranking | NDCG | ||||
| Peak discovery | Normalized Maximum Score@ | ||||
| Peak coverage | Recall@ |
Table 5 : Evaluation objectives, corresponding metrics, and metric applicability across the four tasks. Ranking metrics are not used for T1 because it is a generation task instead of a ranking task.
Figure 3 : Overview of AMixKit, integrating protein features, PLM inference, and mutant construction.
Figure 4 : Task-level and metric-level rank-based model capability. For each task and metric, all methods are ranked according to their performance, with indicating the best method.
| Category | Model | NMS@ | Recall@ |
| Statistic | Random | ||
| PLM | VenusREM | 0.7932 | 0.0427 |
| S3F-MSA | |||
| ProSST-2048 | 0.0407 | ||
| S3F | |||
| ESM-2 |
Table 6: Performance on T1: Single-mutant generation . The best results are in bold and the second-best results are underlined . The same convention is used in the tables below.
| Category | Model | Spearman | NDCG | NMS@ | Recall@ |
| Statistic | Random | ||||
| PLM | VenusREM | ||||
| S3F-MSA | |||||
| ProSST-2048 | |||||
| S3F | |||||
| ESM-2 |
Table 7: Performance on T2: Measurement-free multi-mutant ranking .
| Category | Model | Spearman | NDCG | NMS@ | Recall@ |
| Statistic | Random | ||||
| PLM | VenusREM | ||||
| S3F-MSA | |||||
| ProSST-2048 | |||||
| S3F | |||||
| ESM-2 |
Table 9: Performance on T4: Single-mutant-informed multi-mutant ranking .
Figure 5 : Directional six-class amino-acid transitions across model families. The three panels present wild-type-to-mutant class transition matrices for PLMs, LLMs, and agents. The residue classes comprise nonpolar (NP), polar uncharged (P0), polar positively charged (P+), polar negatively charged (P-), aromatic (Ar), and Gly/Pro (G/P). Each cell denotes the percentage of successful T1 hits within the respective model-family matrix (normalized to ). Color intensity follows an exponential scale, with frequencies below depicted in white. Charge-separated states are preserved, maintaining distinct off-diagonal transitions for acidic-to-basic and basic-to-acidic substitutions.
Figure 6 : Conditional utilization of single-mutant evidence in multi-mutant ranking. Panels (a)–(c) show PLMs, LLMs and agents, respectively, with each point corresponding to one T4 assay. The horizontal axis is the Spearman correlation between the sum of measured component single-mutant fitness and the measured multi-mutant fitness. This is an information-transfer diagnostic, not a direct estimate of a mechanistic epistasis coefficient. Circles mark assays with and squares mark assays below this operational threshold. The line marks a linear fit, while is the ranking correlation between the diagnostic and model performance.
Figure 7 : Common limitations across model families. (a) Sequence search space. Verified T1 top-40 hit counts, stratified by the number of legal variants per assay into three bins ( 2,000; 2,001–8,000; 8,000). Each violin shows the distribution of hit counts across assays for one family (PLMs, LLMs or agents), with a horizontal line marking the median; the three families are offset within each bin and share a common color coding. (b) Mutation depth. Mean assay-level Spearman ranking accuracy for double mutants (solid) versus candidates with more than two substitutions (hatched), shown per family with the mean value labeled above each bar. PLMs, LLMs and agents use the same family colors in both panels.
Figure 8 : Cumulative performance comparison of round-wise single mutant generation. Single-round refers to the original T1: Single-mutant generation setting, where all 40 mutations are proposed in a single pass, and is indicated with dotted horizontal lines. Multi-round refers to the results after a complete four-round feedback-guided generation protocol, with 10 mutations proposed each round, where proposed mutants with an assay-recorded DMS_score are used as scored context for the next model pass, and is depicted in solid lines.
Figure 9 : Confidence distribution and quality alignment. (a) GPT-6 Astra’s self-reported confidence across T1 – T4 . (b) Within-setting Spearman correlations between confidence and ranking-quality metrics.
Figure 10 : Test-time scaling performance across sample counts. The panels show NMS@ (left) and Recall@ (right) as the number of sampled mutation lists increases, with the dotted line representing the sample average.
| Setting | Method | NMS@ | Recall@ | Invalid per Assay |
| Vanilla | Sample Avg | |||
| RRF | 0.7847 | 0.0199 | 1.4472 | |
| Clean | Sample Avg | 1.0531 | ||
| RRF | 0.7845 | 0.0197 | ||
| In-Assay | Sample Avg | |||
| RRF | 0.7842 | 0.0191 |
Table 10: Validity-controlled RRF analysis at . Clean requires 40 unique and sequence-valid mutations. In-Assay further requires all mutations to appear in the assay’s DMS table.
| Model | Inputs and Checkpoint | Core Configuration | Candidate Score |
| ESM-2 [ 1 ] | Wild-type sequence; esm2_t33_650M_UR50D | Masked inference with a 1,024-token context (at most 1,022 residues); deterministic mutation-centered windows for longer chains | Sum of wild-type-context masked-marginal log odds, , over substituted sites |
| ProGen2-base [ 2 ] | Complete wild-type and mutant sequences; progen2-base | Forward and reversed causal sequence scoring with a 2,048-token context; all evaluated chains were processed at full length | Difference between the bidirectional sequence scores of the complete mutant and wild-type chains |
| ProSST-2048 [ 3 ] | Wild-type sequence and AlphaFold 3 structure; AI4Protein/ProSST-2048 | Official GVP quantizer with a 2,048-code structural vocabulary; all evaluated chains were processed at full length within the 2,046-residue limit | Sum of structure-conditioned wild-type-context marginal log odds over substituted sites |
| S3F [ 4 ] | Wild-type sequence, AlphaFold 3 structure, and the corresponding molecular surface; released S3F checkpoint | Mutation-centered windows of at most 1,022 residues; sequence logits replace structure-conditioned logits where AF3 pLDDT is below 70 | Sum of masked-marginal log-odds contributions over substituted sites |
| VenusREM [ 5 ] | Wild-type sequence, ProSST-2048 structural tokens, and a UniRef100/MMseqs2 MSA | aa_seq_aln retrieval; , sampling ratio , and one sampling pass; full-chain inference for all evaluated contexts | Sum of mutant-minus-wild-type values from the fused sequence, structure, and MSA representation |
| S3F-MSA [ 4 , 6 ] | S3F score and an EVE ensemble trained on the corresponding wild-type-chain MSA | Five EVE seeds; 400,000 optimization steps, batch size 256, learning rate ; 20,000 Monte Carlo samples per seed at scoring time | Equal average of query-wise standardized S3F and EVE scores, where the EVE score is the negative mean evolution index across seeds |
Table 11 : Deployment and scoring configurations of the protein-model baselines.
| Category | Model | Baseline | Adaptive | |||
| T1 | T2 | T3 | T4 | |||
| Statistic | Total | 123 | 74 | 67 | 29 | 123 |
| LLM | GPT-6 Astra | 7 | 0 | 0 | 0 | 5 |
| Claude Opus 5 | 12 | 6 | 6 | 1 | 9 | |
| Gemini 3.1 Pro | 0 | 0 | 0 | 0 | 0 | |
| Kimi K3 | 2 | 3 | 2 | 1 | — | |
Table 12: Number of assay-level queries replaced by the deterministic SHA-256 random baseline during evaluation, for the baseline PFArena setting and the multi-round adaptive search setting. A dash indicates that the corresponding task was not run for that model.
Figure 11 : Matched-query analysis of round-wise single-mutant generation. Marker lines show cumulative performance on all 123 assay queries, with deterministic random-baseline substitutions used for missing queries. Thin curves show performance on the common cohort of 113 assays completed by every model in all four rounds. Shaded regions indicate the performance gap between the full and matched evaluations.
Figure 12 : Complementary single-mutant selection in T1 . Left, overlap between evaluable top- selections from VenusREM (PLM) and GPT-6 Astra (LLM) across 116 paired assays. Right, experimental fitness-rank distributions of PLM-only, shared and LLM-only selections. Rank scores were normalized within each assay from 1 (best) to 0 (worst). n denotes mutant–assay selections.
Figure 13 : Complementary correction of extreme ranking errors by GPT-6 Astra and VenusREM across T2 – T4 . The left column shows mutants for which GPT-6 Astra corrected extreme VenusREM ranking errors, and the right column shows the converse. True fitness rank scores range from 0 (worst) to 1 (best), whereas predicted rank scores range from 0 (bottom) to 1 (top). Orange crosses and blue circles denote VenusREM and GPT-6 Astra predictions, respectively; grey arrows connect predictions for the same mutant. Shaded regions indicate true bottom-30% mutants predicted in the top 10% or true top-30% mutants predicted in the bottom 10%. Central labels report the total number of mutants in each direction.