PFArena: Benchmarking Language Models for Protein Modification

Yawen Ouyang; Xinbo Zhang; Ziyuan Ma; Yixin Wu; Wenbin Liao; Feiran Zhang; Wenjie Li; Lihao Wang

cs.AISep 24, 2026

PFArena: Benchmarking Language Models for Protein Modification

Authors: Yawen Ouyang, Xinbo Zhang, Ziyuan Ma, Yixin Wu, Wenbin Liao, Feiran Zhang, Wenjie Li, Lihao Wang, +10 more

Organizations: Shanghai Artificial Intelligence Laboratory · Generative Symbolic Intelligence Lab (GenSI), Tsinghua University · Institute for AI Industry Research (AIR), Tsinghua University · School of Pharmaceutical Sciences, Tsinghua University

Abstract

Protein modification requires navigating an immense sequence space, yet wet-lab validation remains low-throughput and costly. Although computational paradigms including protein language models (PLMs), large language models (LLMs), and LLM-based agents have shown promise in protein modification, their relative efficacy across realistic experimental decision-making settings remains unclear. To bridge this gap, we introduce PFArena, a benchmark comprising four controlled task interfaces that cover single-mutant generation and multi-mutant ranking. By providing varying levels of mutation fitness data, PFArena reflects four representative research scenarios characterized by differing degrees of prior experimental context. We assess six PLMs, six LLMs, and five LLM-based agents using complementary metrics to measure both peak and overall protein modification performance. Our evaluation reveals that model performance shifts systematically with the availability of target-specific experimental evidence: PLMs demonstrate proficiency in open-ended single-mutant generation by leveraging protein-specific priors, whereas LLMs and agents perform strongly in multi-mutant ranking, particularly when target-specific fitness data are available. Nevertheless, all model families face fundamental challenges with increasing search-space size and mutation depth. We release our code and benchmark suite to facilitate reproducible research in model-assisted protein modification.

Figures & tables

Figure 1 : Benchmark performance across the four protein modification tasks in PFArena. Method classes are distinguished by cell background color: protein language models, large language models, and agents. For agents, Claude denotes Claude Opus 5 and GPT denotes GPT-6 Astra. Methods are ranked by Recall@ $40$ for task T1 and by Spearman correlation for tasks T2 – T4 . Results for the remaining metrics are detailed in the experiments section.

Figure 2 : PFArena evaluates protein modification from the perspective of experimental scientists. The benchmark is organized around four representative research scenarios that correspond to the questions scientists ask when deciding which mutants to study next. T1: Single-mutant generation generates high-performing single mutations without candidate input. T2: Measurement-free multi-mutant ranking ranks multi-mutant candidates without prior fitness measurements. T3: Anchor-informed multi-mutant ranking ranks multi-mutant successors conditioned on a tested anchor mutant. T4: Single-mutant-informed multi-mutant ranking ranks multi-mutant candidates using measured single-mutant fitness information. These scenarios span single- and multi-mutant discovery under progressively richer experimental evidence, enabling systematic comparison of PLMs, LLMs, and agents in realistic protein-engineering workflows. The protein structure depicted here corresponds to PDB entry 1RQC [ 10 ] .

Task component	Specification
Definition	Budgeted generation of legal single substitutions without target-specific mutation measurements.
Input	Wild-type sequence $x$ , assay context $c$ and proposed budget $K$ .
Output	Proposed mutation set of at most $K$ single substitutions.
Instance	Input: Wild-type sequence: MSSSLGKE...AMV Assay context: - Primary task class: activity function. - Fitness type: organismal or cellular fitness. - Readout subclass: growth or selection proxy. Proposed budget $K$ : 40 Output: Proposed mutation set: P557L , K369Y , N434D , ... .
Statistics	Assays: 123; measured single mutants: 594,695.

Table 1: Task specification for T1: Single-mutant generation .

Task component	Specification
Definition	Ranking multi-mutant candidates without target-specific mutation measurements.
Input	Wild-type sequence $x$ , assay context $c$ , and a score-blind candidate pool.
Output	Complete ranking of the supplied candidate pool.
Instance	Input: Wild-type sequence: MLEGKVKW...KEA Assay context: - Primary task class: stability. - Fitness type: stability. - Readout subclass: folding free energy stability readout. Candidate pool: { V28C+V63L , V28Q+V63P , V28M+V63F , ... }. Output: Ranked candidate set: V28C+V63L , V28M+V63F , V28Q+V63P , ... .
Statistics	Assays: 74; candidate number: 6,288.

Table 2: Task specification for T2: Measurement-free multi-mutant ranking .

Task component	Specification
Definition	Ranking strict successors of a measured anchor mutant.
Input	Wild-type sequence $x$ , assay context $c$ , a measured anchor mutant $u$ with score $y_{u}$ , and a score-blind successor pool $\mathcal{C}$ .
Output	Complete ranking of the supplied candidate pool.
Instance	Input: Wild-type sequence: QVQLVQSG...VSS Assay context: - Primary task class: binding. - Fitness type: binding. - Readout subclass: binding. Anchor: T28P with score -1.586. Successor pool: { T28P+S30R+N59K+T76A , T28P+S30R+N59K+Q62P+S75F , T28P+S30R+L104V , ... }. Output: Ranked candidate set: T28P+S30R+N59K+Q62P+S75F , T28P+S30R+N59K+T76A , T28P+S30R+L104V , ... .
Statistics	Assays: 67; candidate number: 3,648.

Table 3: Task specification for T3: Anchor-informed multi-mutant ranking .

Task component	Specification
Definition	Multi-mutant ranking with measured context for every component substitution.
Input	Wild-type sequence $x$ , assay context $c$ , a measured single-mutant context map $\mathcal{S}$ , and a score-blind multi-mutant candidate pool $\mathcal{C}$ .
Output	Complete ranking of the supplied candidate pool.
Instance	Input: Wild-type sequence: EVKLDETG...EIK Assay context: - Primary task class: stability. - Fitness type: abundance or expression. - Readout subclass: cellular abundance stability proxy. Single-mutant context: { W108E : -0.354, G109P : -1.062, M34Q : 0.605, N35S : 1.154, ... }. Candidate pool: { W108E+G109P , Y102P+M105K , M34Q+N35S , ... }. Output: Ranked candidate set: M34Q+N35S , Y102P+M105K , W108E+G109P , ... .
Statistics	Assays: 29; candidate number: 2,638; visible single-mutant context rows: 28,609.

Table 4: Task specification for T4: Single-mutant-informed multi-mutant ranking .

Objective	Metric	T1	T2	T3	T4
Global ranking	Spearman Correlation	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
Top-weighted ranking	NDCG	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
Peak discovery	Normalized Maximum Score@ $K$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Peak coverage	Recall@ $K$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Table 5 : Evaluation objectives, corresponding metrics, and metric applicability across the four tasks. Ranking metrics are not used for T1 because it is a generation task instead of a ranking task.

Figure 3 : Overview of AMixKit, integrating protein features, PLM inference, and mutant construction.

Figure 4 : Task-level and metric-level rank-based model capability. For each task and metric, all methods are ranked according to their performance, with $rank=1$ indicating the best method.

Category	Model	NMS@ $40$	Recall@ $40$
Statistic	Random	$0.7749$	$0.0142$
PLM	VenusREM	0.7932	0.0427
	S3F-MSA	$0.7816$	$0.0364$
	ProSST-2048	$0.7847$	0.0407
	S3F	$0.7804$	$0.0364$
	ESM-2	$0.7854$	$0.0370$

Table 6: Performance on T1: Single-mutant generation . The best results are in bold and the second-best results are underlined . The same convention is used in the tables below.

Category	Model	Spearman	NDCG	NMS@ $5$	Recall@ $5$
Statistic	Random	$-0.0215$	$0.8196$	$0.7113$	$0.0811$
PLM	VenusREM	$0.2804$	$0.8684$	$0.8003$	$0.2162$
	S3F-MSA	$0.2803$	$0.8700$	$0.7964$	$0.2027$
	ProSST-2048	$0.2777$	$0.8657$	$0.7829$	$0.2108$
	S3F	$0.2785$	$0.8626$	$0.7751$	$0.1757$
	ESM-2	$0.2539$	$0.8634$	$0.7831$	$0.1892$

Table 7: Performance on T2: Measurement-free multi-mutant ranking .

Category	Model	Spearman	NDCG	NMS@ $5$	Recall@ $5$
Statistic	Random	$-0.0204$	$0.8124$	$0.6371$	$0.0552$
PLM	VenusREM	$0.3821$	$0.8809$	$0.8212$	$0.2552$
	S3F-MSA	$0.3662$	$0.9005$	$0.8686$	$0.2897$
	ProSST-2048	$0.3779$	$0.8725$	$0.8078$	$0.2483$
	S3F	$0.3487$	$0.8836$	$0.8410$	$0.2759$
	ESM-2	$0.3153$	$0.8828$	$0.8129$	$0.2276$

Table 9: Performance on T4: Single-mutant-informed multi-mutant ranking .

Figure 5 : Directional six-class amino-acid transitions across model families. The three panels present $6\times 6$ wild-type-to-mutant class transition matrices for PLMs, LLMs, and agents. The residue classes comprise nonpolar (NP), polar uncharged (P0), polar positively charged (P+), polar negatively charged (P-), aromatic (Ar), and Gly/Pro (G/P). Each cell denotes the percentage of successful T1 hits within the respective model-family matrix (normalized to $100\%$ ). Color intensity follows an exponential scale, with frequencies below $0.5\%$ depicted in white. Charge-separated states are preserved, maintaining distinct off-diagonal transitions for acidic-to-basic and basic-to-acidic substitutions.

Figure 6 : Conditional utilization of single-mutant evidence in multi-mutant ranking. Panels (a)–(c) show PLMs, LLMs and agents, respectively, with each point corresponding to one T4 assay. The horizontal axis is the Spearman correlation between the sum of measured component single-mutant fitness and the measured multi-mutant fitness. This is an information-transfer diagnostic, not a direct estimate of a mechanistic epistasis coefficient. Circles mark assays with

\rho_{\mathrm{sum,combination}}\geq 0

and squares mark assays below this operational threshold. The line marks a linear fit, while

\rho

is the ranking correlation between the diagnostic and model performance.

Figure 7 : Common limitations across model families. (a) Sequence search space. Verified T1 top-40 hit counts, stratified by the number of legal variants per assay into three bins (

\leq

2,000; 2,001–8,000;

>

8,000). Each violin shows the distribution of hit counts across assays for one family (PLMs, LLMs or agents), with a horizontal line marking the median; the three families are offset within each bin and share a common color coding. (b) Mutation depth. Mean assay-level Spearman ranking accuracy for double mutants (solid) versus candidates with more than two substitutions (hatched), shown per family with the mean value labeled above each bar. PLMs, LLMs and agents use the same family colors in both panels.

Figure 8 : Cumulative performance comparison of round-wise single mutant generation. Single-round refers to the original T1: Single-mutant generation setting, where all 40 mutations are proposed in a single pass, and is indicated with dotted horizontal lines. Multi-round refers to the results after a complete four-round feedback-guided generation protocol, with 10 mutations proposed each round, where proposed mutants with an assay-recorded DMS_score are used as scored context for the next model pass, and is depicted in solid lines.

Figure 9 : Confidence distribution and quality alignment. (a) GPT-6 Astra’s self-reported confidence across T1 – T4 . (b) Within-setting Spearman correlations between confidence and ranking-quality metrics.

Figure 10 : Test-time scaling performance across sample counts. The panels show NMS@

40

(left) and Recall@

40

(right) as the number of sampled mutation lists increases, with the dotted line representing the sample average.

Setting	Method	NMS@ $40$ $\uparrow$	Recall@ $40$ $\uparrow$	Invalid per Assay $\downarrow$
Vanilla	Sample Avg	$0.7696$	$0.0186$	$2.2805$
Vanilla	RRF	0.7847	0.0199	1.4472
Clean	Sample Avg	$0.7771$	$0.0189$	1.0531
Clean	RRF	0.7845	0.0197	$1.1057$
In-Assay	Sample Avg	$0.7745$	$0.0172$	$0.0000$
In-Assay	RRF	0.7842	0.0191	$0.0000$

Table 10: Validity-controlled RRF analysis at

n=16

. Clean requires 40 unique and sequence-valid mutations. In-Assay further requires all mutations to appear in the assay’s DMS table.

Model	Inputs and Checkpoint	Core Configuration	Candidate Score
ESM-2 [ 1 ]	Wild-type sequence; esm2_t33_650M_UR50D	Masked inference with a 1,024-token context (at most 1,022 residues); deterministic mutation-centered windows for longer chains	Sum of wild-type-context masked-marginal log odds, $\log p(x_{i}^{\mathrm{mut}})-\log p(x_{i}^{\mathrm{wt}})$ , over substituted sites
ProGen2-base [ 2 ]	Complete wild-type and mutant sequences; progen2-base	Forward and reversed causal sequence scoring with a 2,048-token context; all evaluated chains were processed at full length	Difference between the bidirectional sequence scores of the complete mutant and wild-type chains
ProSST-2048 [ 3 ]	Wild-type sequence and AlphaFold 3 structure; AI4Protein/ProSST-2048	Official GVP quantizer with a 2,048-code structural vocabulary; all evaluated chains were processed at full length within the 2,046-residue limit	Sum of structure-conditioned wild-type-context marginal log odds over substituted sites
S3F [ 4 ]	Wild-type sequence, AlphaFold 3 structure, and the corresponding molecular surface; released S3F checkpoint	Mutation-centered windows of at most 1,022 residues; sequence logits replace structure-conditioned logits where AF3 pLDDT is below 70	Sum of masked-marginal log-odds contributions over substituted sites
VenusREM [ 5 ]	Wild-type sequence, ProSST-2048 structural tokens, and a UniRef100/MMseqs2 MSA	aa_seq_aln retrieval; $\alpha=0.8$ , sampling ratio $1.0$ , and one sampling pass; full-chain inference for all evaluated contexts	Sum of mutant-minus-wild-type values from the fused sequence, structure, and MSA representation
S3F-MSA [ 4 , 6 ]	S3F score and an EVE ensemble trained on the corresponding wild-type-chain MSA	Five EVE seeds; 400,000 optimization steps, batch size 256, learning rate $10^{-4}$ ; 20,000 Monte Carlo samples per seed at scoring time	Equal average of query-wise standardized S3F and EVE scores, where the EVE score is the negative mean evolution index across seeds

Table 11 : Deployment and scoring configurations of the protein-model baselines.

Category	Model	Baseline				Adaptive
Category	Model	T1	T2	T3	T4	Adaptive
Statistic	Total	123	74	67	29	123
LLM	GPT-6 Astra	7	0	0	0	5
	Claude Opus 5	12	6	6	1	9
	Gemini 3.1 Pro	0	0	0	0	0
	Kimi K3	2	3	2	1	—

Table 12: Number of assay-level queries replaced by the deterministic SHA-256 random baseline during evaluation, for the baseline PFArena setting and the multi-round adaptive search setting. A dash indicates that the corresponding task was not run for that model.

Figure 11 : Matched-query analysis of round-wise single-mutant generation. Marker lines show cumulative performance on all 123 assay queries, with deterministic random-baseline substitutions used for missing queries. Thin curves show performance on the common cohort of 113 assays completed by every model in all four rounds. Shaded regions indicate the performance gap between the full and matched evaluations.

Figure 12 : Complementary single-mutant selection in T1 . Left, overlap between evaluable top-

40

selections from VenusREM (PLM) and GPT-6 Astra (LLM) across 116 paired assays. Right, experimental fitness-rank distributions of PLM-only, shared and LLM-only selections. Rank scores were normalized within each assay from 1 (best) to 0 (worst). n denotes mutant–assay selections.

Figure 13 : Complementary correction of extreme ranking errors by GPT-6 Astra and VenusREM across T2 – T4 . The left column shows mutants for which GPT-6 Astra corrected extreme VenusREM ranking errors, and the right column shows the converse. True fitness rank scores range from 0 (worst) to 1 (best), whereas predicted rank scores range from 0 (bottom) to 1 (top). Orange crosses and blue circles denote VenusREM and GPT-6 Astra predictions, respectively; grey arrows connect predictions for the same mutant. Shaded regions indicate true bottom-30% mutants predicted in the top 10% or true top-30% mutants predicted in the bottom 10%. Central labels report the total number of mutants in each direction.

Explore similar work

Jun 1, 2026cs.AI

AgentPLM: Agentic Protein Language Models with Reasoning-Augmented Decoding for Protein Sequence Design

Protein language models (PLMs) are passive oracles: they generate sequences in a single forward pass with no mechanism to consult external biophysical feedback or redirect generation when a candidate violates thermodynamic or structural constraints. We introduce AgentPLM, which addresses this by equipping a pre-trained PLM with i) Reasoning-Augmented Decoding (RAD), which interleaves autoregressive generation with tool calls (ESMFold, FoldX, AutoDock Vina), and ii) Contrastive Agent Policy Optimisation (CAPO), a trajectory-level extension of direct preference optimisation that trains the policy end-to-end to learn when oracle feedback is informative rather than merely imitating high-fitness sequences. We evaluate AgentPLM on benchmark tasks spanning de novo enzyme design, antibody optimisation, thermostability, PPI interface design, and zero-shot fitness prediction with standardised oracle APIs and controlled sequence-identity splits. AgentPLM achieves state-of-the-art results with a gain in antibody top-10% hit rate over the strongest passive baseline, providing mechanistic evidence of online error correction without explicit backtracking.

Sahil Rahman, Maxx Richard Rahman

Bedford College, London, United Kingdom · Saarland University, Saarbrücken, Germany

Protein Design Antibody Language Modeling Search-Augmented Reasoning Physical Activities

Ask PDF

May 29, 2026q-bio.BM

AMix-2: Establishing Protein as a Native Modality in Large Language Models

We present AMix-2, a protein-text foundation model that establishes protein as a native modality in large language models (LLMs), unifying protein understanding and sequence design within a single foundation model. AMix-2 is built upon two key ideas: (1) a unified protein-text formulation that embeds natural language and protein sequence in a shared token space, enabling one model to perform biological reasoning and conditional design instead of separate downstream task-specialized models; and (2) a block-wise diffusion language modeling backbone that combines causal generation across blocks with bidirectional context and iterative refinement within blocks. This scheme better matches the intrinsic nature of proteins than a strict left-to-right factorization. To evaluate protein foundation models under realistic generalization settings, we further introduce ProteinArena, a comprehensive benchmark with time-aware and homology-aware protocols across various understanding and design tasks, and with baselines covering classical bioinformatics tools, protein-specialized models and LLMs. On ProteinArena, AMix-2 outperforms frontier LLMs and demonstrates competitive performance to task-specific protein models. Controlled experiments further show that the diffusion-based paradigm generally surpasses its autoregressive counterpart, highlighting the advantage of flexible generation order for protein sequences. We release both AMix-2 and ProteinArena to facilitate open research in protein foundation models.

Keyue Qiu, Yixin Wu, Lihao Wang +19

1Shanghai Artificial Intelligence Laboratory · Institute for AI Industry Research (AIR), Tsinghua University

Protein Language Modeling Wireless Foundation Models Diffusion Language Models Mix Modalities

Ask PDF

Aug 12, 2026cs.LG

Task- and dataset-specific information in protein language models

Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs). By consensus, embeddings from the models' last layers are used, while the models' internal behavior remains poorly understood. We analyzed 13 PLMs across 15 DTs and 9 datasets to assess the value of embeddings from intermediate PLM layers. We trained probe models on embeddings from each layer, compared their performance, and showed that the last layers of PLMs rarely produced embeddings that led to the best results on downstream tasks. Furthermore, we identified a connection between how models learn a certain DT and the similarity between that DT and the pre-training objective. For example, for residue-level downstream tasks, we observed a steady increase in performance across almost all PLM layers, which we attributed to their similarity to most PLMs' pre-training objectives. To allow the community to capitalize on our findings, we provide PLMSommelier, a Python package that automatically identifies the best PLM layer for a given DT with ~98% accuracy and creates a truncated model using only the early layers up to the best-performing layer. This will help users save time and memory during inference and yield better predictive performance.

Roman Joeres, Ilya Senatorov, Anastasia Kolchina +2

Drug Bioinformatics, Helmholtz Institute for Pharmaceutical Research Saarland, 66123 Saarbruecken, Germany · Center for Bioinformatics, Saarland University, 66123 Saarbruecken, Germany · Saarland Informatics Campus, Saarland University, 66123 Saarbruecken, Germany +2

Language Modeling

Ask PDF