ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins

Yichen Zhou; Jonathan Golob; Amir Karimi; Stefan Bauer; Patrick Schwab

q-bio.QMMar 6, 2026

ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins

Authors: Yichen Zhou, Jonathan Golob, Amir Karimi, Stefan Bauer, Patrick Schwab

Organizations: GlaxoSmithKline · Technical University of Munich · University of Washington · KTH Royal Institute of Technology

Abstract

Protein language models (pLMs) have shown strong potential for zero-shot prediction of missense variant effects, yet systematic benchmarking on viral proteins remains limited, a critical gap given the need for proactive tools that can anticipate emerging mutations ahead of experimental validation. Here we introduce ViroGym, a comprehensive benchmark evaluating pLMs across three tasks: 79 deep mutational scanning (DMS) assays covering eukaryotic viruses with 552,065 mutated sequences across 7 phenotypic readouts, 21 influenza neutralisation tasks, and a real-world pandemic prediction task for SARS-CoV-2. We benchmark well-established pLMs on fitness landscapes, antigenic diversity, and pandemic forecasting, and find that the ProGen2 family consistently achieves the strongest performance across all three tasks. Crucially, DMS and neutralisation performance reliably identifies models that generalise to real-world emergence, even though the mutation sets they surface barely overlap, revealing that complementary in vitro benchmarks capture the evolutionary constraints needed for real-world mutation forecasting.

Figures & tables

Figure 1: ViroGym benchmark framework. The benchmark consists of two major components: in vitro experimental evaluation and real-world prediction tasks. The in vitro evaluation leverages experimental measurements from DMS assays and neutralisation assays to evaluate model performance on protein functional effects. The real-world component evaluates models on SARS-CoV-2 pandemic forecasting using viral sequence data from GISAID database, capturing model generalisation from controlled wet lab settings to natural viral evolution.

Strategy	Recall	Std.	Spearman	Std.
masked	0.1145	0.0454	0.1089	0.1206
wildtype	0.1123	0.0441	0.1063	0.1157
mutation	0.1144	0.0432	0.1083	0.1180
grammar	0.1244	0.0452	0.1108	0.1170
semantic	0.1382	0.0608	0.1678	0.1045
ratio	0.0809	0.0450	0.0912	0.0925

Table 1: Performance of ESM2-650M under different scoring strategies. Results are reported as the average top 10% recall and absolute Spearman’s rank correlation between model predictions and experimental measurements, with standard deviation across 79 tasks.

Model	Recall	Std.	Spearman	Std.
VESPAl	0.1681	0.1279	0.2561	0.1486
VESPA	0.1669	0.0971	0.2380	0.1351
Trancept.	0.1898	0.0857	0.2735	0.1482
ProtGPT2	0.1105	0.0382	0.1019	0.0768
ProGen2	0.1973	0.0913	0.2928	0.1595
ESM1v	0.1461	0.0667	0.1950	0.1018

Table 2: Zero-shot performance on the DMS benchmark. Results are reported as the average top 10% recall and absolute Spearman’s rank correlation between model scores and experimental measurements across all baselines, with standard deviation across 79 tasks.

Model Name	Spearman	Std.
ProtT5	0.1961	0.206
Tranception	0.2316	0.1696
ProtGPT2	0.2018	0.1845
ProGen2	0.2250	0.1852
ESM1v	0.2282	0.2043
ESM1	0.2222	0.2098

Table 3: Zero-shot neutralisation prediction results. Reported values are the average absolute Spearman’s rank correlation between model predictions and experimental measurements across all baseline methods.

Model Name	Recall	Spearman	Precision@3
VESPAl	0.2953	0.3158	0.00
VESPA	0.3238	0.3616	0.00
Tranception L	0.3743	0.4014	0.33
ProtGPT2	0.0968	0.0495	0.00
ProGen2 XL	0.4347	0.4018	0.33
ESM1v	0.2676	0.2682	0.00

Table 4: Zero-shot pandemic prediction results. Metrics reported for all baselines include Top 10% Recall, absolute Spearman’s rank correlation, and Precision@3 between model scores and mutation frequencies from GISAID. Evaluated on 24,187 substitution mutations (deletion mutations excluded for fair cross-model comparison, as VESPA/VESPAl cannot score deletions; results on all 25,460 mutations excluding VESPA/VESPAl are in Appendix Table 12 ). For multi-size families the best-performing variant is shown.

Figure 2: Pseudo-temporal validation across five SARS-CoV-2 variant eras. Each panel shows the metric for the best-performing variant from each model family as a function of the reference era. Deletion mutations are excluded for fair cross-model comparison. The dashed black line indicates the expected performance of a random predictor.

Figure 3: Temporal overlap of top-10% RBD mutations (Spike positions 331–531) across five SARS-CoV-2 variant eras. Blue: Top-10% Recall of ProGen2-XL predictions against is_emergent ground truth. Red: Top-10% Recall of era-matched DMS binding assays against the same ground truth. Green: Jaccard similarity between the ProGen2-XL and DMS top-10% prediction sets.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Phenotype	Sequences	Reference
*ZIKV	viral growth	10,080	Sourisseau et al. [2019]
ZIKV	immune escape	9,576	Sourisseau et al. [2019] , Kikawa et al. [2023]
RABV	immune escape	7,220	Aditham et al. [2025]
RABV	cell entry	8,614	Aditham et al. [2025]
NIPAH	binding	8,654	Larsen et al. [2025]
NIPAH	immune escape	8,690	Larsen et al. [2025]

Appendix

Table 5: Sources of DMS benchmark datasets (part 1 of 2). Mutational effect prediction tasks are based on DMS assays. Datasets marked * overlap with ProteinGym.

Dataset	Phenotype	Sequences	Reference
SCV2 RBD Omicron BA.2	binding	4,002	Starr et al. [2022b]
SCV2 RBD Omicron BA.2	expression	4,002	Starr et al. [2022b]
SCV2 RBD Omicron BQ.1.1	binding	4,203	Taylor and Starr [2023]
SCV2 RBD Omicron BQ.1.1	expression	4,202	Taylor and Starr [2023]
SCV2 RBD Omicron XBB.1.5	binding	4,212	Taylor and Starr [2023]
SCV2 RBD Omicron XBB.1.5	expression	4,212	Taylor and Starr [2023]

Appendix

Table 6: Sources of DMS benchmark datasets (part 2 of 2). Datasets marked * overlap with ProteinGym.

Virus	Assays	Substitution	Indel	Stop/RT	Total
SARS-CoV-2	36	172,394	2,835	0	175,229
Influenza A	17	117,674	0	0	117,674
HIV	11	83,646	353	97	84,096
Nipah	3	27,434	0	0	27,434
Zika	2	19,656	0	0	19,656
Rabies	2	15,834	0	0	15,834

Appendix

Table 7: Per-virus breakdown of DMS entries by mutation type across all 79 assays (552,065 total entries). Substitution : standard single amino acid change ( X#Y ); Indel : deletion or insertion; Combinatorial : two or more simultaneous substitutions (colon-separated); Stop/RT : nonsense (AA

\to

stop) or readthrough (stop

\to

AA) variants.

	Binding	Cell entry	Expression	Immune escape	Stability	Viral growth	Total
SARS-CoV-2	15	4	12	4		1	36
Influenza A		2		3	2	10	17
HIV		2		3		6	11
Nipah	1	1		1			3
Zika				1		1	2
Rabies		1		1			2

Appendix

Table 8: Distribution of viruses and phenotypes across all 79 DMS functional assays.

Dataset	Sera Source	Subtype	Reference
A/Massachusetts/18/2022	ferret	H3N2	Kikawa et al. [2025a]
A/Thailand/8/2022	ferret	H3N2	Kikawa et al. [2025a]
A/DistrictOfColumbia/27/2023	ferret	H3N2	Kikawa et al. [2025a]
A/Croatia/10136RV/2023-egg	ferret	H3N2	Kikawa et al. [2025a]
A/Netherlands/10563/2023	ferret	H3N2	Kikawa et al. [2025a]
A/Lisboa/216/2023	ferret	H3N2	Kikawa et al. [2025a]

Appendix

Table 9: Sources of the 21 influenza A neutralisation assays used for antigenic diversity prediction. Sera from ferret antisera (post-infection) and human donors (post-vaccination) are included; subtypes H3N2 and H1N1 are represented.

Virus	Sub-only tasks	Comb. tasks	Indel/other tasks	Total tasks
SARS-CoV-2	19	0	17	36
Influenza A	17	0	0	17
HIV	6	0	5	11
Nipah	3	0	0	3
Zika	2	0	0	2
Rabies	2	0	0	2

Appendix

Table 10: Classification of the 79 DMS tasks by mutation content. Sub-only : every variant is a single amino-acid substitution ( X#Y ). Comb. : task contains multi-site combinatorial mutants but no deletions (AAV2: 98.7% combinatorial; PESV: 80.6%). Indel/other : task contains deletion, insertion, stop-codon, or readthrough variants. Per-virus sequence counts broken down by mutation type are in Table 7 .

Model Name	Recall	Std.	Spearman	Std.	Spearman	Std.	Recall	Spearman
	DMS				Neutralisation		GISAID
VESPAl	0.1681	0.1279	0.2561	0.1486	0.1965	0.2057	0.2953	0.3158
VESPA	0.1669	0.0971	0.2380	0.1351	0.1965	0.2057	0.3238	0.3616
Tranception S	0.1603	0.0759	0.2358	0.1381	0.1843	0.1810	0.2370	0.2894
Tranception M	0.1734	0.0755	0.2654	0.1346	0.2316	0.1696	0.3143	0.3249
Tranception L	0.1898	0.0857	0.2735	0.1482	0.1927	0.1961	0.3743	0.4014

Appendix

Table 11: Results for all models across tasks.

Model Name	Recall	Spearman	Precision@3
ProGen2 XL	0.4077	0.3170	0.33
ProGen2 BFD90	0.3798	0.2950	0.33
ProGen2	0.3794	0.3245	0.33
Tranception L	0.3523	0.3326	0.33
Tranception M	0.2985	0.3217	0.33
ProGen2 M	0.2639	0.2428	0.00

Appendix

Table 12: Zero-shot pandemic prediction results for all models on all 25,460 mutations (VESPA/VESPAl excluded as they cannot score deletion mutations). Metrics: Top-10% Recall, absolute Spearman’s rank correlation, and Precision@3 against GISAID all-time mutation frequencies.

Model Name	Recall	Improvement (%)	Spearman	Improvement (%)
ESM1 670M UR50D	0.1453	4.2070	0.1842	4.0250
ESM2 3B	0.1485	3.9177	0.1890	13.9151
VESPAl	0.1746	3.8895	0.2586	0.9669
ESM2 150M	0.1351	3.8340	0.1376	20.6038
Tranception S	0.1662	3.6926	0.2502	6.1056
Tranception L	0.1964	3.4928	0.2801	2.4001

Appendix

Table 13: Performance on the 56 substitution-only DMS tasks (no indels). Improvement is the relative percentage change over the full 79-task results, showing modest gains from excluding indel-containing assays.

	Top-10% Recall			$\|\mathrm{Spearman}\|$
Model	Mean	$\Delta$	$p_{\mathrm{adj}}$	Mean	$\Delta$	$p_{\mathrm{adj}}$
ProGen2 XL (ref)	0.1973	—	—	0.2928	—	—
VESPAl	0.1682	$-$ 0.0291	$0.0001^{***}$	0.2561	$-$ 0.0367	$0.0510^{\mathrm{ns}}$
VESPA	0.1669	$-$ 0.0304	$0.0002^{***}$	0.2380	$-$ 0.0548	$0.0000^{***}$
Tranception S	0.1603	$-$ 0.0370	$0.0004^{***}$	0.2357	$-$ 0.0571	$0.0043^{**}$
Tranception M	0.1734	$-$ 0.0239	$0.0002^{***}$	0.2653	$-$ 0.0275	$0.0043^{**}$

Appendix

Table 14: Pairwise statistical significance of DMS benchmark comparisons against ProGen2-XL. For each model,

\Delta

is the mean difference (model

-

ProGen2-XL) across all 79 shared tasks.

p_{\mathrm{adj}}

is the two-sided exact paired sign-test

p

-value after Benjamini–Hochberg correction within each metric (

n=79

tasks for all pairs). Significance:

{}^{***}p<0.001

{}^{**}p<0.01

{}^{*}p<0.05

, ns not significant (

\alpha=0.05

Figure 4: SARS-CoV-2 Spike Protein Mutation Heat Map. This heat map displays the frequency of 21 potential amino acid substitutions across 1273 residues of the SARS-CoV-2 Spike protein, with colour intensity indicating mutation frequency at each position. Data were collected from the GISAID database between January 2020 and May 2025.

Figure 5: Task-wise comparison of ESM2 15B and ProGen2-XL on the DMS benchmark. ESM2 15B scores are computed using the semantic scoring strategy, while ProGen2-XL scores use the negative log-likelihood strategy. Reported values represent the absolute Spearman’s rank correlation between model fitness scores and experimental measurements.

Figure 6: Task-wise comparison of all baselines on the DMS benchmark. Reported values represent the absolute Spearman’s rank correlation between model fitness scores and experimental measurements.

Figure 7: Task-wise comparison of all baselines on the DMS benchmark. Reported values represent the top 10% recall between model fitness scores and experimental measurements.

Figure 8: Task-wise comparison of all baselines on the neutralisation benchmark. Reported values represent the absolute Spearman’s rank correlation between model fitness scores and experimental measurements.

Figure 9: Task-wise performance of Tranception M on the neutralisation benchmark. Antigenicity scores are computed using the negative log-likelihood strategy. Each task corresponds to a vaccine strain representing post-vaccination serum, with colors indicating influenza A subtypes. Performance is measured as the absolute Spearman’s rank correlation between predicted antigenicity scores and experimental measurements, averaged across sera from different animal sources.

Figure 10: Overlap among top-10 RBD mutations (Spike positions 331–531) from computational predictions, in vitro DMS assays (Wuhan-Hu-1 ACE2 binding), and naturally occurring mutations from GISAID. a. ESM2-650M predictions show no overlap with DMS or GISAID mutations. b. ProGen2-XL overlaps

\sim

40% with GISAID and

\sim

20% with DMS, while DMS and GISAID share only

\sim

10%, confirming that pLM and DMS signals are complementary.

Figure 11: In silico fitness score heatmap for ESM1. Colour intensity indicates the predicted mutational effect (semantic distance score) at each position.

Figure 12: In silico fitness score heatmap for ESM2 (semantic distance scoring).

Figure 13: In silico fitness score heatmap for ESM1v (semantic distance scoring).

Figure 14: In silico fitness score heatmap for ProGen2 (negative log-likelihood scoring).

Figure 15: In silico fitness score heatmap for ProtGPT2 (negative log-likelihood scoring).

Figure 16: In silico fitness score heatmap for Tranception (negative log-likelihood scoring).

Figure 17: In silico fitness score heatmap for VESPA (euclidean distance scoring).

Figure 18: In silico fitness score heatmap for VESPAl (euclidean distance scoring).

Protein Language Models

Protein language models learn patterns in amino-acid sequences to predict protein properties, interpret mutations, and design new proteins.

See all 69 papers

Viral Proteins Reveal Geometry of Protein Language Models
Jun 10, 2026Arthur Bigot, Harmon Bhasin, Core Francisco Park +2Protein Language Models Protein Representation Learning
Department of Computer Science, ETH Zurich, Zurich, Switzerland · Department of Chemistry and Chemical Biology, Harvard University, Cambridge, MA, USA · TwentyTwo, USA +2
A General Harness for Protein Foundation Model Fitness Prediction
Sep 28, 2026Yang Tan, Qijia Tian, Gangyu Sun +5Protein Fitness Prediction Protein Language Models
Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory
Better Protein Function Prediction by Modeling Survivorship Bias
May 7, 2026Zhongmou Chao, Poompol Buathong, Ekaterina Selivanovitch +2Protein Function Prediction Weakly Supervised Learning
Smith School of Chemical and Biomolecular Engineering, Cornell University, USA · Center for Applied Mathematics, Cornell University, USA · School of Operations Research and Information Engineering, Cornell University, USA

Jun 10, 2026J/K move · Enter open · S save

Viral Proteins Reveal Geometry of Protein Language Models

Arthur Bigot, Harmon Bhasin, Core Francisco Park, Eugene Shakhnovich+1

Department of Computer Science, ETH Zurich, Zurich, Switzerland · Department of Chemistry and Chemical Biology, Harvard University, Cambridge, MA, USA · TwentyTwo, USA+2

Protein Language Models Protein Representation Learning Molecular Representation Learning

Open paper AskPDF arXiv

Protein language models are trained on highly imbalanced datasets, raising the question of how they represent underrepresented biological sequences. Using viral proteins as a case study across ESM model families, we identify a dominant nativeness axis in embedding space, aligned with masked reconstruction perplexity, that orders sequences from well-modeled cellular proteins through viral proteins to shuffled and random sequences. Scaling contracts this axis unevenly across viral families. Despite this, protein language model embeddings retain viral-specific signal: viral proteins remain linearly separable beyond zero-shot perplexity and shallow sequence features. Together, these results suggest that pLM representations are structured by a general notion of nativeness while preserving information specific to distinct biological groups.

Abstract

Protein Language Models

Explore similar work

Viral Proteins Reveal Geometry of Protein Language Models

A General Harness for Protein Foundation Model Fitness Prediction

Better Protein Function Prediction by Modeling Survivorship Bias