CoPeP: Benchmarking Continual Pretraining for Protein Language Models

Darshan Patil; Pranshu Malviya; Mathieu Reymond; Quentin Fournier; Sarath Chandar

cs.LGFeb 27, 2026

CoPeP: Benchmarking Continual Pretraining for Protein Language Models

Authors: Darshan Patil, Pranshu Malviya, Mathieu Reymond, Quentin Fournier, Sarath Chandar

Organizations: Mila - Quebec AI Institute · Polytechnique Montréal · Chandar Research Lab · Canada CIFAR AI Chair

Abstract

Protein language models (pLMs) have recently gained significant attention for their ability to uncover relationships between sequence, structure, and function from evolutionary statistics, thereby accelerating therapeutic drug discovery. These models learn from large protein databases that are continuously updated by the biology community and whose dynamic nature motivates the application of continual learning, not only to keep up with the ever-growing data, but also as an opportunity to take advantage of the temporal meta-information that is created during this process. As a result, we introduce the Continual Pretraining of Protein Language Models (CoPeP) benchmark, a novel benchmark for evaluating continual learning approaches on pLMs. Specifically, we curate a sequence of protein datasets derived from the UniProt Knowledgebase spanning a decade and define metrics to assess pLM performance across 31 protein understanding tasks. We evaluate several methods from the continual learning literature, including replay, unlearning, and plasticity-based methods, some of which have never been applied to models and data of this scale. Our findings reveal that incorporating temporal meta-information improves perplexity by up to 7% even when compared to training on data from all tasks jointly. Moreover, even at scale, several continual learning methods outperform naive continual pretraining. The CoPeP benchmark offers an exciting opportunity to study these methods at scale in an impactful real-world application.

Figures & tables

Figure 1: Overview of the CoPeP benchmark. The UniProt Knowledgebase is continuously updated by biologists, which reflects the continuous discovery and curation of proteins. For every year’s task, we pull the latest UniRef100 release derived from that year’s UniProtKB release and use it to update the model. We evaluate several continual learning approaches on this sequence of datasets and assess their performance on a variety of protein understanding tasks.

Figure 2: Temporal evolution of UniRef100 from 2015 to 2024. The database shows a steady upward trend in total sequences despite millions of sequences being culled annually.

Panel of Figure 3 — Figure 3: Left: Evolution of sequence identity distributions relative to the validation set. We track the sequence identity of the UniRef100 entries in each year compared to their nearest neighbor in the validation set. The plot shows the change in sequence identity density relative to the 2015 baseline. A systematic shift toward lower identity values (peak at $\sim$ 0.25) indicates that newer sequences are increasingly divergent from the validation set. Right: Year-over-year density shifts. The initial divergence is driven by substantial shifts in early years, whereas later transitions (2020–2024) show smaller, stabilizing fluctuations.

Panel of Figure 4 — Figure 4: Impact of temporal data filtering. Models are trained exclusively on the intersection of sequences retained between two UniRef100 releases. Left shows the validation perplexity for each model and right shows the corresponding dataset sizes. The best performance in each row is boxed in red, and the best performance in each column is boxed in green. Most years show improved validation perplexity on at least one intersection dataset despite a substantial reduction in dataset size. The best performance by any model is achieved by a model trained on the intersection of 2022 and 2024, despite only having 64% of the data of the larger 2024 release.

Panel of Figure 5 — Figure 5: Perplexity (left) and Sequence Recovery (right) on the UniProt validation set with shading representing standard error. Continual learning baselines generally improve over time, and outperform jointly training on all data. Temporal Replay achieves the best performance outperforming other continual methods and single year matched training for all years. Every continual baseline outperforms the naive continual learning baseline, validating many of them for the first time at scale.

Figure 6: Spearman correlation on ProteinGym. 5 of the 6 continual learning methods outperform joint training, with Gradient Ascent and Hare Tortoise achieving the best scores. The best single year performance (2021 matched) is also outperformed by Gradient Ascent.

Figure 7: Win rates on multi-task protein understanding benchmarks for the best checkpoint for each method. Left : On PEER, Shrink and Perturb achieves the highest win rate, with Replay and Single Year Match in second place. Right : On DGEB, Random Labels achieves the highest win rate. Across both benchmarks, specialized continual learning methods consistently match or outperform the naive continual and single year baselines.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: UMAP projection of protein embeddings from all UniRef100 releases (AMPLIFY-350M representations). The plot shows a stable global structure with a dense core and branches, indicating natural groupings of proteins.

Figure 9: Yearly dynamics of UniRef100 embeddings. Top row (blue): proteins added in each year; bottom row (red): proteins removed. While the global organization of protein embeddings is stable, the local shifts such as density increases and cluster expansion indicate yearly shifts in the underlying distribution.

Figure 10: UMAP visualization of protein embeddings across continual learning methods $(2015\textendash 2022)$ with performance metrics (PPL: validation perplexity; PG: ProteinGym mean Spearman correlation). Color shades are based on Single Year clusters. Darker shades indicate denser regions. The same color shades are used in other rows/methods to indicate those centroid/representative protein locations.

Panel of Figure 11 — Figure 11: Two sample QQ-plots of various protein sequence statistics across different years. Each line represents a comparison between a year and the reference year 2015.

Figure 12: Accuracy on the UniProt validation set when training on different filtered datasets. The best performance in each row is boxed in red, and the best performance in each column is boxed in green. Note, the best performance overall ( $2022\cap 2024$ ) corresponds to training on an intersection that is 64% the size of the largest dataset (2024 no filter).

Figure 13: Proportion of different taxonomic groups in the UniProt validation set. Note that in order to make Archaea properly visible, the area for each sector is according to the log of the number of sequences in that lineage.

Panel of Figure 14 — Figure 14: Left: Sequence similarity between different proteins in the validation set. Right: Histogram of closest sequence similarity for each protein in the training set to any protein in the validation set.

Figure 15: Perplexity on the UniProt validation set broken down by taxonomic lineage for each method. In each subplot, the mean perplexity is shown in orange, and the lines for the more common groups are shaded to be darker and thicker.

Panel of Figure 16 — Figure 16: Distribution of Spearman correlations for each method from 2015–2024 on ProteinGym benchmarks.

Figure 17: Spearman correlation on ProteinGym stratified by taxon.

Figure 18: Spearman correlation on ProteinGym stratified by function.

Figure 19: Spearman correlation on ProteinGym stratified by MSA Number of Effective Sequences.

Figure 20: Win rates per year on the PEER benchmark.

Figure 21: Full results on the PEER benchmark.

Figure 22: Win rates per year on the DGEB benchmark.

Figure 23: Full results on the DGEB benchmark.

Figure 24: Both the cosine learning rate schedule and the warmup stable decay achieve approximately the same performance.

Method	Performance
Continual	$6.468\pm 0.043$
Replay	$\mathbf{6.157\pm 0.043}$
Shrink Perturb	$6.331\pm 0.043$
Gradient Ascent	$6.357\pm 0.043$
Hare Tortoise	$6.381\pm 0.043$
Random Labels	$6.344\pm 0.043$

Appendix

Table 1: Results of training for an equivalent number of on a Single Year (2015) compared to continual training across all years.

Figure 25: Data filtering experiments with a larger model (350M parameters).

Method	Hyperparameter	Distribution	Selected Value
Continual	None
Replay	$\lambda_{replay}$	Uniform(0, 1.0)	0.357495045651384
Hare and Tortoise	$\lambda_{ht\_mom}$	Uniform(.5, 1.0)	.931247906596137
	$\lambda_{reset\_freq}$	LogInt(10, 10000)	559
Gradient Ascent	$\lambda_{asc}$	Uniform(0, 1.0)	$0.0150798214665966$
Random Labels	$\lambda_{rand}$	Uniform(0, 1.0)	$0.00176366392582128$

Appendix

Table 2: The hyperparameter ranges and the selected hyperparameters for each method in our study.

Year	Release	Date	# of Proteins	# of Proteins after Deduplication
2015	2015_12	Dec. 9	70,511,308	68,944,139
2016	2016_11	Nov. 30	92,558,090	90,433,115
2017	2017_12	Dec. 20	128,263,573	125,565,575
2018	2018_11	Dec. 5	168,593,206	164,710,596
2019	2019_11	Dec. 11	213,522,593	209,351,328
2020	2020_06	Dec. 2	261,174,669	256,625,754

Appendix

Table 3: The selected UniRef100 releases in our benchmark. The number of proteins listed are taken from the UniRef website, before we do any processing and deduplicating.

Protein Language Models

Protein language models learn patterns in amino-acid sequences to predict protein properties, interpret mutations, and design new proteins.

See all 69 papers

May 8, 2026

ProteinJEPA: Latent prediction improves protein language model pretraining

Protein language models are trained primarily with masked language modeling (MLM), which predicts masked amino-acid identities. Joint-embedding predictive architectures (JEPA) instead predict latent representations, but have not been applied to proteins. ProteinJEPA supplements MLM with a cosine loss for predicting the half-depth hidden states of a teacher given the unmasked sequence. On 19 tasks, with ESM2 at 35M and 150M parameters and three pretraining seeds, MLM+JEPA outperforms compute-matched and step-matched MLM-only continued training in 78 and 76 of 114 comparisons (14 losses, 22 ties). The median compute-matched gain is

+0.0106

on structure- and homology-sensitive tasks versus

+0.0041

elsewhere, led by SCOPe-40 retrieval and remote homology with improvements of 6.1 percentage points in Recall@1 and 2.7 points in accuracy, respectively. Gains on these tasks increase with model size from 8M to 150M. Against the off-the-shelf checkpoint, MLM+JEPA wins 81 of 114 comparisons (median

+0.0068

) without improving MLM loss. In random initialization the gain is smaller and replicates inconsistently across seeds (

p{=}0.059

). The same recipe improves the causal ProGen3 model, beating a compute-matched next-token-prediction control on 12 of 16 tasks. Ablations show that cosine loss beats mean squared error, while adding shallower targets removes most of the task gain. JEPA-only training collapses downstream performance: latent prediction complements MLM rather than replacing it. Code: https://anonymous.4open.science/r/protJepa-FF24

Dan Ofer, Dafna Shahaf, Michal Linial

Department of Biological Chemistry The Hebrew University of Jerusalem · School of Computer Science and Engineering The Hebrew University of Jerusalem

Language Model Pretraining Masked Language Modeling Joint-Embedding Predictive Architecture Protein Language Models Protein Representation Learning

Ask PDF

Feb 24, 2026

Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference

Modern Protein Language Models (PLMs) apply transformer-based model architectures from natural language processing to biological sequences, predicting a variety of protein functions and properties. However, protein language has key differences from natural language, such as a rich functional space despite a vocabulary of only 20 amino acids. These differences motivate research into how transformer-based architectures operate differently in the protein domain and how we can better leverage PLMs to solve protein-related tasks. In this work, we begin by directly comparing how the distribution of information stored across layers of attention heads differs between the protein and natural language domain. Furthermore, we adapt a simple early-exit technique-originally used in the natural language domain to improve efficiency at the cost of performance-to achieve both increased accuracy and substantial efficiency gains in protein non-structural property prediction by allowing the model to automatically select protein representations from the intermediate layers of the PLMs for the specific task and protein at hand. We achieve performance gains ranging from 0.4 to 7.01 percentage points while simultaneously improving efficiency by over 10 percent across models and non-structural prediction tasks. Our work opens up an area of research directly comparing how language models change behavior when moved into the protein domain and advances language modeling in biological domains.

Anna Hart, Chi Han, Jeonghwan Kim +2

Siebel School of Data Science and Computing, University of Illinois Urbana-Champaign, 201 N Goodwin Ave, Urbana, IL 61801, USA · DOE Center for Advanced Bioenergy and Bioproducts Innovation, University of Illinois Urbana-Champaign, 1206 W Gregory Drive, Urbana, IL 61801, USA · Chemical and Biomolecular Engineering, University of Illinois Urbana-Champaign, 600 S Matthews Ave, Urbana, IL 61801, USA

Protein Language Models Protein Representation Learning Bioinformatics Molecular Property Prediction

Ask PDF

Aug 12, 2026

Task- and dataset-specific information in protein language models

Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs). By consensus, embeddings from the models' last layers are used, while the models' internal behavior remains poorly understood. We analyzed 13 PLMs across 15 DTs and 9 datasets to assess the value of embeddings from intermediate PLM layers. We trained probe models on embeddings from each layer, compared their performance, and showed that the last layers of PLMs rarely produced embeddings that led to the best results on downstream tasks. Furthermore, we identified a connection between how models learn a certain DT and the similarity between that DT and the pre-training objective. For example, for residue-level downstream tasks, we observed a steady increase in performance across almost all PLM layers, which we attributed to their similarity to most PLMs' pre-training objectives. To allow the community to capitalize on our findings, we provide PLMSommelier, a Python package that automatically identifies the best PLM layer for a given DT with ~98% accuracy and creates a truncated model using only the early layers up to the best-performing layer. This will help users save time and memory during inference and yield better predictive performance.

Roman Joeres, Ilya Senatorov, Anastasia Kolchina +2

Drug Bioinformatics, Helmholtz Institute for Pharmaceutical Research Saarland, 66123 Saarbruecken, Germany · Center for Bioinformatics, Saarland University, 66123 Saarbruecken, Germany · Saarland Informatics Campus, Saarland University, 66123 Saarbruecken, Germany +2

Protein Language Models Protein Representation Learning Language Model Probing

Ask PDF

Abstract

Protein Language Models

Explore similar work

ProteinJEPA: Latent prediction improves protein language model pretraining

Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference

Task- and dataset-specific information in protein language models