cs.LGFeb 27, 2026

CoPeP: Benchmarking Continual Pretraining for Protein Language Models

Authors: Darshan Patil, Pranshu Malviya, Mathieu Reymond, Quentin Fournier, Sarath Chandar

Organizations: Mila - Quebec AI Institute · Polytechnique Montréal · Chandar Research Lab · Canada CIFAR AI Chair

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Protein language models (pLMs) have recently gained significant attention for their ability to uncover relationships between sequence, structure, and function from evolutionary statistics, thereby accelerating therapeutic drug discovery. These models learn from large protein databases that are continuously updated by the biology community and whose dynamic nature motivates the application of continual learning, not only to keep up with the ever-growing data, but also as an opportunity to take advantage of the temporal meta-information that is created during this process. As a result, we introduce the Continual Pretraining of Protein Language Models (CoPeP) benchmark, a novel benchmark for evaluating continual learning approaches on pLMs. Specifically, we curate a sequence of protein datasets derived from the UniProt Knowledgebase spanning a decade and define metrics to assess pLM performance across 31 protein understanding tasks. We evaluate several methods from the continual learning literature, including replay, unlearning, and plasticity-based methods, some of which have never been applied to models and data of this scale. Our findings reveal that incorporating temporal meta-information improves perplexity by up to 7% even when compared to training on data from all tasks jointly. Moreover, even at scale, several continual learning methods outperform naive continual pretraining. The CoPeP benchmark offers an exciting opportunity to study these methods at scale in an impactful real-world application.

Figures & tables

Figure 1: Overview of the CoPeP benchmark. The UniProt Knowledgebase is continuously updated by biologists, which reflects the continuous discovery and curation of proteins. For every year’s task, we pull the latest UniRef100 release derived from that year’s UniProtKB release and use it to update the model. We evaluate several continual learning approaches on this sequence of datasets and assess their performance on a variety of protein understanding tasks.
Figure 2: Temporal evolution of UniRef100 from 2015 to 2024. The database shows a steady upward trend in total sequences despite millions of sequences being culled annually.
Figure 3: Left: Evolution of sequence identity distributions relative to the validation set. We track the sequence identity of the UniRef100 entries in each year compared to their nearest neighbor in the validation set. The plot shows the change in sequence identity density relative to the 2015 baseline. A systematic shift toward lower identity values (peak at ∼\sim 0.25) indicates that newer sequences are increasingly divergent from the validation set. Right: Year-over-year density shifts. The initial divergence is driven by substantial shifts in early years, whereas later transitions (2020–2024) show smaller, stabilizing fluctuations.
Figure 4: Impact of temporal data filtering. Models are trained exclusively on the intersection of sequences retained between two UniRef100 releases. Left shows the validation perplexity for each model and right shows the corresponding dataset sizes. The best performance in each row is boxed in red, and the best performance in each column is boxed in green. Most years show improved validation perplexity on at least one intersection dataset despite a substantial reduction in dataset size. The best performance by any model is achieved by a model trained on the intersection of 2022 and 2024, despite only having 64% of the data of the larger 2024 release.
Figure 5: Perplexity (left) and Sequence Recovery (right) on the UniProt validation set with shading representing standard error. Continual learning baselines generally improve over time, and outperform jointly training on all data. Temporal Replay achieves the best performance outperforming other continual methods and single year matched training for all years. Every continual baseline outperforms the naive continual learning baseline, validating many of them for the first time at scale.
Figure 6: Spearman correlation on ProteinGym. 5 of the 6 continual learning methods outperform joint training, with Gradient Ascent and Hare Tortoise achieving the best scores. The best single year performance (2021 matched) is also outperformed by Gradient Ascent.
Figure 7: Win rates on multi-task protein understanding benchmarks for the best checkpoint for each method. Left : On PEER, Shrink and Perturb achieves the highest win rate, with Replay and Single Year Match in second place. Right : On DGEB, Random Labels achieves the highest win rate. Across both benchmarks, specialized continual learning methods consistently match or outperform the naive continual and single year baselines.
Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: UMAP projection of protein embeddings from all UniRef100 releases (AMPLIFY-350M representations). The plot shows a stable global structure with a dense core and branches, indicating natural groupings of proteins.
Figure 9: Yearly dynamics of UniRef100 embeddings. Top row (blue): proteins added in each year; bottom row (red): proteins removed. While the global organization of protein embeddings is stable, the local shifts such as density increases and cluster expansion indicate yearly shifts in the underlying distribution.
Figure 10: UMAP visualization of protein embeddings across continual learning methods (2015\textendash2022)(2015\textendash 2022) with performance metrics (PPL: validation perplexity; PG: ProteinGym mean Spearman correlation). Color shades are based on Single Year clusters. Darker shades indicate denser regions. The same color shades are used in other rows/methods to indicate those centroid/representative protein locations.
Figure 11: Two sample QQ-plots of various protein sequence statistics across different years. Each line represents a comparison between a year and the reference year 2015.
Figure 12: Accuracy on the UniProt validation set when training on different filtered datasets. The best performance in each row is boxed in red, and the best performance in each column is boxed in green. Note, the best performance overall ( 2022∩20242022\cap 2024 ) corresponds to training on an intersection that is 64% the size of the largest dataset (2024 no filter).
Figure 13: Proportion of different taxonomic groups in the UniProt validation set. Note that in order to make Archaea properly visible, the area for each sector is according to the log of the number of sequences in that lineage.
Figure 14: Left: Sequence similarity between different proteins in the validation set. Right: Histogram of closest sequence similarity for each protein in the training set to any protein in the validation set.
Figure 15: Perplexity on the UniProt validation set broken down by taxonomic lineage for each method. In each subplot, the mean perplexity is shown in orange, and the lines for the more common groups are shaded to be darker and thicker.
Figure 16: Distribution of Spearman correlations for each method from 2015–2024 on ProteinGym benchmarks.
Figure 17: Spearman correlation on ProteinGym stratified by taxon.
Figure 18: Spearman correlation on ProteinGym stratified by function.
Figure 19: Spearman correlation on ProteinGym stratified by MSA Number of Effective Sequences.
Figure 20: Win rates per year on the PEER benchmark.
Figure 21: Full results on the PEER benchmark.
Figure 22: Win rates per year on the DGEB benchmark.
Figure 23: Full results on the DGEB benchmark.
Figure 24: Both the cosine learning rate schedule and the warmup stable decay achieve approximately the same performance.
MethodPerformance
Continual6.468±0.0436.468\pm 0.043
Replay6.157±0.043\mathbf{6.157\pm 0.043}
Shrink Perturb6.331±0.0436.331\pm 0.043
Gradient Ascent6.357±0.0436.357\pm 0.043
Hare Tortoise6.381±0.0436.381\pm 0.043
Random Labels6.344±0.0436.344\pm 0.043
Appendix
Table 1: Results of training for an equivalent number of on a Single Year (2015) compared to continual training across all years.
Figure 25: Data filtering experiments with a larger model (350M parameters).
MethodHyperparameterDistributionSelected Value
ContinualNone  
Replayλreplay\lambda_{replay}Uniform(0, 1.0)0.357495045651384
Hare and Tortoiseλht_mom\lambda_{ht\_mom}Uniform(.5, 1.0).931247906596137
 λreset_freq\lambda_{reset\_freq}LogInt(10, 10000)559
Gradient Ascentλasc\lambda_{asc}Uniform(0, 1.0)0.01507982146659660.0150798214665966
Random Labelsλrand\lambda_{rand}Uniform(0, 1.0)0.001763663925821280.00176366392582128
Appendix
Table 2: The hyperparameter ranges and the selected hyperparameters for each method in our study.
YearReleaseDate# of Proteins# of Proteins after Deduplication
20152015_12Dec. 970,511,30868,944,139
20162016_11Nov. 3092,558,09090,433,115
20172017_12Dec. 20128,263,573125,565,575
20182018_11Dec. 5168,593,206164,710,596
20192019_11Dec. 11213,522,593209,351,328
20202020_06Dec. 2261,174,669256,625,754
Appendix
Table 3: The selected UniRef100 releases in our benchmark. The number of proteins listed are taken from the UniRef website, before we do any processing and deduplicating.
More in this topic

Protein Language Models

Protein language models learn patterns in amino-acid sequences to predict protein properties, interpret mutations, and design new proteins.

See all 69 papers

Explore similar work

CardsList

  1. ProteinJEPA: Latent prediction improves protein language model pretraining

    May 8, 2026Dan Ofer, Dafna Shahaf, Michal LinialLanguage Model PretrainingMasked Language Modeling

    Department of Biological Chemistry The Hebrew University of Jerusalem · School of Computer Science and Engineering The Hebrew University of Jerusalem

  2. Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference

    Feb 24, 2026Anna Hart, Chi Han, Jeonghwan Kim +2Protein Language ModelsProtein Representation Learning

    Siebel School of Data Science and Computing, University of Illinois Urbana-Champaign, 201 N Goodwin Ave, Urbana, IL 61801, USA · DOE Center for Advanced Bioenergy and Bioproducts Innovation, University of Illinois Urbana-Champaign, 1206 W Gregory Drive, Urbana, IL 61801, USA · Chemical and Biomolecular Engineering, University of Illinois Urbana-Champaign, 600 S Matthews Ave, Urbana, IL 61801, USA

  3. Task- and dataset-specific information in protein language models

    Aug 12, 2026Roman Joeres, Ilya Senatorov, Anastasia Kolchina +2Protein Language ModelsProtein Representation Learning

    Drug Bioinformatics, Helmholtz Institute for Pharmaceutical Research Saarland, 66123 Saarbruecken, Germany · Center for Bioinformatics, Saarland University, 66123 Saarbruecken, Germany · Saarland Informatics Campus, Saarland University, 66123 Saarbruecken, Germany +2