Do Vision CIL Methods Transfer to Protein? A Diagnostic Benchmark of Frozen Protein Encoders

Ziyuan Gao

cs.LGOct 1, 2026

Do Vision CIL Methods Transfer to Protein? A Diagnostic Benchmark of Frozen Protein Encoders

Authors: Ziyuan Gao

Organizations: Department of Computer Science and Technology, University of Cambridge

Abstract

Protein function annotation operates over an expanding label space, making class-incremental learning (CIL) a natural framework for incorporating newly discovered functions without full retraining. Yet existing CIL methods have been developed largely in vision and have not been systematically evaluated over frozen protein representations. We conduct a controlled benchmark of nine representative CIL methods across five sequence-, structure-, and sequence--structure-based protein encoders and two enzyme function datasets at different scales, while keeping the encoders frozen and evaluation settings consistent across representations. We find that no method transfers reliably across encoders, and that these failures persist across alternative phase constructions. Diagnostic analyses reveal that this instability arises from weak gradient-based stability signals, substantial heterogeneity in embedding geometry, and covariance assumptions whose appropriate capacity changes with both dataset scale and encoder. These findings suggest that the representation itself should be treated as a central component of continual-learning design rather than a passive feature extractor. More generally, reliable protein CIL will require encoder-aware methods that adapt their statistical assumptions and capacity to the geometry of the representation on which they operate. Code is available at https://github.com/zygao930/ProteinCIL.

Figures & tables

Figure 1: Motivation for frozen protein CIL. Frozen protein encoders induce distinct class geometries, creating a covariance-capacity trade-off as new classes arrive.

		Enzyme Reaction (348 classes)					EC Number (4788 classes)
		ESM2	ESM2	GearNet			ESM2	ESM2	GearNet
Cat.	Method	650M	150M	Edge	SaProt	ProtBert	650M	150M	Edge	SaProt	ProtBert
Average Incremental Accuracy (%) $\uparrow$
LB	Naive	13.7 $\pm$ 0.5	12.6 $\pm$ 1.5	7.4 $\pm$ 1.4	13.4 $\pm$ 0.7	10.7 $\pm$ 0.5	25.4 $\pm$ 1.6	23.5 $\pm$ 1.4	8.6 $\pm$ 0.3	20.7 $\pm$ 0.8	21.4 $\pm$ 0.2
Joint	Joint MLP	73.9 $\pm$ 0.8	69.1 $\pm$ 0.8	52.1 $\pm$ 1.1	71.2 $\pm$ 0.4	65.5 $\pm$ 0.3	99.5 $\pm$ 0.2	99.5 $\pm$ 0.1	68.4 $\pm$ 0.7	99.4 $\pm$ 0.0	99.5 $\pm$ 0.0
Reg.	EWC	21.8 $\pm$ 4.0	17.7 $\pm$ 4.1	6.9 $\pm$ 1.1	20.8 $\pm$ 4.0	15.0 $\pm$ 1.8	45.2 $\pm$ 5.4	40.2 $\pm$ 2.9	8.7 $\pm$ 1.2	32.6 $\pm$ 1.8	40.2 $\pm$ 1.3

Table 1: Benchmark results across two datasets and five encoders. Encoder-specific results are reported as mean

\pm

std over 3 seeds. Best per column in bold , second underlined (Naive and Joint excluded from ranking). Category abbreviations: Reg. = Regularization, Rep. = Replay, Sub. = Subspace, Proj. = Projection, Proto. = Prototype.

Figure 2: Continual-learning performance on EC Number with ESM2-650M. Left: Average incremental accuracy across phases. Middle: Forgetting across phases. Right: Final accuracy across the seven top-level EC families.

Data- set	Method	Random classes			Finer-EC buckets			Mixed hierarchy
Data- set	Method	Acc $\uparrow$	Fgt $\downarrow$	Plast $\uparrow$	Acc $\uparrow$	Fgt $\downarrow$	Plast $\uparrow$	Acc $\uparrow$	Fgt $\downarrow$	Plast $\uparrow$
Enzyme Reaction	EWC	14.5 $\pm$ 2.8	76.3 $\pm$ 4.6	79.9 $\pm$ 6.5	14.8 $\pm$ 3.1	71.5 $\pm$ 4.0	76.1 $\pm$ 6.5	15.4 $\pm$ 3.3	76.5 $\pm$ 3.5	81.0 $\pm$ 6.2
	FeTrIL	21.4 $\pm$ 2.2	68.3 $\pm$ 6.4	79.9 $\pm$ 7.6	23.4 $\pm$ 3.0	62.6 $\pm$ 4.8	77.0 $\pm$ 7.2	22.1 $\pm$ 2.7	68.1 $\pm$ 5.5	80.5 $\pm$ 7.4
	EASE	39.2 $\pm$ 9.4	14.8 $\pm$ 0.5	51.8 $\pm$ 9.4	42.8 $\pm$ 9.9	12.6 $\pm$ 0.7	53.5 $\pm$ 9.9	39.3 $\pm$ 9.3	14.5 $\pm$ 1.0	51.6 $\pm$ 9.8
	RER	46.6 $\pm$ 6.9	35.2 $\pm$ 2.1	76.6 $\pm$ 7.1	49.3 $\pm$ 8.1	29.2 $\pm$ 1.5	74.1 $\pm$ 7.2	49.4 $\pm$ 7.3	32.8 $\pm$ 1.5	77.5 $\pm$ 7.2
	Replay	46.5 $\pm$ 6.9	35.5 $\pm$ 2.0	76.8 $\pm$ 6.7	48.9 $\pm$ 8.2	29.8 $\pm$ 2.0	74.3 $\pm$ 6.7	49.1 $\pm$ 6.9	33.1 $\pm$ 0.9	77.6 $\pm$ 6.7

Table 2: Protocol sanity check with baselines under alternative phase constructions. We report encoder-average Avg Acc, Forgetting, and Plasticity (%, mean

\pm

std over the five encoders; per-encoder values are averaged over three seeds) under the random, finer-EC, and mixed phase-construction protocols.

		Gradient signal (Hypo.A)		Embedding geometry (Hypo.B)
Dataset	Encoder	Conflict cosine	Fisher concentration	Embedding dimension	Intrinsic rank $r_{95}$	Effective rank	Condition number	Class separability
Enzyme Reaction (348 classes)	ESM2-650M	$-0.047$	0.90	1280	179	17.2	$1.5\!\times\!10^{5}$	0.205
	ESM2-150M	$-0.044$	0.88	640	60	7.5	$1.6\!\times\!10^{5}$	0.203
	GearNet-Edge	$-0.048$	0.96	3072	84	31.7	$7.1\!\times\!10^{9}$	0.241
	SaProt	$-0.045$	0.78	1280	177	29.4	$6.9\!\times\!10^{6}$	0.209
	ProtBert	$-0.046$	0.92	1024	75	9.6	$8.6\!\times\!10^{4}$	0.165

Table 3: Diagnostic statistics for gradient signal (Hypothesis A) and embedding geometry (Hypothesis B) across datasets and encoders, averaged over three seeds. Gradient diagnostics report inter-phase conflict cosine and top-1% Fisher concentration on the MLP head. Geometry diagnostics report embedding dimension, intrinsic rank

r_{95}

, effective rank, within-class covariance condition number, and between/within class separability.

Figure 3: Representative diagnostic patterns across methods and protein encoders. (a) Phase-wise accuracy matrices $R$ for RanPAC, FeCAM, EWC, and FeTrIL on EC Number with ESM2-650M. (b)(c) EC hierarchy error decomposition across encoders on Enzyme Reaction and EC Number.

ESM2-150M				SaProt
True	Pred.	# mis.	Example	True	Pred.	# mis.	Example
3 (hydro.)	2 (transf.)	461	3.1.4.35 $\to$ 2.4.1.229	2 (transf.)	3 (hydro.)	84	2.3.2.27 $\to$ 3.2.2.6
2 (transf.)	3 (hydro.)	270	2.7.2.3 $\to$ 3.5.1.76	3 (hydro.)	2 (transf.)	77	3.5.4.26 $\to$ 2.1.1.85
2 (transf.)	1 (oxidored.)	170	2.7.1.148 $\to$ 1.1.1.193	3 (hydro.)	1 (oxidored.)	39	3.5.4.19 $\to$ 1.1.1.23
3 (hydro.)	1 (oxidored.)	159	3.1.21.4 $\to$ 1.13.11.58	7 (transl.)	1 (oxidored.)	33	7.1.1.9 $\to$ 1.7.2.5
2 (transf.)	4 (lyases)	117	2.4.2.9 $\to$ 4.2.1.91	2 (transf.)	1 (oxidored.)	28	2.7.2.3 $\to$ 1.2.1.28

Table 4: Representative cross-family confusions under a single shared covariance on EC Number. ESM2-150M and SaProt provide contrasting examples of cross-family error structure across encoders. Each row reports a true EC family, its predicted family, the number of misclassified instances, and one representative EC-level confusion.

		No covariance assumption			Mahalanobis (covariance granularity)
Dataset	Encoder	Euclidean NCM	Cosine NCM	Ridge classifier	Shared (diagonal)	Full shared	Grouped $K\!=\!8$	Per- class
Enzyme Reaction (348 classes)	ESM2-650M	0.130	0.167	0.506	0.594	0.610	0.490	0.541
	ESM2-150M	0.108	0.147	0.484	0.586	0.591	0.478	0.523
	GearNet-Edge	0.183	0.198	0.295	0.371	0.472	0.267	0.248
	SaProt	0.220	0.236	0.474	0.603	0.615	0.566	0.552
	ProtBert	0.058	0.074	0.451	0.576	0.604	0.451	0.473

Table 5: Effect of covariance granularity on prototype classification across datasets and encoders. Classification accuracy is reported on 256-dimensional PCA projections for three covariance-free controls and four Mahalanobis variants, averaged over three seeds. Best result for each dataset–encoder pair is shown in bold .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Classes	Train	Test	Train/class	Phases
Enzyme Reaction	348	29,215	5,651	$\sim$ 84	7
EC Number	4,788	73,259	9,112	$\sim$ 15	7

Appendix

Table 6: Dataset statistics. Counts report the final cached sequence-encoder splits; structure-dependent encoders may exclude samples without matched structures.

Encoder	Type	Input	Dim.	Pooling
ESM2-650M [ 6 ]	Sequence PLM	AA sequence	1280	Mean pool, final layer
ESM2-150M [ 6 ]	Sequence PLM	AA sequence	640	Mean pool, final layer
ProtBert [ 10 ]	Sequence PLM	Space-separated AA	1024	Attention-weighted mean
SaProt [ 11 ]	Seq.+Struct.	AA + 3Di tokens	1280	Attention-weighted mean
GearNet-Edge [ 7 ]	Structure GNN	Residue graph from 3D coords	3072	Node mean; L2 norm.

Appendix

Table 7: Frozen encoders and embedding extraction details.

Method	Strategy	Core Mechanism	Key Hyperparameters
Naive	Lower bound	Train on new phase only	Hidden=1024, lr= $10^{-3}$ , 100 epochs
Joint MLP	Joint reference	Retrain on all data	Same as Naive
EWC [ 5 ]	Regularization	Fisher-weighted L2 penalty	$\lambda{=}2000$
Replay [ 20 ]	Replay	Fixed-size reservoir buffer	Buffer=2,000
iCaRL [ 4 ]	Replay + Distillation	Exemplar replay + distillation	Buffer=2,000
DER++ [ 21 ]	Replay + Distillation	Logit replay + label replay	Buffer=2,000, $\alpha{=}0.5$ , $\beta{=}0.5$

Appendix

Table 8: Baseline CIL methods and configurations.

Dataset	Method	Random classes			Finer-EC buckets			Mixed hierarchy
Dataset	Method	Acc $\uparrow$	Fgt $\downarrow$	Plast $\uparrow$	Acc $\uparrow$	Fgt $\downarrow$	Plast $\uparrow$	Acc $\uparrow$	Fgt $\downarrow$	Plast $\uparrow$
ESM2-150M
Enzyme Reaction	EWC	14.8 $\pm$ 0.8	78.9 $\pm$ 1.8	82.4 $\pm$ 2.3	15.7 $\pm$ 2.5	73.2 $\pm$ 3.0	78.4 $\pm$ 0.1	15.8 $\pm$ 1.8	78.0 $\pm$ 2.0	82.6 $\pm$ 0.5
	FeTrIL	22.4 $\pm$ 5.0	70.7 $\pm$ 6.8	83.0 $\pm$ 1.7	24.4 $\pm$ 5.2	62.9 $\pm$ 6.0	78.3 $\pm$ 0.4	23.3 $\pm$ 4.9	69.9 $\pm$ 5.0	83.2 $\pm$ 0.6
	EASE	41.8 $\pm$ 1.9	14.2 $\pm$ 1.6	53.8 $\pm$ 1.7	45.1 $\pm$ 1.2	11.4 $\pm$ 1.8	54.7 $\pm$ 0.7	41.0 $\pm$ 4.1	14.6 $\pm$ 3.9	53.5 $\pm$ 1.5
	RER	47.0 $\pm$ 3.4	36.8 $\pm$ 2.6	78.5 $\pm$ 2.2	52.8 $\pm$ 1.6	28.0 $\pm$ 2.1	76.7 $\pm$ 0.4	51.6 $\pm$ 2.6	33.1 $\pm$ 2.0	80.0 $\pm$ 1.0

Appendix

Table 9: Per-encoder breakdown of the protocol sanity check (part 1/2: ESM2-150M, ESM2-650M), in the same format as Table 2 but reported separately for each encoder instead of averaged over them. Each encoder block contains both datasets. Values are mean

\pm

std over three seeds (%). Acc = Avg Acc (

\uparrow

), Fgt = Forgetting (

\downarrow

), Plast = Plasticity (

\uparrow

Dataset	Method	Random classes			Finer-EC buckets			Mixed hierarchy
Dataset	Method	Acc $\uparrow$	Fgt $\downarrow$	Plast $\uparrow$	Acc $\uparrow$	Fgt $\downarrow$	Plast $\uparrow$	Acc $\uparrow$	Fgt $\downarrow$	Plast $\uparrow$
GearNet-Edge
Enzyme Reaction	EWC	10.5 $\pm$ 0.7	68.3 $\pm$ 2.0	69.1 $\pm$ 2.4	9.9 $\pm$ 1.2	65.0 $\pm$ 1.8	65.5 $\pm$ 1.3	10.0 $\pm$ 0.3	70.7 $\pm$ 2.0	70.6 $\pm$ 1.9
	FeTrIL	17.5 $\pm$ 3.9	57.3 $\pm$ 4.1	66.6 $\pm$ 2.8	18.5 $\pm$ 2.9	54.3 $\pm$ 4.7	64.6 $\pm$ 1.5	17.7 $\pm$ 4.8	58.6 $\pm$ 4.6	67.9 $\pm$ 0.9
	EASE	24.0 $\pm$ 3.2	15.0 $\pm$ 4.1	36.9 $\pm$ 1.1	27.1 $\pm$ 0.9	12.8 $\pm$ 2.8	37.8 $\pm$ 1.9	25.0 $\pm$ 2.6	13.3 $\pm$ 3.3	36.4 $\pm$ 0.4
	RER	36.6 $\pm$ 2.8	33.6 $\pm$ 2.2	64.7 $\pm$ 2.8	36.3 $\pm$ 1.4	31.2 $\pm$ 2.1	62.3 $\pm$ 2.1	37.5 $\pm$ 2.1	32.2 $\pm$ 2.7	65.1 $\pm$ 0.3

Appendix

Table 10: Per-encoder breakdown of the protocol sanity check (part 2/2: GearNet-Edge, ProtBert, SaProt). Same format as Table 9 . Values are mean

\pm

std over three seeds (%). Acc = Avg Acc (

\uparrow

), Fgt = Forgetting (

\downarrow

), Plast = Plasticity (

\uparrow

Enzyme Reaction				EC Number
True	Pred.	# mis.	Example	True	Pred.	# mis.	Example
ESM2-150M				ESM2-150M
2 (transf.)	3 (hydro.)	96	2.1.1.- $\to$ 3.1.21.4	3 (hydro.)	2 (transf.)	461	3.1.4.35 $\to$ 2.4.1.229
3 (hydro.)	2 (transf.)	48	3.4.19.12 $\to$ 2.7.10.-	2 (transf.)	3 (hydro.)	270	2.7.2.3 $\to$ 3.5.1.76
2 (transf.)	4 (lyases)	34	2.7.7.- $\to$ 4.2.1.-	2 (transf.)	1 (oxidored.)	170	2.7.1.148 $\to$ 1.1.1.193
3 (hydro.)	5 (isomer.)	16	3.6.4.- $\to$ 5.6.1.-	3 (hydro.)	1 (oxidored.)	159	3.1.21.4 $\to$ 1.13.11.58

Appendix

Table 11: Cross-family confusion breakdowns under a single shared covariance for all five protein encoders on both datasets. For each encoder and dataset, we report the six most frequent cross-family misclassifications together with one representative EC-level confusion.

Figure 5: Per-encoder continual-learning results on EC Number (4,788 classes, 7 EC top-level groups). Left: Average incremental accuracy across phases. Middle: Forgetting across phases. Right: Final accuracy broken down by EC class.

Figure 6: Phase-wise accuracy matrices across encoder–dataset configurations. Within each configuration, matrices from left to right correspond to FeTrIL, RanPAC, and FeCAM. Rows indicate the training phase and columns the evaluated phase.

Protein Function Prediction

See all 9 papers

CoPeP: Benchmarking Continual Pretraining for Protein Language Models
Feb 27, 2026Darshan Patil, Pranshu Malviya, Mathieu Reymond +2Protein Language Models Replay-Based Continual Learning
Mila - Quebec AI Institute · Polytechnique Montréal · Chandar Research Lab +1
Protein Fold Classification at Scale: Benchmarking and Pretraining
May 18, 2026Dexiong Chen, Andrei Manolache, Mathias Niepert +1Masked Autoencoders Self-Supervised Pre-Training
Max Planck Institute of Biochemistry, Martinsried, Germany · Computer Science Department, University of Stuttgart, Germany · Bitdefender, Romania
ProtAlign: Contrastive learning paradigm for Sequence and structure alignment
Mar 6, 2026Aditya Ranganath, Hasin Us Sami, Kowshik Thopalli +2Cross-Modal Alignment Protein Representation Learning
Center for Applied Scientific Computing, Lawrence Livermore National Laboratory

Feb 27, 2026J/K move · Enter open · S save

CoPeP: Benchmarking Continual Pretraining for Protein Language Models

Darshan Patil, Pranshu Malviya, Mathieu Reymond, Quentin Fournier+1

Mila - Quebec AI Institute · Polytechnique Montréal · Chandar Research Lab+1

Protein Language Models Replay-Based Continual Learning

Open paper AskPDF arXiv

Protein language models (pLMs) have recently gained significant attention for their ability to uncover relationships between sequence, structure, and function from evolutionary statistics, thereby accelerating therapeutic drug discovery. These models learn from large protein databases that are continuously updated by the biology community and whose dynamic nature motivates the application of continual learning, not only to keep up with the ever-growing data, but also as an opportunity to take advantage of the temporal meta-information that is created during this process. As a result, we introduce the Continual Pretraining of Protein Language Models (CoPeP) benchmark, a novel benchmark for evaluating continual learning approaches on pLMs. Specifically, we curate a sequence of protein datasets derived from the UniProt Knowledgebase spanning a decade and define metrics to assess pLM performance across 31 protein understanding tasks. We evaluate several methods from the continual learning literature, including replay, unlearning, and plasticity-based methods, some of which have never been applied to models and data of this scale. Our findings reveal that incorporating temporal meta-information improves perplexity by up to 7% even when compared to training on data from all tasks jointly. Moreover, even at scale, several continual learning methods outperform naive continual pretraining. The CoPeP benchmark offers an exciting opportunity to study these methods at scale in an impactful real-world application.

Abstract

Protein Function Prediction

Explore similar work

CoPeP: Benchmarking Continual Pretraining for Protein Language Models

Protein Fold Classification at Scale: Benchmarking and Pretraining

ProtAlign: Contrastive learning paradigm for Sequence and structure alignment