ProtModernBERT: Compute-Efficient Protein Language Modeling

Alberto Martínez; Natalia Flechas Manrique; Llorenç Espinosa-Portalés; Luc Andrea

cs.LGOct 14, 2026

ProtModernBERT: Compute-Efficient Protein Language Modeling

Authors: Alberto Martínez, Natalia Flechas Manrique, Llorenç Espinosa-Portalés, Luc Andrea

Organizations: Multiverse Computing, Parque Científico y Tecnológico de Gipuzkoa, Paseo de Miramón 170, 20014 Donostia-San Sebastián, Spain. · Multiverse Computing, 7 rue de la Croix Martre, 91120 Palaiseau, Paris, France

Abstract

Large protein language models provide a powerful framework for learning sequence-function relationships and guiding experimental design. However, architectural improvements are underexplored, especially in low-compute regimes of relevance for resource-constrained research laboratories. We show that the training cost for encoder-only transformer architectures can be reduced by leveraging modern architectural refinements, offering an encouraging performance / compute-efficiency trade-off. We introduce ProtModernBERT, a modernized BERT-style model pre-trained on amino acid sequences, and compare it with ProtBert, Amplify, and ESM2 across a set of supervised structural tasks -encompassing secondary structure prediction, disorder prediction, and subcellular localization- and ProteinGym's zero-shot functional benchmark. We find that ProtModernBERT matches or closely tracks the performance of Amplify-120M and ProtBert with up to 80-90% reduction in pre-training FLOPs, and narrows the gap to ESM2 when controlling for pre-training. Our analysis using a cluster-based validation protocol for hyperparameter selection and randomly initialized models disentangles the contributions of architecture and data, showing that modern components provide stronger inductive biases that are more consequential than parameter count in low-compute regimes.

Figures & tables

Hyperparameter	ModernBERT (Base, 140M)	Amplify (Base, 120M)	ProtModernBERT
Optimizer	StableAdamW ( $\beta{=}(0.90,0.98)$ , $\epsilon{=}1\text{e-}6$ )	AdamW ( $\beta{=}(0.90,0.95)$ , $\epsilon{=}1\text{e-}8$ )	AdamW ( $\beta{=}(0.90,0.99)$ , $\epsilon{=}1\text{e-}8$ )
Learning rate	$8\text{e-}4$ (pretrain), $3\text{e-}4$ (context)	0.001	$8\text{e-}4$
Weight decay	$1\text{e-}5$	0.01	0.01
LR schedule	Trapezoidal	Cosine decay (Stage 1), none (Stage 2)	Trapezoidal
Global Batch size	4,608 (pretrain), 72 (context)	4,096	12288
Precision	fp16/bf16 (via FlashAttention)	bf16 (mixed precision)	bf16 (via FlashAttention)

TABLE I: Key architectural and optimization hyperparameters for ModernBERT (Base), Amplify (Base), and ProtModernBERT . These models are shown because they directly inform the architectural design and hyperparameter selection of ProtModernBERT .

Dataset	Task	Train	Test
DeepLoc	Localization (10-class)	6,622	1,842
	Membrane (binary)	6,622	1,842
Train_HHblits	Sec. structure	10,792	–
CASP12	Sec. structure	–	20
TS115	Sec. structure	–	115
CB513	Sec. structure	–	511

TABLE II: Overview of benchmark datasets.

Fig. 1: Accuracy benchmark results under the pre-trained and random initialization conditions for ProtBert , Amplify-120M , Amplify-350M , and ProtModernBERT . The dummy majority-class baseline is shown on the far left for scale.

Fig. 2: F1 benchmark results under the pre-trained and random initialization conditions for ProtBert , Amplify-120M , Amplify-350M , and ProtModernBERT . As in the previous figure, the dummy majority-class baseline is shown on the far left for scale.

Model	Parameters	FLOPs
ProtBert	420M	$6.0\times 10^{21}$
ESM2-150M	150M	$9\times 10^{20}$ * * * Training FLOPs for ESM2-150M are not reported in original ESM2 publications. ESM2-150M does not seem to be the most computationally expensive model in our comparison: based on our estimates, its total pre-training budget lies in the range $9\times 10^{20}$ – $1\times 10^{21}$ FLOPs, we report the most conservative estimate and further discuss this in the Appendix ( A-D ).
Amplify-120M	120M	$1.6\times 10^{21}$
Amplify-350M	350M	$4.6\times 10^{21}$
ProtModernBERT	140M	$5.8\times 10^{20}$
Early ProtModernBERT	140M	$1.3\times 10^{20}$

TABLE III: Comparison of model size and pre-training compute.

Model	Total (Overall)	Org. Fitness	Binding	Expression	Activity	Stability
ProtBert	0.376	0.357	0.211	0.338	0.363	0.448
ProtModernBERT	0.373	0.379	0.282	0.385	0.342	0.400
Early ProtModernBERT	0.327	0.331	0.288	0.354	0.316	0.331
ESM2-150M	0.387	0.305	0.326	0.402	0.391	0.510

TABLE IV: ProteinGym zero-shot substitution results (Spearman correlation). ProtModernBERT demonstrates competitive functional modeling across categories, notably achieving the highest score in Organismal Fitness despite significantly lower pre-training compute.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 3: Training loss during the four preliminary pre-training runs varying weight decay (wd) and local batch size (bs). Lower weight decay exhibits occasional loss spikes, while weight decay = 0.01 is more stable.

Model	LR	Epochs	Weight Decay
Amplify 120M	3e-5	4	0.02
Amplify 120M RandInit	3e-5	4	0.02
Amplify 350M	3e-5	4	0.02
Amplify 350M RandInit	3e-5	4	0.02
ESM150M	5e-5	4	0.10
ESM150M RandInit	5e-5	4	0.10

Appendix

TABLE V: Best hyperparameters used for each model when the model was fine-tuned under the cluster-based validation protocol.

Fig. 4: ProtModernBERT fine-tuning accuracy results.

Fig. 5: ProtModernBERT fine-tuning macro-F1 results.

Label	Learning Rate	Epochs	Weight Decay
Fine-tune 0	3e-5	4	0.02
Fine-tune 1	3e-5	4	0.10
Fine-tune 2	3e-5	6	0.02
Fine-tune 3	3e-5	6	0.10
Fine-tune 4	5e-5	4	0.02
Fine-tune 5	5e-5	4	0.10

Appendix

TABLE VI: Mapping of labels and hyperparameters.

Protein Language Models

Protein language models learn patterns in amino-acid sequences to predict protein properties, interpret mutations, and design new proteins.

See all 69 papers

Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference
Feb 24, 2026Anna Hart, Chi Han, Jeonghwan Kim +2Protein Language Models Protein Representation Learning
Siebel School of Data Science and Computing, University of Illinois Urbana-Champaign, 201 N Goodwin Ave, Urbana, IL 61801, USA · DOE Center for Advanced Bioenergy and Bioproducts Innovation, University of Illinois Urbana-Champaign, 1206 W Gregory Drive, Urbana, IL 61801, USA · Chemical and Biomolecular Engineering, University of Illinois Urbana-Champaign, 600 S Matthews Ave, Urbana, IL 61801, USA
ProtSent: Protein Sentence Transformers
May 7, 2026Dan Ofer, Oriel Perets, Michal Linial +1Contrastive Learning Protein Language Models
Department of Biological Chemistry The Hebrew University of Jerusalem · Department of Computer and Information Science Ben-Gurion University of the Negev
ProtLingo: Efficient Protein Language Modeling via Conditional Memory and Expert Routing
Sep 4, 2026Mingrui Li, Sixian Shen, Minzhang Li +4Mixture-of-Experts Language Models Protein Fitness Prediction
ShanghaiTech University

Feb 24, 2026J/K move · Enter open · S save

Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference

Anna Hart, Chi Han, Jeonghwan Kim, Huimin Zhao+1

Siebel School of Data Science and Computing, University of Illinois Urbana-Champaign, 201 N Goodwin Ave, Urbana, IL 61801, USA · DOE Center for Advanced Bioenergy and Bioproducts Innovation, University of Illinois Urbana-Champaign, 1206 W Gregory Drive, Urbana, IL 61801, USA · Chemical and Biomolecular Engineering, University of Illinois Urbana-Champaign, 600 S Matthews Ave, Urbana, IL 61801, USA

Protein Language Models Protein Representation Learning Bioinformatics Molecular Property Prediction

Open paper AskPDF arXiv

Modern Protein Language Models (PLMs) apply transformer-based model architectures from natural language processing to biological sequences, predicting a variety of protein functions and properties. However, protein language has key differences from natural language, such as a rich functional space despite a vocabulary of only 20 amino acids. These differences motivate research into how transformer-based architectures operate differently in the protein domain and how we can better leverage PLMs to solve protein-related tasks. In this work, we begin by directly comparing how the distribution of information stored across layers of attention heads differs between the protein and natural language domain. Furthermore, we adapt a simple early-exit technique-originally used in the natural language domain to improve efficiency at the cost of performance-to achieve both increased accuracy and substantial efficiency gains in protein non-structural property prediction by allowing the model to automatically select protein representations from the intermediate layers of the PLMs for the specific task and protein at hand. We achieve performance gains ranging from 0.4 to 7.01 percentage points while simultaneously improving efficiency by over 10 percent across models and non-structural prediction tasks. Our work opens up an area of research directly comparing how language models change behavior when moved into the protein domain and advances language modeling in biological domains.

Abstract

Protein Language Models

Explore similar work

Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference

ProtSent: Protein Sentence Transformers

ProtLingo: Efficient Protein Language Modeling via Conditional Memory and Expert Routing