Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference

Anna Hart; Chi Han; Jeonghwan Kim; Huimin Zhao; Heng Ji

cs.LGFeb 24, 2026

Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference

Authors: Anna Hart, Chi Han, Jeonghwan Kim, Huimin Zhao, Heng Ji

Organizations: Siebel School of Data Science and Computing, University of Illinois Urbana-Champaign, 201 N Goodwin Ave, Urbana, IL 61801, USA · DOE Center for Advanced Bioenergy and Bioproducts Innovation, University of Illinois Urbana-Champaign, 1206 W Gregory Drive, Urbana, IL 61801, USA · Chemical and Biomolecular Engineering, University of Illinois Urbana-Champaign, 600 S Matthews Ave, Urbana, IL 61801, USA

Abstract

Modern Protein Language Models (PLMs) apply transformer-based model architectures from natural language processing to biological sequences, predicting a variety of protein functions and properties. However, protein language has key differences from natural language, such as a rich functional space despite a vocabulary of only 20 amino acids. These differences motivate research into how transformer-based architectures operate differently in the protein domain and how we can better leverage PLMs to solve protein-related tasks. In this work, we begin by directly comparing how the distribution of information stored across layers of attention heads differs between the protein and natural language domain. Furthermore, we adapt a simple early-exit technique-originally used in the natural language domain to improve efficiency at the cost of performance-to achieve both increased accuracy and substantial efficiency gains in protein non-structural property prediction by allowing the model to automatically select protein representations from the intermediate layers of the PLMs for the specific task and protein at hand. We achieve performance gains ranging from 0.4 to 7.01 percentage points while simultaneously improving efficiency by over 10 percent across models and non-structural prediction tasks. Our work opens up an area of research directly comparing how language models change behavior when moved into the protein domain and advances language modeling in biological domains.

Figures & tables

Figure 1: The scheme for early-exit, based on Schwartz et al. ( Schwartz et al., 2020 ) . The input protein sequence is fed into the PLM. At each layer, an MLP makes a prediction for the downstream task and the confidence of this prediction is calculated. When the confidence reaches a predetermined threshold, the model will output the result from the current layer and cease further execution.

Figure 2: Many PLMs display more variability in their attention focus than the corresponding NLM. The heat map displays how attention heads distribute their focus between positional and semantic information across 1,000 inputs, plotting each head for each input by ratio of positional to semantic information focus. These plots are generated for NLMs BERT ( Devlin et al., 2019 ) , AlBERT ( Lan et al., 2020 ) , T5 encoder ( Raffel et al., 2020 ) , and XLNet ( Yang et al., 2019 ) and their corresponding PLMs ( Elnaggar et al., 2022 ) . The y axis represents the ratio of positional:semantic information captured by the attention heads, and the color represents the number of attention heads in that layer per ratio bin. All attention heads, for each of 1,000 inputs, are accounted for in each layer. As shown in the figure, more variability in the attention focus is displayed in the protein versions of BERT, ALBERT, and T5, with XLNet as an exception.

Model		Input-Dependent Variance		Layer-Dependent Variance		Head-Dependent Variance
NLM	PLM	NLM	PLM	NLM	PLM	NLM	PLM
BERT	ProtBERT	0.493 (± 0.040)	1.262 (± 0.095)	2.973 (± 0.034)	7.317(± 0.167)	2.412 (± 0.041)	4.620(± 0.099)
ALBERT	ProtALBERT	0.288 (± 0.021)	0.752 (± 0.075)	2.040 (± 0.010)	2.986(± 0.038)	2.056 (± 0.019)	3.851 (± 0.044)
T5	ProtT5-UniProt	0.440 (± 0.021)	0.878 (± 0.042)	1.456(± 0.010)	2.567 (± 0.033)	2.658 (± 0.015)	3.438 (± 0.023)
XLNet	ProtXLNet	0.828 (± 0.068)	0.451 (± 0.033)	3.459 (± 0.079)	2.390 (± 0.025)	2.464 (± 0.062)	1.732 (± 0.017)

Table 1: Many PLMs show greater input, layer, and attention-head dependent variability in attention focus than the corresponding NLM. This table reports the mean and standard deviation of the estimated variance in attention ratio (semantic vs. positional) from 10 disjoint samples of 100 inputs each. These statistics are computed across NLMs BERT ( Devlin et al., 2019 ) , AlBERT ( Lan et al., 2020 ) , T5 encoder ( Raffel et al., 2020 ) , and XLNet ( Yang et al., 2019 ) and their corresponding PLMs ( Elnaggar et al., 2022 ) . PLMs ProtBERT, ProtAlBERT, and ProtT5 have higher input, layer, and attention-head dependent variance than the correponding NLMs—BERT, ALBERT, and T5, with XLNet as an exception

NLP Model	Correlation	Protein Model	Correlation
Model	Coefficient	Model	Coefficient
BERT	0.770 (± 0.029)	ProtBERT	0.733 (± 0.050)
ALBERT	0.903 (± 0.011)	ProtALBERT	0.769 (± 0.034)
T5	0.601 (±0.032)	ProtT5-UniProt	0.708 (± 0.037)
XLNet	0.638 (±0.053)	ProtXLNet	0.7047(±0.025)

Table 2: Correlation of original attention matrix with of attention matrix reconstructed from 3-component decomposition for NLMs BERT ( Devlin et al., 2019 ) , AlBERT ( Lan et al., 2020 ) , T5 encoder ( Raffel et al., 2020 ) , and XLNet ( Yang et al., 2019 ) and their corresponding PLMs ( Elnaggar et al., 2022 ) . The results indicate that the decomposed components explain a major portion of the information encoded in the self-attention.

Figure 3: Early-Exit Improves both Performance and Efficiency in Non-Structural Tasks across Multiple PLMs. The total number of computed layers is used as a proxy for efficiency. The trade-offs between model performance and efficiency are calculated for: (1) Individual Layer Performance, (2) Early-Exit Last Layer Fallback , and (3) Early-exit Most Confident Layer Fallback . The baseline performance of the last layer is drawn across with a black line. Computations are done for ESM2 ( Lin et al., 2023 ) , ProtBERT, and ProtALBERT ( Elnaggar et al., 2022 ) . Early-exit Most Confident Layer Fallback outperforms both the last-layer performance baseline and early-exit Last Layer Fallback regarding both performance and efficiency in non-structural tasks. For the secondary structure prediction, early-exit allows efficiency gains but harms performance.

Figure 4: Walltimes versus total number of computed layers The walltime for the testing set on 1 V100 GPU versus the average number of computed layers is plotted across all models and tasks. Early-exit Most Confident Layer Fallback is plotted. A diamond marker at the final layer denotes the baseline walltime. We see that walltime corresponds linearly with the number of computed layers

Figure 5: Confidence calibration . A lower excess AURC score ( Geifman et al., 2019 ) denotes a better calibrated confidence metric. We see that, for all models, confidence is well calibrated across layers for EC, is well calibrated in middle and later layers for CL, and is poorly calibrated for GO tasks.

Explore similar work

Task- and dataset-specific information in protein language models
Aug 12, 2026Roman Joeres, Ilya Senatorov, Anastasia Kolchina +2Protein Language Models Protein Representation Learning
Drug Bioinformatics, Helmholtz Institute for Pharmaceutical Research Saarland, 66123 Saarbruecken, Germany · Center for Bioinformatics, Saarland University, 66123 Saarbruecken, Germany · Saarland Informatics Campus, Saarland University, 66123 Saarbruecken, Germany +2
ProtSent: Protein Sentence Transformers
May 7, 2026Dan Ofer, Oriel Perets, Michal Linial +1Contrastive Learning Protein Language Models
Department of Computer and Information Science Ben-Gurion University of the Negev
From Words to Amino Acids: Does the Curse of Depth Persist?
Feb 25, 2026Aleena Siji, Amir Mohammad Karimi Mamaghan, Ferdinand Kapl +9Protein Language Models Protein Representation Learning
Technical University of Munich · Helmholtz AI, Munich · KTH Royal Institute of Technology +2

Aug 12, 2026J/K move · Enter open · S save

Task- and dataset-specific information in protein language models

Roman Joeres, Ilya Senatorov, Anastasia Kolchina, Dietrich Klakow+1

Drug Bioinformatics, Helmholtz Institute for Pharmaceutical Research Saarland, 66123 Saarbruecken, Germany · Center for Bioinformatics, Saarland University, 66123 Saarbruecken, Germany · Saarland Informatics Campus, Saarland University, 66123 Saarbruecken, Germany+2

Protein Language Models Protein Representation Learning Language Model Probing

Open paper AskPDF arXiv

Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs). By consensus, embeddings from the models' last layers are used, while the models' internal behavior remains poorly understood. We analyzed 13 PLMs across 15 DTs and 9 datasets to assess the value of embeddings from intermediate PLM layers. We trained probe models on embeddings from each layer, compared their performance, and showed that the last layers of PLMs rarely produced embeddings that led to the best results on downstream tasks. Furthermore, we identified a connection between how models learn a certain DT and the similarity between that DT and the pre-training objective. For example, for residue-level downstream tasks, we observed a steady increase in performance across almost all PLM layers, which we attributed to their similarity to most PLMs' pre-training objectives. To allow the community to capitalize on our findings, we provide PLMSommelier, a Python package that automatically identifies the best PLM layer for a given DT with ~98% accuracy and creates a truncated model using only the early layers up to the best-performing layer. This will help users save time and memory during inference and yield better predictive performance.

Abstract

Explore similar work

Task- and dataset-specific information in protein language models

ProtSent: Protein Sentence Transformers

From Words to Amino Acids: Does the Curse of Depth Persist?