Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference
Authors: Anna Hart, Chi Han, Jeonghwan Kim, Huimin Zhao, Heng Ji
Organizations: Siebel School of Data Science and Computing, University of Illinois Urbana-Champaign, 201 N Goodwin Ave, Urbana, IL 61801, USA · DOE Center for Advanced Bioenergy and Bioproducts Innovation, University of Illinois Urbana-Champaign, 1206 W Gregory Drive, Urbana, IL 61801, USA · Chemical and Biomolecular Engineering, University of Illinois Urbana-Champaign, 600 S Matthews Ave, Urbana, IL 61801, USA
Modern Protein Language Models (PLMs) apply transformer-based model architectures from natural language processing to biological sequences, predicting a variety of protein functions and properties. However, protein language has key differences from natural language, such as a rich functional space despite a vocabulary of only 20 amino acids. These differences motivate research into how transformer-based architectures operate differently in the protein domain and how we can better leverage PLMs to solve protein-related tasks. In this work, we begin by directly comparing how the distribution of information stored across layers of attention heads differs between the protein and natural language domain. Furthermore, we adapt a simple early-exit technique-originally used in the natural language domain to improve efficiency at the cost of performance-to achieve both increased accuracy and substantial efficiency gains in protein non-structural property prediction by allowing the model to automatically select protein representations from the intermediate layers of the PLMs for the specific task and protein at hand. We achieve performance gains ranging from 0.4 to 7.01 percentage points while simultaneously improving efficiency by over 10 percent across models and non-structural prediction tasks. Our work opens up an area of research directly comparing how language models change behavior when moved into the protein domain and advances language modeling in biological domains.
Figures & tables
Figure 1: The scheme for early-exit, based on Schwartz et al. ( Schwartz et al., 2020 ) . The input protein sequence is fed into the PLM. At each layer, an MLP makes a prediction for the downstream task and the confidence of this prediction is calculated. When the confidence reaches a predetermined threshold, the model will output the result from the current layer and cease further execution.
Figure 2: Many PLMs display more variability in their attention focus than the corresponding NLM. The heat map displays how attention heads distribute their focus between positional and semantic information across 1,000 inputs, plotting each head for each input by ratio of positional to semantic information focus. These plots are generated for NLMs BERT ( Devlin et al., 2019 ) , AlBERT ( Lan et al., 2020 ) , T5 encoder ( Raffel et al., 2020 ) , and XLNet ( Yang et al., 2019 ) and their corresponding PLMs ( Elnaggar et al., 2022 ) . The y axis represents the ratio of positional:semantic information captured by the attention heads, and the color represents the number of attention heads in that layer per ratio bin. All attention heads, for each of 1,000 inputs, are accounted for in each layer. As shown in the figure, more variability in the attention focus is displayed in the protein versions of BERT, ALBERT, and T5, with XLNet as an exception.
Model
Input-Dependent Variance
Layer-Dependent Variance
Head-Dependent Variance
NLM
PLM
NLM
PLM
NLM
PLM
NLM
PLM
BERT
ProtBERT
0.493 (± 0.040)
1.262 (± 0.095)
2.973 (± 0.034)
7.317(± 0.167)
2.412 (± 0.041)
4.620(± 0.099)
ALBERT
ProtALBERT
0.288 (± 0.021)
0.752 (± 0.075)
2.040 (± 0.010)
2.986(± 0.038)
2.056 (± 0.019)
3.851 (± 0.044)
T5
ProtT5-UniProt
0.440 (± 0.021)
0.878 (± 0.042)
1.456(± 0.010)
2.567 (± 0.033)
2.658 (± 0.015)
3.438 (± 0.023)
XLNet
ProtXLNet
0.828 (± 0.068)
0.451 (± 0.033)
3.459 (± 0.079)
2.390 (± 0.025)
2.464 (± 0.062)
1.732 (± 0.017)
Table 1: Many PLMs show greater input, layer, and attention-head dependent variability in attention focus than the corresponding NLM. This table reports the mean and standard deviation of the estimated variance in attention ratio (semantic vs. positional) from 10 disjoint samples of 100 inputs each. These statistics are computed across NLMs BERT ( Devlin et al., 2019 ) , AlBERT ( Lan et al., 2020 ) , T5 encoder ( Raffel et al., 2020 ) , and XLNet ( Yang et al., 2019 ) and their corresponding PLMs ( Elnaggar et al., 2022 ) . PLMs ProtBERT, ProtAlBERT, and ProtT5 have higher input, layer, and attention-head dependent variance than the correponding NLMs—BERT, ALBERT, and T5, with XLNet as an exception
NLP Model
Correlation
Protein Model
Correlation
Model
Coefficient
Model
Coefficient
BERT
0.770 (± 0.029)
ProtBERT
0.733 (± 0.050)
ALBERT
0.903 (± 0.011)
ProtALBERT
0.769 (± 0.034)
T5
0.601 (±0.032)
ProtT5-UniProt
0.708 (± 0.037)
XLNet
0.638 (±0.053)
ProtXLNet
0.7047(±0.025)
Table 2: Correlation of original attention matrix with of attention matrix reconstructed from 3-component decomposition for NLMs BERT ( Devlin et al., 2019 ) , AlBERT ( Lan et al., 2020 ) , T5 encoder ( Raffel et al., 2020 ) , and XLNet ( Yang et al., 2019 ) and their corresponding PLMs ( Elnaggar et al., 2022 ) . The results indicate that the decomposed components explain a major portion of the information encoded in the self-attention.
Figure 3: Early-Exit Improves both Performance and Efficiency in Non-Structural Tasks across Multiple PLMs. The total number of computed layers is used as a proxy for efficiency. The trade-offs between model performance and efficiency are calculated for: (1) Individual Layer Performance, (2) Early-Exit Last Layer Fallback , and (3) Early-exit Most Confident Layer Fallback . The baseline performance of the last layer is drawn across with a black line. Computations are done for ESM2 ( Lin et al., 2023 ) , ProtBERT, and ProtALBERT ( Elnaggar et al., 2022 ) . Early-exit Most Confident Layer Fallback outperforms both the last-layer performance baseline and early-exit Last Layer Fallback regarding both performance and efficiency in non-structural tasks. For the secondary structure prediction, early-exit allows efficiency gains but harms performance.
Figure 4: Walltimes versus total number of computed layers The walltime for the testing set on 1 V100 GPU versus the average number of computed layers is plotted across all models and tasks. Early-exit Most Confident Layer Fallback is plotted. A diamond marker at the final layer denotes the baseline walltime. We see that walltime corresponds linearly with the number of computed layers
Figure 5: Confidence calibration . A lower excess AURC score ( Geifman et al., 2019 ) denotes a better calibrated confidence metric. We see that, for all models, confidence is well calibrated across layers for EC, is well calibrated in middle and later layers for CL, and is poorly calibrated for GO tasks.
Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs). By consensus, embeddings from the models' last layers are used, while the models' internal behavior remains poorly understood. We analyzed 13 PLMs across 15 DTs and 9 datasets to assess the value of embeddings from intermediate PLM layers. We trained probe models on embeddings from each layer, compared their performance, and showed that the last layers of PLMs rarely produced embeddings that led to the best results on downstream tasks. Furthermore, we identified a connection between how models learn a certain DT and the similarity between that DT and the pre-training objective. For example, for residue-level downstream tasks, we observed a steady increase in performance across almost all PLM layers, which we attributed to their similarity to most PLMs' pre-training objectives. To allow the community to capitalize on our findings, we provide PLMSommelier, a Python package that automatically identifies the best PLM layer for a given DT with ~98% accuracy and creates a truncated model using only the early layers up to the best-performing layer. This will help users save time and memory during inference and yield better predictive performance.
Roman Joeres, Ilya Senatorov, Anastasia Kolchina +2
Drug Bioinformatics, Helmholtz Institute for Pharmaceutical Research Saarland, 66123 Saarbruecken, Germany · Center for Bioinformatics, Saarland University, 66123 Saarbruecken, Germany · Saarland Informatics Campus, Saarland University, 66123 Saarbruecken, Germany +2
Protein language models (pLMs) produce per-residue representations that capture evolutionary and structural information, yet their mean-pooled sequence embeddings are not explicitly trained to reflect functional, evolutionary or structural similarity between proteins. We present Protein Sentence Transformers (ProtSent), a contrastive fine-tuning framework for adapting PLMs into general-purpose embedding models. ProtSent trains with MultipleNegativesRankingLoss across five protein-pair datasets: Pfam families, structurally derived hard negatives, AlphaFold DB structural pairs, and StringDB protein--protein interactions, and Deep Mutational Scanning data. We evaluate on 23~downstream tasks using frozen embeddings with a k-nearest-neighbor probe to measure embedding neighborhood quality. On ESM-2 150M, ProtSent improves 15 of 23 tasks, with gains of +105% on remote homology detection, +17% on variant effect prediction, and +19.9% Recall@1 on SCOPe-40 structural retrieval. The 35M variant improves 16 of 23 tasks with +40.5% on remote homology and +15.5% Recall@1 on SCOPe-40. Contrastive fine-tuning restructures the embedding space to better capture protein function and structure, without any task-specific supervision. We release the models, public data, and training recipe and code.
Dan Ofer, Oriel Perets, Michal Linial +1
Department of Computer and Information Science Ben-Gurion University of the Negev
Protein language models (PLMs) have become widely adopted as general-purpose models, demonstrating strong performance in protein engineering and de novo design. Like large language models (LLMs), they are typically trained as deep transformers with next-token or masked-token prediction objectives on massive sequence corpora and are scaled by increasing model depth. Recent work on autoregressive LLMs has identified the Curse of Depth: many later layers contribute little to the final output predictions. These findings naturally raise the question of whether a similar depth inefficiency also appears in PLMs, where many widely used models are not autoregressive, and some are multimodal, accepting both protein sequence and structure as input. In this work, we present a depth analysis of seven popular PLM families across model scales, spanning autoregressive, masked, and diffusion objectives, and quantify how layer contributions evolve with depth using a unified set of probing-, perturbation-, and downstream-evaluation measurements. Across models, we observe consistent depth-dependent patterns that extend prior findings on LLMs: a large fraction of task-relevant computation is concentrated in a subset of layers, while the remaining layers mainly provide incremental refinement of the final prediction. These trends persist beyond sequence-only settings and also appear in multimodal PLMs. Taken together, our results suggest that depth inefficiency is a common feature of modern PLMs, motivating future work on more depth-efficient architectures and training methods.
Aleena Siji, Amir Mohammad Karimi Mamaghan, Ferdinand Kapl +9
Technical University of Munich · Helmholtz AI, Munich · KTH Royal Institute of Technology +2