Interpreting Protein Language Model Embeddings via Orthogonal Projection for Protein Fitness Prediction

Paulo Yanez Sarmiento; Pia Francesca Rissom; Manuel Pfeuffer; Marco Simnacher; Jordan F. Safer; Sumaiya Iqbal; Henrike O. Heyne; Nadja Klein

cs.LGAug 26, 2026

Interpreting Protein Language Model Embeddings via Orthogonal Projection for Protein Fitness Prediction

Authors: Paulo Yanez Sarmiento, Pia Francesca Rissom, Manuel Pfeuffer, Marco Simnacher, Jordan F. Safer, Sumaiya Iqbal, Henrike O. Heyne, Nadja Klein,

Organizations: Hasso Plattner Institute, University of Potsdam · Bioinformatics and Machine Learning, Broad Institute of MIT and Harvard · School of Business and Economics, Humboldt University of Berlin · Scientific Computing Center, Karlsruhe Institute of Technology

Abstract

Recently, there has been a growing adoption of protein language models (PLMs) in biomedical science. Their embeddings provide a rich numerical representation of protein sequences which achieve state-of-the-art performance on several downstream tasks including protein fitness prediction. However, PLM embeddings are not directly interpretable and, thereby, it remains unclear what features they encode. To gain insight into which biochemical properties of the protein are driving the prediction, we leverage an orthogonal projection technique that removes linear effects of known tabular features from embeddings and extend it to high-order and interaction effects. In this way, we remove the effects of interpretable biochemical features from PLM embeddings. In an ablation study, we show that this leads to a decrease in performance for a downstream classifier trained only on the embeddings to predict protein fitness. In an additional evaluation, we find that these biochemical features explain a substantial part of the variance in the predictions of this classifier. Hence, we can show that PLM embeddings encode patterns correlated with biochemical properties and quantify their contribution to predicting protein fitness. This computationally efficient approach is not limited to the features or embeddings considered here and is readily transferable to problem settings beyond protein fitness prediction.

Figures & tables

Figure 1: Overview of methodology: Two data modalities are considered: protein sequences $Z$ and corresponding tabular biochemical features $X$ . Every data point refers to a variant (or mutation), i.e., where a single amino acid has been substituted compared to the reference protein (or wild-type). The sequences are embedded via a protein language model (PLM), i.e., mapped onto a numerical vector $u\in\mathbb{R}^{d}$ . For the tabular features, a feature selection via $L1$ regularization is applied. These selected features $\widetilde{X}$ are either (a) directly used to project the embeddings $U$ onto their orthogonal complement or (b) a multi-layer perceptron (MLP) is trained on these features and its hidden layer representation $\widehat{X}$ is used for the orthogonal projection $P^{\perp}_{X}$ . A logistic regression as downstream classifier for protein fitness is trained separately on the embeddings $U$ and the orthogonally projected embeddings $\widetilde{U}$ . In an ablation study their predictive performance is compared.

Figure 2: Raw embeddings (left) of ESM-2 150M and and their differences to the corresponding reference wild-type embedding (right) after UMAP dimensionality reduction for the subset Activity . In the first row, data points are colored based on protein fitness label (positive: orange, negative: blue). The second row displays the same data but the data points are colored based on their corresponding reference wild-type. The raw embeddings cluster by their reference wild-type, the embedding differences are centered.

Figure 3: Predictive performance measured in Matthews correlation coefficient (MCC) of linear classifiers trained on different data modalities: tabular features (dark green), hidden-layer representation of tabular features (light green), PLM embeddings (blue), and their respective concatenation (dark yellow: embeddings and tabular features, light yellow: embeddings and representation of tabular features) for the four protein functions and PLMs ESM-1v 650M, ESM-2 35M, ESM-2 150M, and ESM-2 650M. The values are averages across folds with corresponding estimated standard deviation as black error bars.

Protein Function	Single amino acid features		Chemical pro- perty features
Protein Function	$\widetilde{p}$	%	$\widetilde{p}$	%
Org. Fitness	204 (24)	53	53 (1)	80
Activity	249 (20)	64	43 (2)	64
Expression	260 (10)	67	45 (1)	67
Binding	193 (30)	50	30 (6)	44

Table 1: Feature selection results using

L1

regularization for four protein functions. The columns contain the results for the substitution either on single amino acid level or grouped by chemical property. The number

\widetilde{p}

denotes average number of features with non-zero coefficients over the five folds with standard deviation in brackets. Additionally, the percentage of the respective total number of features is provided.

PLM	Protein Function	$U\perp y\mid X$	$X\perp y\mid U$
ESM-1v 650M	Organismal Fitness	2.91e-7	$2.54\text{\times}{10}^{-1}$
	Activity	9.21e-6	1.59e-3
	Expression	2.04e-7	$1.70\text{\times}{10}^{-1}$
	Binding	5.04e-3	$7.03\text{\times}{10}^{-1}$
ESM-2 35M	Organismal Fitness	8.19e-7	$7.68\text{\times}{10}^{-3}$
	Activity	$1.40\text{\times}{10}^{-2}$	1.20e-3

Table 2:

p

-values for CITs for tabular features (

X

) and embeddings (

U

) with protein function (

y

), using

L2

-penalized prediction models within the CIT. Rows are protein language models; columns are protein functions. Each entry is the combination of the

p

-values across folds into a single

p

-value, obtained by

\min(1,\,2\cdot\mathrm{mean}(p))

[ 36 ] . Bold values are significant after Bonferroni correction (

4

embeddings

\times

2

hypotheses;

\alpha=0.05/8=0.00625

Figure 4: Predictive performance measured in Matthews correlation coefficient (MCC) of linear classifiers trained on PLM embeddings (blue) and PLM embeddings orthogonally projected w.r.t. randomized tabular features (dark gray), tabular features (dark red), their randomized representation (light gray) or their representation (light red), respectively, for the four protein functions and PLMs ESM-1v 650M, ESM-2 35M, ESM-2 150M, and ESM-2 650M. The values are averages across folds with corresponding estimated standard deviation as black error bars. For all protein functions and models, the orthogonal projection w.r.t. tabular features (dark red) leads to substantial decrease in predictive performance exceeding the standard deviation observed across folds. If the projection is w.r.t. representation of tabular features (light red), the effect is even stronger. In contrast, for the orthogonal projection w.r.t. to randomized features or representations (dark and light gray), there is only a negligible change in performance.

Protein Function	Adjusted $R^{2}$
	Tabular			Repr. Tab.
	No ortho.	Rand. ortho.	Ortho.	No ortho.	Rand. ortho.	Ortho.
Org. Fitness	0.189 (0.006)	0.189 (0.006)	0.002 (0.001)	0.234 (0.005)	0.234 (0.005)	0.007 (0.001)
Activity	0.220 (0.006)	0.221 (0.005)	0.004 (0.002)	0.250 (0.005)	0.251 (0.004)	0.006 (0.003)
Expression	0.262 (0.004)	0.262 (0.004)	0.008 (0.003)	0.289 (0.012)	0.288 (0.012)	0.009 (0.003)
Binding	0.153 (0.024)	0.153 (0.021)	0.018 (0.009)	0.206 (0.014)	0.203 (0.008)	0.011 (0.005)

Table 3: Results of evaluation model for four protein functions: Adjusted

R^{2}

for regressing embedding-based predicted logits (ESM-1v 650M) on tabular features or representation of tabular features, respectively, for embeddings without orthogonal projection (‘No ortho’), after applying the randomized orthogonal projection (‘Rand. ortho.’), and after (regular) orthogonal projection (‘Ortho.’). The values are averages across folds with corresponding estimated standard deviation in brackets.

Feature	Protein Function
Feature	Org. Fitness	Activity	Expression	Binding
phi	$-0.051$ *	$-0.021$	$-0.127$ *	$-0.145$ *
psi	$0.003$	$0.052$	$-0.011$	$0.010$
plddt	$-0.139$ *	$-0.138$ *	$-0.117$ *	$-0.097$
rsa	$0.316$ *	$0.425$ *	$0.467$ *	$0.339$ *
strand	—	$0.026$	$-0.274$	$-0.606$ *

Table 4: Coefficients of structural features of evaluation model for four protein functions: average coefficients across folds for regressing embedding-based predicted logits (ESM-1v 650M) on structural tabular features. Significance refers to whether estimate was significant across all folds. Blanks refer to variables that were excluded during features selection.

Figure 5: Heatmaps of coefficients of (binary) substitution features of evaluation model, i.e., fitted on the embedding-based predictions (ESM-1v 650M), for different protein functions. The reference amino acid (row) of the wild-type protein was substituted for the alternative amino acid (column). Upper plots illustrate coefficients for single amino acid substitution features, lower ones for substitution features grouped by chemical property. Every entry corresponds to coefficient with either positive (red) or negative (blue) contribution to the predicting the positive class. Coefficients of features excluded by the feature selection during training are set to zero (white). The values of coefficients are averages across folds.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Predictive performance measured in Matthews correlation coefficient (MCC) of linear classifiers trained on different data modalities: tabular features where the substitution features where grouped by biochemical property (dark green), hidden-layer representation of tabular features (light green), PLM embeddings (blue), and their respective concatenation (dark yellow: embeddings and tabular features, light yellow: embeddings and representation of tabular features) for the four protein functions and PLMs ESM-1v 650M, ESM-2 35M, ESM-2 150M, and ESM-2 650M. The values are averages across folds with corresponding estimated standard deviation as black error bars.

Figure 7: Predictive performance measured in Matthews correlation coefficient (MCC) of linear classifiers trained on PLM embeddings (blue) and PLM embeddings orthogonally projected w.r.t. randomized tabular features (dark gray), tabular features (dark red), their randomized representation (light gray) or their representation (light red), respectively, for the four protein functions and PLMs ESM-1v 650M, ESM-2 35M, ESM-2 150M, and ESM-2 650M. The substitution features of the tabular features are grouped by biochemical property. The values are averages across folds with corresponding estimated standard deviation as black error bars. For all protein functions and models, the orthogonal projection w.r.t. tabular features (dark red) leads to substantial decrease in predictive performance exceeding the standard deviation observed across folds. If the projection is w.r.t. representation of tabular features (light red), the effect is even stronger. In contrast, for the orthogonal projection w.r.t. to randomized features or representations (dark and light gray), there is only a negligible change in performance.

ESM-2 size	PF	Adjusted $R^{2}$
		Tabular			Repr. Tab.
		No ortho.	Rand. ortho.	Ortho.	No ortho.	Rand. ortho.	Ortho.
35M	Org.	0.170 (0.007)	0.169 (0.007)	0.001 (0.001)	0.189 (0.004)	0.190 (0.004)	0.004 (0.001)
	Act.	0.217 (0.008)	0.217 (0.008)	0.006 (0.002)	0.224 (0.008)	0.224 (0.008)	0.006 (0.002)
	Expr.	0.212 (0.006)	0.211 (0.006)	0.010 (0.004)	0.220 (0.007)	0.218 (0.007)	0.008 (0.003)
	Bind.	0.157 (0.017)	0.156 (0.017)	0.010 (0.011)	0.210 (0.020)	0.208 (0.021)	0.014 (0.002)

Appendix

Table 5: Results of evaluation model for four protein functions (‘PF’): Adjusted

R^{2}

for regressing embedding-based predicted logits (ESM-2 35M, 150M, and 650M) on tabular features or representation of tabular features, respectively, for embeddings without orthogonal projection (‘No ortho’), after applying the randomized orthogonal projection (‘Rand. ortho.’), and after (regular) orthogonal projection (‘Ortho.’). The values are averages across folds with corresponding estimated standard deviation in brackets.

Feature	Protein Function
Feature	Org. Fitness	Activity	Expression	Binding
phi	$-0.052$ **	—	$-0.125$ *	$-0.082$
psi	$0.002$	$0.060$	—	$-0.023$
plddt	$-0.134$ *	$-0.138$ *	$-0.061$	$-0.132$
rsa	$0.348$ *	$0.395$ *	$0.494$ *	$0.415$ *
strand	—	$-0.096$	$-0.298$	$-0.427$

Appendix

Table 6: Coefficients of structural features of evaluation model for four protein functions: average coefficients across folds for regressing embedding-based predicted logits (ESM-2 650M) on structural tabular features. Significance refers to whether estimate was significant across all folds. Blanks refer to variables that were excluded during features selection.

Figure 8: Heatmaps of coefficients of (binary) substitution features of evaluation model, i.e., fitted on the embedding-based predictions (ESM-2 650M), for different protein functions. The reference amino acid (row) of the wild-type protein was substituted for the alternative amino acid (column). Upper plots illustrate coefficients for single amino acid substitution features, lower ones for substitution features grouped by chemical property. Every entry corresponds to coefficient with either positive (red) or negative (blue) contribution to the predicting the positive class. Coefficients of features excluded by the feature selection during training are set to zero (white). The values of coefficients are averages across folds.

Protein Language Models

Protein language models learn patterns in amino-acid sequences to predict protein properties, interpret mutations, and design new proteins.

See all 69 papers

Aug 12, 2026

Task- and dataset-specific information in protein language models

Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs). By consensus, embeddings from the models' last layers are used, while the models' internal behavior remains poorly understood. We analyzed 13 PLMs across 15 DTs and 9 datasets to assess the value of embeddings from intermediate PLM layers. We trained probe models on embeddings from each layer, compared their performance, and showed that the last layers of PLMs rarely produced embeddings that led to the best results on downstream tasks. Furthermore, we identified a connection between how models learn a certain DT and the similarity between that DT and the pre-training objective. For example, for residue-level downstream tasks, we observed a steady increase in performance across almost all PLM layers, which we attributed to their similarity to most PLMs' pre-training objectives. To allow the community to capitalize on our findings, we provide PLMSommelier, a Python package that automatically identifies the best PLM layer for a given DT with ~98% accuracy and creates a truncated model using only the early layers up to the best-performing layer. This will help users save time and memory during inference and yield better predictive performance.

Roman Joeres, Ilya Senatorov, Anastasia Kolchina +2

Drug Bioinformatics, Helmholtz Institute for Pharmaceutical Research Saarland, 66123 Saarbruecken, Germany · Center for Bioinformatics, Saarland University, 66123 Saarbruecken, Germany · Saarland Informatics Campus, Saarland University, 66123 Saarbruecken, Germany +2

Protein Language Models Protein Representation Learning Language Model Probing

Ask PDF

Feb 24, 2026

Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference

Modern Protein Language Models (PLMs) apply transformer-based model architectures from natural language processing to biological sequences, predicting a variety of protein functions and properties. However, protein language has key differences from natural language, such as a rich functional space despite a vocabulary of only 20 amino acids. These differences motivate research into how transformer-based architectures operate differently in the protein domain and how we can better leverage PLMs to solve protein-related tasks. In this work, we begin by directly comparing how the distribution of information stored across layers of attention heads differs between the protein and natural language domain. Furthermore, we adapt a simple early-exit technique-originally used in the natural language domain to improve efficiency at the cost of performance-to achieve both increased accuracy and substantial efficiency gains in protein non-structural property prediction by allowing the model to automatically select protein representations from the intermediate layers of the PLMs for the specific task and protein at hand. We achieve performance gains ranging from 0.4 to 7.01 percentage points while simultaneously improving efficiency by over 10 percent across models and non-structural prediction tasks. Our work opens up an area of research directly comparing how language models change behavior when moved into the protein domain and advances language modeling in biological domains.

Anna Hart, Chi Han, Jeonghwan Kim +2

Siebel School of Data Science and Computing, University of Illinois Urbana-Champaign, 201 N Goodwin Ave, Urbana, IL 61801, USA · DOE Center for Advanced Bioenergy and Bioproducts Innovation, University of Illinois Urbana-Champaign, 1206 W Gregory Drive, Urbana, IL 61801, USA · Chemical and Biomolecular Engineering, University of Illinois Urbana-Champaign, 600 S Matthews Ave, Urbana, IL 61801, USA

Protein Language Models Protein Representation Learning Bioinformatics Molecular Property Prediction

Ask PDF

May 7, 2026

ProtSent: Protein Sentence Transformers

Protein language models produce rich per-residue representations, but the sequence embeddings obtained by mean-pooling them are never trained to place related proteins near one another, so the geometry that retrieval, clustering and nearest-neighbor transfer depend on is never optimized. We present ProtSent, a contrastive post-training framework that supervises a protein language model with three types of biological relation at once, Pfam family co-membership, AlphaFold DB structural clusters and STRING interactions, and with deep mutational scanning fitness ordering as an optional fourth. It needs no structure at training or inference, trains on corpora filtered at 40% sequence identity against the benchmark test sets they most resemble, and is applied to ESM-2 (35M and 150M parameters) and ESM-C (300M). Post-training makes structural family membership recoverable from the frozen embedding without a trained readout: on SCOPe-40, the adjusted Rand index of the 35M model rises from 0.054 (near chance) to 0.507, and family retrieval gains +0.185, +0.190 and +0.424 Recall@1 over the three frozen backbones, with paired bootstrap intervals excluding zero. On the CATH midnight-zone benchmark, homologous-superfamily accuracy rises from 40.7% to 56.7% at 35M and from 43.3% to 62.7% at 150M, more than the 12 points a CATH-supervised model gains over its own base. Against a maximally sensitive HMMER search, alignment keeps the best single hit at both ESM-2 scales, but the embedding leads at ranking depth (MAP 0.705 against 0.607 at 150M) and returns a ranking for every query, including those where alignment finds none. Under a trained linear probe, however, the same embeddings fall below the untuned backbone, as do structure-distilled embeddings against theirs, so the gain in neighborhood structure comes at a cost in linear-probe accuracy. Code and models: https://github.com/oriel9p/ProtSent

Dan Ofer, Oriel Perets, Michal Linial +1

Department of Biological Chemistry The Hebrew University of Jerusalem · Department of Computer and Information Science Ben-Gurion University of the Negev

Contrastive Learning Protein Language Models Protein Representation Learning Molecular Representation Learning Bioinformatics

Ask PDF

Abstract

Protein Language Models

Explore similar work

Task- and dataset-specific information in protein language models

Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference

ProtSent: Protein Sentence Transformers