Interpreting Protein Language Model Embeddings via Orthogonal Projection for Protein Fitness Prediction

Paulo Yanez Sarmiento; Pia Francesca Rissom; Manuel Pfeuffer; Marco Simnacher; Jordan F. Safer; Sumaiya Iqbal; Henrike O. Heyne; Nadja Klein

cs.LGAug 26, 2026

Interpreting Protein Language Model Embeddings via Orthogonal Projection for Protein Fitness Prediction

Authors: Paulo Yanez Sarmiento, Pia Francesca Rissom, Manuel Pfeuffer, Marco Simnacher, Jordan F. Safer, Sumaiya Iqbal, Henrike O. Heyne, Nadja Klein,

Organizations: Hasso Plattner Institute, University of Potsdam · Bioinformatics and Machine Learning, Broad Institute of MIT and Harvard · School of Business and Economics, Humboldt University of Berlin · Scientific Computing Center, Karlsruhe Institute of Technology

Abstract

Recently, there has been a growing adoption of protein language models (PLMs) in biomedical science. Their embeddings provide a rich numerical representation of protein sequences which achieve state-of-the-art performance on several downstream tasks including protein fitness prediction. However, PLM embeddings are not directly interpretable and, thereby, it remains unclear what features they encode. To gain insight into which biochemical properties of the protein are driving the prediction, we leverage an orthogonal projection technique that removes linear effects of known tabular features from embeddings and extend it to high-order and interaction effects. In this way, we remove the effects of interpretable biochemical features from PLM embeddings. In an ablation study, we show that this leads to a decrease in performance for a downstream classifier trained only on the embeddings to predict protein fitness. In an additional evaluation, we find that these biochemical features explain a substantial part of the variance in the predictions of this classifier. Hence, we can show that PLM embeddings encode patterns correlated with biochemical properties and quantify their contribution to predicting protein fitness. This computationally efficient approach is not limited to the features or embeddings considered here and is readily transferable to problem settings beyond protein fitness prediction.

Figures & tables

Figure 1: Overview of methodology: Two data modalities are considered: protein sequences $Z$ and corresponding tabular biochemical features $X$ . Every data point refers to a variant (or mutation), i.e., where a single amino acid has been substituted compared to the reference protein (or wild-type). The sequences are embedded via a protein language model (PLM), i.e., mapped onto a numerical vector $u\in\mathbb{R}^{d}$ . For the tabular features, a feature selection via $L1$ regularization is applied. These selected features $\widetilde{X}$ are either (a) directly used to project the embeddings $U$ onto their orthogonal complement or (b) a multi-layer perceptron (MLP) is trained on these features and its hidden layer representation $\widehat{X}$ is used for the orthogonal projection $P^{\perp}_{X}$ . A logistic regression as downstream classifier for protein fitness is trained separately on the embeddings $U$ and the orthogonally projected embeddings $\widetilde{U}$ . In an ablation study their predictive performance is compared.

Figure 2: Raw embeddings (left) of ESM-2 150M and and their differences to the corresponding reference wild-type embedding (right) after UMAP dimensionality reduction for the subset Activity . In the first row, data points are colored based on protein fitness label (positive: orange, negative: blue). The second row displays the same data but the data points are colored based on their corresponding reference wild-type. The raw embeddings cluster by their reference wild-type, the embedding differences are centered.

Figure 3: Predictive performance measured in Matthews correlation coefficient (MCC) of linear classifiers trained on different data modalities: tabular features (dark green), hidden-layer representation of tabular features (light green), PLM embeddings (blue), and their respective concatenation (dark yellow: embeddings and tabular features, light yellow: embeddings and representation of tabular features) for the four protein functions and PLMs ESM-1v 650M, ESM-2 35M, ESM-2 150M, and ESM-2 650M. The values are averages across folds with corresponding estimated standard deviation as black error bars.

Protein Function	Single amino acid features		Chemical pro- perty features
Protein Function	$\widetilde{p}$	%	$\widetilde{p}$	%
Org. Fitness	204 (24)	53	53 (1)	80
Activity	249 (20)	64	43 (2)	64
Expression	260 (10)	67	45 (1)	67
Binding	193 (30)	50	30 (6)	44

Table 1: Feature selection results using

L1

regularization for four protein functions. The columns contain the results for the substitution either on single amino acid level or grouped by chemical property. The number

\widetilde{p}

denotes average number of features with non-zero coefficients over the five folds with standard deviation in brackets. Additionally, the percentage of the respective total number of features is provided.

PLM	Protein Function	$U\perp y\mid X$	$X\perp y\mid U$
ESM-1v 650M	Organismal Fitness	2.91e-7	$2.54\text{\times}{10}^{-1}$
	Activity	9.21e-6	1.59e-3
	Expression	2.04e-7	$1.70\text{\times}{10}^{-1}$
	Binding	5.04e-3	$7.03\text{\times}{10}^{-1}$
ESM-2 35M	Organismal Fitness	8.19e-7	$7.68\text{\times}{10}^{-3}$
	Activity	$1.40\text{\times}{10}^{-2}$	1.20e-3

Table 2:

p

-values for CITs for tabular features (

X

) and embeddings (

U

) with protein function (

y

), using

L2

-penalized prediction models within the CIT. Rows are protein language models; columns are protein functions. Each entry is the combination of the

p

-values across folds into a single

p

-value, obtained by

\min(1,\,2\cdot\mathrm{mean}(p))

[ 36 ] . Bold values are significant after Bonferroni correction (

4

embeddings

\times

2

hypotheses;

\alpha=0.05/8=0.00625

Figure 4: Predictive performance measured in Matthews correlation coefficient (MCC) of linear classifiers trained on PLM embeddings (blue) and PLM embeddings orthogonally projected w.r.t. randomized tabular features (dark gray), tabular features (dark red), their randomized representation (light gray) or their representation (light red), respectively, for the four protein functions and PLMs ESM-1v 650M, ESM-2 35M, ESM-2 150M, and ESM-2 650M. The values are averages across folds with corresponding estimated standard deviation as black error bars. For all protein functions and models, the orthogonal projection w.r.t. tabular features (dark red) leads to substantial decrease in predictive performance exceeding the standard deviation observed across folds. If the projection is w.r.t. representation of tabular features (light red), the effect is even stronger. In contrast, for the orthogonal projection w.r.t. to randomized features or representations (dark and light gray), there is only a negligible change in performance.

Protein Function	Adjusted $R^{2}$
	Tabular			Repr. Tab.
	No ortho.	Rand. ortho.	Ortho.	No ortho.	Rand. ortho.	Ortho.
Org. Fitness	0.189 (0.006)	0.189 (0.006)	0.002 (0.001)	0.234 (0.005)	0.234 (0.005)	0.007 (0.001)
Activity	0.220 (0.006)	0.221 (0.005)	0.004 (0.002)	0.250 (0.005)	0.251 (0.004)	0.006 (0.003)
Expression	0.262 (0.004)	0.262 (0.004)	0.008 (0.003)	0.289 (0.012)	0.288 (0.012)	0.009 (0.003)
Binding	0.153 (0.024)	0.153 (0.021)	0.018 (0.009)	0.206 (0.014)	0.203 (0.008)	0.011 (0.005)

Table 3: Results of evaluation model for four protein functions: Adjusted

R^{2}

for regressing embedding-based predicted logits (ESM-1v 650M) on tabular features or representation of tabular features, respectively, for embeddings without orthogonal projection (‘No ortho’), after applying the randomized orthogonal projection (‘Rand. ortho.’), and after (regular) orthogonal projection (‘Ortho.’). The values are averages across folds with corresponding estimated standard deviation in brackets.

Feature	Protein Function
Feature	Org. Fitness	Activity	Expression	Binding
phi	$-0.051$ *	$-0.021$	$-0.127$ *	$-0.145$ *
psi	$0.003$	$0.052$	$-0.011$	$0.010$
plddt	$-0.139$ *	$-0.138$ *	$-0.117$ *	$-0.097$
rsa	$0.316$ *	$0.425$ *	$0.467$ *	$0.339$ *
strand	—	$0.026$	$-0.274$	$-0.606$ *

Table 4: Coefficients of structural features of evaluation model for four protein functions: average coefficients across folds for regressing embedding-based predicted logits (ESM-1v 650M) on structural tabular features. Significance refers to whether estimate was significant across all folds. Blanks refer to variables that were excluded during features selection.

Figure 5: Heatmaps of coefficients of (binary) substitution features of evaluation model, i.e., fitted on the embedding-based predictions (ESM-1v 650M), for different protein functions. The reference amino acid (row) of the wild-type protein was substituted for the alternative amino acid (column). Upper plots illustrate coefficients for single amino acid substitution features, lower ones for substitution features grouped by chemical property. Every entry corresponds to coefficient with either positive (red) or negative (blue) contribution to the predicting the positive class. Coefficients of features excluded by the feature selection during training are set to zero (white). The values of coefficients are averages across folds.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Predictive performance measured in Matthews correlation coefficient (MCC) of linear classifiers trained on different data modalities: tabular features where the substitution features where grouped by biochemical property (dark green), hidden-layer representation of tabular features (light green), PLM embeddings (blue), and their respective concatenation (dark yellow: embeddings and tabular features, light yellow: embeddings and representation of tabular features) for the four protein functions and PLMs ESM-1v 650M, ESM-2 35M, ESM-2 150M, and ESM-2 650M. The values are averages across folds with corresponding estimated standard deviation as black error bars.

Figure 7: Predictive performance measured in Matthews correlation coefficient (MCC) of linear classifiers trained on PLM embeddings (blue) and PLM embeddings orthogonally projected w.r.t. randomized tabular features (dark gray), tabular features (dark red), their randomized representation (light gray) or their representation (light red), respectively, for the four protein functions and PLMs ESM-1v 650M, ESM-2 35M, ESM-2 150M, and ESM-2 650M. The substitution features of the tabular features are grouped by biochemical property. The values are averages across folds with corresponding estimated standard deviation as black error bars. For all protein functions and models, the orthogonal projection w.r.t. tabular features (dark red) leads to substantial decrease in predictive performance exceeding the standard deviation observed across folds. If the projection is w.r.t. representation of tabular features (light red), the effect is even stronger. In contrast, for the orthogonal projection w.r.t. to randomized features or representations (dark and light gray), there is only a negligible change in performance.

ESM-2 size	PF	Adjusted $R^{2}$
		Tabular			Repr. Tab.
		No ortho.	Rand. ortho.	Ortho.	No ortho.	Rand. ortho.	Ortho.
35M	Org.	0.170 (0.007)	0.169 (0.007)	0.001 (0.001)	0.189 (0.004)	0.190 (0.004)	0.004 (0.001)
	Act.	0.217 (0.008)	0.217 (0.008)	0.006 (0.002)	0.224 (0.008)	0.224 (0.008)	0.006 (0.002)
	Expr.	0.212 (0.006)	0.211 (0.006)	0.010 (0.004)	0.220 (0.007)	0.218 (0.007)	0.008 (0.003)
	Bind.	0.157 (0.017)	0.156 (0.017)	0.010 (0.011)	0.210 (0.020)	0.208 (0.021)	0.014 (0.002)

Appendix

Table 5: Results of evaluation model for four protein functions (‘PF’): Adjusted

R^{2}

for regressing embedding-based predicted logits (ESM-2 35M, 150M, and 650M) on tabular features or representation of tabular features, respectively, for embeddings without orthogonal projection (‘No ortho’), after applying the randomized orthogonal projection (‘Rand. ortho.’), and after (regular) orthogonal projection (‘Ortho.’). The values are averages across folds with corresponding estimated standard deviation in brackets.

Feature	Protein Function
Feature	Org. Fitness	Activity	Expression	Binding
phi	$-0.052$ **	—	$-0.125$ *	$-0.082$
psi	$0.002$	$0.060$	—	$-0.023$
plddt	$-0.134$ *	$-0.138$ *	$-0.061$	$-0.132$
rsa	$0.348$ *	$0.395$ *	$0.494$ *	$0.415$ *
strand	—	$-0.096$	$-0.298$	$-0.427$

Appendix

Table 6: Coefficients of structural features of evaluation model for four protein functions: average coefficients across folds for regressing embedding-based predicted logits (ESM-2 650M) on structural tabular features. Significance refers to whether estimate was significant across all folds. Blanks refer to variables that were excluded during features selection.

Figure 8: Heatmaps of coefficients of (binary) substitution features of evaluation model, i.e., fitted on the embedding-based predictions (ESM-2 650M), for different protein functions. The reference amino acid (row) of the wild-type protein was substituted for the alternative amino acid (column). Upper plots illustrate coefficients for single amino acid substitution features, lower ones for substitution features grouped by chemical property. Every entry corresponds to coefficient with either positive (red) or negative (blue) contribution to the predicting the positive class. Coefficients of features excluded by the feature selection during training are set to zero (white). The values of coefficients are averages across folds.

Protein Language Models

Protein language models learn patterns in amino-acid sequences to predict protein properties, interpret mutations, and design new proteins.

See all 69 papers

Task- and dataset-specific information in protein language models
Aug 12, 2026Roman Joeres, Ilya Senatorov, Anastasia Kolchina +2Protein Language Models Protein Representation Learning
Drug Bioinformatics, Helmholtz Institute for Pharmaceutical Research Saarland, 66123 Saarbruecken, Germany · Center for Bioinformatics, Saarland University, 66123 Saarbruecken, Germany · Saarland Informatics Campus, Saarland University, 66123 Saarbruecken, Germany +2
Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference
Feb 24, 2026Anna Hart, Chi Han, Jeonghwan Kim +2Protein Language Models Protein Representation Learning
Siebel School of Data Science and Computing, University of Illinois Urbana-Champaign, 201 N Goodwin Ave, Urbana, IL 61801, USA · DOE Center for Advanced Bioenergy and Bioproducts Innovation, University of Illinois Urbana-Champaign, 1206 W Gregory Drive, Urbana, IL 61801, USA · Chemical and Biomolecular Engineering, University of Illinois Urbana-Champaign, 600 S Matthews Ave, Urbana, IL 61801, USA
ProtSent: Protein Sentence Transformers
May 7, 2026Dan Ofer, Oriel Perets, Michal Linial +1Contrastive Learning Protein Language Models
Department of Biological Chemistry The Hebrew University of Jerusalem · Department of Computer and Information Science Ben-Gurion University of the Negev

Aug 12, 2026J/K move · Enter open · S save

Task- and dataset-specific information in protein language models

Roman Joeres, Ilya Senatorov, Anastasia Kolchina, Dietrich Klakow+1

Drug Bioinformatics, Helmholtz Institute for Pharmaceutical Research Saarland, 66123 Saarbruecken, Germany · Center for Bioinformatics, Saarland University, 66123 Saarbruecken, Germany · Saarland Informatics Campus, Saarland University, 66123 Saarbruecken, Germany+2

Protein Language Models Protein Representation Learning Language Model Probing

Open paper AskPDF arXiv

Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs). By consensus, embeddings from the models' last layers are used, while the models' internal behavior remains poorly understood. We analyzed 13 PLMs across 15 DTs and 9 datasets to assess the value of embeddings from intermediate PLM layers. We trained probe models on embeddings from each layer, compared their performance, and showed that the last layers of PLMs rarely produced embeddings that led to the best results on downstream tasks. Furthermore, we identified a connection between how models learn a certain DT and the similarity between that DT and the pre-training objective. For example, for residue-level downstream tasks, we observed a steady increase in performance across almost all PLM layers, which we attributed to their similarity to most PLMs' pre-training objectives. To allow the community to capitalize on our findings, we provide PLMSommelier, a Python package that automatically identifies the best PLM layer for a given DT with ~98% accuracy and creates a truncated model using only the early layers up to the best-performing layer. This will help users save time and memory during inference and yield better predictive performance.

Abstract

Protein Language Models

Explore similar work

Task- and dataset-specific information in protein language models

Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference

ProtSent: Protein Sentence Transformers