From Words to Amino Acids: Does the Curse of Depth Persist?

Aleena Siji; Amir Mohammad Karimi Mamaghan; Ferdinand Kapl; Tobias Höppe; Emmanouil Angelis; Andrea Dittadi; Maurice Brenner; Michael Heinzinger

cs.LGFeb 25, 2026

From Words to Amino Acids: Does the Curse of Depth Persist?

Authors: Aleena Siji, Amir Mohammad Karimi Mamaghan, Ferdinand Kapl, Tobias Höppe, Emmanouil Angelis, Andrea Dittadi, Maurice Brenner, Michael Heinzinger, +4 more

Organizations: Technical University of Munich · Helmholtz AI, Munich · KTH Royal Institute of Technology · Institute of Computational Biology, Helmholtz Munich · Google, Paradigms of Intelligence Team

Abstract

Protein language models (PLMs) have become widely adopted as general-purpose models, demonstrating strong performance in protein engineering and de novo design. Like large language models (LLMs), they are typically trained as deep transformers with next-token or masked-token prediction objectives on massive sequence corpora and are scaled by increasing model depth. Recent work on autoregressive LLMs has identified the Curse of Depth: many later layers contribute little to the final output predictions. These findings naturally raise the question of whether a similar depth inefficiency also appears in PLMs, where many widely used models are not autoregressive, and some are multimodal, accepting both protein sequence and structure as input. In this work, we present a depth analysis of seven popular PLM families across model scales, spanning autoregressive, masked, and diffusion objectives, and quantify how layer contributions evolve with depth using a unified set of probing-, perturbation-, and downstream-evaluation measurements. Across models, we observe consistent depth-dependent patterns that extend prior findings on LLMs: a large fraction of task-relevant computation is concentrated in a subset of layers, while the remaining layers mainly provide incremental refinement of the final prediction. These trends persist beyond sequence-only settings and also appear in multimodal PLMs. Taken together, our results suggest that depth inefficiency is a common feature of modern PLMs, motivating future work on more depth-efficient architectures and training methods.

Figures & tables

Figure 1: Maximum propagated effect of skipping each layer on future-token computations, alongside layer-layer CKA similarity, in ESM2. Here, “future” refers to a held-out subset of non-intervened masked tokens. Even at 35M, skipping later layers produces relatively weak propagated effects compared to skipping early layers. From 150M onward, this separation becomes clear: a substantial fraction of late layers can be skipped with only minor effects on subsequent computations on future tokens. Moreover, the low-effect regions also report high layer-layer similarity. This pattern strengthens with scale, indicating that downstream sensitivity increasingly concentrates in early-to-middle layers, while later layers mainly refine the final prediction. We also observe localized low-effect regions among early layers. This aligns with the stage-wise view of Lad et al. [21] and suggests that depth is organized into multiple inference stages with weaker dependencies across certain layer ranges.

Figure 2: TunedLens analysis for ESM2 across depth: KL divergence to the final output distribution (left) and top-1 overlap with the full-model prediction (right). Across ESM2 variants, KL divergence slowly decreases toward later layers, indicating that deeper layers make increasingly incremental updates that bring the distribution closer to the final output. Consistently, top-1 overlap is lower in early layers and increases toward the end of the network, with the most pronounced gains in the final layers for larger variants. Together, these trends suggest that the top prediction becomes increasingly stable with depth, while later layers mainly refine confidence over tokens rather than change the predicted token.

Figure 3: Layer-wise ProteinGym performance for ESM2. Average Spearman correlation as a function of relative depth, normalized to

[0,1]

, where predictions are taken from each layer. Performance improves with depth for all model sizes, but the largest models exhibit diminishing returns in the final layers, suggesting that earlier layers already capture much of the signal and later layers mainly provide small refinements to the final predictions.

Panel of Figure 4 — Figure 4: Depth analysis results for ESM3 for the structure stream. Left: skiplayer propagation and layer-layer similarity, and Right: LogitLens readouts. Aligned with the sequence-stream results, ESM3 shows weaker propagated effects in early layers, with stronger contributions emerging in mid-to-late layers.

Figure 5: Cross-modal layer–layer cosine similarity between sequence-only and structure-only representations for ESM3. Each cell compares a sequence-layer representation to a structure-layer representation for the same proteins, with similarities averaged across proteins. The first $\sim$ 60% of layers show high cross-modal similarity, suggesting strong early sequence-structure alignment. After this point, similarity becomes more localized, indicating a transition toward more modality-specific representations in later layers.

Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Maximum propagated effect of skipping each layer on future-token computations for DPLM.

Figure 7: Maximum propagated effect of skipping each layer on future-token computations for DPLM2.

Figure 8: Maximum propagated effect of skipping each layer on future-token computations for Profluent-E1.

Figure 9: Maximum propagated effect of skipping each layer on later-layer representations (future tokens only), for ESM3.

Figure 10: Maximum propagated effect of skipping each layer on future-token computations for ProGen2.

Figure 11: Maximum propagated effect of skipping each layer on future-token computations for ProGen3.

Figure 12: Maximum change in ESM2 output probabilities under layer skipping, restricted to future tokens only.

Figure 13: Maximum change in DPLM output probabilities under layer skipping, restricted to future tokens only.

Figure 14: Maximum change in DPLM2 output probabilities under layer skipping, restricted to future tokens only.

Figure 15: Maximum change in Profluent-E1 output probabilities under layer skipping, restricted to future tokens only.

Figure 16: Maximum change in ESM3 output probabilities under layer skipping, restricted to future tokens only.

Figure 17: Maximum change in ProGen2 output probabilities under layer skipping, restricted to future tokens only.

Figure 18: Maximum change in ProGen3 output probabilities under layer skipping, restricted to future tokens only.

Figure 19: TunedLens analysis for DPLM across depth: KL divergence between the layer-wise output distribution and the final output distribution (left), and top-1 overlap between the layer-wise prediction and the full-model prediction (right).

Figure 20: TunedLens analysis for DPLM2 across depth: KL divergence between the layer-wise output distribution and the final output distribution (left), and top-1 overlap between the layer-wise prediction and the full-model prediction (right).

Figure 21: TunedLens analysis for E1 across depth: KL divergence between the layer-wise output distribution and the final output distribution (left), and top-1 overlap between the layer-wise prediction and the full-model prediction (right).

Figure 22: TunedLens analysis for ESM3 across depth: KL divergence between the layer-wise output distribution and the final output distribution (left), and top-1 overlap between the layer-wise prediction and the full-model prediction (right).

Figure 23: TunedLens analysis for ProGen2 across depth: KL divergence between the layer-wise output distribution and the final output distribution (left), and top-1 overlap between the layer-wise prediction and the full-model prediction (right).

Figure 24: TunedLens analysis for ProGen3 across depth: KL divergence between the layer-wise output distribution and the final output distribution (left), and top-1 overlap between the layer-wise prediction and the full-model prediction (right).

Figure 25: Average Spearman correlation for DPLM on ProteinGym, calculated at each layer. The relative depth is normalized to

[0,1]

Figure 26: Average Spearman correlation for DPLM2 on ProteinGym, calculated at each layer. The relative depth is normalized to

[0,1]

Figure 27: Average Spearman correlation for Profluent-E1 on ProteinGym, calculated at each layer. The relative depth is normalized to

[0,1]

Figure 28: Average Spearman correlation for ESM3 on ProteinGym, calculated at each layer. The relative depth is normalized to

[0,1]

Figure 29: Average Spearman correlation for ProGen2 on ProteinGym, calculated at each layer. The relative depth is normalized to

[0,1]

Figure 30: Average Spearman correlation for ProGen3 on ProteinGym, calculated at each layer. The relative depth is normalized to

[0,1]

Figure 31: Average Spearman correlation for ESM2 on ProteinGym, computed at each layer and shown separately by phenotype. Relative depth is normalized to

[0,1]

Figure 32: Average Spearman correlation for DPLM on ProteinGym, computed at each layer and shown separately by phenotype. Relative depth is normalized to

[0,1]

Figure 33: Average Spearman correlation for DPLM2 on ProteinGym, computed at each layer and shown separately by phenotype. Relative depth is normalized to

[0,1]

Figure 34: Average Spearman correlation for Profluent-E1 on ProteinGym, computed at each layer and shown separately by phenotype. Relative depth is normalized to

[0,1]

Figure 35: Average Spearman correlation for ESM3 on ProteinGym, computed at each layer and shown separately by phenotype. Relative depth is normalized to

[0,1]

Figure 36: Average Spearman correlation for ProGen2 on ProteinGym, computed at each layer and shown separately by phenotype. Relative depth is normalized to

[0,1]

Figure 37: Average Spearman correlation for ProGen3 on ProteinGym, computed at each layer and shown separately by phenotype. Relative depth is normalized to

[0,1]

Figure 38: Maximum propagated effect of skipping each layer on future-token computations of the multimodal (sequence-structure) stream for ESM3.

Figure 39: Maximum propagated effect of skipping each layer on future-token computations of the structure stream for DPLM2.

Figure 40: Maximum propagated effect of skipping each layer on future-token computations of the multimodal (sequence-structure) stream for DPLM2.

Figure 41: LogitLens analysis for ESM3 across depth on the multimodal (sequence-structure) stream: KL divergence between the layer-wise output distribution and the final output distribution (left), and top-1 overlap between the layer-wise prediction and the full-model prediction (right).

Figure 42: LogitLens analysis for DPLM2 across depth on the structure stream: KL divergence between the layer-wise output distribution and the final output distribution (left), and top-1 overlap between the layer-wise prediction and the full-model prediction (right).

Figure 43: LogitLens analysis for DPLM2 across depth on the multimodal (sequence-structure) stream: KL divergence between the layer-wise output distribution and the final output distribution (left), and top-1 overlap between the layer-wise prediction and the full-model prediction (right).

Figure 44: Cross-modal layer–layer cosine similarity between sequence-only and structure-only representations for DPLM2, averaged across proteins. Each cell compares a sequence-layer representation to a structure-layer representation for the same proteins.

Explore similar work

Aug 12, 2026

Task- and dataset-specific information in protein language models

Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs). By consensus, embeddings from the models' last layers are used, while the models' internal behavior remains poorly understood. We analyzed 13 PLMs across 15 DTs and 9 datasets to assess the value of embeddings from intermediate PLM layers. We trained probe models on embeddings from each layer, compared their performance, and showed that the last layers of PLMs rarely produced embeddings that led to the best results on downstream tasks. Furthermore, we identified a connection between how models learn a certain DT and the similarity between that DT and the pre-training objective. For example, for residue-level downstream tasks, we observed a steady increase in performance across almost all PLM layers, which we attributed to their similarity to most PLMs' pre-training objectives. To allow the community to capitalize on our findings, we provide PLMSommelier, a Python package that automatically identifies the best PLM layer for a given DT with ~98% accuracy and creates a truncated model using only the early layers up to the best-performing layer. This will help users save time and memory during inference and yield better predictive performance.

Roman Joeres, Ilya Senatorov, Anastasia Kolchina +2

Drug Bioinformatics, Helmholtz Institute for Pharmaceutical Research Saarland, 66123 Saarbruecken, Germany · Center for Bioinformatics, Saarland University, 66123 Saarbruecken, Germany · Saarland Informatics Campus, Saarland University, 66123 Saarbruecken, Germany +2

Protein Language Models Protein Representation Learning Language Model Probing

Ask PDF

Feb 24, 2026

Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference

Modern Protein Language Models (PLMs) apply transformer-based model architectures from natural language processing to biological sequences, predicting a variety of protein functions and properties. However, protein language has key differences from natural language, such as a rich functional space despite a vocabulary of only 20 amino acids. These differences motivate research into how transformer-based architectures operate differently in the protein domain and how we can better leverage PLMs to solve protein-related tasks. In this work, we begin by directly comparing how the distribution of information stored across layers of attention heads differs between the protein and natural language domain. Furthermore, we adapt a simple early-exit technique-originally used in the natural language domain to improve efficiency at the cost of performance-to achieve both increased accuracy and substantial efficiency gains in protein non-structural property prediction by allowing the model to automatically select protein representations from the intermediate layers of the PLMs for the specific task and protein at hand. We achieve performance gains ranging from 0.4 to 7.01 percentage points while simultaneously improving efficiency by over 10 percent across models and non-structural prediction tasks. Our work opens up an area of research directly comparing how language models change behavior when moved into the protein domain and advances language modeling in biological domains.

Anna Hart, Chi Han, Jeonghwan Kim +2

Siebel School of Data Science and Computing, University of Illinois Urbana-Champaign, 201 N Goodwin Ave, Urbana, IL 61801, USA · DOE Center for Advanced Bioenergy and Bioproducts Innovation, University of Illinois Urbana-Champaign, 1206 W Gregory Drive, Urbana, IL 61801, USA · Chemical and Biomolecular Engineering, University of Illinois Urbana-Champaign, 600 S Matthews Ave, Urbana, IL 61801, USA

Protein Language Models Protein Representation Learning Molecular Property Prediction

Ask PDF

Mar 8, 2026

Reverse Distillation: Consistently Scaling Protein Language Model Representations

Unlike the predictable scaling laws in natural language processing and computer vision, protein language models (PLMs) scale poorly: for many tasks, models within the same family plateau or even decrease in performance, with mid-sized models often outperforming the largest in the family. We introduce Reverse Distillation, a principled framework that decomposes large PLM representations into orthogonal subspaces guided by smaller models of the same family. The resulting embeddings have a nested, Matryoshka-style structure: the first k dimensions of a larger model's embedding are exactly the representation from the smaller model. This ensures that larger reverse-distilled models consistently outperform smaller ones. A motivating intuition is that smaller models, constrained by capacity, preferentially encode broadly-shared protein features. Reverse distillation isolates these shared features and orthogonally extracts additional contributions from larger models, preventing interference between the two. On ProteinGym benchmarks, reverse-distilled ESM-2 variants outperform their respective baselines at the same embedding dimensionality, with the reverse-distilled 15 billion parameter model achieving the strongest performance. Our framework is generalizable to any model family where scaling challenges persist. Code and trained models are available at https://github.com/rohitsinghlab/plm_reverse_distillation.

Darius Catrina, Christian Bepler, Samuel Sledzieski +1

Department of Computer Science, Duke University, Durham, NC, USA · Depts of Biostats. and Bioinformatics; and Cell Biology, Duke University, Durham, NC, USA

Protein Language Models Protein Representation Learning Knowledge Distillation Matryoshka Representation Learning

Ask PDF