cs.LGFeb 25, 2026
SaveFrom Words to Amino Acids: Does the Curse of Depth Persist?
Organizations: Technical University of Munich · Helmholtz AI, Munich · KTH Royal Institute of Technology · Institute of Computational Biology, Helmholtz Munich · Google, Paradigms of Intelligence Team
Abstract
Protein language models (PLMs) have become widely adopted as general-purpose models, demonstrating strong performance in protein engineering and de novo design. Like large language models (LLMs), they are typically trained as deep transformers with next-token or masked-token prediction objectives on massive sequence corpora and are scaled by increasing model depth. Recent work on autoregressive LLMs has identified the Curse of Depth: many later layers contribute little to the final output predictions. These findings naturally raise the question of whether a similar depth inefficiency also appears in PLMs, where many widely used models are not autoregressive, and some are multimodal, accepting both protein sequence and structure as input. In this work, we present a depth analysis of seven popular PLM families across model scales, spanning autoregressive, masked, and diffusion objectives, and quantify how layer contributions evolve with depth using a unified set of probing-, perturbation-, and downstream-evaluation measurements. Across models, we observe consistent depth-dependent patterns that extend prior findings on LLMs: a large fraction of task-relevant computation is concentrated in a subset of layers, while the remaining layers mainly provide incremental refinement of the final prediction. These trends persist beyond sequence-only settings and also appear in multimodal PLMs. Taken together, our results suggest that depth inefficiency is a common feature of modern PLMs, motivating future work on more depth-efficient architectures and training methods.
Figures & tables
Figure 1: Maximum propagated effect of skipping each layer on future-token computations, alongside layer-layer CKA similarity, in ESM2. Here, “future” refers to a held-out subset of non-intervened masked tokens. Even at 35M, skipping later layers produces relatively weak propagated effects compared to skipping early layers. From 150M onward, this separation becomes clear: a substantial fraction of late layers can be skipped with only minor effects on subsequent computations on future tokens. Moreover, the low-effect regions also report high layer-layer similarity. This pattern strengthens with scale, indicating that downstream sensitivity increasingly concentrates in early-to-middle layers, while later layers mainly refine the final prediction. We also observe localized low-effect regions among early layers. This aligns with the stage-wise view of Lad et al. [21] and suggests that depth is organized into multiple inference stages with weaker dependencies across certain layer ranges.
Figure 2: TunedLens analysis for ESM2 across depth: KL divergence to the final output distribution (left) and top-1 overlap with the full-model prediction (right). Across ESM2 variants, KL divergence slowly decreases toward later layers, indicating that deeper layers make increasingly incremental updates that bring the distribution closer to the final output. Consistently, top-1 overlap is lower in early layers and increases toward the end of the network, with the most pronounced gains in the final layers for larger variants. Together, these trends suggest that the top prediction becomes increasingly stable with depth, while later layers mainly refine confidence over tokens rather than change the predicted token.
Figure 3: Layer-wise ProteinGym performance for ESM2. Average Spearman correlation as a function of relative depth, normalized to , where predictions are taken from each layer. Performance improves with depth for all model sizes, but the largest models exhibit diminishing returns in the final layers, suggesting that earlier layers already capture much of the signal and later layers mainly provide small refinements to the final predictions.
Figure 4: Depth analysis results for ESM3 for the structure stream. Left: skiplayer propagation and layer-layer similarity, and Right: LogitLens readouts. Aligned with the sequence-stream results, ESM3 shows weaker propagated effects in early layers, with stronger contributions emerging in mid-to-late layers.
Figure 5: Cross-modal layer–layer cosine similarity between sequence-only and structure-only representations for ESM3. Each cell compares a sequence-layer representation to a structure-layer representation for the same proteins, with similarities averaged across proteins. The first 60% of layers show high cross-modal similarity, suggesting strong early sequence-structure alignment. After this point, similarity becomes more localized, indicating a transition toward more modality-specific representations in later layers.
Appendix figures & tables39 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Maximum propagated effect of skipping each layer on future-token computations for DPLM.
Figure 7: Maximum propagated effect of skipping each layer on future-token computations for DPLM2.
Figure 8: Maximum propagated effect of skipping each layer on future-token computations for Profluent-E1.
Figure 9: Maximum propagated effect of skipping each layer on later-layer representations (future tokens only), for ESM3.
Figure 10: Maximum propagated effect of skipping each layer on future-token computations for ProGen2.
Figure 11: Maximum propagated effect of skipping each layer on future-token computations for ProGen3.
Figure 12: Maximum change in ESM2 output probabilities under layer skipping, restricted to future tokens only.
Figure 13: Maximum change in DPLM output probabilities under layer skipping, restricted to future tokens only.
Figure 14: Maximum change in DPLM2 output probabilities under layer skipping, restricted to future tokens only.
Figure 15: Maximum change in Profluent-E1 output probabilities under layer skipping, restricted to future tokens only.
Figure 16: Maximum change in ESM3 output probabilities under layer skipping, restricted to future tokens only.
Figure 17: Maximum change in ProGen2 output probabilities under layer skipping, restricted to future tokens only.
Figure 18: Maximum change in ProGen3 output probabilities under layer skipping, restricted to future tokens only.
Figure 19: TunedLens analysis for DPLM across depth: KL divergence between the layer-wise output distribution and the final output distribution (left), and top-1 overlap between the layer-wise prediction and the full-model prediction (right).
Figure 20: TunedLens analysis for DPLM2 across depth: KL divergence between the layer-wise output distribution and the final output distribution (left), and top-1 overlap between the layer-wise prediction and the full-model prediction (right).
Figure 21: TunedLens analysis for E1 across depth: KL divergence between the layer-wise output distribution and the final output distribution (left), and top-1 overlap between the layer-wise prediction and the full-model prediction (right).
Figure 22: TunedLens analysis for ESM3 across depth: KL divergence between the layer-wise output distribution and the final output distribution (left), and top-1 overlap between the layer-wise prediction and the full-model prediction (right).
Figure 23: TunedLens analysis for ProGen2 across depth: KL divergence between the layer-wise output distribution and the final output distribution (left), and top-1 overlap between the layer-wise prediction and the full-model prediction (right).
Figure 24: TunedLens analysis for ProGen3 across depth: KL divergence between the layer-wise output distribution and the final output distribution (left), and top-1 overlap between the layer-wise prediction and the full-model prediction (right).
Figure 25: Average Spearman correlation for DPLM on ProteinGym, calculated at each layer. The relative depth is normalized to .
Figure 26: Average Spearman correlation for DPLM2 on ProteinGym, calculated at each layer. The relative depth is normalized to .
Figure 27: Average Spearman correlation for Profluent-E1 on ProteinGym, calculated at each layer. The relative depth is normalized to .
Figure 28: Average Spearman correlation for ESM3 on ProteinGym, calculated at each layer. The relative depth is normalized to .
Figure 29: Average Spearman correlation for ProGen2 on ProteinGym, calculated at each layer. The relative depth is normalized to .
Figure 30: Average Spearman correlation for ProGen3 on ProteinGym, calculated at each layer. The relative depth is normalized to .
Figure 31: Average Spearman correlation for ESM2 on ProteinGym, computed at each layer and shown separately by phenotype. Relative depth is normalized to .
Figure 32: Average Spearman correlation for DPLM on ProteinGym, computed at each layer and shown separately by phenotype. Relative depth is normalized to .
Figure 33: Average Spearman correlation for DPLM2 on ProteinGym, computed at each layer and shown separately by phenotype. Relative depth is normalized to .
Figure 34: Average Spearman correlation for Profluent-E1 on ProteinGym, computed at each layer and shown separately by phenotype. Relative depth is normalized to .
Figure 35: Average Spearman correlation for ESM3 on ProteinGym, computed at each layer and shown separately by phenotype. Relative depth is normalized to .
Figure 36: Average Spearman correlation for ProGen2 on ProteinGym, computed at each layer and shown separately by phenotype. Relative depth is normalized to .
Figure 37: Average Spearman correlation for ProGen3 on ProteinGym, computed at each layer and shown separately by phenotype. Relative depth is normalized to .
Figure 38: Maximum propagated effect of skipping each layer on future-token computations of the multimodal (sequence-structure) stream for ESM3.
Figure 39: Maximum propagated effect of skipping each layer on future-token computations of the structure stream for DPLM2.
Figure 40: Maximum propagated effect of skipping each layer on future-token computations of the multimodal (sequence-structure) stream for DPLM2.
Figure 41: LogitLens analysis for ESM3 across depth on the multimodal (sequence-structure) stream: KL divergence between the layer-wise output distribution and the final output distribution (left), and top-1 overlap between the layer-wise prediction and the full-model prediction (right).
Figure 42: LogitLens analysis for DPLM2 across depth on the structure stream: KL divergence between the layer-wise output distribution and the final output distribution (left), and top-1 overlap between the layer-wise prediction and the full-model prediction (right).
Figure 43: LogitLens analysis for DPLM2 across depth on the multimodal (sequence-structure) stream: KL divergence between the layer-wise output distribution and the final output distribution (left), and top-1 overlap between the layer-wise prediction and the full-model prediction (right).
Figure 44: Cross-modal layer–layer cosine similarity between sequence-only and structure-only representations for DPLM2, averaged across proteins. Each cell compares a sequence-layer representation to a structure-layer representation for the same proteins.