q-bio.QMNov 12, 2026

Protein Language Models Fail To Capture Contact Predictions Under Invariant Relabeling

Authors: Paarth Batra, Aryan Keluskar

Organizations: The Biodesign Institute Tempe, AZ 85281, USA

Abstract

Protein language models (pLMs) produce highly accurate contact maps from single-sequence inputs. People often take this accuracy as proof that these models implicitly learn invariant coevolutionary statistics, extracting the same signals as classical alignment-based methods like Potts. In this work, we introduce a diagnostic framework around a global bijective relabeling of the amino-acid alphabet. This transformation permutes residue letters while leaving all alignment-level information, column entropies, and pairwise edit distances unchanged. Pseudolikelihood Potts models and inverse-covariance estimators remain invariant under this transformation, but pLMs fail the test. Across multiple architectures (ESM-2, MSA Transformer, AMPLIFY) and model scales (8M to 3B parameters), pLM contact maps collapse under arbitrary relabeling. Using Hamming distance controls and biochemical grouping interventions, we attempt to isolate what likely causes this degradation and find that these models rely on explicit, canonical residue identities (``alphabet anchoring'') rather than pure coevolutionary statistics. Increasing scale makes models more responsive to biochemical classes, but their performance still degrades under relabeling. Our findings show a gap between how pLMs and classical coevolutionary models compute contacts. More broadly, our work shows how diagnostic perturbations that leave classical models provably unchanged can expose hidden representation biases and prevent over-reliance on neural black boxes in biological modeling.

Explore similar work

CardsList
  1. recAMPLIFY: Large-Scale Pretraining of Recursive Protein Language Models

    Oct 19, 2026Saurav Jha, Qianli Liao, Robert Vernon +2Protein Language ModelsRecursive Language Models

  2. Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference

    Feb 24, 2026Anna Hart, Chi Han, Jeonghwan Kim +2Protein Language ModelsProtein Representation Learning

  3. Task- and dataset-specific information in protein language models

    Aug 12, 2026Roman Joeres, Ilya Senatorov, Anastasia Kolchina +2Protein Language ModelsProtein Representation Learning