Protein Representation Learning

Follow this topic Ask about this topic

Protein representation learning studies how to encode amino-acid sequences and three-dimensional structures so models can compare proteins and predict properties such as function, folding, interactions, and mutation effects. Common approaches include protein language models trained with masked-token prediction, geometric and graph encoders of residue contacts, structure-aware masking, and contrastive learning that aligns sequence with structure or function. These representations are tested on tasks such as remote homology, protein fitness, structure prediction, and binding, using both pooled protein embeddings and residue-level features. A central challenge is making representations transferable across proteins and data splits: larger pretrained models can help, but useful information varies by task and layer, and structural or evolutionary features may offer gains with less data or computation.

Latest papers

69 papers12 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Reverse Distillation: Consistently Scaling Protein Language Model Representations
Mar 8, 2026Darius Catrina, Christian Bepler, Samuel Sledzieski +1Protein Language Models Protein Representation Learning
Department of Computer Science, Duke University, Durham, NC, USA · Depts of Biostats. and Bioinformatics; and Cell Biology, Duke University, Durham, NC, USA
ProtAlign: Contrastive learning paradigm for Sequence and structure alignment
Mar 6, 2026Aditya Ranganath, Hasin Us Sami, Kowshik Thopalli +2Cross-Modal Alignment Protein Representation Learning
Center for Applied Scientific Computing, Lawrence Livermore National Laboratory
Rigidity-Aware Geometric Pretraining for Protein Design and Conformational Ensembles
Mar 2, 2026Zhanghan Ni, Yanjing Li, Zeju Qiu +4Protein Structure Generation Generative Modeling
University of Illinois Urbana-Champaign · University of Washington · MPI for Intelligent Systems, Tübingen +3
Induction Meets Biology: Mechanisms of Repeat Detection in Protein Language Models
Feb 26, 2026Gal Pomerants, Yaniv Nikankin, Anja Reusch +3Protein Language Models LLM Interpretability
Technion - Israel Institute of Technology · The Hebrew University of Jerusalem, Israel · Kempner Institute, Harvard University
From Words to Amino Acids: Does the Curse of Depth Persist?
Feb 25, 2026Aleena Siji, Amir Mohammad Karimi Mamaghan, Ferdinand Kapl +9Protein Language Models Protein Representation Learning
Technical University of Munich · Helmholtz AI, Munich · KTH Royal Institute of Technology +2
Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference
Feb 24, 2026Anna Hart, Chi Han, Jeonghwan Kim +2Protein Language Models Protein Representation Learning
Siebel School of Data Science and Computing, University of Illinois Urbana-Champaign, 201 N Goodwin Ave, Urbana, IL 61801, USA · DOE Center for Advanced Bioenergy and Bioproducts Innovation, University of Illinois Urbana-Champaign, 1206 W Gregory Drive, Urbana, IL 61801, USA · Chemical and Biomolecular Engineering, University of Illinois Urbana-Champaign, 600 S Matthews Ave, Urbana, IL 61801, USA
Exploring the limits of pre-trained embeddings in machine-guided protein design: a case study on predicting AAV vector viability
Feb 16, 2026Ana F. Rodrigues, Lucas Ferraz, Laura Balbi +2Protein Representation Learning Protein Language Models
LASIGE, Faculdade de Ciências da Universidade de Lisboa, Lisboa, Portugal
When do cheap embeddings beat protein language models? A theoretically-grounded hashing sketch for biological sequence classification
Dec 10, 2025Sarwan Ali, Taslim Murad, Imdadullah Khan +1Protein Language Models Randomized Sketching
Columbia University, NY, USA · Institute of Business Administration, Karachi, Pakistan · Lahore University of Management Sciences, Lahore, Pakistan +1
PUMA: Learning a Mutation-Aware Vocabulary of Protein Units
Mar 11, 2025Burak Suyunu, Özdeniz Dolu, Ibukunoluwa Abigail Olaosebikan +2Protein Language Models Protein Representation Learning
Department of Computer Engineering, Boaziçi University, 34342, Bebek, Istanbul, Turkey · C. Eugene Bennett Department of Chemistry, West Virginia University, Morgantown, West Virginia 26505, United States · Department of Pharmaceutical Chemistry, Istanbul Medipol University, Faculty of Pharmacy, 34815 Beykoz, Istanbul, Turkey +1

Mar 8, 2026J/K move · Enter open · S save

Reverse Distillation: Consistently Scaling Protein Language Model Representations

Darius Catrina, Christian Bepler, Samuel Sledzieski, Rohit Singh

Department of Computer Science, Duke University, Durham, NC, USA · Depts of Biostats. and Bioinformatics; and Cell Biology, Duke University, Durham, NC, USA

Protein Language Models Protein Representation Learning Knowledge Distillation Bioinformatics Matryoshka Representation Learning

Open paper AskPDF arXiv

Unlike the predictable scaling laws in natural language processing and computer vision, protein language models (PLMs) scale poorly: for many tasks, models within the same family plateau or even decrease in performance, with mid-sized models often outperforming the largest in the family. We introduce Reverse Distillation, a principled framework that decomposes large PLM representations into orthogonal subspaces guided by smaller models of the same family. The resulting embeddings have a nested, Matryoshka-style structure: the first k dimensions of a larger model's embedding are exactly the representation from the smaller model. This ensures that larger reverse-distilled models consistently outperform smaller ones. A motivating intuition is that smaller models, constrained by capacity, preferentially encode broadly-shared protein features. Reverse distillation isolates these shared features and orthogonally extracts additional contributions from larger models, preventing interference between the two. On ProteinGym benchmarks, reverse-distilled ESM-2 variants outperform their respective baselines at the same embedding dimensionality, with the reverse-distilled 15 billion parameter model achieving the strongest performance. Our framework is generalizable to any model family where scaling challenges persist. Code and trained models are available at https://github.com/rohitsinghlab/plm_reverse_distillation.