Protein Representation Learning

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 36

All topics

ProteinJEPA: Latent prediction improves protein language model pretraining
May 8, 2026Dan Ofer, Dafna Shahaf, Michal LinialLanguage Model Pretraining Masked Language Modeling
Department of Biological Chemistry The Hebrew University of Jerusalem · School of Computer Science and Engineering The Hebrew University of Jerusalem
ProtSent: Protein Sentence Transformers
May 7, 2026Dan Ofer, Oriel Perets, Michal Linial +1Contrastive Learning Protein Language Models
Department of Computer and Information Science Ben-Gurion University of the Negev
Retrieval and competition: how a protein foundation model starts a protein
May 5, 2026Piotr Jedryszek, Oliver M. CrookProtein Language Models Mechanistic Interpretability
Department of Biology, University of Oxford, Oxford, UK · Kavli Institute for Nanoscience Discovery, University of Oxford, Oxford, UK · Department of Chemistry, University of Oxford, Oxford, UK
PRIME: Protein Representation via Physics-Informed Multiscale Equivariant Hierarchies
May 2, 2026Viet Thanh Duy Nguyen, John K. Johnstone, Truong-Son HyGraph Representation Learning Hierarchical Representation Learning
University of Alabama at Birmingham Birmingham, Alabama, USA
PUMA: Learning a Mutation-Aware Vocabulary of Protein Units
Mar 11, 2025Burak Suyunu, Özdeniz Dolu, Ibukunoluwa Abigail Olaosebikan +2Protein Language Models Protein Representation Learning
Department of Computer Engineering, Boaziçi University, 34342, Bebek, Istanbul, Turkey · C. Eugene Bennett Department of Chemistry, West Virginia University, Morgantown, West Virginia 26505, United States · Department of Pharmaceutical Chemistry, Istanbul Medipol University, Faculty of Pharmacy, 34815 Beykoz, Istanbul, Turkey +1
When do cheap embeddings beat protein language models? A theoretically-grounded hashing sketch for biological sequence classification
Date pendingSarwan Ali, Taslim Murad, Imdadullah Khan +1Protein Language Models Randomized Sketching
Columbia University, NY, USA · Institute of Business Administration, Karachi, Pakistan · Lahore University of Management Sciences, Lahore, Pakistan +1

May 8, 2026, cs.LGJ/K move · Enter open · S save

ProteinJEPA: Latent prediction improves protein language model pretraining

Dan Ofer, Dafna Shahaf, Michal Linial

Department of Biological Chemistry The Hebrew University of Jerusalem · School of Computer Science and Engineering The Hebrew University of Jerusalem

Language Model Pretraining Masked Language Modeling Joint-Embedding Predictive Architecture Protein Language Models Protein Representation Learning

Open paper AskPDF arXiv

Protein language models are trained primarily with masked language modeling (MLM), which predicts masked amino-acid identities. Joint-embedding predictive architectures (JEPA) instead predict latent representations, but have not been applied to proteins. ProteinJEPA supplements MLM with a cosine loss for predicting the half-depth hidden states of a teacher given the unmasked sequence. On 19 tasks, with ESM2 at 35M and 150M parameters and three pretraining seeds, MLM+JEPA outperforms compute-matched and step-matched MLM-only continued training in 78 and 76 of 114 comparisons (14 losses, 22 ties). The median compute-matched gain is

+0.0106

on structure- and homology-sensitive tasks versus

+0.0041

elsewhere, led by SCOPe-40 retrieval and remote homology with improvements of 6.1 percentage points in Recall@1 and 2.7 points in accuracy, respectively. Gains on these tasks increase with model size from 8M to 150M. Against the off-the-shelf checkpoint, MLM+JEPA wins 81 of 114 comparisons (median

+0.0068

) without improving MLM loss. In random initialization the gain is smaller and replicates inconsistently across seeds (

p{=}0.059

). The same recipe improves the causal ProGen3 model, beating a compute-matched next-token-prediction control on 12 of 16 tasks. Ablations show that cosine loss beats mean squared error, while adding shallower targets removes most of the task gain. JEPA-only training collapses downstream performance: latent prediction complements MLM rather than replacing it. Code: https://anonymous.4open.science/r/protJepa-FF24

Momentum

Related topics

Latest papers 36

ProteinJEPA: Latent prediction improves protein language model pretraining

ProtSent: Protein Sentence Transformers

Retrieval and competition: how a protein foundation model starts a protein

PRIME: Protein Representation via Physics-Informed Multiscale Equivariant Hierarchies

PUMA: Learning a Mutation-Aware Vocabulary of Protein Units

When do cheap embeddings beat protein language models? A theoretically-grounded hashing sketch for biological sequence classification