Joint-Embedding Predictive Architecture

Also known as JEPA

Follow this topic Ask about this topic

Momentum

32 papers in the last four weeks, up 146% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 158

All topics

Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning
Apr 16, 2026Aimen BoukhariRepresentation Geometry Masked Language Modeling
´Ecole Nationale Sup´erieure d’Informatique (ESI), Algiers , Algeria
US-JEPA: A Joint Embedding Predictive Architecture for Ultrasound
Feb 22, 2026Ashwath Radhachandran, Vedrana Ivezić, Shreeram Athreya +2Joint-Embedding Predictive Architecture Ultrasound Imaging
Biomedical Artificial Intelligence Research Lab, University of California, Los Angeles, Los Angeles, United States of America · Medical Informatics, University of California, Los Angeles, Los Angeles, United States of America · Department of Electrical and Computer Engineering, University of California, Los Angeles, Los Angeles, United States of America +3
JEPA-Bisim: Learning Robust Visual Representations for Planning with Joint-Embedding Predictive World Models
Feb 20, 2026Leonardo F. Toso, Davit Shadunts, Yunyang Lu +5Visual Representation Learning World Models for Robotics
Department of Electrical Engineering, Columbia University, New York, USA. · Capital One, USA.
SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos
Feb 5, 2026Jinlin Wu, Felix Holm, Chuxi Chen +18Joint-Embedding Predictive Architecture Surgical Video Understanding
Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany +7
Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving
Jan 29, 2026Linhan Wang, Zichong Yang, Chen Bai +6End-to-End Autonomous Driving Joint-Embedding Predictive Architecture
Virginia Tech · Purdue University · XPENG Motors +1
RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture
Jan 22, 2026Anas Anwarul Haq Khan, Mariam Husain, Pratik Jalan +1Radiology Report Generation Medical Imaging
Department of Computer Science (CSE), Indian Institute of Technology Bombay, India · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Department of Biomedical Engineering, Johns Hopkins University, USA +1
What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?
Dec 30, 2025Basile Terver, Tsung-Yen Yang, Jean Ponce +2World Model-Based Planning Joint-Embedding Predictive Architecture
Meta FAIR · Inria Paris · Ecole normale supérieure / PSL +1
ACT-JEPA: Novel Joint-Embedding Predictive Architecture for Efficient Policy Representation Learning
Jan 24, 2025Aleksandar Vujinovic, Aleksandar KovacevicRepresentation Learning World Model Learning
Faculty of Technical Sciences, University of Novi Sad

Apr 16, 2026, cs.CLJ/K move · Enter open · S save

Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning

Aimen Boukhari

´Ecole Nationale Sup´erieure d’Informatique (ESI), Algiers , Algeria

Representation Geometry Masked Language Modeling Self-Supervised Learning Joint-Embedding Predictive Architecture Predictive Representation Learning Representation Geometry in Language Models

Open paper AskPDF arXiv

Masked language modelling (MLM) has been the dominant pre-training objective for text encoders since BERT, yet it encourages representations that are strongly anchored to surface-form token identity rather than deeper semantic structure. Inspired by the success of Joint Embedding Predictive Architectures (JEPA) (LeCun, 2022) in vision and audio, we propose a hybrid pre-training objective that combines a JEPA-style latent-space prediction loss with a standard MLM objective over a single shared encoder. A learnable scalar parameter continuously balances the two objectives during training. We pre-train both a hybrid model and a pure-MLM baseline on English Wikipedia using identical architectures and compute budgets (NVIDIA H100). Extensive representation analysis across five GLUE benchmarks (SST-2, MRPC, MNLI, CoLA, STS-B) using four pooling strategies reveals that the hybrid encoder produces significantly more uniform embeddings (uniformity less than -0.16 vs -0.05 for MLM), exhibits richer spectral geometry under max pooling, encodes less surface-level lexical information, and achieves a better semantic-to-lexical balance. Despite similar linear-probe downstream accuracy, the geometric differences are consistent and significant, suggesting that the JEPA predictive objective reshapes the latent space in ways that standard accuracy metrics alone cannot capture.

Momentum

Related topics

Latest papers 158