Self-Supervised Pre-Training

Follow this topic Ask about this topic

Self-supervised pre-training studies how models can learn useful structure from unlabeled data, making it possible to adapt them to new tasks with fewer annotations. Common approaches include masked autoencoders, contrastive learning, teacher-student distillation, and joint-embedding predictive architectures such as JEPA, applied to images, signals, video, text, and scientific data. Researchers test these representations through frozen-feature probes, fine-tuning, and few-label or cross-domain transfer on benchmarks in areas such as medical imaging, speech, and autonomous driving. Results show that pre-training can improve data efficiency and robustness, but its benefits depend on the match between objective, data, and downstream task; greater model size or training time does not reliably help.

Latest papers

251 papers34 papers in the last four weeks, level with the four weeks before. 0.2% of all new papers.

Tracking the emergence of linguistic structure in self-supervised models learning from speech
Apr 2, 2026Marianne de Heer Kloots, Martijn Bentum, Hosein Mohebbi +3Self-Supervised Speech Representation Learning Self-Supervised Pre-Training
Institute for Logic, Language and Computation, University of Amsterdam, The Netherlands · Centre for Language Studies, Radboud University, The Netherlands · Cognitive Science and Artificial Intelligence, Tilburg University, The Netherlands
NEMESIS: Noise-suppressed Efficient MAE with Enhanced Superpatch Integration Strategy
Apr 2, 2026Kyeonghun Kim, Hyeonseok Jung, Youngung Han +9Medical Image Representation Learning Masked Autoencoders
OUTTA · Chung-Ang University · Seoul National University +3
PI-JEPA: Label-Free Surrogate Pretraining for Coupled Multiphysics Simulation via Operator-Split Latent Prediction
Apr 1, 2026Brandon Yee, Pairie KohMultiphysics Simulation PDE Surrogate Modeling
Physics Lab, Yee Collins Research Group · Graduate School of Business, Stanford University
MAESIL: Masked Autoencoder for Enhanced Self-supervised Medical Image Learning
Apr 1, 2026Kyeonghun Kim, Hyeonseok Jung, Youngung Han +14Self-Supervised Pre-Training Medical Image Representation Learning
NVIDIA ATC, Taiwan
Mean Masked Autoencoder with Flow-Mixing for Encrypted Traffic Classification
Mar 31, 2026Xiao Liu, Xiaowei Fu, Fuxiang Huang +1Encrypted Traffic Classification Encrypted Network Traffic Analysis
School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China · School of Data Science, Lingnan University, Hong Kong, China
The Surprising Effectiveness of Noise Pretraining for Implicit Neural Representations
Mar 30, 2026Kushal Vyas, Alper Kayabasi, Daniel Kim +3Implicit Neural Representations Neural Network Initialization
Rice University · University of California, Riverside
On the Asymptotics of Self-Supervised Pre-training: Two-Stage M-Estimation and Representation Symmetry
Mar 29, 2026Mohammad Tinati, Stephen TuSelf-Supervised Pre-Training Invariant Representation Learning
Department of Electrical and Computer Engineering, University of Southern California
MOOZY: A Patient-First Foundation Model for Computational Pathology
Mar 27, 2026Yousef Kotp, Vincent Quoc-Huy Trinh, Christopher Pal +1Computational Pathology Self-Supervised Pre-Training
CSSE, Concordia University, Montréal, Canada · Mila – Québec AI Institute, Montréal, Canada · CHUM, Université de Montréal, Montréal, Canada +4
A Human-Inspired Decoupled Architecture for Efficient Audio Representation Learning
Mar 27, 2026Harunori Kawano, Takeshi SasakiAudio Representation Learning Self-Supervised Pre-Training
Faculty of Engineering and Information Technology, University of Technology Sydney · Dept. of Electrical Engineering and Computer Science, Shibaura Institute of Technology
Rethinking Masking Strategies for Masked Prediction-based Audio Self-supervised Learning
Mar 25, 2026Daisuke Niizumi, Daiki Takeuchi, Masahiro Yasuda +3Audio Representation Learning Self-Supervised Pre-Training
Tokyo Metropolitan University, Tokyo, Japan · NTT Inc., Atsugi, Japan
Rethinking UMM Visual Generation: Masked Modeling for Efficient Image-Only Pre-training
Mar 17, 2026Peng Sun, Jun Xie, Tao LinUnified Multimodal Models Multimodal Pretraining
Zhejiang University · Westlake University · Shanghai Innovation Institute
A Tutorial on ALOS2 SAR Utilization: Dataset Preparation, Self-Supervised Pretraining, and Semantic Segmentation
Mar 16, 2026Nevrez Imamoglu, Ali Caglayan, Toru KouyamaSelf-Supervised Pre-Training Masked Autoencoders
National Institute of Advanced Industrial Science and Technology
Learning Generalizable 3D Medical Image Representations from Mask-Guided Self-Supervision
Mar 14, 2026Yunhe Gao, Yabin Zhang, Chong Wang +5Medical Image Representation Learning Self-Supervised Pre-Training
Stanford University
LLM2Vec-Gen: Generative Embeddings from Large Language Models
Mar 11, 2026Parishad BehnamGhader, Vaibhav Adlakha, Fabian David Schmidt +3Text Embedding Models Self-Supervised Pre-Training
McGill University · Mila–Quebec AI Institute · ServiceNow Research +3
Where Do Flow Semantics Reside? A Protocol-Native Tabular Pretraining Paradigm for Encrypted Traffic Classification
Mar 9, 2026Sizhe Huang, Zitong Li, Shujie YangEncrypted Traffic Classification Encrypted Network Traffic Analysis
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China.
SIGMAE: A Spectral-Index-Guided Foundation Model for Multispectral Remote Sensing
Mar 8, 2026Xiaokang Zhang, Bo Li, Chufeng Zhou +2Remote Sensing Image Understanding Remote Sensing
School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China · School of Artificial Intelligence and Automation, Wuhan University of Science and Technology, Wuhan, 430081, China · School of Electronic Information, Wuhan University of Science and Technology, Wuhan, 430081, China +2
Efficient Chest X-ray Representation Learning via Semantic-Partitioned Contrastive Learning
Mar 7, 2026Wangyu Feng, Shawn Young, Lijian XuMedical Image Representation Learning Self-Supervised Pre-Training
Shenzhen University of Advanced Technology, Shenzhen, China
ICHOR: A Robust Representation Learning Approach for ASL CBF Maps with Self-Supervised Masked Autoencoders
Mar 5, 2026Xavier Beltran-Urbano, Yiran Li, Xinglin Zeng +10Medical Image Representation Learning Self-Supervised Pre-Training
University of Pennsylvania, Philadelphia, USA · University of Maryland School of Medicine, Baltimore, USA
Scaling Dense Event-Stream Pretraining from Visual Foundation Models
Mar 4, 2026Zhiwen Chen, Junhui Hou, Zhiyu Zhu +2Event-Based Vision Representation Learning
City University of Hong Kong · Xidian University
Utonia: Toward One Encoder for All Point Clouds
Mar 3, 2026Yujia Zhang, Xiaoyang Wu, Yunhan Yang +63D Foundation Models 3D ViTs
The University of Hong Kong · Xiaomi · The Chinese University of Hong Kong
ScribeTokens: Fixed-Vocabulary Tokenization of Digital Ink
Mar 3, 2026Douglass WangOnline Handwriting Recognition Self-Supervised Pre-Training
Independent Researcher
NeighborMAE: Exploiting Spatial Dependencies between Neighboring Earth Observation Images in Masked Autoencoders Pretraining
Mar 3, 2026Liang Zeng, Valerio Marsocci, Wufan Zhao +2Self-Supervised Pre-Training Masked Autoencoders
KU Leuven · ESA Φ-lab · HKUST(GZ) +1
Rigidity-Aware Geometric Pretraining for Protein Design and Conformational Ensembles
Mar 2, 2026Zhanghan Ni, Yanjing Li, Zeju Qiu +4Protein Structure Generation Generative Modeling
University of Illinois Urbana-Champaign · University of Washington · MPI for Intelligent Systems, Tübingen +3
OSF: On Pre-training and Scaling of Sleep Foundation Models
Feb 27, 2026Zitao Shuai, Zongzhe Xu, David Yang +2Self-Supervised Pre-Training Scaling Laws
University of California, Los Angeles · Emory University
An Analysis of Role Semantics in Directed Dynamic Graph Modeling for Edge Classification
Feb 26, 2026Tyler Bonnet, Marek ReiGraph Representation Learning Self-Supervised Pre-Training
Imperial College London
RhythmBERT: A Self-Supervised Language Model Based on Latent Representations of ECG Waveforms for Heart Disease Detection
Feb 26, 2026Xin Wang, Burcu Ozek, Aruna Mohan +3Electrocardiogram Representation Learning Electrocardiogram Foundation Models
Virginia Tech, VA, USA · Walkky LLC, Boston, MA, USA · Northeastern University, Boston, MA, USA +1
Learning to Drive is a Free Gift: Large-Scale Label-Free Autonomy Pretraining from Unposed In-The-Wild Videos
Feb 25, 2026Matthew Strong, Wei-Jer Chang, Quentin Herau +4Autonomous Driving Autonomous Driving Perception
Applied Intuition · Stanford University · UC Berkeley
GeoPT: Scaling Physics Simulation via Lifted Geometric Pre-Training
Feb 23, 2026Haixu Wu, Minghao Guo, Zongyi Li +4Physics-Based Simulation Neural Surrogate Modeling
MIT CSAIL · Tsinghua University
Faster Training, Fewer Labels: Self-Supervised Pretraining for Fine-Grained BEV Segmentation
Feb 20, 2026Daniel Busch, Christian Bohn, Thomas Kurbiel +3Bird's-Eye-View Segmentation Autonomous Driving Perception
University of Wuppertal · APTIV
MeDUET: Disentangled Unified Pretraining for 3D Medical Image Synthesis and Analysis
Feb 19, 2026Junkai Liu, Ling Shao, Le ZhangDisentangled Representation Learning Self-Supervised Pre-Training
School of Engineering, College of Engineering and Physical Sciences, University of Birmingham, Birmingham, England, United Kingdom · UCAS-Terminus AI Laboratory, University of Chinese Academy of Sciences, Beijing 100049, China

Apr 2, 2026J/K move · Enter open · S save

Tracking the emergence of linguistic structure in self-supervised models learning from speech

Marianne de Heer Kloots, Martijn Bentum, Hosein Mohebbi, Charlotte Pouw+2

Institute for Logic, Language and Computation, University of Amsterdam, The Netherlands · Centre for Language Studies, Radboud University, The Netherlands · Cognitive Science and Artificial Intelligence, Tilburg University, The Netherlands

Self-Supervised Speech Representation Learning Self-Supervised Pre-Training

Open paper AskPDF arXiv

Self-supervised speech models learn effective representations of spoken language, which have been shown to reflect various aspects of linguistic structure. But when does such structure emerge in model training? We study the encoding of a wide range of linguistic structures, across layers and intermediate checkpoints of six Wav2Vec2 and HuBERT models trained on spoken Dutch. We find that different levels of linguistic structure show notably distinct layerwise patterns as well as learning trajectories, which can partially be explained by differences in their degree of abstraction from the acoustic signal and the timescale at which information from the input is integrated. Moreover, we find that the level at which pre-training objectives are defined strongly affects both the layerwise organization and the learning trajectories of linguistic structures, with greater parallelism induced by higher-order prediction tasks (i.e. iteratively refined pseudo-labels).