Self-Supervised Pre-Training

Ask about this topic

Overview

Self-supervised pre-training studies how models can learn useful structure from unlabeled data, making it possible to adapt them to new tasks with fewer annotations. Common approaches include masked autoencoders, contrastive learning, teacher-student distillation, and joint-embedding predictive architectures such as JEPA, applied to images, signals, video, text, and scientific data. Researchers test these representations through frozen-feature probes, fine-tuning, and few-label or cross-domain transfer on benchmarks in areas such as medical imaging, speech, and autonomous driving. Results show that pre-training can improve data efficiency and robustness, but its benefits depend on the match between objective, data, and downstream task; greater model size or training time does not reliably help.

Latest papers

CardsList

251 papers34 papers in the last four weeks, level with the four weeks before. 0.2% of all new papers.

  1. SpectralTF-MAE: A Pre-trained Spectral Foundation Model for Tokamak Fusion Plasma Diagnostics

    Oct 28, 2026Xiao Wang, Zhijie Lu, Duoqing Yang +7Representation LearningSelf-Supervised Pre-Training

    School of Computer Science and Technology, Anhui University, Hefei 230601, China · Institute of Plasma Physics, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China · New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China

  2. Systematic Morphometric Brain Autoencoding for Representation Learning in Neurodegenerative Diseases

    Oct 27, 2026Nuria Balbás-Esteban, Albert Belenguer-Llorens, Elaheh Moradi +3Medical Image Representation LearningSelf-Supervised Pre-Training

    Department of Signal Processing and Communications, Universidad Carlos III de Madrid, Leganés, 28911, Madrid, Spain · Department of Health Sciences and Technology (DHEST), ETH Zurich, 8092, Zürich, Switzerland · SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland +2

  3. SuperJEPA: Pretraining with a Supervision-Augmented Joint-Embedding Predictive Architecture for High Energy Physics

    Oct 26, 2026Dylan Sheldon Rankin, Ho Fung TsoiSelf-Supervised Pre-TrainingJoint-Embedding Predictive Architecture

    Department of Physics and Astronomy, University of Pennsylvania, Philadelphia, PA, USA

  4. Self-Supervised Pretraining for PHM: What Transfers, What Does Not, and What Federating It Costs

    Oct 26, 2026José Sanz-Durán, David Solís-MartínSelf-Supervised Pre-TrainingPredictive Maintenance

    Department of Computer Science and Artificial Intelligence, Universidad de Sevilla, Seville, Spain

  5. QECGBERT: A Scalable ECG Foundation Model Based on Self-Supervised Language Learning via VQ-VAE Waveform Tokenization

    Oct 25, 2026Aruna Mohan, Burcu Ozek, Or Zilbershot +1Electrocardiogram Foundation ModelsElectrocardiogram Representation Learning

    Walkky LLC, Boston, MA, USA · Clemson University, Clemson, SC, USA

  6. Exploring Joint-Embedding Predictive Architectures for Self-Supervised Representation Learning of Powder X-ray Diffraction Patterns

    Oct 23, 2026Ming-Chiang Chang, Brian DeCostRepresentation LearningSelf-Supervised Pre-Training

    National Institute of Standards and Technology 100 Bureau Dr Gaithersburg, MD 20899 USA

  7. Random Initialization as a Baseline for Time Series Foundation Models

    Oct 22, 2026Pinar Sungu Isiacik, Georgiana IfrimTime-Series Foundation ModelsSelf-Supervised Pre-Training

    University College Dublin, Dublin, Ireland

  8. BrainCodes: A Lightweight EEG Foundation Model over Discrete Neural Codes

    Oct 21, 2026Kleanthis Avramidis, Ivan Tashev, Dimitra EmmanouilidouEEG Foundation ModelsRepresentation Learning

    Signal Analysis and Interpretation Lab, University of Southern California, Los Angeles, CA 90007, USA · Microsoft Research, Redmond, WA, USA

  9. Brain2Vec: A Self-Supervised 3D Foundation Model for Structural Brain MRI

    Oct 20, 2026Spiros Maggioros, Guray Erus, Gareth Harman +4Medical Imaging Foundation ModelsMedical Image Representation Learning

    University of Pennsylvania

  10. Foundation-Style Pretraining for Scalable and Transferable Distributed PV Forecasting

    Oct 20, 2026Dayin Chen, Xiaodan Shi, Xin Xu +6Solar Power ForecastingTime-Series Foundation Models

    Department of Building Environment and Energy Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China · International Centre of Urban Energy Nexus, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong SAR, China · Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo, 315200, China +2

  11. MiCLA: Mixture-to-Constituent Latent Alignment for Polyphonic Audio Representation Learning

    Oct 20, 2026Kohei Yamamoto, Kosuke OkusaAudio Representation LearningSelf-Supervised Pre-Training

    Research & Development Center, Technology Division, Oki Electric Industry Co., Ltd., Saitama, Japan. · Department of Data Science for Business Innovation, Chuo University, Tokyo, Japan.

  12. PRISM: Prior Relational Information for Self-supervised Modeling to Enhance Solubility OOD Generalization

    Oct 19, 2026Xinyi Chen, Jiahuan Pang, Titus Chima +2Molecular Property PredictionSelf-Supervised Pre-Training

    Global College, Shanghai Jiao Tong University, Shanghai, China. · Digital Innovation Research Center, Duke Kunshan University, Kunshan, Jiangsu, China. · Zu Chongzhi Center, Duke Kunshan University, Kunshan, Jiangsu, China. +1

  13. Data-Efficient Classifier Pretraining Through Diffusion Models

    Oct 16, 2026Qinyu Li, Yee Whye Teh, Chris Holmes +1Data-Efficient LearningSelf-Supervised Pre-Training

    Department of Statistics, University of Oxford · Faculty of Medicine and Health Technology, Tampere University · Tampere Institute for Advanced Study, Tampere University

  14. Data-Efficient ConvNeXt Pretraining for High-Resolution Sentinel-1 GRD SAR Representation Learning

    Oct 15, 2026Yuki Natsume, Sang-Ho Yun, Chai Kiat YeoSelf-Supervised Pre-TrainingMasked Autoencoders

    College of Computing and Data Science, Nanyang Technological University, Singapore · Earth Observatory of Singapore, Nanyang Technological University, Singapore · Asian School of the Environment, Nanyang Technological University, Singapore +1

  15. L2GSSL: Label-Efficient Local-to-Global Self-Supervised Learning for Volumetric Healthcare Data

    Oct 15, 2026Po-Cheng Cheng, Hung-Hsuan ChenLung Nodule DetectionMedical Image Representation Learning

    Computer Science and Information Engineering, National Central University

  16. GeoNIRS: Geometry-Aware Pretraining for Heterogeneous fNIRS Representation Learning

    Oct 13, 2026Zongsheng Li, Shengyuan Zhu, Zimo Zhang +3Representation LearningNeuroimaging

    Southern University of Science and Technology, Shenzhen, China · The Chinese University of Hong Kong, Shenzhen, China · Omni-Intelligence, Shenzhen, China +1

  17. Benchmarking Self-Supervised 3D Medical Vision Foundation Models and Transferability Estimation

    Oct 13, 2026Jiaqi Tang, Yu Wang, Yang Liu +7Medical Imaging Foundation ModelsMedical Image Benchmarks

    National Institution of Health Data Science, Peking University, Beijing, China. · Institute of Medical Technology, Peking University, Beijing, China. · State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China. +5

  18. On Sleep-Wake Disorder Identification from Polysomnographic Representations

    Oct 12, 2026Alvise Dei Rossi, Luigi Fiorillo, Giuliana Monachino +6Self-Supervised Pre-TrainingSelf-Supervised Time Series Representation Learning

    Faculty of Informatics, Università della Svizzera Italiana · MeDiTech, SUPSI · Ente Ospedaliero Cantonale (EOC) +2

  19. Sequence Composition in EHR Foundation Model Pretraining

    Oct 12, 2026Joshua Placidi, Inigo Stibbe, Alfred J. Balston +2EHR Foundation ModelsElectronic Health Records

    Department of Computing, Imperial College London

  20. Growing to See: Efficient Pretraining of Self-Supervised Vision Transformers

    Oct 12, 2026Amir Mohammad Karimi Mamaghan, Ferdinand Kapl, Andrea Dittadi +4Self-Supervised Visual Representation LearningSelf-Supervised Pre-Training

    KTH Royal Institute of Technology · Helmholtz AI, Munich · Technical University of Munich +3

  21. HyBRA-EEG: Hyperbolic Global-Local Brain-Region Aggregation for Self-Supervised EEG Representation Learning

    Oct 12, 2026Xin Liu, Feng Lu, Wei LiElectroencephalographySelf-Supervised Pre-Training

  22. Learning Transferable World-Action Models from Task-Paired Human Videos

    Oct 9, 2026Jeongeun Park, Taekyung Kim, Dongyoon Han +1World Model LearningAction-Conditioned World Models

    NAVER AI Lab · Georgia Institute of Technology

  23. Where to Look Matters: Rethinking Sub-Volume Sampling in 3D Medical Self-supervised Learning

    Oct 9, 2026Junkai Liu, Le ZhangMedical Image Representation LearningSelf-Supervised Pre-Training

    School of Engineering, University of Birmingham

  24. Mid-Training Language Models on Raw Video

    Oct 8, 2026Jaedong Hwang, Xiaoqian Shen, Ernie Chang +9Multimodal Large Language ModelsVideo-Language Models

    Meta AI · Massachusetts Institute of Technology · Work done at Meta +2

  25. MorphCL: Morphological Contrastive Learning for Inertial-based Human Activity Recognition

    Oct 7, 2026Marius Bock, Yuwei Zhang, Juergen Gall +3Human Activity RecognitionMotion Representation Learning

    University of Bonn · University of Cambridge · Lamarr Institute for Machine Learning and Artificial Intelligence +1

  26. HuC-VideoMAE: Human-Centric Video Masked Autoencoding from synthetic data

    Oct 6, 2026Ricardo Pizarro, Roberto Valle, José M. Buenaposada +2Synthetic Data PretrainingMasked Autoencoders

    Universidad de Alcal´a, Alcal´a de Henares, Spain · Universidad Polit´ecnica de Madrid, Madrid, Spain · Universidad Rey Juan Carlos, M´ostoles, Spain

  27. SpecBraM: What Should an EEG Foundation Model Predict? Masked Band-Power Prediction versus Waveform Reconstruction

    Oct 5, 2026Peng Xie, Yequan Bie, Jianda Mao +1ElectroencephalographySleep Stage Classification

    The Hong Kong University of Science and Technology

  28. dIon: Fragmentation-Based Invariance for Self-Supervised Learning of Tandem Mass Spectra

    Oct 5, 2026Alfred Nilsson, Joel Lapin, Samuel H. Payne +2Self-Supervised Pre-TrainingMass Spectrometry

    Science for Life Laboratory, KTH Royal Institute of Technology Stockholm, Sweden · Computational Mass Spectrometry, TUM School of Life Sciences Technical University of Munich, Freising, Germany · Biology Department, Brigham Young University Provo, Utah 84602, United States +1

  29. Lightweight Semantic EEG Foundation Model for Frozen Cross-Disorder Transfer

    Oct 4, 2026Rita Huan-Ting Peng, Nhat BuiEEG Seizure DetectionSelf-Supervised Pre-Training

    University of Illinois Urbana–Champaign, Urbana, IL, USA · Carle Foundation Hospital, Urbana, IL, USA

  30. DIVINE: Simple Cross-Market Stock Pretraining via Diverse Indicator Reconstruction

    Oct 2, 2026Kuan-Yu Chen, Shu-Cheng Zheng, Yu-Chen Den +2Financial Time SeriesSelf-Supervised Pre-Training

    SinoPac Holdings, Taipei, Taiwan