Self-Supervised Pre-Training

Overview

Self-supervised pre-training studies how models can learn useful structure from unlabeled data, making it possible to adapt them to new tasks with fewer annotations. Common approaches include masked autoencoders, contrastive learning, teacher-student distillation, and joint-embedding predictive architectures such as JEPA, applied to images, signals, video, text, and scientific data. Researchers test these representations through frozen-feature probes, fine-tuning, and few-label or cross-domain transfer on benchmarks in areas such as medical imaging, speech, and autonomous driving. Results show that pre-training can improve data efficiency and robustness, but its benefits depend on the match between objective, data, and downstream task; greater model size or training time does not reliably help.

Latest papers

CardsList

265 papers23 papers in the last four weeks, down 36% on the four weeks before. 0.2% of all new papers.

  1. BRepA: Graph-Based B-Rep Pretraining for Unified CAD Understanding and Generation

    Nov 17, 2026Dekai Zhu, Zhaiyu Chen, Stefan Gavranovic +13D Representation LearningSelf-Supervised Pre-Training

  2. Open-fMIND: A Large-Scale Open-Access fMRI Corpus for Pretraining Brain Foundation Models

    Nov 16, 2026Carlo Alberto Barbano, Giovanni Marraffini, Demian WassermannfMRI Foundation ModelsfMRI Representation Learning

  3. ScaleBrain: A Scale-Consistent Framework for EEG Foundation Models

    Nov 10, 2026Minyu Zhou, Yajing Wu, Yongqiang Tang +2Axial AttentionEEG Foundation Models

  4. ZeoBench: A Benchmark for Self-Supervised Learning on 3D Zeolite Representations

    Nov 6, 2026Aaron Sun, Yachan Liu, Ping Yang +23D Representation LearningSelf-Supervised Pre-Training

  5. Self-Supervised Learning of Universal Latent Spaces for Operator Learning

    Nov 3, 2026Sepehr Mousavi, Levi Lingsch, Siddhartha Mishra +1PDE Operator LearningNeural Operators

  6. Self-Supervised Pretraining for Label-Efficient Facial Expression Recognition using Smart Glasses

    Nov 3, 2026Zhengtao Gao, Francesco Bombassei De Bona, Leonardo Alchieri +6Facial Expression RecognitionData-Efficient Learning

  7. LaBraM at Scale: Scaling Data and Models for EEG Pretraining

    Nov 2, 2026Jun-Yu Pan, Ruicheng Yin, Bao-Liang Lu +3EEG Foundation ModelsSelf-Supervised Pre-Training

  8. Cross-Domain Joint-Embedding Pretraining For Heterogeneous Industrial Time Series

    Nov 2, 2026Jonathan Adam Rico, Nagarajan Raghavan, Senthilnath JayaveluSelf-Supervised Time Series Representation LearningSelf-Supervised Pre-Training

  9. SensOFormer: Foundation Models for Particle Accelerators

    Nov 1, 2026Mahindra Rautela, Alexander ScheinkerSelf-Supervised Time Series Representation LearningTime-Series Foundation Models

  10. A Self-Supervised Pose Estimation Approach for Construction Machines

    Oct 30, 2026Alaa Alshubbak, Daniel GörgesObject Pose EstimationSelf-Supervised Pre-Training

  11. SpectralTF-MAE: A Pre-trained Spectral Foundation Model for Tokamak Fusion Plasma Diagnostics

    Oct 28, 2026Xiao Wang, Zhijie Lu, Duoqing Yang +7Representation LearningSelf-Supervised Pre-Training

  12. Systematic Morphometric Brain Autoencoding for Representation Learning in Neurodegenerative Diseases

    Oct 27, 2026Nuria Balbás-Esteban, Albert Belenguer-Llorens, Elaheh Moradi +3Medical Image Representation LearningSelf-Supervised Pre-Training

  13. SuperJEPA: Pretraining with a Supervision-Augmented Joint-Embedding Predictive Architecture for High Energy Physics

    Oct 26, 2026Dylan Sheldon Rankin, Ho Fung TsoiSelf-Supervised Pre-TrainingJoint-Embedding Predictive Architecture

  14. Self-Supervised Pretraining for PHM: What Transfers, What Does Not, and What Federating It Costs

    Oct 26, 2026José Sanz-Durán, David Solís-MartínSelf-Supervised Pre-TrainingPredictive Maintenance

  15. Exploring Joint-Embedding Predictive Architectures for Self-Supervised Representation Learning of Powder X-ray Diffraction Patterns

    Oct 23, 2026Ming-Chiang Chang, Brian DeCostRepresentation LearningSelf-Supervised Pre-Training

  16. Random Initialization as a Baseline for Time Series Foundation Models

    Oct 22, 2026Pinar Sungu Isiacik, Georgiana IfrimTime-Series Foundation ModelsSelf-Supervised Pre-Training

  17. BrainCodes: A Lightweight EEG Foundation Model over Discrete Neural Codes

    Oct 21, 2026Kleanthis Avramidis, Ivan Tashev, Dimitra EmmanouilidouEEG Foundation ModelsRepresentation Learning

  18. Foundation-Style Pretraining for Scalable and Transferable Distributed PV Forecasting

    Oct 20, 2026Dayin Chen, Xiaodan Shi, Xin Xu +6Solar Power ForecastingTime-Series Foundation Models

  19. MiCLA: Mixture-to-Constituent Latent Alignment for Polyphonic Audio Representation Learning

    Oct 20, 2026Kohei Yamamoto, Kosuke OkusaAudio Representation LearningSelf-Supervised Pre-Training

  20. PRISM: Prior Relational Information for Self-supervised Modeling to Enhance Solubility OOD Generalization

    Oct 19, 2026Xinyi Chen, Jiahuan Pang, Titus Chima +2Molecular Property PredictionSelf-Supervised Pre-Training

  21. Data-Efficient Classifier Pretraining Through Diffusion Models

    Oct 16, 2026Qinyu Li, Yee Whye Teh, Chris Holmes +1Data-Efficient LearningSelf-Supervised Pre-Training

  22. Data-Efficient ConvNeXt Pretraining for High-Resolution Sentinel-1 GRD SAR Representation Learning

    Oct 15, 2026Yuki Natsume, Sang-Ho Yun, Chai Kiat YeoSelf-Supervised Pre-TrainingMasked Autoencoders

  23. L2GSSL: Label-Efficient Local-to-Global Self-Supervised Learning for Volumetric Healthcare Data

    Oct 15, 2026Po-Cheng Cheng, Hung-Hsuan ChenLung Nodule DetectionMedical Image Representation Learning

  24. GeoNIRS: Geometry-Aware Pretraining for Heterogeneous fNIRS Representation Learning

    Oct 13, 2026Zongsheng Li, Shengyuan Zhu, Zimo Zhang +3Representation LearningNeuroimaging

  25. Benchmarking Self-Supervised 3D Medical Vision Foundation Models and Transferability Estimation

    Oct 13, 2026Jiaqi Tang, Yu Wang, Yang Liu +7Medical Imaging Foundation ModelsMedical Image Benchmarks