Self-Supervised Pre-Training

Follow this topic Ask about this topic

Self-supervised pre-training studies how models can learn useful structure from unlabeled data, making it possible to adapt them to new tasks with fewer annotations. Common approaches include masked autoencoders, contrastive learning, teacher-student distillation, and joint-embedding predictive architectures such as JEPA, applied to images, signals, video, text, and scientific data. Researchers test these representations through frozen-feature probes, fine-tuning, and few-label or cross-domain transfer on benchmarks in areas such as medical imaging, speech, and autonomous driving. Results show that pre-training can improve data efficiency and robustness, but its benefits depend on the match between objective, data, and downstream task; greater model size or training time does not reliably help.

Latest papers

251 papers34 papers in the last four weeks, level with the four weeks before. 0.2% of all new papers.

TinySSL: Distilled Self-Supervised Pretraining for Sub-Megabyte MCU Models
May 7, 2026Bibin WilsonSelf-Supervised Learning Self-Supervised Pre-Training
Attention-Based Chaotic Self-Supervision for Medical Image Classification
May 6, 2026Joao Batista Florindo, Amanda Pontes de Oliveira OrnelasDenoising Autoencoders Self-Supervised Pre-Training
Institute of Mathematics, Statistics and Scientific Computing - University of Campinas, Rua S´ergio Buarque de Holanda, 651, Campinas, Brasil
Diffusion Masked Pretraining for Dynamic Point Cloud
May 5, 2026Zhuoyue Zhang, Yiding Sun, Haozhe Cheng +4Self-Supervised Pre-Training Point Cloud Learning
School of Software Engineering, Xi’an Jiaotong University, Xi’an, China · School of Artificial Intelligence and Robotics, Hunan University, Changsha, China · Department of Computer Science and Software Engineering, University of Western Australia, Perth, Australia
SignMAE: Segmentation-Driven Self-Supervised Learning for Sign Language Recognition
May 3, 2026Kunyuan Xie, Zhixi Cai, Kalin StefanovSelf-Supervised Pre-Training Sign Language Recognition
Monash University, Australia
PointCSP: Cross-Sample Semantic Propagation and Stability Preservation in Self-Supervised Point Cloud Learning
May 3, 2026Xinxing Yu, Ajian Liu, Sunyuan Qiang +5Self-Supervised Point Cloud Learning Self-Supervised Pre-Training
Faculty of Innovation Engineering, Macau University of Science and Technology, Macau, China · MAIS, The Institute of Automation of the Chinese Academy of Sciences, Beijing, China · Southwest Institute of Technical Physics, Chengdu, China +1
Adaptive Texture-aware Masking for Self-Supervised Learning in 3D Dental CBCT Analysis
May 3, 2026Xinquan Yang, Jianfeng Ren, Xuguang Li +4Cone-Beam CT Self-Supervised Pre-Training
School of Artiﬁcial Intelligence, Shenzhen University, Shenzhen, China. · School of Computer Science, University of Notting-ham Ningbo China, Ningbo, China. · Department of Stomatology, Shenzhen University General Hospital, Shenzhen, China.
GraphSculptor: Sculpting Pre-training Coreset for Graph Self-supervised Learning
May 2, 2026Chuang Liu, Zelin Yao, Xueqi Ma +4Graph Representation Learning Self-Supervised Pre-Training
Sangfor Technologies Inc. · Wuhan University · The University of Melbourne +2
Rhamba: Region-Aware Hybrid Attention-Mamba Framework for Self-Supervised Learning in Resting-State fMRI
May 2, 2026Ruthwik Reddy Doodipala, Pankaj Pandey, Pratheek Eranki +3Self-Supervised Pre-Training Self-Supervised Time Series Representation Learning
1St. Jude Children’s Research Hospital · 2The University of Memphis
UniBCI: Towards a Unified Pretrained Model for Invasive Brain-Computer Interfaces
Apr 30, 2026Binjie Hong, Rui Xiong, Liyuan Han +1Brain-Computer Interfaces Self-Supervised Pre-Training
Brain Science Data Center · CEBSIT, Institute of Neuroscience
LA-Pose: Latent Action Pretraining Meets Pose Estimation
Apr 30, 2026Zhengqing Wang, Saurabh Nair, Prajwal Chidananda +4Self-Supervised Pre-Training Camera Pose Estimation
1Wayve · 2Simon Fraser University
BrainDINO: A Brain MRI Foundation Model for Generalizable Clinical Representation Learning
Apr 30, 2026Yizhou Wu, Shansong Wang, Yuheng Li +5Medical Imaging Foundation Models Self-Supervised Pre-Training
Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA. · Department of Radiation and Cellular Oncology, The University of Chicago, Chicago, IL, USA. · Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA, USA. +2
CLLAP: Contrastive Learning-based LiDAR-Augmented Pretraining for Enhanced Radar-Camera Fusion
Apr 27, 2026Bingyi Liu, Chuanhui Zhu, Hongfei Xue +5Camera-Radar Fusion Self-Supervised Pre-Training
Wuhan University of Technology · University of North Carolina at Charlotte · The Hong Kong Polytechnic University +2
SS3D: End2End Self-Supervised 3D from Web Videos
Apr 24, 2026Marwane Hariat, Gianni Franchi, David Filliat +1Monocular Depth Estimation Depth Estimation
U2IS, ENSTA – Institut Polytechnique de Paris, Palaiseau, France · Pôle Recherche, Agence Ministérielle pour l’IA de Défense, Palaiseau, France
A Nationwide Japanese Medical Claims Foundation Model: Balancing Model Scaling and Task-Specific Computational Efficiency
Apr 24, 2026Nanae Aratake, Taisei Tosaki, Yuji Okamoto +5Self-Supervised Pre-Training Clinical Outcome Prediction
1Graduate School of Medicine, Kyoto University, Kyoto, Japan. · 2Medical Data Vision Co., Ltd., Tokyo, Japan. · 3IQVIA Solutions Japan G.K., Tokyo, Japan. +1
Generalizing Numerical Reasoning in Table Data through Operation Sketches and Self-Supervised Learning
Apr 23, 2026Hanjun Cho, Gahyun Yoo, Hanseong Kim +1Table QA Self-Supervised Pre-Training
Seoul National University · Soongsil University
MAE-Based Self-Supervised Pretraining for Data-Efficient Medical Image Segmentation Using nnFormer
Apr 22, 2026R. M. Krishna Sureddi, T. Satyanarayana Murthy, Nomula Varsha Reddy +2Self-Supervised Pre-Training Medical Image Segmentation
Information Technology, Chaitanya Bharathi Institute of Technology, Hyderabad, India
Self-supervised pretraining for an iterative image size agnostic vision transformer
Apr 22, 2026Nedyalko Prisadnikov, Danda Pani Paudel, Yuqian Fu +1Self-Supervised Pre-Training Vision Transformer
INSAIT, Sofia University "St. Kliment Ohridski", Bulgaria
PolarMAE: Efficient Fetal Ultrasound Pre-training via Semantic Screening and Polar-Guided Masking
Apr 17, 2026Meng Lv, Yapeng Li, Hang Su +2Masked Autoencoders Self-Supervised Pre-Training
Wuhan University
Co-distilled attention guided masked image modeling with noisy teacher for self-supervised learning on medical images
Apr 16, 2026Jue Jiang, Aneesh Rangnekar, Harini VeeraraghavanMasked Autoencoders Self-Supervised Pre-Training
Memorial Sloan Kettering Cancer Center
Brain-DiT: A Universal Multi-state fMRI Foundation Model with Metadata-Conditioned Pretraining
Apr 14, 2026Junfeng Xia, Wenhao Ye, Xuanye Pan +3Medical Imaging Foundation Models Diffusion Transformer
Department of Biomedical Engineering, Southern University of Science and Technology, China · School of Biomedical Engineering, Shenzhen University, China
Towards Brain MRI Foundation Models for the Clinic: Findings from the FOMO25 Challenge
Apr 13, 2026Asbjørn Munk, Stefano Cerri, Vardan Nersesjan +81Medical Imaging Foundation Models Self-Supervised Pre-Training
Using Deep Learning Models Pretrained by Self-Supervised Learning for Protein Localization
Apr 13, 2026Ben Isselmann, Dilara Göksu, Heinz Neumann +1Supervised Fine-Tuning Self-Supervised Pre-Training
Department of Mathematics and Natural Sciences, Hochschule Darmstadt, Schoeﬀerstraße 3, Darmstadt, 64295, Hessen, Germany. · Department of Chemical Engineering and Biotechnology, Hochschule Darmstadt, Stephanstrasse 7, Darmstadt, 64295, Hessen, Germany. · European University of Technology, European Union. +1
Self-supervised Pretraining of Cell Segmentation Models
Apr 12, 2026Kaden Stillwagon, Alexandra Dunnum VandeLoo, Benjamin Magondu +1Instance Segmentation Microscopy Image Analysis
School of Computer Science, Georgia Institute of Technology · School of Materials Science and Engineering, Georgia Institute of Technology · Department of Biomedical Engineering, Georgia Institute of Technology +1
Masked Contrastive Pre-Training Improves Music Audio Key Detection
Apr 11, 2026Ori Yonay, Tracy Hammond, Tianbao YangMusic Information Retrieval Audio Representation Learning
Department of Computer Science and Engineering, Texas A&M University
PRISM-CTG: A Foundation Model for Cardiotocography Analysis with Multi-View SSL
Apr 9, 2026Sheng Wong, Ravi Shankar, Beth Albert +5Self-Supervised Time Series Representation Learning Self-Supervised Learning
Nuffield Department of Women’s & Reproductive Health, University of Oxford, United Kingdom · OXDHL, University of Oxford, Oxford, United Kingdom · OATML, Department of Computer Science, University of Oxford, United Kingdom +1
Towards foundation-style models for energy-frontier heterogeneous neutrino detectors via self-supervised pre-training
Apr 8, 2026Saúl Alonso-Monsalve, Fabio Cufino, Umut Kose +2Self-Supervised Pre-Training High-Energy Physics
IPA, ETH Zürich, Otto Stern Weg 5, Zurich, 8093, Switzerland.
VAMAE: Vessel-Aware Masked Autoencoders for OCT Angiography
Apr 8, 2026Ilerioluwakiiye Abolade, Prince Mireku, Kelechi Chibundu +4Masked Autoencoders Self-Supervised Pre-Training
Federal University of Agriculture Abeokuta, Nigeria · ML Collective · Ashesi University, Ghana +1
Entropy, Disagreement, and the Limits of Foundation Models in Genomics
Apr 5, 2026Maxime Rochkoulets, Lovro Vrček, Mile ŠikićGenomic Foundation Models Genomics
Genome Institute of Singapore, A*STAR, Singapore · KU Leuven, Belgium · Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia
PRIME: Prototype-Driven Multimodal Pretraining for Cancer Prognosis with Missing Modalities
Apr 5, 2026Kai Yu, Shuang Zhou, Yiran Song +9Multimodal Pretraining Missing-Modality Learning
Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, MN, USA. · University of North Carolina at Chapel Hill, Chapel Hill, NC, USA. · North Carolina State University, Raleigh, NC, USA. +2
CalM: A Self-Supervised Foundation Model for Population Dynamics in Calcium Imaging Data
Apr 3, 2026Xinhong Xu, Yimeng Zhang, Qichen Qian +1Self-Supervised Pre-Training Time-Series Foundation Models
Department of Automation, Tsinghua University, Beijing, China · School of Life Sciences, Tsinghua University, Beijing, China

May 7, 2026J/K move · Enter open · S save

TinySSL: Distilled Self-Supervised Pretraining for Sub-Megabyte MCU Models

Bibin Wilson

Self-Supervised Learning Self-Supervised Pre-Training Tiny ML Knowledge Distillation Self-Supervised Visual Representation Learning

Open paper AskPDF arXiv

Self-supervised learning (SSL) has transformed representation learning for large models, yet remains unexplored for microcontroller (MCU)-class models with fewer than 500K parameters. We identify three obstacles at this scale -- projection head dominance, representation bottleneck, and augmentation sensitivity -- and propose Capacity-Aware Distilled Self-Supervised Learning (CA-DSSL), a teacher-guided framework that overcomes them without labels or text supervision. CA-DSSL combines asymmetric distillation from a frozen DINO ViT-S/16 teacher, multi-scale feature distillation for spatial representations, and a progressive augmentation curriculum. On a MobileNetV2-0.35 backbone (396K parameters) pretrained on CIFAR-100, CA-DSSL reaches 62.7 0.5% linear-probe accuracy (3-seed mean) -- surpassing SimCLR-Tiny by 18 pp, matching SEED (61.7%) with 10 fewer projection parameters (426K vs. 3.15M), and reaching 94.0% of a supervised upper bound. Standard SSL methods (BYOL-Tiny, DINO-Tiny) collapse entirely at this scale. On Pascal VOC detection, CA-DSSL achieves 2.3 the mAP of random initialization and +3 pp over SEED, though SimCLR-Tiny matches CA-DSSL on detection mAP. The deployed backbone occupies 378 KB (INT8) with no inference overhead from pretraining. Preliminary ImageNet-100 experiments reveal that CA-DSSL's advantage is specific to small-data regimes; scaling to ImageNet-1K is discussed as future work.