Self-Supervised Pre-Training

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 64% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 155

All topics

Rethinking Transfer Learning for Industrial Inspection: DINOv3 vs. ImageNet Pretraining Across RGB and X-ray Tasks
May 22, 2026Mehdi Gharbage, Céline Teulière, Pierre Bouges +1Self-Supervised Pre-Training Industrial Anomaly Detection
1Michelin Tyres Manufacturer · 2Universit´e Clermont Auvergne, CNRS, Institut Pascal, Clermont-Ferrand, France
RelPrism: A Multi-Faceted Pre-training Framework with Self-Generated Tasks for Relational Databases
May 22, 2026Jinyu Yang, Cheng Yang, Junze Chen +4Representation Learning Self-Supervised Pre-Training
Beijing University Of Posts and Telecommunications Beijing, China · Peng Cheng Laboratory Shenzhen, China · China Telecom Bestpay Beijing, China +1
Towards a General Intelligence and Interface for Wearable Health Data
May 21, 2026Girish Narayanswamy, Maxwell A. Xu, A. Ali Heydari +37Self-Supervised Pre-Training Clinical Outcome Prediction
1Google Research · University of Washington · University of Illinois Urbana-Champaign +2
Atom-level Protein Representation Learning Improves Protein Structure Prediction
May 21, 2026Taewon Kim, Hyosoon Jang, Hyunjin Seo +6Protein Structure Prediction Self-Supervised Pre-Training
Graduate School of AI, KAIST · Department of Chemistry, KAIST · HITS Inc. +3
Entropy-Guided Self-Supervised Learning for Medical Image Classification
May 21, 2026Joao Florindo, Viviane MouraSelf-Supervised Pre-Training Medical Image Classification
Institute of Mathematics, Statistics and Scientific Computing Department of Applied Mathematics, University of Campinas Rua Sergio Buarque de Holanda, 651, Campinas, 13083-859, Sao Paulo, Brazil
Towards Understanding Self-Pretraining for Sequence Classification
May 20, 2026Omar Coser, Loredana Zollo, Paolo Soda +1Self-Attention Masked Language Modeling
TriForces: Augmenting Atomistic GNNs for Transferable Representations
May 20, 2026Ali Ramlaoui, Alexandre Duval, Hannah Bull +4Disentangled Representation Learning Machine Learning Interatomic Potentials
Entalpic, Paris, France · Universit´e Paris-Saclay, CentraleSup´elec, Inria, Gif-sur-Yvette, France
Hierarchical Contrastive Learning for Multi-Domain Protein-Ligand Binding
May 19, 2026Shuo Zhang, Rongqi Hong, Huifeng Zhang +1Protein-Ligand Binding Affinity Prediction Contrastive Learning
University of Birmingham, UK
When Does Self-Supervised Learning Transfer to Time-Series Tasks?
May 19, 2026Noam Major, Kathy Razmadze, Yoli ShavitSelf-Supervised Pre-Training Time Series Classification
Faculty of Engineering, Bar-Ilan University, Israel
PilotWiMAE: Pilot-Native Representation Learning for Wireless Channels
May 19, 2026Berkay Guler, Giovanni Geraci, Hamid JafarkhaniRepresentation Learning Masked Autoencoders
Center for Pervasive Communications and Computing, University of California, Irvine CA, USA · Nokia and Universitat Pompeu Fabra, Spain
Knowing When Not to Predict: Self Supervised Learning and Abstention for Safer DR Screening
May 18, 2026Muskaan Chopra, Lorenz Sparrenberg, Jan H. Terheyden +1Selective Prediction Self-Supervised Pre-Training
Rheinische Friedrich-Wilhelms-Universität Bonn, Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany · University Hospital Bonn - Department of Ophthalmology, Germany +1
Protein Fold Classification at Scale: Benchmarking and Pretraining
May 18, 2026Dexiong Chen, Andrei Manolache, Mathias Niepert +1Masked Autoencoders Self-Supervised Pre-Training
Max Planck Institute of Biochemistry, Martinsried, Germany · Computer Science Department, University of Stuttgart, Germany · Bitdefender, Romania
Benchmarking transferability of SSL pretraining to same and different modality segmentation tasks
May 18, 2026Jue Jiang, Harini VeeraraghavanCross-Modality Medical Image Segmentation Self-Supervised Pre-Training
Department of Medical Physics, Memorial Sloan Kettering Cancer Center
DARE-EEG: A Foundation Model for Mining Dual-Aligned Representation of EEG
May 18, 2026Yang Shao, Peiliang Gong, Qun Dai +1Masked Autoencoders Self-Supervised Pre-Training
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics Nanjing, Jiangsu, China
Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Framework
May 17, 2026Yumeng Yao, Jingzhi Dong, Haowen Gu +43D ViTs Self-Supervised Pre-Training
Nanjing University of Science and Technology · Hangzhou City University · East China Normal University +1
How Do Electrocardiogram Models Scale?
May 17, 2026Jiawei Li, Fabio Bonassi, Ming Jin +4Self-Supervised Pre-Training Electrocardiography
Uppsala University · Griffith University
VolTA-3D: Self-Supervised Learning for Brain MRI using 3D Volumetric Token Alignment
May 16, 2026Amy Makawana, Abhijeet Parida, Marius George Linguraru +23D ViTs Self-Supervised Pre-Training
Institute of Health Informatics, University College London, London, UK · Sheikh Zayed Institute for Pediatric Surgical Innovation, Children’s National Hospital, Washington, DC USA · School of Medicine and Health Sciences, George Washington University, Washington, DC, USA
Multi-level Self-supervised Pretraining on Compositional Hierarchical Graph for Molecular Property Prediction
May 15, 2026Xiayu Liu, Zhengyi Lu, Hou-biao LiSelf-Supervised Pre-Training Hierarchical Representation Learning
School of Mathematical Sciences, University of Electronic Science and Technology of China, No.2006, Xiyuan Avenue, West Hi-tech Zone,2026 611731, Chengdu, China · Department of Computer Science and Engineering, Oakland University, 201 Meadow Brook RD, Rochester Hills, 48309, MI, USA
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
May 14, 2026Tan Pan, Shuhao Mei, Yixuan Sun +8Representation Learning Multimodal Robustness
Fudan University, China · Shanghai Academy of AI for Science, China · Bioinformatics Institute (BII), Agency for Science, Technology and Research (A*STAR), Singapore +2
SpectraFlow: Unifying Structural Pretraining and Frequency Adaptation for Medical Image Segmentation
May 14, 2026Zhiquan Chen, Haitao Wang, Guowei Zou +1Flow Matching Image Segmentation
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · Guangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou, China
AudioMosaic: Contrastive Masked Audio Representation Learning
May 14, 2026Hanxun Huang, Qizhou Wang, Xingjun Ma +3Audio Representation Learning Self-Supervised Pre-Training
School of Computing and Information Systems, The University of Melbourne, Australia · Institute of Trustworthy Embodied AI, Fudan University, China · Baskin School of Engineering, University of California, Santa Cruz, USA
CSI-JEPA: Towards Foundation Representations for Ubiquitous Sensing with Minimal Supervision
May 13, 2026Xuanhao Luo, Zhizhen Li, Yuchen LiuSelf-Supervised Pre-Training Self-Supervised Time Series Representation Learning
North Carolina State University, USA
Unified Pix Token And Word Token Generative Language Model
May 13, 2026Haun Leung, ZiNan WangImage Tokenization Self-Supervised Pre-Training
AOI-SSL: Self-Supervised Framework for Efficient Segmentation of Wire-bonded Semiconductors In Optical Inspection
May 12, 2026Joaquín Figueira, Rob Van Gastel, Giacomo D'Amicantonio +4Self-Supervised Pre-Training Semiconductor Manufacturing
AIMS Lab, Eindhoven University of Technology · CoC, ASMPT · iCIS, Radboud University
MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification
May 12, 2026Bo Zheng, Yudong Chen, Zihua Xiong +4Tabular Foundation Models Self-Supervised Pre-Training
Zhejiang University · MyBank, Ant Group
TRACE: Temporal Routing with Autoregressive Cross-channel Experts for EEG Representation Learning
May 12, 2026Fan Ma, Qier An, Peng Chen +6Representation Learning Self-Supervised Pre-Training
Department of Biomedical Informatics and Data Science, Yale University
Clin-JEPA: A Multi-Phase Co-Training Framework for Joint-Embedding Predictive Pretraining on EHR Patient Trajectories
May 11, 2026Yixuan Yang, Mehak Arora, Ryan Zhang +10Self-Supervised Pre-Training Healthcare
Duke University, Durham, NC, USA
From pre-training to downstream performance: Does domain-specific pre-training make sense?
May 9, 2026Felix KronesSelf-Supervised Pre-Training Domain-Adaptive Pretraining
Oxford Internet Institute, University of Oxford, Oxford, UK
When Losses Align: Gradient-Based Composite Loss Weighting for Efficient Pretraining
May 8, 2026Ivan Karpukhin, Andrey SavchenkoSelf-Supervised Pre-Training Adaptive Loss Weighting
Sber AI Lab · HSE University ISP RAS Research Center for Trusted Artificial Intelligence
ProtoSSL: Self-Supervised Pretraining and Downstream Transfer for Projection-Based Prototype Models
May 7, 2026Steven Song, Sahil Sethi, Brett Beaulieu-Jones +1Self-Supervised Learning Self-Supervised Pre-Training
Department of Computer Science, University of Chicago, Chicago, IL 60637, USA · Center for Translational Data Science, University of Chicago, Chicago, IL 60637, USA · Medical Scientist Training Program, University of Chicago, Chicago, IL 60637, USA +3

May 22, 2026, cs.CVJ/K move · Enter open · S save

Rethinking Transfer Learning for Industrial Inspection: DINOv3 vs. ImageNet Pretraining Across RGB and X-ray Tasks

Mehdi Gharbage, Céline Teulière, Pierre Bouges, Thierry Chateau

1Michelin Tyres Manufacturer · 2Universit´e Clermont Auvergne, CNRS, Institut Pascal, Clermont-Ferrand, France

Self-Supervised Pre-Training Industrial Anomaly Detection Vision Foundation Model Adaptation Industrial Inspection Transfer Learning

Open paper AskPDF arXiv

Vision foundation models pretrained on web-scale data have recently shown strong transfer capabilities on many downstream tasks, but their effectiveness for industrial visual inspection remains unclear. Industrial data differ substantially from web-data and often require fine-grained dense prediction, raising the question of whether modern self-supervised pretraining can improve over the conventional transfer-learning paradigm based on supervised ImageNet initialization. In this work, we compare ConvNeXt backbones pretrained with supervised ImageNet classification or DINOv3 distillation, and relate them to the conventional ResNet-50 baseline. We evaluate semantic segmentation, instance segmentation, and object detection across four downstream datasets spanning RGB surface-defect inspection and X-ray defect detection. We further study both frozen and fully finetuned adaptation regimes. Our results show that DINOv3 offers no clear advantage in frozen transfer, but provides a stronger initialization after full finetuning on RGB tasks, yielding faster convergence and better final performance. Under X-ray modality shift, however, supervised ImageNet pretraining remains more effective in both frozen and finetuned settings. Overall, our findings suggest that modern vision foundation models are promising for supervised RGB industrial inspection, but their transferability is strongly conditioned by downstream adaptation and target modality.

Momentum

Related topics

Latest papers 155