Self Supervised Learning Method

Self-supervised learning (SSL) aims to train deep learning models using unlabeled data by creating proxy tasks that leverage inherent data relationships. Current research focuses on developing and comparing various SSL methods, often employing contrastive learning, masked image modeling, and self-distillation techniques, within architectures like Vision Transformers and Siamese networks, and adapting them to diverse data types including images, videos, time series, and tabular data. This approach is particularly valuable in domains with limited labeled data, such as medical imaging and remote sensing, offering a path towards more efficient and robust model training for various applications. The resulting improvements in model performance and reduced reliance on manual annotation have significant implications across numerous scientific fields and practical applications.

Papers

August 30, 2024

A Survey of the Self Supervised Learning Mechanisms for Vision Transformers
Asifullah Khan, Anabia Sohail, Mustansar Fiaz, Mehdi Hassan, Tariq Habib Afridi, Sibghat Ullah Marwat, Farzeen Munir, Safdar Ali, Hannan Naseem, Muhammad Zaigham Zaheer, Kamran Ali, Tangina Sultana, Ziaurrehman Tanoli, Naeem Akhter
Timely Survey Vision Transformer Computer Vision Supervised Learning Vision Task Self Supervision Deep Learning Algorithm Self Supervised Learning Method

August 16, 2024

Unsupervised Transfer Learning via Adversarial Contrastive Training
Chenguang Duan, Yuling Jiao, Huazhen Lin, Wensen Ma, Jerry Zhijian Yang
Data Augmentation Transfer Learning Self Supervised Learning Method Downstream Classification Adversarial Contrastive

July 8, 2024

An Experimental Comparison of Transfer Learning against Self-supervised Learning
Zehui Zhao, Laith Alzubaidi, Jinglan Zhang, Ye Duan, Usman Naseem, Yuantong Gu
Self Supervised Learning Transfer Learning Source Domain Self Supervised Learning Method Experimental Comparison

June 15, 2024

Self-Supervised Representation Learning with Spatial-Temporal Consistency for Sign Language Recognition
Weichao Zhao, Wengang Zhou, Hezhen Hu, Min Wang, Houqiang Li
Self Supervised Self Supervised Representation Learning Sign Language Recognition Self Supervised Learning Method Spatiotemporal Consistency

May 23, 2024

Harmony: A Joint Self-Supervised and Weakly-Supervised Framework for Learning General Purpose Visual Representations
Mohammed Baharoon, Jonathan Klein, Dominik L. Michels
Contrastive Learning Weakly Supervised Self Supervised Learning Method General Purpose Representation Learning

May 8, 2024

A review on discriminative self-supervised learning methods
Nikolaos Giakoumoglou, Tania Stathaki
Self Supervised Learning Computer Vision Narrative Review Self Supervised Learning Method ImageNet Benchmark Discriminative Approach

February 20, 2024

Indiscriminate Data Poisoning Attacks on Pre-trained Feature Extractors
Yiwei Lu, Matthew Y. R. Yang, Gautam Kamath, Yaoliang Yu
Transfer Learning Poisoning Attack Data Poisoning Attack Self Supervised Learning Method Pre Trained Feature Extractor Untargeted Attack

January 26, 2024

Deep Learning with Tabular Data: A Self-supervised Approach
Tirth Kiranbhai Vyas
Deep Learning Tabular Data Tabular Datasets Self Supervised Approach Self Supervised Learning Method TabTransformer Model

January 10, 2024

Siamese Networks with Soft Labels for Unsupervised Lesion Detection and Patch Pretraining on Screening Mammograms
Kevin Van Vorst, Li Shen
Self Supervised Learning Siamese Network Patch Based Soft Label Self Supervised Learning Method Mammogram Classification Digital Mammogram Different Embeddings

November 19, 2023

Self-Distilled Representation Learning for Time Series
Felix Pieper, Konstantin Ditschuneit, Martin Genzel, Alexandra Lindt, Johannes Otterbach
Contrastive Learning Self Supervised Time Series Self Supervised Learning Method Input Time Series

November 16, 2023

From Pretext to Purpose: Batch-Adaptive Self-Supervised Learning
Jiansong Zhang, Linlin Shen, Peizhong Liu
Contrastive Learning Self Supervised Contrastive Learning Self Supervised Learning Method Pretext Task Self Supervised Paradigm

September 18, 2023

Self-supervised Multi-view Clustering in Computer Vision: A Survey
Jiatai Wang, Zhiwei Xu, Xuewen Yang, Hailong Li, Bo Li, Xuying Meng
Contrastive Learning Timely Survey Self Supervised Learning Computer Vision Representation Learning Multi View Clustering Self Supervised Learning Method

September 5, 2023

A Survey of the Impact of Self-Supervised Pretraining for Diagnostic Tasks with Radiological Images
Blake VanBerlo, Jesse Hoey, Alexander Wong
Timely Survey Self Supervised Learning Transfer Learning Global Impact Supervised Learning Self Supervised Pretraining Self Supervised Learning Method Radiology Image Diagnosis Task

September 1, 2023

Remixing-based Unsupervised Source Separation from Scratch
Kohei Saijo, Tetsuji Ogawa
Pre Trained Model Source Separation Scratch Project Self Supervised Learning Method Separation Model Monaural Speech Separation

August 19, 2023

Efficient Representation Learning for Healthcare with Cross-Architectural Self-Supervision
Pranav Singh, Jacopo Cirrone
Convolutional Neural Network Self Supervised Representation Learning Deep Learning Architecture Healthcare System Self Supervised Learning Method Self Supervised Technique

July 31, 2023

Can Self-Supervised Representation Learning Methods Withstand Distribution Shifts and Corruptions?
Prakash Chandra Chhipa, Johan Rodahl Holmgren, Kanjar De, Rajkumar Saini, Marcus Liwicki
Self Supervised Learning Self Supervised Self Supervised Representation Learning Self Supervised Learning Method Self Supervised Representation Corruption Level Self Supervised Paradigm

May 24, 2023

Delving Deeper into Data Scaling in Masked Image Modeling
Cheng-Ze Lu, Xiaojie Jin, Qibin Hou, Jun Hao Liew, Ming-Ming Cheng, Jiashi Feng
Image Modeling ImageNet Dataset Masked Image Modeling Training Image Self Supervised Learning Method Large Scale Model Deep Depth Data Scaling

April 8, 2023

EMP-SSL: Towards Self-Supervised Learning in One Training Epoch
Shengbang Tong, Yubei Chen, Yi Ma, Yann Lecun
Self Supervised Learning Supervised Learning Self Supervised Method Tiny ImageNet Self Supervised Learning Method Training Epoch

March 27, 2023

On the Stepwise Nature of Self-Supervised Learning
James B. Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J. Fetterman, Joshua Albrecht
Self Supervised Learning Dimensional Embeddings Self Supervised Learning Method

February 27, 2023

MPS-AMS: Masked Patches Selection and Adaptive Masking Strategy Based Self-Supervised Medical Image Segmentation
Xiangtao Wang, Ruizhi Wang, Biao Tian, Jiaojiao Zhang, Shuo Zhang, Junyang Chen, Thomas Lukasiewicz, Zhenghua Xu
Self Supervised Learning Method Masking Strategy Medical Image Segmentation Datasets Patch Selection Self Supervised Medical Lesion Classification