Self Supervised Learning

Self-supervised learning (SSL) aims to train machine learning models using unlabeled data by formulating pretext tasks that encourage the model to learn useful representations. Current research focuses on improving SSL's performance and generalization across diverse data types (images, audio, graphs, point clouds) and downstream tasks, employing techniques like contrastive learning, masked autoencoders, and generative models within various architectures such as transformers and convolutional neural networks. These advancements are significant because they reduce the reliance on expensive and time-consuming data labeling, enabling the development of robust models for applications ranging from medical image analysis and speech recognition to geospatial AI and protein function prediction. The efficiency gains from SSL are also a key focus, with research exploring optimal model and data sizes for given computational budgets.

Papers

February 14, 2023

Self-supervised learning of Split Invariant Equivariant representations
Quentin Garrido, Laurent Najman, Yann Lecun
Self Supervised Learning Latent Space Visual Representation Inter Part Equivariance Equivariant Map Equivariant Representation

February 13, 2023

Label-efficient Time Series Representation Learning: A Review
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee-Keong Kwoh, Xiaoli Li
Deep Learning Model Self Supervised Learning Transfer Learning Semi Supervised Learning Narrative Review Time Series Representation Learning

February 12, 2023

February 11, 2023

Anatomical Invariance Modeling and Semantic Alignment for Self-supervised Learning in 3D Medical Image Analysis
Yankai Jiang, Mingze Sun, Heng Guo, Xiaoyu Bai, Ke Yan, Le Lu, Minfeng Xu
Self Supervised Learning Self Supervised 3D Medical Image Semantic Alignment

February 10, 2023

February 7, 2023

Delving Deep into Simplicity Bias for Long-Tailed Image Recognition
Xiu-Shen Wei, Xuhao Sun, Yang Shen, Anqi Xu, Peng Wang, Faen Zhang
Self Supervised Learning Generalization Performance Simplicity Bias Diving Deep Long Tailed Image Recognition

February 3, 2023

Blockwise Self-Supervised Learning at Scale
Shoaib Ahmed Siddiqui, David Krueger, Yann LeCun, Stéphane Deny
Neural Network Self Supervised Learning Back Propagation Visual Analogue Scale State of the Art Deep Pairwise Approach

January 30, 2023

PointSmile: Point Self-supervised Learning via Curriculum Mutual Information
Xin Li, Mingqiang Wei, Songcan Chen
Point Cloud Self Supervised Learning Self Supervised Method Point Cloud Processing Point Supervised

January 29, 2023

SeeGera: Self-supervised Semi-implicit Graph Variational Auto-encoders with Masking
Xiang Li, Tiandi Ye, Caihua Shan, Dongsheng Li, Ming Gao
Self Supervised Learning Node Embeddings Variational Graph Graph Generative Model Variational Graph Auto

January 27, 2023

January 25, 2023

STERLING: Synergistic Representation Learning on Bipartite Graphs
Baoyu Jing, Yuchen Yan, Kaize Ding, Chanyoung Park, Yada Zhu, Huan Liu, Hanghang Tong
Self Supervised Learning Node Embeddings Bipartite Graph Informative Latent Representation Synergistic Representation Learning Bipartite Graph Learning

January 23, 2023

January 22, 2023

Unifying Synergies between Self-supervised Learning and Dynamic Computation
Tarun Krishna, Ayush K Rai, Alexandru Drimbarean, Eric Arazo, Paul Albert, Alan F Smeaton, Kevin McGuinness, Noel E O'Connor
Knowledge Distillation Self Supervised Learning Self Supervised Large Pre Trained Model Emerging Synergy

January 21, 2023

Slice Transformer and Self-supervised Learning for 6DoF Localization in 3D Point Cloud Maps
Muhammad Ibrahim, Naveed Akhtar, Saeed Anwar, Michael Wise, Ajmal Mian
Self Supervised Learning Autonomous Vehicle Localization Focus LiDAR Data LiDAR Scan Point Cloud Map Large Scale LiDAR

January 20, 2023

Self-Supervised Learning for Data Scarcity in a Fatigue Damage Prognostic Problem
Anass Akrim, Christian Gogu, Rob Vingerhoeds, Michel Salaün
Self Supervised Learning Data Scarcity Prognostic Prediction Self Supervised Pre Trained Model Fatigue Life Prediction

Self Supervised Learning

Papers

Self-supervised learning of Split Invariant Equivariant representations

Label-efficient Time Series Representation Learning: A Review

ASR Bundestag: A Large-Scale political debate dataset in German

Transformer models: an introduction and catalog

Anatomical Invariance Modeling and Semantic Alignment for Self-supervised Learning in 3D Medical Image Analysis

A Review of Predictive and Contrastive Self-supervised Learning for Medical Images

AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations

Language-Aware Multilingual Machine Translation with Self-Supervised Learning

Delving Deep into Simplicity Bias for Long-Tailed Image Recognition

Blockwise Self-Supervised Learning at Scale

PointSmile: Point Self-supervised Learning via Curriculum Mutual Information

SeeGera: Self-supervised Semi-implicit Graph Variational Auto-encoders with Masking

Cross-Architectural Positive Pairs improve the effectiveness of Self-Supervised Learning

Leveraging the Third Dimension in Contrastive Learning

STERLING: Synergistic Representation Learning on Bipartite Graphs

Optimizing the Noise in Self-Supervised Learning: from Importance Sampling to Noise-Contrastive Estimation

Self-Supervised Image Representation Learning: Transcending Masking with Paired Image Overlay

Unifying Synergies between Self-supervised Learning and Dynamic Computation

Slice Transformer and Self-supervised Learning for 6DoF Localization in 3D Point Cloud Maps

Self-Supervised Learning for Data Scarcity in a Fatigue Damage Prognostic Problem