Self Supervised Pre Trained Model

Self-supervised pre-trained models (SSPMs) leverage vast amounts of unlabeled data to learn robust feature representations, which are then fine-tuned for various downstream tasks, improving performance, especially in data-scarce scenarios. Current research focuses on optimizing SSPM architectures for specific data types (e.g., time series, images, speech) and exploring effective strategies for knowledge transfer and model compression, including techniques like adapters and structured pruning. The resulting improvements in accuracy and efficiency across diverse applications, such as speech recognition, image clustering, and medical image classification, highlight the significant impact of SSPMs on machine learning.

Papers

August 15, 2024

A Systematic Evaluation of Generated Time Series and Their Effects in Self-Supervised Pretraining
Audrey Der, Chin-Chia Michael Yeh, Xin Dai, Huiyuan Chen, Yan Zheng, Yujie Fan, Zhongfang Zhuang, Vivian Lai, Junpeng Wang, Liang Wang, Wei Zhang, Eamonn Keogh
Mixed Effect Self Supervised Pretraining Time Series Generation Self Supervised Pre Trained Model

August 4, 2024

Image Clustering Algorithm Based on Self-Supervised Pretrained Models and Latent Feature Distribution Optimization
Qiuyu Zhu, Liheng Hu, Sijin Wang
Nearest Neighbor Deep Clustering Clustering Algorithm Latent Space Optimization Image Clustering Self Supervised Pre Trained Model State of the Art Clustering

June 12, 2024

SE/BN Adapter: Parametric Efficient Domain Adaptation for Speaker Recognition
Tianhao Wang, Lantian Li, Dong Wang
Batch Normalization Speaker Recognition Speech Encoder Model Adaptation Self Supervised Pre Trained Model Efficient Domain Adaptation Squeeze and Excitation

March 6, 2024

Bridging Diversity and Uncertainty in Active learning with Self-Supervised Pre-Training
Paul Doucet, Benjamin Estermann, Till Aczel, Roger Wattenhofer
Active Learning High Uncertainty Anticipation Self Supervised Pre Training Diverse Datasets Diversity Enhancement Self Supervised Pre Trained Model Sample Diversity Uncertainty Sampling

February 17, 2024

Target Speech Extraction with Pre-trained Self-supervised Learning Models
Junyi Peng, Marc Delcroix, Tsubasa Ochiai, Oldrich Plchot, Shoko Araki, Jan Cernocky
Supervised Learning Speaker Embeddings Speech Encoder Target Speech Extraction Self Supervised Pre Trained Model

October 28, 2023

Exploring Data Augmentations on Self-/Semi-/Fully- Supervised Pre-trained Models
Shentong Mo, Zhun Sun, Chao Li
Data Augmentation Pre Trained Model Augmentation Technique Self Supervised Pre Trained Model

October 20, 2023

Yet Another Model for Arabic Dialect Identification
Ajinkya Kulkarni, Hanan Aldarmaki
Full Model Acoustic Feature Arabic Speaker Self Supervised Pre Trained Model Time Delay Neural Network Arabic Dialect Identification

October 15, 2023

MERTech: Instrument Playing Technique Detection Using Self-Supervised Pretrained Model With Multi-Task Finetuning
Dichucheng Li, Yinghao Ma, Weixing Wei, Qiuqiang Kong, Yulun Wu, Mingjin Che, Fan Xia, Emmanouil Benetos, Wei Li
Fine Tuning Multi Task Audio Data Self Supervised Pre Trained Model Multi Task Fine Tuning Onset Detection Playing Technique

September 21, 2023

Bridging the Gap: Learning Pace Synchronization for Open-World Semi-Supervised Learning
Bo Ye, Kai Gan, Tong Wei, Min-Ling Zhang
Pseudo Label Self Supervised Pre Trained Model Contrastive Clustering Asynchronous Learning Open World Semi Supervised

September 8, 2023

Unsupervised Object Localization with Representer Point Selection
Yeonghwan Song, Seokwoo Jang, Dina Katabi, Jeany Son
Object Localization Class Activation Point Representation Self Supervised Pre Trained Model Unsupervised Object Localization

July 27, 2023

Federated Model Aggregation via Self-Supervised Priors for Highly Imbalanced Medical Image Classification
Marawan Elbatel, Hualiang Wang, Robert Martí, Huazhu Fu, Xiaomeng Li
Model Aggregation Auxiliary Task Client Model Self Supervised Pre Trained Model Imbalanced Medical Image Self Supervised Auxiliary Task

May 24, 2023

Spoofing Attacker Also Benefits from Self-Supervised Pretrained Model
Aoi Ito, Shota Horiguchi
Self Supervised Learning Anti Spoofing Self Supervised Pre Trained Model Pseudo Victim AttackER Pretrained Model Spoofed Speech

May 1, 2023

Learning Self-Prior for Mesh Inpainting Using Self-Supervised Graph Convolutional Networks
Shota Hattori, Tatsuya Yatagawa, Yutaka Ohtake, Hiromasa Suzuki
High Quality Mesh Self Supervised Pre Trained Model Supervised Graph Neural Network

March 7, 2023

MAST: Masked Augmentation Subspace Training for Generalizable Self-Supervised Priors
Chen Huang, Hanlin Goh, Jiatao Gu, Josh Susskind
Data Augmentation Self Supervised Feature Representation Self Supervised Pre Trained Model

February 27, 2023

Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding
Yifan Peng, Kwangyoun Kim, Felix Wu, Prashant Sridhar, Shinji Watanabe
Speech Recognition Human Understanding Model Compression Convolutional Layer Speech Model Structured Pruning Self Supervised Speech Representation Learning Self Supervised Pre Trained Model Task Specific Structured Pruning

January 20, 2023

Self-Supervised Learning for Data Scarcity in a Fatigue Damage Prognostic Problem
Anass Akrim, Christian Gogu, Rob Vingerhoeds, Michel Salaün
Self Supervised Learning Data Scarcity Prognostic Prediction Self Supervised Pre Trained Model Fatigue Life Prediction

December 11, 2022

SEPT: Towards Scalable and Efficient Visual Pre-Training
Yiqi Lin, Huabin Zheng, Huaping Zhong, Jinjing Zhu, Weijia Li, Conghui He, Lin Wang
Self Supervised Pre Trained Potential Scalability Self Supervised Pre Training Self Supervised Pre Trained Model Visual Pre Training Large Scale Unlabeled Data

November 23, 2022

Distilling Knowledge from Self-Supervised Teacher by Embedding Graph Alignment
Yuchen Ma, Yanbei Chen, Zeynep Akata
Self Supervised Learning Self Supervised Graph Alignment Self Supervised Pre Trained Model Distilling Knowledge Semi Supervised Teacher Student

November 10, 2022

A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understanding
Yifan Peng, Siddhant Arora, Yosuke Higuchi, Yushi Ueda, Sujay Kumar, Karthik Ganesan, Siddharth Dalmia, Xuankai Chang, Shinji Watanabe
Language Model Automatic Speech Recognition Supervised Learning Spoken Language Understanding Speech Model Self Supervised Pre Trained Model Self Supervised Speech

September 28, 2022

Speech Enhancement Using Self-Supervised Pre-Trained Model and Vector Quantization
Xiao-Ying Zhao, Qiu-Shi Zhu, Jie Zhang
Pre Trained Model Speech Enhancement Speech Representation Vector Quantization Clean Speech Neural Speech Enhancement Self Supervised Pre Trained Model

Self Supervised Pre Trained Model

Papers

A Systematic Evaluation of Generated Time Series and Their Effects in Self-Supervised Pretraining

Image Clustering Algorithm Based on Self-Supervised Pretrained Models and Latent Feature Distribution Optimization

SE/BN Adapter: Parametric Efficient Domain Adaptation for Speaker Recognition

Bridging Diversity and Uncertainty in Active learning with Self-Supervised Pre-Training

Target Speech Extraction with Pre-trained Self-supervised Learning Models

Exploring Data Augmentations on Self-/Semi-/Fully- Supervised Pre-trained Models

Yet Another Model for Arabic Dialect Identification

MERTech: Instrument Playing Technique Detection Using Self-Supervised Pretrained Model With Multi-Task Finetuning

Bridging the Gap: Learning Pace Synchronization for Open-World Semi-Supervised Learning

Unsupervised Object Localization with Representer Point Selection

Federated Model Aggregation via Self-Supervised Priors for Highly Imbalanced Medical Image Classification

Spoofing Attacker Also Benefits from Self-Supervised Pretrained Model

Learning Self-Prior for Mesh Inpainting Using Self-Supervised Graph Convolutional Networks

MAST: Masked Augmentation Subspace Training for Generalizable Self-Supervised Priors

Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding

Self-Supervised Learning for Data Scarcity in a Fatigue Damage Prognostic Problem

SEPT: Towards Scalable and Efficient Visual Pre-Training

Distilling Knowledge from Self-Supervised Teacher by Embedding Graph Alignment

A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understanding

Speech Enhancement Using Self-Supervised Pre-Trained Model and Vector Quantization