Self Supervised Transformer

Self-supervised transformers are revolutionizing various fields by learning powerful representations from unlabeled data, thereby reducing the reliance on large, manually annotated datasets. Current research focuses on adapting transformer architectures for diverse tasks, including speech recognition, image segmentation, and medical image analysis, often employing techniques like masked autoencoders and contrastive learning. These advancements are significantly impacting numerous applications, from improving the accuracy of medical diagnoses to enabling more efficient and robust object detection in various domains. The ability to learn effectively from unlabeled data promises to accelerate progress in areas previously limited by data scarcity.

Papers

June 6, 2023

DenseDINO: Boosting Dense Self-Supervised Learning with Token-Based Point-Level Consistency
Yike Yuan, Xinghe Fu, Yunlong Yu, Xi Li
Semantic Segmentation Self Supervised Learning Self Supervised Transformer Dense Prediction Task Level Consistency Dense Self Supervised Learning Dense Visual Representation

June 1, 2023

Affinity-based Attention in Self-supervised Transformers Predicts Dynamics of Object Grouping in Humans
Hossein Adeli, Seoyoung Ahn, Nikolaus Kriegeskorte, Gregory Zelinsky
Human Attention Arbitrary Object Real Human Visual Representation Learning Self Supervised Vision Transformer Self Supervised Transformer Reaction Time Feature Affinity

May 8, 2023

AnomalyBERT: Self-Supervised Transformer for Time Series Anomaly Detection using Data Degradation Scheme
Yungi Jeong, Eunseok Yang, Jung Hyun Ryu, Imseong Park, Myungjoo Kang
Multivariate Time Series Self Supervised Transformer Complex Time Series Synthetic Outlier Degradation Feature

April 27, 2023

SoGAR: Self-supervised Spatiotemporal Attention-based Social Group Activity Recognition
Naga VS Raviteja Chappa, Pha Nguyen, Alexander H Nelson, Han-Seok Seo, Xin Li, Page Daniel Dobbs, Khoa Luu
Transformer Based Self Supervised Transformer Group Activity Recognition Social Group Activity Recognition

April 25, 2023

DuETT: Dual Event Time Transformer for Electronic Health Records
Alex Labach, Aslesha Pokhrel, Xiao Shi Huang, Saba Zuberi, Seung Eun Yi, Maksims Volkovs, Tomi Poutanen, Rahul G. Krishnan
Transformer Based Model Electronic Health Record Self Supervised Transformer Event Transformer

April 11, 2023

MOST: Multiple Object localization with Self-supervised Transformers for object discovery
Sai Saketh Rambhatla, Ishan Misra, Rama Chellappa, Abhinav Shrivastava
Object Localization Object Discovery Semi Supervised Object Detection Self Supervised Transformer Unsupervised Object Localization

March 15, 2023

SeqCo-DETR: Sequence Consistency Training for Self-Supervised Object Detection with Transformers
Guoqiang Jin, Fan Yang, Mingshan Sun, Ruyi Zhao, Yakun Liu, Wei Li, Tianpeng Bao, Liwei Wu, Xingyu Zeng, Rui Zhao
Transformer Megatron Decepticons Self Supervised Detection Task Self Supervised Transformer Consistency Training Self Supervised Object Detection

March 6, 2023

ST-KeyS: Self-Supervised Transformer for Keyword Spotting in Historical Handwritten Documents
Sana Khamekhem Jemni, Sourour Ammar, Mohamed Ali Souibgui, Yousri Kessentini, Abbas Cheddad
Pre Trained Human Annotation Keyword Spotting Self Supervised Transformer Auto Encoder Model Annotated Corpus Annotated Training Data Secret Key Handwritten Manuscript

March 2, 2023

STDepthFormer: Predicting Spatio-temporal Depth from Video with a Self-supervised Transformer Model
Houssem Boulahbal, Adrian Voicila, Andrew Comport
Source Video Temporal Attention Depth Prediction Self Supervised Transformer Spatio Temporal Context Single Image Depth Estimation Scene Knowledge

February 3, 2023

Self-Supervised Transformer Architecture for Change Detection in Radio Access Networks
Igor Kozlov, Dmitriy Rivkin, Wei-Di Chang, Di Wu, Xue Liu, Gregory Dudek
Time Series Change Detection Self Supervised Transformer Runtime Monitoring Self Supervised Learning Framework

January 14, 2023

Gated Self-supervised Learning For Improving Supervised Learning
Erland Hilman Fuadi, Aristo Renaldo Ruslim, Putu Wahyu Kusuma Wardhana, Novanto Yudistira
Self Supervised Learning Image Classification Supervised Learning Self Supervised Transformer Mixture Density Network Channel Data Augmentation

December 5, 2022

Location-Aware Self-Supervised Transformers for Semantic Segmentation
Mathilde Caron, Neil Houlsby, Cordelia Schmid
Semantic Segmentation Pixel Level Self Supervised Contrastive Learning Self Supervised Transformer Dense Pseudo Label

October 25, 2022

MOFormer: Self-Supervised Transformer model for Metal-Organic Framework Property Prediction
Zhonglin Cao, Rishikesh Magar, Yuyang Wang, Amir Barati Farimani
Self Supervised Transformer Metal Organic Framework MOF Structure Crystal Graph Convolutional Neural Network

October 4, 2022

MTSMAE: Masked Autoencoders for Multivariate Time-Series Forecasting
Peiwang Tang, Xianchao Zhang
Supervised Autoencoder Time Series Forecasting Multivariate Time Series Masked Autoencoders Self Supervised Transformer

September 9, 2022

Discriminative Sampling of Proposals in Self-Supervised Transformers for Weakly Supervised Object Localization
Shakeeb Murtaza, Soufiane Belharbi, Marco Pedersoli, Aydin Sarraf, Eric Granger
Image Level Label Proposal Balance Refinement Self Supervised Vision Transformer Self Supervised Transformer Weakly Supervised Object Localization Drone Based

September 1, 2022

TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut
Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, James L Crowley, Dominique Vaufreydaz
Segmentation Task Self Supervised Transformer Segment Object Unsupervised Video Object Segmentation Video Detection Token Reduction Unsupervised Salient Object Detection Normalized Cut

August 29, 2022

SB-SSL: Slice-Based Self-Supervised Transformers for Knee Abnormality Classification from MRI
Sara Atito, Syed Muhammad Anwar, Muhammad Awais, Josef Kitler
Self Supervised Ground Truth Supervised Learning Self Supervised Transformer Clinical Datasets Knee Magnetic Resonance

July 20, 2022

Hierarchically Self-Supervised Transformer for Human Skeleton Representation Learning
Yuxiao Chen, Long Zhao, Jianbo Yuan, Yu Tian, Zhaoyang Xia, Shijie Geng, Ligong Han, Dimitris N. Metaxas
Self Supervised Transformer Skeleton Sequence Skeleton Representation Skeleton Annotation Skeleton Encoder

July 4, 2022

Masked Autoencoders in 3D Point Cloud Representation Learning
Jincen Jiang, Xuequan Lu, Lizhi Zhao, Richard Dazeley, Meili Wang
Point Cloud Masked Autoencoders Point Cloud Completion Self Supervised Transformer Patch Transformer Point Cloud Representation Learning

July 1, 2022

Improving Low-Resource Speech Recognition with Pretrained Speech Models: Continued Pretraining vs. Semi-Supervised Training
Mitchell DeHaven, Jayadev Billa
Automatic Speech Recognition Audio Data Self Supervised Transformer Pre Trained Speech Model Semi Supervised Training Low Resource Speech Recognition

Self Supervised Transformer

Papers

DenseDINO: Boosting Dense Self-Supervised Learning with Token-Based Point-Level Consistency

Affinity-based Attention in Self-supervised Transformers Predicts Dynamics of Object Grouping in Humans

AnomalyBERT: Self-Supervised Transformer for Time Series Anomaly Detection using Data Degradation Scheme

SoGAR: Self-supervised Spatiotemporal Attention-based Social Group Activity Recognition

DuETT: Dual Event Time Transformer for Electronic Health Records

MOST: Multiple Object localization with Self-supervised Transformers for object discovery

SeqCo-DETR: Sequence Consistency Training for Self-Supervised Object Detection with Transformers

ST-KeyS: Self-Supervised Transformer for Keyword Spotting in Historical Handwritten Documents

STDepthFormer: Predicting Spatio-temporal Depth from Video with a Self-supervised Transformer Model

Self-Supervised Transformer Architecture for Change Detection in Radio Access Networks

Gated Self-supervised Learning For Improving Supervised Learning

Location-Aware Self-Supervised Transformers for Semantic Segmentation

MOFormer: Self-Supervised Transformer model for Metal-Organic Framework Property Prediction

MTSMAE: Masked Autoencoders for Multivariate Time-Series Forecasting

Discriminative Sampling of Proposals in Self-Supervised Transformers for Weakly Supervised Object Localization

TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut

SB-SSL: Slice-Based Self-Supervised Transformers for Knee Abnormality Classification from MRI

Hierarchically Self-Supervised Transformer for Human Skeleton Representation Learning

Masked Autoencoders in 3D Point Cloud Representation Learning

Improving Low-Resource Speech Recognition with Pretrained Speech Models: Continued Pretraining vs. Semi-Supervised Training