Self-Supervised Pre-Training

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 64% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 155

All topics

DINO-A: Adapting Self-Distillation Vision Transformers to General Audio Representation Learning
Aug 11, 2026Tomasz Radzikowski, Mateusz Modrzejewski, Przemysław RokitaAudio Representation Learning Self-Supervised Pre-Training
Warsaw University of Technology, Warsaw, Poland
DegradeQuery: Counterfactual Tuple Pretraining for Context-Aware PROTAC Degradation Prediction
Aug 11, 2026Dong Xu, Zhangfan Yang, Jiantao Wu +3Contrastive Learning Self-Supervised Pre-Training
School of Artificial Intelligence, Shenzhen University · EasternDawn · School of Computer Science, University of Nottingham Ningbo
The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora
Aug 10, 2026E. M. FreeburgLanguage Model Pretraining Self-Supervised Pre-Training
Independent Researcher
Vernata: Self-Supervised Learning of LiDAR Point Representations
Aug 7, 2026Oliver Lemke, Alexander Liniger, Abel Gawel +1Self-Supervised Learning Self-Supervised Pre-Training
1Robotics and AI Institute · 2ETH Zurich
CellWorld: From Gene-Level Reconstruction to Latent Cell Prediction in Spatial Transcriptomics Foundation Models
Aug 7, 2026Haiping Liu, Qian Zhao, Lijing Lin +2Spatial Transcriptomics Self-Supervised Pre-Training
University of Manchester
Lesion Detection in CT with Frozen Self-Distilled Features: SALT, a Spatially Adaptive Label-Guided Temperature
Aug 5, 2026Mahmut S. Gokmen, Evan W. Damron, Mitchell A. Klusty +3Self-Supervised Pre-Training Medical Image Analysis
University of Kentucky Lexington, KY, USA · University of Louisville Louisville, KY, USA
Helping Music Co-Creation Agents 'Listen' Well: Hierarchical Self-Supervised World Models for Understanding and Generation
Aug 5, 2026Scott H. HawleyRepresentation Learning Self-Supervised Pre-Training
Department of Chemistry & Physics Belmont University Nashville, TN, USA
ED-DiT: Physics-Guided Diffusion Pretraining for Transferable Molecular Representations from Electron Density
Aug 4, 2026Liang Shuang, Haocheng Wang, Jiayi Song +2Diffusion Transformer Self-Supervised Pre-Training
Fudan University, Shanghai, China · Shenzhen Loop Area Institute, Shenzhen, China · The Chinese University of Hong Kong, Hong Kong SAR, China
Understanding and Correcting Low-Frequency Bias in EEG Foundation Model
Aug 3, 2026Junjie Yu, Zihan Deng, Jianyu Zhang +8Masked Autoencoders Self-Supervised Pre-Training
Department of Biomedical Engineering, Southern University of Science and Technology, Shenzhen, China · Omni-Intelligence, Shenzhen, China · College of Design and Engineering, National University of Singapore, Singapore +4
Beyond Gene Reconstruction: Learning Cell Representations through Complementary Transcriptomic Views
Aug 2, 2026Jiaqi Xiong, Yuntao hu, Yu Zheng +3Contrastive Learning Representation Learning
University of Oxford, Oxford, United Kingdom · Tongji University, Shanghai, China · Hunan University, Changsha, China +2
Geometric Self-Supervised Pre-training for Neural Combinatorial Optimization
Jul 31, 2026David Aguado, Daniel Fuertes, Carlos R. del-Blanco +1Traveling Salesperson Problem Self-Supervised Learning
Grupo de Tratamiento de Imágenes, Information Processing and Telecomunications Center, ETSI Telecomunicación, Universidad Politécnica de Madrid, 28040, Madrid, Spain
On the Efficacy of Self-Supervised Point Cloud Encoders for Efficient 3D Large Language Models
Jul 31, 2026Yao Zheng, Tian ZhangSelf-Supervised Pre-Training Point Cloud Learning
Beijing University of Posts and Telecommunications
A Lightweight Foundation Model for Collider Physics with Multi-Domain Adaptation
Jul 29, 2026Liangyu Wu, Qibin Liu, Alexander Yue +1Self-Supervised Pre-Training High-Energy Physics
Department of Physics, Stanford University, 450 Jane Stanford Way, Stanford, 94305, CA, USA. · SLAC National Accelerator Laboratory, 2575 Sand Hill Road, Menlo Park, 94025, CA, USA.
SARATR-X-v2: Scale-Aware Structural Pre-Training for SAR Foundation Models
Jul 25, 2026Weijie Li, Yafei Song, Yongxiang Liu +7Self-Supervised Pre-Training Synthetic Aperture Radar
College of Electronic Science and Technology, National University of Defense Technology, Changsha 410073, China
IQ-JEPA: A Joint-Embedding Predictive Architecture with a Hermitian Vision Transformer for Sound Speed and Attenuation Estimation from Ultrasound IQ Data
Jul 24, 2026Masashi Sode, Gianmarco PintonSelf-Supervised Pre-Training Joint-Embedding Predictive Architecture
Joint Department of Biomedical Engineering, University of North Carolina at Chapel Hill and North Carolina State University, Chapel Hill, NC 27599 USA
MSBraM: A Multi-scale Self-supervised Brain Foundation Model for Hierarchical EEG Dynamics Learning
Jul 23, 2026Tao Zhou, Jing Han, Lingyu Shu +1Electroencephalography Self-Supervised Pre-Training
Hunan University Changsha, China
Multimodal Pretraining for Generalizable EEG Representation Learning
Jul 23, 2026Targol Bakhtiarvand, Jugal Kalita, Adham AtyabiCross-Subject EEG Decoding Electroencephalography
Department of Computer Science, University of Colorado Colorado Springs, Colorado Springs, CO, USA
Self-supervision drives representational convergence in medical foundation models more than clinical supervision
Jul 22, 2026Soroosh Tayebi Arasteh, Sebastian Ziegelmayer, Mahshad Lotfinia +4Medical Imaging Foundation Models Self-Supervised Pre-Training
Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany · Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany · Department of Diagnostic and Interventional Radiology, TUM University Clinic, School of Medicine and Health, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany +4
Not All Patches are Equal: Sampling Matters for Visible-Infrared Pre-Training
Jul 22, 2026Qiwei Ma, Bin Deng, Junjie Zhu +5Visual Representation Learning Self-Supervised Pre-Training
School of Artificial Intelligence and Robotics, Hunan University, Changsha, 410082, Hunan, China · Intelligent Game and Decision Lab, Beijing, 100166, Beijing, China · Yuelushan Center for Industrial Innovation, Changsha, 410082, Hunan, China
On the Effectiveness of Pretraining for Graph Combinatorial Optimization
Jul 21, 2026David Aguado, Daniel Fuertes, Carlos R. del-Blanco +1Traveling Salesperson Problem Self-Supervised Pre-Training
Grupo de Tratamiento de Imágenes, Information Processing and Telecomunications Center, ETSI Telecomunicación, Universidad Politécnica de Madrid, 28040, Madrid, Spain
BrainNext: A General-Purpose Self-Supervised Foundation Model for Brain MRI Analysis
Jul 20, 2026Moona Mazher, Abdul Qayyum, Steven A. Niederer +1Medical Imaging Foundation Models Self-Supervised Pre-Training
UCL Hawkes Institute, Department of Computer Science, University College London, London, United Kingdom · National Heart & Lung Institute, Imperial College London, London, United Kingdom · Division of Cardiovascular Medicine, Stanford Cardiovascular Institute, Stanford University, California, USA
A Framework for Early Sepsis Prediction via Self-Supervised (JEPA) and Federated Representation Learning
Jul 18, 2026Umair bin Mansoor, Munaf Rashid, Roomi NaqviSelf-Supervised Pre-Training Joint-Embedding Predictive Architecture
Electrical Engg. Department, DHA Suffa University, Karachi, Pakistan · Salim Habib University, Karachi, Sindh, Pakistan · Intel Foundry Services, Intel, Tempe, Arizona, United States
Multimodality as Supervision: Self-Supervised Specialization to the Test Environment via Multimodality
Jul 16, 2026Kunal Pratap Singh, Ali Garjani, Rishubh Singh +6Cross-Modal Learning Cross-Modal Representation Learning
Swiss Federal Institute of Technology Lausanne (EPFL) · Apple
MIDI-RAE-JEPA: Hierarchical Representation Learning and Generation for Symbolic Music
Jul 16, 2026Scott H. HawleyRepresentation Learning Self-Supervised Pre-Training
Department of Chemistry & Physics, Belmont University, Nashville, TN, USA
Leveraging unlabelled data for generalizable neural population decoding
Jul 15, 2026Ximeng Mao, Nanda H. Krishna, Avery Hee-Woon Ryoo +2Masked Autoencoders Self-Supervised Pre-Training
Mila – Quebec AI Institute · Université de Montréal · Canada CIFAR AI Chair
Self-supervised training for high-resolution close-range multispectral remote sensing imagery
Jul 13, 2026Leon-Friedrich Thomas, Mikael Änäkkälä, Antti LajunenAgricultural Image Analysis Remote Sensing Image Segmentation
University of Helsinki, Department of Agricultural Sciences, P.O. Box 28, Fi-00014 Helsinki, Finland · University of Helsinki, Department of Forest Sciences, P.O. Box 27, Fi-00014 Helsinki, Finland
Scalable Visual Pretraining for Language Intelligence
Jul 10, 2026Yiming Zhang, Zhonghan Zhao, Wenwei Zhang +13Language Model Pretraining Self-Supervised Pre-Training
Shanghai Artificial Intelligence Laboratory · University of Science and Technology of China · Zhejiang University +1
CoCoT-EEG: Contrastive-Pretrained Multiscale Convolutional Transformer for EEG Decoding
Jul 10, 2026Gabriel Mahuas, Victoria Shevchenko, Ugo Tanielian +2Contrastive Learning Hybrid CNN-Transformer Architectures
Sigma Nova Paris, France · Goethe University Frankfurt Frankfurt, Germany
JEPA for AI-Native 6G: Predictive Representations and Open Challenges
Jul 9, 2026Sheikh Salman Hassan, Irshad A. Meer, Almoatssimbillah Saifaldawla +96G Networks Self-Supervised Pre-Training
IDCoM, University of Edinburgh, Edinburgh EH9 3BF, UK · KTH Royal Institute of Technology, SE-100 44 Stockholm, Sweden · SnT, University of Luxembourg, L-1855 Luxembourg +6
Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence
Jul 8, 2026Shuailei Ma, Jiaqi Liao, Xinyang Wang +24Video Diffusion Models Self-Supervised Pre-Training

Aug 11, 2026, cs.SDJ/K move · Enter open · S save

DINO-A: Adapting Self-Distillation Vision Transformers to General Audio Representation Learning

Tomasz Radzikowski, Mateusz Modrzejewski, Przemysław Rokita

Warsaw University of Technology, Warsaw, Poland

Audio Representation Learning Self-Supervised Pre-Training Self-Distillation Audio Classification

Open paper AskPDF arXiv

We present DINO-A, an adaptation of self-distillation from vision to general audio representation learning. While DINO has become a canonical method in self-supervised vision and prior audio work has explored latent prediction (BYOL-A) and masked modeling (Audio-MAE, BEATs), no prior work has brought canonical DINO to general audio classification in the way BYOL-A brought BYOL. DINO-A retains DINO's multi-crop, EMA teacher, and high-dimensional projection, replacing only the input modality and augmentations with log-mel spectrograms and the BYOL-A v2 augmentation block. We pretrain three backbones, two Vision Transformers with 8x8 and 16x16 patches and a convolutional encoder, on FSD50K and evaluate them with linear probing on ESC-50, Speech Commands v2, UrbanSound8K, and GTZAN. Three findings characterize the resulting representations. Patch resolution within the Vision Transformer family has consistent effect on representation quality, with smaller patches winning across all four tasks. The choice between Vision Transformer and convolutional backbone interacts with task type: convolutional networks lead on speech while Vision Transformers lead on environmental sounds and music. Under identical pretraining and evaluation conditions, DINO-A and BYOL-A v2 differ by 11.96 percentage points on average, and we trace this difference to two mechanisms: the interaction between DINO's high-dimensional projection space and FSD50K's limited scale, and the additional cost of multi-crop augmentation, which DINO uses but BYOL-A v2 does not. The high-dimensional projection space, central to DINO's success in vision, becomes a liability at FSD50K scale.

Momentum

Related topics

Latest papers 155