Self-Supervised Pre-Training

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 64% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 155

All topics

Automatic Echocardiography Segmentation via Transition Probability Correlation for Stable Semantic Extraction
Jul 8, 2026Xinran Chen, Xiyuan Wang, Guangquan Zhou +1Ultrasound Image Segmentation Self-Supervised Pre-Training
School of Biological Science and Medical Engineering, Southeast University, Nanjing, China
Self-Supervised Pretraining Improves Cross-Site and Cross-Scale Robustness of Point Cloud Leaf-Wood Segmentation
Jul 8, 2026Heeju Mun, Tackang Yang, Yunsoo Nam +1Self-Supervised Pre-Training Domain Generalization
Department of Landscape Architecture and Rural Systems Engineering, Seoul National University, Seoul, South Korea · Interdisciplinary Program in Landscape Architecture, Seoul National University, South Korea · Integrated Major in Smart City Global Convergence, Seoul National University, Seoul, Republic of Korea +1
XRFormer: Multiscale Tokenization for XRF Representation Learning
Jul 7, 2026Sofiane Daimellah, Sylvie Le Hégarat-Mascle, Clotilde BoustRepresentation Learning Hybrid CNN-Transformer Architectures
Université Paris-Saclay, Gif-sur-Yvette, France · Centre de Recherche et de Restauration des Musées de France, Paris, France
Separating Representation from Reconstruction Enables Scalable Text Encoders
Jul 4, 2026Megi Dervishi, Mathurin Videau, Yann LeCunMasked Language Modeling Self-Supervised Pre-Training
FAIR · 3New York University
Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs
Jul 2, 2026Junhao Shi, Siyin Wang, Xiaopeng Yu +3Self-Supervised Pre-Training Efficient VLA Models
1Fudan University · 2Shanghai Innovation Institute
From Pixels to Temporal Correlations: Learning Informative Representations for Reinforcement Learning Pre-training
Jul 1, 2026Jinwen Wang, Youfang Lin, Xiaobo Hu +4Contrastive Learning Reinforcement Learning
School of Computer Science & Technology, Beijing Jiaotong University, Beijing, China · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing Jiaotong University, Beijing, China · Beijing Jiaotong University, Beijing, China
SAMBA: A Scatter-Guided Masked Bidirectional Mamba Foundation Model for SAR Target Recognition
Jun 30, 2026Ke Wang, Xiaoyi Pan, Zhaoyu Gu +4Mamba Self-Supervised Pre-Training
College of Electronic Science and Technology, National University of Defense Technology, Changsha 410073, China
BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations
Jun 29, 2026Ludovic K. Tuncay, Etienne Labbé, Thomas PellegriniAudio Representation Learning Self-Supervised Pre-Training
IRIT-SAMoVA · IRIT, Université de Toulouse, CNRS, Toulouse INP, Toulouse, France
SPOTR: Spatio-temporal Pooling One-Token Reconstruction for Universal Physiological Signal Self-supervised Learning
Jun 20, 2026Yiyu Gui, Mingzhi Chen, Yuesheng Zhu +2Representation Learning Self-Supervised Learning
New Cornerstone Science Laboratory, Guangdong Provincial Key Laboratory of In-Memory Computing Chips, Shenzhen Graduate School, Peking University, Shenzhen, China · Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Shenzhen Graduate School, Peking University, Shenzhen, China
CoughPhase-CLR: Designing an acoustics-informed foundation model for coughing sound classification
Jun 19, 2026Marius Moldovan, Anton Batliner, Thomas M. Berghaus +2Contrastive Learning Audio Representation Learning
CHI – the Chair of Health Informatics at the TUM University Hospital, Munich, Germany · MCML – the Munich Center for Machine Learning and MDSI – the Munich Data Science Institute, Munich, Germany · University Hospital Augsburg at the University of Augsburg, Augsburg, Germany +2
SOHET: Sequence Of Heterogeneous Events Transformer with Self-Supervised Pre-Training
Jun 19, 2026Kees Jan de Vries, Mustafa Radha, Mathijs de JongTransformer Self-Supervised Pre-Training
Booking.com, Amsterdam, The Netherlands
HilDA: Hierarchical Distillation with Diffusion for Advancing Self-Supervised LiDAR Pre-training
Jun 18, 2026Maciej Wozniak, Jesper Ericsson, Hariprasath Govindarajan +4Cross-Modal Knowledge Distillation Self-Supervised Pre-Training
KTH Royal Institute of Technology, Sweden · TRATON AB, Sweden · Linköping University, Sweden +1
Multi-Modal Contrastive Learning for Implicit Earth Embeddings via Location Tying
Jun 18, 2026Jonathan Hecht, Lukas Arzoumanidis, Ziyue Li +1Self-Supervised Pre-Training Geospatial Representation Learning
Computational Methods Lab, HafenCity University Hamburg, Germany. · Institute of Geodesy and Geoinformation, University of Bonn, Germany. · Dept. of Operations & Technology, Technical University of Munich; Heilbronn Data Science Center; Munich Data Science Institute.
Pre-Training for Simulation-Based Science: A Study on Jet Foundation Model Training Objectives
Jun 12, 2026Ibrahim Elsharkawy, Joschka Birk, Vinicius Mikuni +3Self-Supervised Pre-Training High-Energy Physics
Department of Physics, University of Toronto and Vector Institute, Toronto, ON, Canada · NERSC, Lawrence Berkeley National Laboratory, Berkeley, California, USA · Institut für Experimentalphysik, Universität Hamburg, 22761 Hamburg, Germany +3
Masked and Predictive Self-Supervised Foundation Models for 3D Brain MRI
Jun 11, 2026Esra Ergün, Hersh Chandarana, Dan Sodickson +1Medical Imaging Foundation Models Masked Autoencoders
Department of Computer Engineering, Istanbul Technical University, Istanbul, 34469, Türkiye · Department of Radiology, NYU Grossman School of Medicine, New York, NY, 10016, United States · Department of AI and Data Engineering, Istanbul Technical University, Istanbul, 34469, Türkiye
Probabilistic Contrastive Pretraining for Multi-task ADME Property Prediction
Jun 9, 2026Yifan Xue, Srimukh Prasad Veccham, Saee Paliwal +2Contrastive Learning Self-Supervised Pre-Training
NVIDIA Santa Clara, CA 95051
Exploring Adaptive Masked Reconstruction for Self-Supervised Skeleton-Based Action Recognition
Jun 9, 2026Shengkai Sun, Zhiyong Cheng, Zefan Zhang +3Skeleton-Based Action Recognition Masked Autoencoders
1Hefei University of Technology · 2Jilin University · 3Zhejiang Gongshang University +1
DECSELFMASK: Leveraging Unlabeled Text via Self-Relevance-Guided Masking for Decoder-Only Classification
Jun 8, 2026Pietro Ferrazzi, Matteo Merler, Giovanni Bonetta +2Self-Supervised Pre-Training Decoder-Only Language Models
Fondazione Bruno Kessler, Trento, Italy · University of Padova, Italy
How Much MRI Preprocessing Is Enough? A Cost-Utility Study for Brain MRI Foundation Models
Jun 6, 2026Jiangshuan Pang, Wangyang Tang, Jing Yan +5Medical Imaging Foundation Models Self-Supervised Pre-Training
School of Artificial Intelligence, University of the Chinese Academy of Sciences, Beijing 101408, China · Key Laboratory of Spatial Omics of Zhejiang Province, BGI Research, Hangzhou 310030, China · Key Laboratory of Brain Cell Mapping of Zhejiang Province, BGI Research, Hangzhou 310030, China
DALE-CT: Depth-Aware Foundation Models for Computed Tomography
Jun 5, 2026Evan W. Damron, Mahmut S. Gokmen, Mitchell A. Klusty +3Medical Imaging Foundation Models Self-Supervised Pre-Training
College of Medicine Office of Research, University of Kentucky, Lexington, KY 40506 USA
LatentWave: JEPA Pretraining for Wireless Foundation Models
Jun 4, 2026Ahmed Mohamed, Ahmed Aboulfotouh, Hatem Abou-ZeidWireless Foundation Models Self-Supervised Pre-Training
Department of Electrical and Software Engineering, University of Calgary, Canada
Structure-Guided Mixed Masked Pretraining and Spatial Continuity Regularization for Printed Circuit Board Defect Detection
Jun 2, 2026Peitong Wang, Nuo Wang, Enxin Qin +3Self-Supervised Pre-Training Small Object Detection
School of Big Data and Statistics, Anhui University, Hefei, 230601, China · School of Computer Science and Technology, Anhui University, Hefei, 230601, China · School of Artificial Intelligence and Data Science, University of Science and Technology of China, Hefei, 230026, China +1
Segment-driven Structural Induction and Semantic Alignment for Heterogeneous Tabular Representation
Jun 1, 2026Woojun Jung, Susik YoonTabular Foundation Models Self-Supervised Pre-Training
Department of Computer Science & Engineering, Korea University, Seoul, South Korea.
Latent Diffusion Pretraining for Crystal Property Prediction
May 30, 2026Shrimon Mukherjee, Kishalay Das, Partha Basuchowdhuri +2Latent Diffusion Models Self-Supervised Pre-Training
School of Mathematical & Computational Sciences, Indian Association for the Cultivation of Science, Kolkata, India · Computer Science and Engineering, Indian Institute of Technology Kharagpur, India.
SlideCheck: Guiding Self-Supervised Pretraining of Pathology Foundation Models via Dataset Distributions
May 28, 2026Mingyi He, Xinyi Guo, Xitong Ling +7Self-Supervised Pre-Training Pathology Foundation Models
Beijing University of Chemical Technology · South China Normal University · Tsinghua University
Entropy-aware Masking for Masked Language Modeling
May 27, 2026Gokul Srinivasagan, Kai Hartung, Munir GeorgesMasked Language Modeling Self-Supervised Pre-Training
AImotion Bavaria, Technische Hochschule Ingolstadt, Germany
Chaos-SSL: An Attention-Based Self-Supervised Learning Framework with Chaotic Transformation for Medical Image Classification
May 26, 2026Joao Batista FlorindoContrastive Learning Data Augmentation
Institute of Mathematics, Statistics and Scientific Computing, University of Campinas, Rua Sergio Buarque de Holanda, 651, Campinas, Brazil
UNATE: UNsupervised ATomic Embedding for crystal structures property prediction
May 25, 2026Laura Solà-Garcia, Àlex Solé, Javier Ruiz-HidalgoRepresentation Learning Denoising Autoencoders
Signal Theory and Comms. Dept. UPC-Univ. Polit`ecnica de Catalunya Barcelona, Spain
NITP: Next Implicit Token Prediction for LLM Pre-training
May 24, 2026Xiangdong Zhang, Debing Zhang, Shaofeng Zhang +3Language Model Pretraining Representation Geometry
School of AI, Shanghai Jiao Tong University · Dots Studio, Xiaohongshu Inc. · University of Science and Technology of China +1
MuellerPT: Decomposition Driven Pretraining for Dense Learning in Mueller Polarimetry
May 22, 2026Adam Tlemsani, Yingdian Li, Maxime Giot +4Self-Supervised Pre-Training Medical Image Classification
Department of Computing, Imperial College London · Hamlyn Centre for Robotic Surgery, Imperial College London · Department of Surgery and Cancer, Imperial College London +2
Rethinking Transfer Learning for Industrial Inspection: DINOv3 vs. ImageNet Pretraining Across RGB and X-ray Tasks
May 22, 2026Mehdi Gharbage, Céline Teulière, Pierre Bouges +1Self-Supervised Pre-Training Industrial Anomaly Detection
1Michelin Tyres Manufacturer · 2Universit´e Clermont Auvergne, CNRS, Institut Pascal, Clermont-Ferrand, France
RelPrism: A Multi-Faceted Pre-training Framework with Self-Generated Tasks for Relational Databases
May 22, 2026Jinyu Yang, Cheng Yang, Junze Chen +4Representation Learning Self-Supervised Pre-Training
Beijing University Of Posts and Telecommunications Beijing, China · Peng Cheng Laboratory Shenzhen, China · China Telecom Bestpay Beijing, China +1
Towards a General Intelligence and Interface for Wearable Health Data
May 21, 2026Girish Narayanswamy, Maxwell A. Xu, A. Ali Heydari +37Self-Supervised Pre-Training Clinical Outcome Prediction
1Google Research · University of Washington · University of Illinois Urbana-Champaign +2
Atom-level Protein Representation Learning Improves Protein Structure Prediction
May 21, 2026Taewon Kim, Hyosoon Jang, Hyunjin Seo +6Protein Structure Prediction Self-Supervised Pre-Training
Graduate School of AI, KAIST · Department of Chemistry, KAIST · HITS Inc. +3
Entropy-Guided Self-Supervised Learning for Medical Image Classification
May 21, 2026Joao Florindo, Viviane MouraSelf-Supervised Pre-Training Medical Image Classification
Institute of Mathematics, Statistics and Scientific Computing Department of Applied Mathematics, University of Campinas Rua Sergio Buarque de Holanda, 651, Campinas, 13083-859, Sao Paulo, Brazil
Towards Understanding Self-Pretraining for Sequence Classification
May 20, 2026Omar Coser, Loredana Zollo, Paolo Soda +1Self-Attention Masked Language Modeling
TriForces: Augmenting Atomistic GNNs for Transferable Representations
May 20, 2026Ali Ramlaoui, Alexandre Duval, Hannah Bull +4Disentangled Representation Learning Machine Learning Interatomic Potentials
Entalpic, Paris, France · Universit´e Paris-Saclay, CentraleSup´elec, Inria, Gif-sur-Yvette, France
Hierarchical Contrastive Learning for Multi-Domain Protein-Ligand Binding
May 19, 2026Shuo Zhang, Rongqi Hong, Huifeng Zhang +1Protein-Ligand Binding Affinity Prediction Contrastive Learning
University of Birmingham, UK
When Does Self-Supervised Learning Transfer to Time-Series Tasks?
May 19, 2026Noam Major, Kathy Razmadze, Yoli ShavitSelf-Supervised Pre-Training Time Series Classification
Faculty of Engineering, Bar-Ilan University, Israel
PilotWiMAE: Pilot-Native Representation Learning for Wireless Channels
May 19, 2026Berkay Guler, Giovanni Geraci, Hamid JafarkhaniRepresentation Learning Masked Autoencoders
Center for Pervasive Communications and Computing, University of California, Irvine CA, USA · Nokia and Universitat Pompeu Fabra, Spain
Knowing When Not to Predict: Self Supervised Learning and Abstention for Safer DR Screening
May 18, 2026Muskaan Chopra, Lorenz Sparrenberg, Jan H. Terheyden +1Selective Prediction Self-Supervised Pre-Training
Rheinische Friedrich-Wilhelms-Universität Bonn, Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany · University Hospital Bonn - Department of Ophthalmology, Germany +1
Protein Fold Classification at Scale: Benchmarking and Pretraining
May 18, 2026Dexiong Chen, Andrei Manolache, Mathias Niepert +1Masked Autoencoders Self-Supervised Pre-Training
Max Planck Institute of Biochemistry, Martinsried, Germany · Computer Science Department, University of Stuttgart, Germany · Bitdefender, Romania
Benchmarking transferability of SSL pretraining to same and different modality segmentation tasks
May 18, 2026Jue Jiang, Harini VeeraraghavanCross-Modality Medical Image Segmentation Self-Supervised Pre-Training
Department of Medical Physics, Memorial Sloan Kettering Cancer Center
DARE-EEG: A Foundation Model for Mining Dual-Aligned Representation of EEG
May 18, 2026Yang Shao, Peiliang Gong, Qun Dai +1Masked Autoencoders Self-Supervised Pre-Training
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics Nanjing, Jiangsu, China
Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Framework
May 17, 2026Yumeng Yao, Jingzhi Dong, Haowen Gu +43D ViTs Self-Supervised Pre-Training
Nanjing University of Science and Technology · Hangzhou City University · East China Normal University +1
How Do Electrocardiogram Models Scale?
May 17, 2026Jiawei Li, Fabio Bonassi, Ming Jin +4Self-Supervised Pre-Training Electrocardiography
Uppsala University · Griffith University
VolTA-3D: Self-Supervised Learning for Brain MRI using 3D Volumetric Token Alignment
May 16, 2026Amy Makawana, Abhijeet Parida, Marius George Linguraru +23D ViTs Self-Supervised Pre-Training
Institute of Health Informatics, University College London, London, UK · Sheikh Zayed Institute for Pediatric Surgical Innovation, Children’s National Hospital, Washington, DC USA · School of Medicine and Health Sciences, George Washington University, Washington, DC, USA
Multi-level Self-supervised Pretraining on Compositional Hierarchical Graph for Molecular Property Prediction
May 15, 2026Xiayu Liu, Zhengyi Lu, Hou-biao LiSelf-Supervised Pre-Training Hierarchical Representation Learning
School of Mathematical Sciences, University of Electronic Science and Technology of China, No.2006, Xiyuan Avenue, West Hi-tech Zone,2026 611731, Chengdu, China · Department of Computer Science and Engineering, Oakland University, 201 Meadow Brook RD, Rochester Hills, 48309, MI, USA
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
May 14, 2026Tan Pan, Shuhao Mei, Yixuan Sun +8Representation Learning Multimodal Robustness
Fudan University, China · Shanghai Academy of AI for Science, China · Bioinformatics Institute (BII), Agency for Science, Technology and Research (A*STAR), Singapore +2
SpectraFlow: Unifying Structural Pretraining and Frequency Adaptation for Medical Image Segmentation
May 14, 2026Zhiquan Chen, Haitao Wang, Guowei Zou +1Flow Matching Image Segmentation
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · Guangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou, China
AudioMosaic: Contrastive Masked Audio Representation Learning
May 14, 2026Hanxun Huang, Qizhou Wang, Xingjun Ma +3Audio Representation Learning Self-Supervised Pre-Training
School of Computing and Information Systems, The University of Melbourne, Australia · Institute of Trustworthy Embodied AI, Fudan University, China · Baskin School of Engineering, University of California, Santa Cruz, USA
CSI-JEPA: Towards Foundation Representations for Ubiquitous Sensing with Minimal Supervision
May 13, 2026Xuanhao Luo, Zhizhen Li, Yuchen LiuSelf-Supervised Pre-Training Self-Supervised Time Series Representation Learning
North Carolina State University, USA
Unified Pix Token And Word Token Generative Language Model
May 13, 2026Haun Leung, ZiNan WangImage Tokenization Self-Supervised Pre-Training
AOI-SSL: Self-Supervised Framework for Efficient Segmentation of Wire-bonded Semiconductors In Optical Inspection
May 12, 2026Joaquín Figueira, Rob Van Gastel, Giacomo D'Amicantonio +4Self-Supervised Pre-Training Semiconductor Manufacturing
AIMS Lab, Eindhoven University of Technology · CoC, ASMPT · iCIS, Radboud University
MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification
May 12, 2026Bo Zheng, Yudong Chen, Zihua Xiong +4Tabular Foundation Models Self-Supervised Pre-Training
Zhejiang University · MyBank, Ant Group
TRACE: Temporal Routing with Autoregressive Cross-channel Experts for EEG Representation Learning
May 12, 2026Fan Ma, Qier An, Peng Chen +6Representation Learning Self-Supervised Pre-Training
Department of Biomedical Informatics and Data Science, Yale University
Clin-JEPA: A Multi-Phase Co-Training Framework for Joint-Embedding Predictive Pretraining on EHR Patient Trajectories
May 11, 2026Yixuan Yang, Mehak Arora, Ryan Zhang +10Self-Supervised Pre-Training Healthcare
Duke University, Durham, NC, USA
From pre-training to downstream performance: Does domain-specific pre-training make sense?
May 9, 2026Felix KronesSelf-Supervised Pre-Training Domain-Adaptive Pretraining
Oxford Internet Institute, University of Oxford, Oxford, UK
When Losses Align: Gradient-Based Composite Loss Weighting for Efficient Pretraining
May 8, 2026Ivan Karpukhin, Andrey SavchenkoSelf-Supervised Pre-Training Adaptive Loss Weighting
Sber AI Lab · HSE University ISP RAS Research Center for Trusted Artificial Intelligence
ProtoSSL: Self-Supervised Pretraining and Downstream Transfer for Projection-Based Prototype Models
May 7, 2026Steven Song, Sahil Sethi, Brett Beaulieu-Jones +1Self-Supervised Learning Self-Supervised Pre-Training
Department of Computer Science, University of Chicago, Chicago, IL 60637, USA · Center for Translational Data Science, University of Chicago, Chicago, IL 60637, USA · Medical Scientist Training Program, University of Chicago, Chicago, IL 60637, USA +3

Jul 8, 2026, cs.CVJ/K move · Enter open · S save

Automatic Echocardiography Segmentation via Transition Probability Correlation for Stable Semantic Extraction

Xinran Chen, Xiyuan Wang, Guangquan Zhou, Chuan Chen

School of Biological Science and Medical Engineering, Southeast University, Nanjing, China

Ultrasound Image Segmentation Self-Supervised Pre-Training Echocardiography Medical Image Segmentation Semantic Segmentation

Open paper AskPDF arXiv

While echocardiography is essential for cardiovascular diagnosis, inherent speckle noise and low signal-to-noise ratio often lead to ambiguous semantic features and fragmented boundaries. These limitations significantly hinder the segmentation accuracy of deep learning models in complex clinical cases. Moreover, temporal motion of the heart plays a critical role in recognizing anatomical structures. To address these challenges, we designed a STLSF module which comprises a window-matching-based semantic correction component and a semantics-guided texture enhancement component. By leveraging local transition probability correlations to correct semantics and employing semantics-guided texture enhancement, the STLSF module effectively mitigates texture instability and ambiguous semantic interpretations caused by disadvantaged echocardiography quality. Additionally, to facilitate the encoder's adaptation to the intrinsic priors of ultrasound-specific imaging patterns, we propose a frequency-aware denoising pre-training method. The entire work builds a convolution-based network with locality inductive bias and long-range dependencies. Extensive experiments confirm our SOTA performance, achieving 93.87% Dice on CAMUS and 92.62% on EchoNet-Dynamic, with respective HD95 values of 3.29mm and 2.73mm.

Momentum

Related topics

Latest papers 155