Self-Supervised Pre-Training

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 64% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 155

All topics

Automatic Echocardiography Segmentation via Transition Probability Correlation for Stable Semantic Extraction
Jul 8, 2026Xinran Chen, Xiyuan Wang, Guangquan Zhou +1Ultrasound Image Segmentation Self-Supervised Pre-Training
School of Biological Science and Medical Engineering, Southeast University, Nanjing, China
Self-Supervised Pretraining Improves Cross-Site and Cross-Scale Robustness of Point Cloud Leaf-Wood Segmentation
Jul 8, 2026Heeju Mun, Tackang Yang, Yunsoo Nam +1Self-Supervised Pre-Training Domain Generalization
Department of Landscape Architecture and Rural Systems Engineering, Seoul National University, Seoul, South Korea · Interdisciplinary Program in Landscape Architecture, Seoul National University, South Korea · Integrated Major in Smart City Global Convergence, Seoul National University, Seoul, Republic of Korea +1
XRFormer: Multiscale Tokenization for XRF Representation Learning
Jul 7, 2026Sofiane Daimellah, Sylvie Le Hégarat-Mascle, Clotilde BoustRepresentation Learning Hybrid CNN-Transformer Architectures
Université Paris-Saclay, Gif-sur-Yvette, France · Centre de Recherche et de Restauration des Musées de France, Paris, France
Separating Representation from Reconstruction Enables Scalable Text Encoders
Jul 4, 2026Megi Dervishi, Mathurin Videau, Yann LeCunMasked Language Modeling Self-Supervised Pre-Training
FAIR · 3New York University
Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs
Jul 2, 2026Junhao Shi, Siyin Wang, Xiaopeng Yu +3Self-Supervised Pre-Training Efficient VLA Models
1Fudan University · 2Shanghai Innovation Institute
From Pixels to Temporal Correlations: Learning Informative Representations for Reinforcement Learning Pre-training
Jul 1, 2026Jinwen Wang, Youfang Lin, Xiaobo Hu +4Contrastive Learning Reinforcement Learning
School of Computer Science & Technology, Beijing Jiaotong University, Beijing, China · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing Jiaotong University, Beijing, China · Beijing Jiaotong University, Beijing, China
SAMBA: A Scatter-Guided Masked Bidirectional Mamba Foundation Model for SAR Target Recognition
Jun 30, 2026Ke Wang, Xiaoyi Pan, Zhaoyu Gu +4Mamba Self-Supervised Pre-Training
College of Electronic Science and Technology, National University of Defense Technology, Changsha 410073, China
BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations
Jun 29, 2026Ludovic K. Tuncay, Etienne Labbé, Thomas PellegriniAudio Representation Learning Self-Supervised Pre-Training
IRIT-SAMoVA · IRIT, Université de Toulouse, CNRS, Toulouse INP, Toulouse, France
SPOTR: Spatio-temporal Pooling One-Token Reconstruction for Universal Physiological Signal Self-supervised Learning
Jun 20, 2026Yiyu Gui, Mingzhi Chen, Yuesheng Zhu +2Representation Learning Self-Supervised Learning
New Cornerstone Science Laboratory, Guangdong Provincial Key Laboratory of In-Memory Computing Chips, Shenzhen Graduate School, Peking University, Shenzhen, China · Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Shenzhen Graduate School, Peking University, Shenzhen, China
CoughPhase-CLR: Designing an acoustics-informed foundation model for coughing sound classification
Jun 19, 2026Marius Moldovan, Anton Batliner, Thomas M. Berghaus +2Contrastive Learning Audio Representation Learning
CHI – the Chair of Health Informatics at the TUM University Hospital, Munich, Germany · MCML – the Munich Center for Machine Learning and MDSI – the Munich Data Science Institute, Munich, Germany · University Hospital Augsburg at the University of Augsburg, Augsburg, Germany +2
SOHET: Sequence Of Heterogeneous Events Transformer with Self-Supervised Pre-Training
Jun 19, 2026Kees Jan de Vries, Mustafa Radha, Mathijs de JongTransformer Self-Supervised Pre-Training
Booking.com, Amsterdam, The Netherlands
HilDA: Hierarchical Distillation with Diffusion for Advancing Self-Supervised LiDAR Pre-training
Jun 18, 2026Maciej Wozniak, Jesper Ericsson, Hariprasath Govindarajan +4Cross-Modal Knowledge Distillation Self-Supervised Pre-Training
KTH Royal Institute of Technology, Sweden · TRATON AB, Sweden · Linköping University, Sweden +1
Multi-Modal Contrastive Learning for Implicit Earth Embeddings via Location Tying
Jun 18, 2026Jonathan Hecht, Lukas Arzoumanidis, Ziyue Li +1Self-Supervised Pre-Training Geospatial Representation Learning
Computational Methods Lab, HafenCity University Hamburg, Germany. · Institute of Geodesy and Geoinformation, University of Bonn, Germany. · Dept. of Operations & Technology, Technical University of Munich; Heilbronn Data Science Center; Munich Data Science Institute.
Pre-Training for Simulation-Based Science: A Study on Jet Foundation Model Training Objectives
Jun 12, 2026Ibrahim Elsharkawy, Joschka Birk, Vinicius Mikuni +3Self-Supervised Pre-Training High-Energy Physics
Department of Physics, University of Toronto and Vector Institute, Toronto, ON, Canada · NERSC, Lawrence Berkeley National Laboratory, Berkeley, California, USA · Institut für Experimentalphysik, Universität Hamburg, 22761 Hamburg, Germany +3
Masked and Predictive Self-Supervised Foundation Models for 3D Brain MRI
Jun 11, 2026Esra Ergün, Hersh Chandarana, Dan Sodickson +1Medical Imaging Foundation Models Masked Autoencoders
Department of Computer Engineering, Istanbul Technical University, Istanbul, 34469, Türkiye · Department of Radiology, NYU Grossman School of Medicine, New York, NY, 10016, United States · Department of AI and Data Engineering, Istanbul Technical University, Istanbul, 34469, Türkiye
Probabilistic Contrastive Pretraining for Multi-task ADME Property Prediction
Jun 9, 2026Yifan Xue, Srimukh Prasad Veccham, Saee Paliwal +2Contrastive Learning Self-Supervised Pre-Training
NVIDIA Santa Clara, CA 95051
Exploring Adaptive Masked Reconstruction for Self-Supervised Skeleton-Based Action Recognition
Jun 9, 2026Shengkai Sun, Zhiyong Cheng, Zefan Zhang +3Skeleton-Based Action Recognition Masked Autoencoders
1Hefei University of Technology · 2Jilin University · 3Zhejiang Gongshang University +1
DECSELFMASK: Leveraging Unlabeled Text via Self-Relevance-Guided Masking for Decoder-Only Classification
Jun 8, 2026Pietro Ferrazzi, Matteo Merler, Giovanni Bonetta +2Self-Supervised Pre-Training Decoder-Only Language Models
Fondazione Bruno Kessler, Trento, Italy · University of Padova, Italy
How Much MRI Preprocessing Is Enough? A Cost-Utility Study for Brain MRI Foundation Models
Jun 6, 2026Jiangshuan Pang, Wangyang Tang, Jing Yan +5Medical Imaging Foundation Models Self-Supervised Pre-Training
School of Artificial Intelligence, University of the Chinese Academy of Sciences, Beijing 101408, China · Key Laboratory of Spatial Omics of Zhejiang Province, BGI Research, Hangzhou 310030, China · Key Laboratory of Brain Cell Mapping of Zhejiang Province, BGI Research, Hangzhou 310030, China
DALE-CT: Depth-Aware Foundation Models for Computed Tomography
Jun 5, 2026Evan W. Damron, Mahmut S. Gokmen, Mitchell A. Klusty +3Medical Imaging Foundation Models Self-Supervised Pre-Training
College of Medicine Office of Research, University of Kentucky, Lexington, KY 40506 USA
LatentWave: JEPA Pretraining for Wireless Foundation Models
Jun 4, 2026Ahmed Mohamed, Ahmed Aboulfotouh, Hatem Abou-ZeidWireless Foundation Models Self-Supervised Pre-Training
Department of Electrical and Software Engineering, University of Calgary, Canada
Structure-Guided Mixed Masked Pretraining and Spatial Continuity Regularization for Printed Circuit Board Defect Detection
Jun 2, 2026Peitong Wang, Nuo Wang, Enxin Qin +3Self-Supervised Pre-Training Small Object Detection
School of Big Data and Statistics, Anhui University, Hefei, 230601, China · School of Computer Science and Technology, Anhui University, Hefei, 230601, China · School of Artificial Intelligence and Data Science, University of Science and Technology of China, Hefei, 230026, China +1
Segment-driven Structural Induction and Semantic Alignment for Heterogeneous Tabular Representation
Jun 1, 2026Woojun Jung, Susik YoonTabular Foundation Models Self-Supervised Pre-Training
Department of Computer Science & Engineering, Korea University, Seoul, South Korea.
Latent Diffusion Pretraining for Crystal Property Prediction
May 30, 2026Shrimon Mukherjee, Kishalay Das, Partha Basuchowdhuri +2Latent Diffusion Models Self-Supervised Pre-Training
School of Mathematical & Computational Sciences, Indian Association for the Cultivation of Science, Kolkata, India · Computer Science and Engineering, Indian Institute of Technology Kharagpur, India.
SlideCheck: Guiding Self-Supervised Pretraining of Pathology Foundation Models via Dataset Distributions
May 28, 2026Mingyi He, Xinyi Guo, Xitong Ling +7Self-Supervised Pre-Training Pathology Foundation Models
Beijing University of Chemical Technology · South China Normal University · Tsinghua University
Entropy-aware Masking for Masked Language Modeling
May 27, 2026Gokul Srinivasagan, Kai Hartung, Munir GeorgesMasked Language Modeling Self-Supervised Pre-Training
AImotion Bavaria, Technische Hochschule Ingolstadt, Germany
Chaos-SSL: An Attention-Based Self-Supervised Learning Framework with Chaotic Transformation for Medical Image Classification
May 26, 2026Joao Batista FlorindoContrastive Learning Data Augmentation
Institute of Mathematics, Statistics and Scientific Computing, University of Campinas, Rua Sergio Buarque de Holanda, 651, Campinas, Brazil
UNATE: UNsupervised ATomic Embedding for crystal structures property prediction
May 25, 2026Laura Solà-Garcia, Àlex Solé, Javier Ruiz-HidalgoRepresentation Learning Denoising Autoencoders
Signal Theory and Comms. Dept. UPC-Univ. Polit`ecnica de Catalunya Barcelona, Spain
NITP: Next Implicit Token Prediction for LLM Pre-training
May 24, 2026Xiangdong Zhang, Debing Zhang, Shaofeng Zhang +3Language Model Pretraining Representation Geometry
School of AI, Shanghai Jiao Tong University · Dots Studio, Xiaohongshu Inc. · University of Science and Technology of China +1
MuellerPT: Decomposition Driven Pretraining for Dense Learning in Mueller Polarimetry
May 22, 2026Adam Tlemsani, Yingdian Li, Maxime Giot +4Self-Supervised Pre-Training Medical Image Classification
Department of Computing, Imperial College London · Hamlyn Centre for Robotic Surgery, Imperial College London · Department of Surgery and Cancer, Imperial College London +2

Jul 8, 2026, cs.CVJ/K move · Enter open · S save

Automatic Echocardiography Segmentation via Transition Probability Correlation for Stable Semantic Extraction

Xinran Chen, Xiyuan Wang, Guangquan Zhou, Chuan Chen

School of Biological Science and Medical Engineering, Southeast University, Nanjing, China

Ultrasound Image Segmentation Self-Supervised Pre-Training Echocardiography Medical Image Segmentation Semantic Segmentation

Open paper AskPDF arXiv

While echocardiography is essential for cardiovascular diagnosis, inherent speckle noise and low signal-to-noise ratio often lead to ambiguous semantic features and fragmented boundaries. These limitations significantly hinder the segmentation accuracy of deep learning models in complex clinical cases. Moreover, temporal motion of the heart plays a critical role in recognizing anatomical structures. To address these challenges, we designed a STLSF module which comprises a window-matching-based semantic correction component and a semantics-guided texture enhancement component. By leveraging local transition probability correlations to correct semantics and employing semantics-guided texture enhancement, the STLSF module effectively mitigates texture instability and ambiguous semantic interpretations caused by disadvantaged echocardiography quality. Additionally, to facilitate the encoder's adaptation to the intrinsic priors of ultrasound-specific imaging patterns, we propose a frequency-aware denoising pre-training method. The entire work builds a convolution-based network with locality inductive bias and long-range dependencies. Extensive experiments confirm our SOTA performance, achieving 93.87% Dice on CAMUS and 92.62% on EchoNet-Dynamic, with respective HD95 values of 3.29mm and 2.73mm.

Momentum

Related topics

Latest papers 155