Self-Supervised Learning

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 429% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 274

Apr 20, 2026cs.LG

Randomly Initialized Networks Can Learn from Peer-to-Peer Consensus

In self-supervised learning, self-distilled methods have shown impressive performance, learning representations useful for downstream tasks and even displaying emergent properties. However, state-of-the-art methods usually rely on ensembles of complex mechanisms, with many design choices that are empirically motivated and not well understood. In this work, we explore the role of self-distillation within learning dynamics. Specifically, we isolate the effect of self-distillation by training a group of randomly initialized networks, removing all other common components such as projectors, predictors, and even pretext tasks. Our findings show that even this minimal setup can lead to learned representations with non-trivial improvements over a random baseline on downstream tasks. We also demonstrate how this effect varies with different hyperparameters and present a short analysis of what is being learned by the models under this setup.

Esteban Rodríguez-Betancourt, Edgar Casasola-Murillo

Posgrado en Computación e Informática Universidad de Costa Rica · Escuela de Ciencias de la Computación Universidad de Costa Rica

Self-Supervised Learning Self-Distillation Framework Emergence

Apr 17, 2026cs.CV

iDocV2: Leveraging Self-Supervision and Open-Set Detection for Improving Pattern Spotting in Historical Documents

Considering the imminent massification of digital books, it has become critical to facilitate searching collections through graphical patterns. Current strategies for document retrieval and pattern spotting in historical documents still need to be improved. State-of-the-art strategies achieve an overall precision of

0.494

for pattern spotting, where the precision for small non-square queries reaches 0.427. In addition, the processing time is excessive, requiring up to 7 seconds for searching in the DocExplore dataset due to a dense-based strategy used by SOTA models. Therefore, we propose a new model based on a better encoder (iDoc), trained under a self-supervised strategy, and an open-set detector to accelerate searching. Our model achieves competitive results with state-of-the-art pattern spotting and document retrieval, improving speed by 10x. Furthermore, our model reaches a new SOTA performance on the small non-square queries, achieving a new precision of 0.612.Different from the previous version, this leverages non-maximum suppression to reduce false positives.

Jose M. Saavedra, Crhistopher Stears, Marcelo Pizarro +2

Facultad de Ingeniería y Ciencias Aplicadas, Universidad de los Andes, Chile, Av. Monseñor Álvaro del Portillo, 12455, Santiago, 7620086, RM, Chile.

Open-Set Recognition Self-Supervised Learning

Apr 17, 2026cs.CV

Stylistic-STORM (ST-STORM) : Perceiving the Semantic Nature of Appearance

One of the dominant paradigms in self-supervised learning (SSL), illustrated by MoCo or DINO, aims to produce robust representations by capturing features that are insensitive to certain image transformations such as illumination, or geometric changes. This strategy is appropriate when the objective is to recognize objects independently of their appearance. However, it becomes counterproductive as soon as appearance itself constitutes the discriminative signal. In weather analysis, for example, rain streaks, snow granularity, atmospheric scattering, as well as reflections and halos, are not noise: they carry the essential information. In critical applications such as autonomous driving, ignoring these cues is risky, since grip and visibility depend directly on ground conditions and atmospheric conditions. We introduce ST-STORM, a hybrid SSL framework that treats appearance (style) as a semantic modality to be disentangled from content. Our architecture explicitly separates two latent streams, regulated by gating mechanisms. The Content branch aims at a stable semantic representation through a JEPA scheme coupled with a contrastive objective, promoting invariance to appearance variations. In parallel, the Style branch is constrained to capture appearance signatures (textures, contrasts, scattering) through feature prediction and reconstruction under an adversarial constraint. We evaluate ST-STORM on several tasks, including object classification (ImageNet-1K), fine-grained weather characterization, and melanoma detection (ISIC 2024 Challenge). The results show that the Style branch effectively isolates complex appearance phenomena (F1=97% on Multi-Weather and F1=94% on ISIC 2024 with 10% labeled data), without degrading the semantic performance (F1=80% on ImageNet-1K) of the Content branch, and improves the preservation of critical appearance

Hamed Ouattara, Pierre Duthon, Pascal Houssam Salmane +2

Intelligent Transportation Systems Research Team, Cerema, 8 rue Bernard Palissy, 63017 Clermont-Ferrand, France · Intelligent Transportation Systems Research Team, Cerema, 1 avenue du Colonel Roche, 31400 Toulouse, France · Clermont Auvergne INP, Universit´e Clermont Auvergne, Institut Pascal, CNRS, 4 Avenue Blaise Pascal, 63178 Clermont-Ferrand, France

Self-Supervised Learning Appearance Stylistic Fidelity Signatures

Apr 17, 2026cs.CV

SSMamba: A Self-Supervised Hybrid State Space Model for Pathological Image Classification

Pathological diagnosis is highly reliant on image analysis, where Regions of Interest (ROIs) serve as the primary basis for diagnostic evidence, while whole-slide image (WSI)-level tasks primarily capture aggregated patterns. To extract these critical morphological features, ROI-level Foundation Models (FMs) based on Vision Transformers (ViTs) and large-scale self-supervised learning (SSL) have been widely adopted. However, three core limitations remain in their application to ROI analysis: (1) cross-magnification domain shift, as fixed-scale pretraining hinders adaptation to diverse clinical settings; (2) inadequate local-global relationship modeling, wherein the ViT backbone of FMs suffers from high computational overhead and imprecise local characterization; (3) insufficient fine-grained sensitivity, as traditional self-attention mechanisms tend to overlook subtle diagnostic cues. To address these challenges, we propose SSMamba, a hybrid SSL framework that enables effective fine-grained feature learning without relying on large external datasets. This framework incorporates three domain-adaptive components: Mamba Masked Image Modeling (MAMIM) for mitigating domain shift, a Directional Multi-scale (DMS) module for balanced local-global modeling, and a Local Perception Residual (LPR) module for enhanced fine-grained sensitivity. Employing a two-stage pipeline, SSL pretraining on target ROI datasets followed by supervised fine-tuning (SFT), SSMamba outperforms 11 state-of-the-art (SOTA) pathological FMs on 10 public ROI datasets and surpasses 8 SOTA methods on 6 public WSI datasets. These results validate the superiority of task-specific architectural designs for pathological image analysis.

Enhui Chai, Sicheng Chen, Tianyi Zhang +2

School of Computer Science, Northwest University · PuzzleLogic Pte Ltd, Singapore 229594, Singapore · Department of Electrical & Computer Engineering, National University of Singapore +1

Mamba Self-Supervised Learning Wireless Foundation Models Mitigating Domain Shift State Space Models State Space

Apr 17, 2026cs.LG

Graph self-supervised learning based on frequency corruption

Graph self-supervised learning can reduce the need for labeled graph data and has been widely used in recommendation, social networks, and other web applications. However, existing methods often underuse high-frequency signals and may overfit to specific local patterns, which limits representation quality and generalization. We propose Frequency-Corrupt Based Graph Self-Supervised Learning (FC-GSSL), a method that builds corrupted graphs biased toward high-frequency information by corrupting nodes and edges according to their low-frequency contributions. These corrupted graphs are used as inputs to an autoencoder, while low-frequency and general features are reconstructed as supervision targets, forcing the model to fuse information from multiple frequency bands. We further design multiple sampling strategies and generate diverse corrupted graphs from the intersections and unions of the sampling results. By aligning node representations from these views, the model can discover useful frequency combinations, reduce reliance on specific high-frequency components, and improve robustness. Experiments on 14 datasets across node classification, graph prediction, and transfer learning show that FC-GSSL consistently improves performance and generalization.

Haojie Li, Mengjiao Zhang, Guanfeng Liu +3

School of Data Science, Qingdao University of Science and Technology Qingdao, China · School of Information Science and Technology, Qingdao University of Science and Technology Qingdao, China · School of Computing, Macquarie University Sydney, Australia

Graph Representation Learning Self-Supervised Learning Graph Representations High-Frequency Reconstruction

Apr 17, 2026cs.CV

SPLIT: Self-supervised Partitioning for Learned Inversion in Nonlinear Tomography

Machine learning has achieved impressive performance in tomographic reconstruction, but supervised training requires paired measurements and ground-truth images that are often unavailable. This has motivated self-supervised approaches, which have primarily addressed denoising and, more recently, linear inverse problems. We address nonlinear inverse problems and introduce SPLIT (Self-supervised Partitioning for Learned Inversion in Nonlinear Tomography), a self-supervised machine-learning framework for reconstructing images from nonlinear, incomplete, and noisy projection data without any samples of ground-truth images. SPLIT enforces cross-partition consistency and measurement-domain fidelity while exploiting complementary information across multiple partitions. Our main theoretical result shows that, under mild conditions, the proposed self-supervised objective is equivalent to its supervised counterpart in expectation. We regularize training with an automatic stopping rule that halts optimization when a no-reference image-quality surrogate saturates. As a concrete application, we derive SPLIT variants for multispectral computed tomography. Experiments on sparse-view acquisitions demonstrate high reconstruction quality and robustness to noise, surpassing classical iterative reconstruction and recent self-supervised baselines.

Markus Haltmeier, Lukas Neumann, Nadja Gruber +1

Image Reconstruction Bayesian Inverse Problems Cone-Beam Computed Tomography Self-Supervised Learning Partitioning Inverse Design+1

Apr 16, 2026cs.CV

Co-distilled attention guided masked image modeling with noisy teacher for self-supervised learning on medical images

Masked image modeling (MIM) is a highly effective self-supervised learning (SSL) approach to extract useful feature representations from unannotated data. Predominantly used random masking methods make SSL less effective for medical images due to the contextual similarity of neighboring patches, leading to information leakage and SSL simplification. Hierarchical shifted window (Swin) transformer, a highly effective approach for medical images cannot use advanced masking methods as it lacks a global [CLS] token. Hence, we introduced an attention guided masking mechanism for Swin within a co-distillation learning framework to selectively mask semantically co-occurring and discriminative patches, to reduce information leakage and increase the difficulty of SSL pretraining. However, attention guided masking inevitably reduces the diversity of attention heads, which negatively impacts downstream task performance. To address this, we for the first time, integrate a noisy teacher into the co-distillation framework (termed DAGMaN) that performs attentive masking while preserving high attention head diversity. We demonstrate the capability of DAGMaN on multiple tasks including full- and few-shot lung nodule classification, immunotherapy outcome prediction, tumor segmentation, and unsupervised organs clustering.

Jue Jiang, Aneesh Rangnekar, Harini Veeraraghavan

Memorial Sloan Kettering Cancer Center

Semi-Supervised Medical Image Segmentation Swin Medical Images Self-Supervised Learning

Apr 13, 2026cs.CV

Using Deep Learning Models Pretrained by Self-Supervised Learning for Protein Localization

Background: Task-specific microscopy datasets are often small, making it difficult to train deep learning models that learn robust features. While self-supervised learning (SSL) has shown promise through pretraining on large, domain-specific datasets, generalizability across datasets with differing staining protocols and channel configurations remains underexplored. We investigated the generalizability of SSL models pretrained on ImageNet-1k and HPA FOV, evaluating their embeddings on OpenCell with and without fine-tuning, two channel-mismatch strategies, and varying fine-tuning data fractions. We additionally analyzed single-cell embeddings on a labeled OpenCell subset. Result: DINO-based ViT backbones pretrained on HPA FOV or ImageNet-1k transfer well to OpenCell even without fine-tuning. Fine-tuning further improved performance to 0.704

\pm

0.027 (17 classes, multi-class labels). At the single-cell level, the HPA single-cell-pretrained model achieved the highest k-nearest neighbor performance across all neighborhood sizes (macro

F_1

\geq

0.515). Conclusion: SSL methods like DINO, pretrained on large domain-relevant datasets, enable effective use of deep learning features for fine-tuning on small, task-specific microscopy datasets.

Ben Isselmann, Dilara Göksu, Heinz Neumann +1

Department of Mathematics and Natural Sciences, Hochschule Darmstadt, Schoeﬀerstraße 3, Darmstadt, 64295, Hessen, Germany. · Department of Chemical Engineering and Biotechnology, Hochschule Darmstadt, Stephanstrasse 7, Darmstadt, 64295, Hessen, Germany. · European University of Technology, European Union. +1

Microscopy Self-Supervised Learning Deep Learning Vision Transformer

Apr 2, 2026cs.CV

Control-DINO: Feature Space Conditioning for Controllable Image-to-Video Diffusion

Video diffusion models have recently been applied with success to problems in content generation, novel view synthesis, and, more broadly, world simulation. Many applications in generation and transfer rely on conditioning these models, typically through perceptual, geometric, or simple semantic signals, fundamentally using them as generative renderers. At the same time, high-dimensional features obtained from large-scale self-supervised learning on images or point clouds are increasingly used as a general-purpose interface for vision models. The connection between the two has been explored for subject specific editing, aligning and training video diffusion models, but not in the role of a dense conditioning signal for pretrained video diffusion models. Features obtained through self-supervised learning like DINOv3, contain a lot of entangled information about style, lighting and semantics of the scene. This makes them great at reconstruction tasks but limits their generative capabilities. In this paper, we show how we can use the features for tasks such as video domain transfer and video-from-3D generation. We introduce a lightweight control architecture and training strategy that decouples appearance from other features that we wish to preserve, enabling robust control for appearance changes such as stylization and relighting. Furthermore, we show that low spatial resolution can be compensated by higher feature dimensionality, improving controllability in generative rendering from explicit spatial representations.

Edoardo A. Dominici, Thomas Deixelberger, Konstantinos Vardis +1

Huawei, Switzerland · Huawei, Austria · Graz University of Technology, Austria

Video Diffusion Models Self-Supervised Learning

Mar 31, 2026cs.CL

Learning Concepts, Not Tokens: Self-Supervised Semantic Alignment for Language Models

The next-token prediction (NTP) objective trains language models to predict a single token at each step, even though many continuations can express the same meaning. For example, in the sentence ``this sticker can be placed here'', positioned, attached, or put are all plausible alternatives. While standard NTP training treats these alternatives as mutually exclusive targets, we explore a self-supervised framework that encourages models to predict concepts, approximated as sets of semantically equivalent tokens. Models trained with this concept supervision align better with human similarity judgments, improve classification, clustering, and reranking performance, and achieve comparable or stronger downstream reasoning. These gains come with lower perplexity on semantically meaningful words (Section 3.2) and only minimal increases in global perplexity, suggesting that concepts enhance semantic alignment while preserving language modeling quality. Our code is available at https://github.com/christine-zhang1/learning-concepts

Christine Zhang, Dan Jurafsky, Chen Shani

Stanford University

Next-Token Prediction Hierarchical Semantic Alignment Token-Level Supervision Language Modeling Self-Supervised Learning Downstream Reasoning

Mar 29, 2026cs.LG

On the Asymptotics of Self-Supervised Pre-training: Two-Stage M-Estimation and Representation Symmetry

Self-supervised pre-training, where large corpora of unlabeled data are used to learn representations for downstream fine-tuning, has become a cornerstone of modern machine learning. While a growing body of theoretical work has begun to analyze this paradigm, existing bounds leave open the question of how sharp the current rates are, and whether they accurately capture the complex interaction between pre-training and fine-tuning. In this paper, we address this gap by developing an asymptotic theory of pre-training via two-stage M-estimation. A key challenge is that the pre-training estimator is often identifiable only up to a group symmetry, a feature common in representation learning that requires careful treatment. We address this issue using tools from Riemannian geometry to study the intrinsic parameters of the pre-training representation, which we link with the downstream predictor through a notion of orbit-invariance, precisely characterizing the limiting distribution of the downstream test risk. We apply our main result to several case studies, including spectral pre-training, factor models, and Gaussian mixture models, and obtain substantial improvements in problem-specific factors over prior art when applicable.

Mohammad Tinati, Stephen Tu

Department of Electrical and Computer Engineering, University of Southern California

Self-Supervised Learning Pretraining Downstream Fine-Tuning Finite Time Analysis Representation Space

Mar 26, 2026eess.IV

Learning to Recorrupt: Noise Distribution Agnostic Self-Supervised Image Denoising

Self-supervised image denoising methods have traditionally relied on architectural constraints, pseudo-pair constructions, or specialized loss functions to avoid the trivial identity mapping. Among these, approaches such as Noisier2Noise or R2R create training pairs by adding synthetic noise to noisy images. While effective, these recorruption-based approaches require precise knowledge of the noise distribution, which is often unavailable. We present Learning to Recorrupt (L2R), a self-supervised framework that does not require exact knowledge of the noise distribution. Our method introduces a learnable recorruptor jointly optimized with the denoiser through a min--max saddle-point objective. The proposed method achieves state-of-the-art performance among methods without prior knowledge of the noise distribution across unconventional and heavy-tailed noise distributions, such as log-gamma and Laplace, as well as spatially correlated noise, while obtaining competitive results on real-world denoising benchmarks characterized by signal-dependent noise.

Brayan Monroy, Jorge Bacca, Julián Tachella

Universidad Industrial de Santander, Colombia · CNRS, ENS de Lyon, France

Noise-Aware Iterative Denoising Methods Self-Supervised Learning Adversarial Corruption Recurrent Model Distributions

Mar 14, 2026cs.CV

Human-like Object Grouping in Self-supervised Vision Transformers

Vision foundation models trained with self-supervised objectives achieve strong performance across diverse tasks and exhibit emergent object segmentation properties. However, their alignment with human object perception remains poorly understood. Here, we introduce a behavioral benchmark in which participants make same/different object judgments for dot pairs on naturalistic scenes, scaling up a classical psychophysics paradigm to over 1000 trials. We test a diverse set of vision models using a simple readout from their representations to predict subjects' reaction times. We observe a steady improvement across model generations, with both architecture and training objective contributing to alignment, and transformer-based models trained with the DINO self-supervised objective showing the strongest performance. To investigate the source of this improvement, we propose a novel metric to quantify the object-centric component of representations by measuring patch similarity within and between objects. Across models, stronger object-centric structure predicts human segmentation behavior more accurately. We further show that matching the Gram matrix of supervised transformer models, capturing similarity structure across image patches, with that of a self-supervised model through distillation improves their alignment with human behavior, converging with the prior finding that Gram anchoring improves DINOv3's feature quality. Together, these results demonstrate that self-supervised vision models capture object structure in a behaviorally human-like manner, and that Gram matrix structure plays a role in driving perceptual alignment.

Hossein Adeli, Seoyoung Ahn, Andrew Luo +3

Zuckerman Mind Brain Behavior Institute, Columbia University, New York · Department of Social Science and AI, Hankuk University of Foreign Studies, Seoul · University of Hong Kong, Hong Kong +2

Self-Supervised Vision Transformers Self-Supervised Learning Photometric Supervision Object-Centric Representations Transformer Architectures Objects

Feb 10, 2026cs.LG

Learning to Discover Iterative Spectral Algorithms

We introduce AutoSpec, a neural network framework for discovering iterative spectral algorithms for large-scale numerical linear algebra and numerical optimization. Our self-supervised models adapt to input operators using coarse spectral information (e.g., eigenvalue estimates and residual norms), and predict recurrence coefficients for computing or applying a matrix polynomial tailored to a downstream task. The effectiveness of AutoSpec relies on three ingredients: an architecture whose inference pass implements short, executable numerical linear algebra recurrences; efficient training on small synthetic problems with transfer to large-scale real-world operators; and task-defined objectives that enforce the desired approximation or preconditioning behavior across the range of spectral profiles represented in the training set. We apply AutoSpec to discovering algorithms for representative tasks on spd matrices: accelerating matrix function approximation; accelerating sparse linear solvers; and spectral filtering/preconditioning for eigenvalue computations. On real-world matrices, the learned procedures deliver up to order-of-magnitude improvements in accuracy and/or reductions in iteration count, relative to spectrum-agnostic baselines. We find clear connections to classical theory: the induced polynomials may exhibit equioscillation behavior characteristic of Chebyshev polynomial approximation. The code is available at: https://github.com/zihanghliu/AutoSpec .

Zihang Liu, Oleg Balabanov, Yaoqing Yang +1

1International Computer Science Institute · University of California, Berkeley · 4Lawrence Berkeley National Laboratory +1

Spectral Representation Method Self-Supervised Learning

Feb 5, 2026cs.CV

Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching

Dataset distillation seeks to synthesize a compact surrogate dataset that enables performance comparable to training on the original dataset for downstream tasks. For the scenario where pre-trained self-supervised models serve as priors, traditional Linear Gradient Matching optimizes synthetic images by encouraging them to mimic the gradient updates induced by real images on the linear probe. However, this batch-level formulation requires loading thousands of real images and applying multiple differentiable augmentations to synthetic images at each distillation step, leading to substantial computational and memory overheads. In this paper, we revisit the linear gradient and theoretically derive that it is essentially a local relative distribution directed from target class centers toward non-target class centers, which we term flow. This property causes suboptimality and instability, often necessitating expensive multiple augmentations to compensate. To address this, we introduce Statistical Flow Matching, an optimal, stable, and efficient supervised learning framework that optimizes synthetic images by aligning global statistical flows in the original data. Our approach loads raw statistics only once and performs a single augmentation pass on the synthetic data, achieving performance comparable to or better than the state-of-the-art method with 10x less GPU memory usage and 4x faster distillation time. Moreover, increasing the number of augmentations for our method yields further performance gains while incurring lower additional cost. Our code is publicly available at https://github.com/einsteinxia/SFM.

Qianxin Xia, Jiawei Du, Yuhan Zhang +6

University of Electronic Science and Technology of China, Chengdu, China · CFAR, Singapore · SWJTU, Chengdu, China

Diffusion-Based Dataset Distillation Self-Supervised Learning Contrastive Flow Matching Conditional Flow Matching Synthetic Data Bilevel Optimization

Feb 2, 2026cs.LG

Self-Supervised Learning from Structural Invariance

Joint-embedding self-supervised learning (SSL), the key paradigm for unsupervised representation learning from visual data, learns from invariances between semantically-related data pairs. We study the one-to-many mapping problem in SSL, where each datum may be mapped to multiple valid targets. This arises when data pairs come from naturally occurring generative processes, e.g., successive video frames. We show that existing methods struggle to flexibly capture this conditional uncertainty. As a remedy, we introduce a latent variable to account for this uncertainty and derive a variational lower bound on the mutual information between paired embeddings. Our derivation yields a simple regularization term for standard SSL objectives. The resulting method, which we call AdaSSL, applies to both contrastive and distillation-based SSL objectives, and we empirically show its versatility in causal representation learning, fine-grained image understanding, and world modeling on videos.

Yipeng Zhang, Hafez Ghaemi, Jungyoon Lee +3

Mila - Qu´ebec AI Institute · Universit´e de Montr´eal · CHU Sainte-Justine +2

Self-Supervised Learning Latent Representation Alignment Latent Variable Causal Representation Learning

Jan 29, 2026cs.CV

Hypersolid: Emergent Vision Representations via Short-Range Repulsion

A central problem in self-supervised learning is preventing representation collapse. Most methods avoid it through global mechanisms, such as contrastive expansion, variance constraints, decorrelating dimensions, or enforcing certain output distributions. In this work, we study a different design: short-range repulsion. We introduce Hypersolid, a self-supervised objective that combines view alignment with local collision avoidance. Our method induces a latent geometry of compact, semantically aligned neighborhoods with low anisotropy. This geometry is especially effective for unsupervised clustering and fine-grained separation, although it comes at the cost of weaker transferability.

Esteban Rodríguez-Betancourt, Edgar Casasola-Murillo

Posgrado en Computación e Informática, Universidad de Costa Rica, San José, Costa Rica · Escuela de Ciencias de la Computación e Informática, Universidad de Costa Rica, San José, Costa Rica

Self-Supervised Representations Self-Supervised Learning Latent Representation Alignment Repulsive Distribution Collision Avoidance

Dec 2, 2025cs.SD

Pianist Transformer: Towards Expressive Piano Performance Rendering via Scalable Self-Supervised Pre-Training

Existing methods for expressive music performance rendering, a conditional generation task that aims to generate a human-like performance from a symbolic score, rely on supervised learning over small labeled datasets, which limits scaling of both data volume and model size, despite the availability of vast unlabeled music, as in vision and language. To address this gap, we introduce Pianist Transformer, with three key contributions: 1) introducing large-scale self-supervised learning into expressive piano performance rendering through a unified Musical Instrument Digital Interface (MIDI) representation, enabling pre-training on 10B tokens of unlabeled MIDI data; 2) an efficient asymmetric Transformer with note-level compression, substantially improving training efficiency, memory usage, and inference speed for long-context music modeling; 3) a state-of-the-art rendering model with an editable workflow, achieving strong objective and subjective results and enabling integration into real-world music production workflows. Overall, Pianist Transformer outlines a scalable path toward human-like performance synthesis in the music domain. Code, audio samples, and model checkpoints are available on our project page: https://yhj137.github.io/pianist-transformer-demo/.

Hong-Jie You, Jie-Jing Shao, Xiao-Wen Yang +3

State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China · School of Intelligence Science and Technology, Nanjing University, Suzhou 215163, China

Instrument Pretraining Self-Supervised Learning

Nov 26, 2025cs.CV

FIELDS: Face reconstruction with accurate Inference of Expression using Learning with Direct Supervision

Monocular 3D face reconstruction estimates a 3D morphable model (3DMM) representation from a single image, providing geometry-aware expression codes that are useful for facial expression analysis and affect understanding. Despite strong progress, most pipelines are trained with image-level self-supervision and evaluated primarily by geometric fidelity, which does not necessarily maximize the affective utility of the learned expression representation and may encourage intensity-amplifying shortcuts when affect supervision is naively coupled. We propose FIELDS (Face reconstruction with accurate Inference of Expression using Learning with Direct Supervision), a task-driven framework that learns FLAME expression codes for facial expression recognition (FER) under a geometric plausibility constraint. Using hybrid 2D/3D supervision, FIELDS improves affect prediction in both in-domain and external evaluations while maintaining competitive geometric fidelity on held-out and out-of-domain 3D benchmarks.

Chen Ling, Henglin Shi, Hedvig Kjellström

KTH Royal Institute of Technology, Sweden · Linköping University, Sweden

Blind Face Restoration Facial Expression Recognition Expression Geometry-Aware Self-Supervised Learning

Nov 25, 2025cs.LG

Pre-train to Gain: Robust Learning Without Clean Labels

Training deep networks with noisy labels leads to poor generalization and degraded accuracy due to overfitting to label noise. Existing approaches for learning with noisy labels often rely on the availability of a clean subset of data. By pre-training a feature extractor on the target dataset without labels using in-domain self-supervised learning (SSL), followed by standard supervised training on the same noisy dataset, we can train a more noise robust model without requiring a subset with clean labels. We evaluate both contrastive and non-contrastive SSL pre-training methods across datasets with synthetic and real-world label noise, demonstrating the broad applicability of our approach across large-scale datasets, diverse downstream tasks, and model architectures. Across all noise rates, in-domain self-supervised pre-training consistently improves classification accuracy and downstream label-error detection (F1 and Balanced Accuracy) compared with supervised training from scratch. The performance gap widens as the noise rate increases, demonstrating improved robustness. Notably, our approach achieves comparable results to ImageNet and DinoV2 pre-trained models at low noise levels, while substantially outperforming them under high noise conditions.

David Szczecina, Nicholas Pellegrino, Paul Fieguth

Systems Design Engineering University of Waterloo Waterloo, Ontario, Canada

Noisy Labels Self-Supervised Learning Pretraining Learnability Contrastive Learning Deep Network

Nov 21, 2025cs.LG

PrismSSL: One Interface, Many Modalities; A Single-Interface Library for Multimodal Self-Supervised Learning

We present PrismSSL, a Python library that unifies state-of-the-art self-supervised learning (SSL) methods across audio, vision, graphs, and cross-modal settings in a single, modular codebase. The goal of the demo is to show how researchers and practitioners can: (i) install, configure, and run pretext training with a few lines of code; (ii) reproduce compact benchmarks; and (iii) extend the framework with new modalities or methods through clean trainer and dataset abstractions. PrismSSL is packaged on PyPI, released under the MIT license, integrates tightly with HuggingFace Transformers, and provides quality-of-life features such as distributed training in PyTorch, Optuna-based hyperparameter search, LoRA fine-tuning for Transformer backbones, animated embedding visualizations for sanity checks, Weights & Biases logging, and colorful, structured terminal logs for improved usability and clarity. In addition, PrismSSL offers a graphical dashboard - built with Flask and standard web technologies - that enables users to configure and launch training pipelines with minimal coding. The artifact (code and data recipes) will be publicly available and reproducible.

Melika Shirian, Kianoosh Vadaei, Kian Majlessi +3

dept. Computer Engineering University of Isfahan Isfahan, Iran · dept. Computer Engineering University of Texas at Dallas Isfahan, Iran · dept. Artificial Intelligence University of Isfahan Isfahan, Iran

Self-Supervised Learning Multimodal Pretraining Neural Audio

Sep 29, 2025cs.LG

Graph Optimization Foundation Model: Tokenizing Graph via A Language-Model Paradigm

The pretrain-transfer paradigm, which underpins the success of large language models (LLMs), has demonstrated the immense power of creating foundation models that learn generalizable representations from vast datasets. However, extending this paradigm to Operations Research (OR) problems on graph structures remains challenging due to the fundamental conflict between the statistical flexibility of language and the strict combinatorial constraints of graphs. To bridge this gap, we introduce the Graph Foundation Model (GFM), the first framework capable of solving all distance-based optimization problems on graph structures. By introducing the LLM-like self-supervised pre-training paradigm on the paths generated from random walks in the graph, GFM is compelled to internalize the graph's complex topological and combinatorial rules, where the connectivity of the structure itself can be treated as the supervisory signal. Unlike existing neural methods that learn complex and task-specific solving policies, our approach leverages the pre-trained GFM as a foundational model of the graph's intrinsic structure, which in turn enables a simple generative heuristic to tackle a diverse range of optimization challenges effectively. Comprehensive experiments on networks ranging from 20 to 893 nodes demonstrate that GFM achieves competitive performance against specialized solvers across a variety of distinct optimization task classes, while maintaining significantly faster inference times. Our work establishes a new paradigm of adapting the pretrain-transfer framework to graph optimization, opening the door for applying foundation model innovations to OR.

Yunhao Liang, Pujun Zhang, Yuan Qu +3

Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, China · Costello College of Business, George Mason University, VA, USA · Faculty of Business and Economics, The University of Hong Kong, Hong Kong, China +1

Graph Foundation Model Combinatorial Optimization Shortest Paths Self-Supervised Learning

Sep 22, 2025cs.LG

SingLEM: Single-Channel Large EEG Model

Current deep learning models for electroencephalography (EEG) are often task-specific and depend on large labeled datasets, limiting their adaptability. Although EEG foundation models seek broader applicability, many still rely on predefined multi-channel inputs, electrode-layout assumptions, or model-specific channel handling. To address these limitations, we introduce the Single-Channel Large EEG Model (SingLEM), a self-supervised foundation model whose hybrid convolutional--Transformer encoder maps each channel independently to a reusable representation capturing local and long-range temporal structure. These representations can be used individually or combined through channel-wise feature concatenation. We assembled 71 public EEG datasets comprising approximately 9,200 subjects and 357,000 single-channel hours. For leakage-controlled evaluation, downstream results were obtained with a model pretrained on 68 datasets after excluding the three source datasets underlying the six tasks. A model pretrained on all 71 datasets is provided for general reuse. Across six motor imagery and cognitive tasks under strict leave-one-subject-out (LOSO) evaluation, the leakage-controlled model with concatenated representations and a support vector machine (SVM) classifier achieved the best overall performance among the compared pretrained and classical feature-based methods. Additional classifier and subject-adapted analyses supported the robustness of its representations. These findings support single-channel self-supervised learning as a montage-flexible foundation for reusable EEG feature extraction and electrode-level spatial analysis. The source code and pretrained models are available at https://github.com/ttlabtuat/SingLEM.

Jamiyan Sukhbaatar, Satoshi Imamura, Ibuki Inoue +5

Department of Electronic and Information Engineering, Tokyo University of Agriculture and Technology, Koganei-Shi 184–8588, Japan · Department of Electronics and Communication Engineering, National University of Mongolia, Ulaanbaatar, 14200 Mongolia

Electroencephalography Foundation Models Electroencephalography Electrocardiogram Foundation Models Self-Supervised Learning Wireless Foundation Models

Sep 12, 2025cs.LG

Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Exploration

For groups of autonomous agents to achieve a particular goal, they must engage in coordination and long-horizon reasoning. Rather than relying on complex reward functions and explicit cooperation mechanisms, we ask what minimal ingredients are required for effective coordination and exploration to emerge in multi-agent settings. We investigate this question through self-supervised goal-reaching, where agents aim to maximize the likelihood of visiting a goal state rather than maximizing a reward. Despite a sparse feedback signal, we present empirical results that show self-supervised goal-reaching techniques enable agents to learn from such feedback. On MARL benchmarks, self-supervised goal-reaching outperforms alternative approaches that have access to the same sparse reward signal. Furthermore, we empirically demonstrate that multi-agent self-supervised goal-reaching approaches can be more robust than single-agent strategies. While there is no explicit exploration mechanism, this approach explores nontrivial intermediate coordination strategies in sparse settings where alternative approaches fail to achieve a single success.

Chirayu Nimonkar, Shlok Shah, Catherine Ji +1

Princeton University

Multi-Agent Coordination Autonomous Agents Global Coordination Efficient Exploration Exploration Self-Supervised Learning

Aug 22, 2025cs.LG

FairSSL: Fair Multimodal Self-Supervised Learning

Prevalent multimodal self-supervised learning (SSL) methods rely on the redundancy assumption: that different views share substantial task-relevant information. We argue that this assumption fails in complex, real-world settings characterized by heterogeneity (e.g., variable-length healthcare or behavioral data), where enforcing strict alignment can discard unique, modality-specific signals and inadvertently amplify bias. In this work, we propose FairSSL, a framework that leverages data heterogeneity as a resource for fairness rather than a hindrance. Unlike standard contrastive approaches, FairSSL uses a subject-aware Variance-Invariance-Covariance Regularization objective, where alignment is enforced across segments drawn from the same subject. We introduce a segment-based pooling strategy to handle variable-length modalities, and we regularize representations to encourage (i) sufficient within-subject variability, (ii) cross-modal and cross-subject invariance, and (iii) representation decorrelation. Theoretical analysis shows that our objective bounds the score gap between protected groups. Empirically, FairSSL significantly outperforms existing baselines on heterogeneous multimodal datasets, improving fairness without sacrificing downstream predictive performance. Code available at: https://github.com/abtinmU/FairSSL

Jiaee Cheong, Abtin Mogharabin, Paul Liang +2

Harvard University · University of Cambridge · Middle East Technical University +1

Multimodal Learning Self-Supervised Learning Modalities Multimodal Dataset

May 18, 2025cs.LG

AdaDim: Dimensionality Adaptation for SSL Representational Dynamics

A key factor in effective Self-Supervised learning (SSL) is preventing dimensional collapse, where higher-dimensional representation spaces (

R

) span a lower-dimensional subspace. Therefore, SSL optimization strategies involve guiding a model to produce

R

with a higher dimensionality (

H(R)

) through objectives that encourage decorrelation of features or sample uniformity in

R

. A higher

H(R)

indicates that

R

has greater feature diversity which is useful for generalization to downstream tasks. Alongside dimensionality optimization, SSL algorithms also utilize a projection head that maps

R

into an embedding space

Z

. Recent work has characterized the projection head as a filter of noisy or irrelevant features from the SSL objective by reducing the mutual information

I(R;Z)

. Therefore, the current literature's view is that a good SSL representation space should have a high

H(R)

and a low

I(R;Z)

. However, this view of SSL is lacking in terms of an understanding of the underlying training dynamics that influences the relationship between both terms. Our analysis shows that the best performing SSL models do not have the highest

H(R)

nor the lowest

I(R;Z)

, but effectively arrive at a balance between both. To take advantage of this analysis, we introduce AdaDim, a training strategy that leverages SSL training dynamics by adaptively balancing between increasing

H(R)

through feature decorrelation and sample uniformity as well as gradual regularization of

I(R;Z)

as training progresses. We show performance improvements of up to 3% over common SSL baselines despite our method not utilizing expensive techniques such as queues, clustering, predictor networks, or student-teacher architectures.

Kiran Kokilepersaud, Mohit Prabhushankar, Ghassan AlRegib

Department of Electrical and Computer Engineering Georgia Institute of Technology Atlanta, GA 30308

Self-Supervised Representations Self-Supervised Learning Projection Heads

Apr 22, 2025cs.CV

Clinician-Friendly Foundation Models for Ophthalmic Image Diagnostics without Fine-Tuning or Technical Barriers

Artificial intelligence (AI) shows remarkable potential in medical imaging diagnostics, yet most current models require retraining when applied across different clinical settings, limiting their scalability. We developed GlobeReady, a deployment-oriented platform powered by the RetiGlobe foun- dation model and local feature augmentation. RetiGlobe was pretrained in two stages: 1) self-supervised learning using DINOv2 on 38 million synthetic ophthalmic images, and 2) contrastive learning using CLIP on 475,845 real image-text pairs spanning diverse ethnicities, imaging devices, and geographic regions worldwide. We evaluate GlobeReady on 488,448 ophthalmic images, including color fundus photographs (CFPs) and optical coherence tomography scans, from multi-centres in China, Singapore, Vietnam and the UK. Prospective testing included usability assessment with 31 ophthalmologists. Exploratory analyses evaluated domain generalisability, Bayesian uncertainty quantification, out-of-distribution (OOD) detection, and feature-based case retrieval.

Meng Wang, Tian Lin, Qingshan Hou +69

Ophthalmology Fundus Images Recent Vision Foundation Models Medical Vision-Language Models Dinov3 Foundation Model+3

Mar 22, 2025cs.NE

Meta-Representational Predictive Coding: Neuroscience-Informed Self-Supervised Learning

Self-supervised learning has become an increasingly important paradigm in the domain of machine intelligence. Furthermore, evidence for self-supervised adaptation, such as contrastive formulations, has emerged in recent computational neuroscience and brain-inspired research. Nevertheless, current work on self-supervised learning relies on biologically implausible credit assignment -- in the form of backpropagation of errors -- and feedforward inference, typically a forward-locked pass. Predictive coding, in its mechanistic form, offers a biologically plausible means to sidestep these backprop-specific limitations. However, unsupervised predictive coding rests on learning a generative model of raw input (akin to "generative AI" approaches), which entails predicting a potentially high dimensional input; on the other hand, supervised predictive coding, which learns a mapping between inputs to target labels, requires human annotation, and thus incurs the drawbacks of supervised learning. In this work, we present a scheme for self-supervised learning, specifically for an emerging research sub-domain that we label as neuroscience-informed self-supervised learning (NeuroSSL), within a neurobiologically plausible framework that appeals to the free energy principle, constructing a new form of predictive coding that we call meta-representational predictive coding (MPC). MPC sidesteps the need for learning a generative model of sensory input (e.g., pixel-level features) by learning to predict representations of the input across parallel streams, resulting in an encoder-only learning and inference scheme. This formulation notably rests on active inference (in the form of sensory glimpsing) to drive the learning of representations, i.e., the representational dynamics are driven by sequences of decisions made by the model to sample informative portions of its sensorium.

Alexander Ororbia, Karl Friston, Rajesh P. N. Rao

Rochester Institute of Technology · Rochester, New York, USA · VERSES AI Research Lab +3

Predictive Coding Self-Supervised Learning Neuroscience Sensory Feedback Representational Capacity

Mar 3, 2025eess.IV

Hyperspectral Image Restoration and Super-resolution with Physics-Aware Deep Learning for Biomedical Applications

Hyperspectral imaging is a powerful bioimaging tool which can uncover novel insights, thanks to its sensitivity to the intrinsic properties of materials. However, this enhanced contrast comes at the cost of system complexity, constrained by an inherent trade-off between spatial, spectral, and temporal resolution. To overcome this limitation, we present a self-supervised deep learning-based approach that restores and enhances pixel resolution post-acquisition without requiring external training data beyond the images to be restored. Fine-tuned using metrics aligned with the imaging model, our physics-aware method achieves a 16

\times

pixel super-resolution enhancement and a 12

\times

imaging speedup without the need of additional training data for transfer learning. Applied to both synthetic and experimental data from five different sample types, including healthy and diseased tissues, we demonstrate that the model preserves biological integrity, as we did not detect systematic loss of biological features or biologically consequential hallucinations in tested datasets. We also concretely demonstrate the model's ability to reveal disease-associated metabolic changes that would otherwise remain undetectable. Furthermore, we provide physical insights into the model's inner workings, paving the way for future refinements that could potentially reveal novel high resolution features in an explainable manner. All methods are available as open-source software on GitHub.

Yuchen Xiang, Zhaolu Liu, Monica Emili Garcia-Segura +12

Department of Metabolism, Digestion and Reproduction, Imperial College London, London, UK · Department of Mathematics, Imperial College London, London, UK · Department of Clinical Neurosciences and NIHR Biomedical Research Centre, University of Cambridge, Cambridge, UK +4

Hyperspectral Image Super-Resolution Hyperspectral Image Image Restoration Microscopy Self-Supervised Learning Deep Learning+2

Feb 7, 2025cs.LG

Stochastic Optimal Control for Continuous-Time fMRI Representation Learning

Learning robust representations from functional magnetic resonance imaging (fMRI) is fundamentally challenged by the temporal irregularity and noise inherent in data from heterogeneous sources. Existing self-supervised learning (SSL) methods often discard critical temporal information by discretizing or averaging fMRI signals. To address this, we introduce a novel framework that reframes SSL as a Stochastic Optimal Control (SOC) problem. Our approach models brain activity as continuous-time latent dynamics, learning a robust representation of brain dynamics by optimizing a control policy that is agnostic to the temporal irregularity. This SOC framework naturally unifies masked autoencoding (MAE) and joint-embedding prediction (JEPA) to extract compact, control-derived representations. Furthermore, a simulation-free inference strategy ensures computational efficiency and scalability for large-scale fMRI datasets. Our model demonstrates state-of-the-art performance across diverse downstream applications, highlighting the potential of the SOC-based continuous-time representation learning framework.

Joonhyeong Park, Byoungwoo Park, Chang-Bae Bang +4

KAIST · Yonsei University · EverEx +1

Functional Magnetic Resonance Imaging Representation Learning Self-Supervised Learning Latent Dynamics Stochastic Optimal Control Optimal Control

Jan 19, 2024cs.CV

Learning to Visually Connect Actions and their Effects

We introduce the novel concept of visually Connecting Actions and Their Effects (CATE) in video understanding. CATE can have applications in areas like task planning and learning from demonstration. We identify and explore two different aspects of the concept of CATE: Action Selection (AS) and Effect-Affinity Assessment (EAA), where video understanding models connect actions and effects at semantic and fine-grained levels, respectively. We design various baseline models for AS and EAA. Despite the intuitive nature of the task, we observe that models struggle, and humans outperform them by a large margin. Our experiments show that in solving AS and EAA, models learn intuitive properties like object tracking and pose encoding without explicit supervision. We demonstrate that CATE can be an effective self-supervised task for learning video representations from unlabeled videos. The study aims to showcase the fundamental nature and versatility of CATE, with the hope of inspiring advanced formulations and models.

Paritosh Parmar, Eric Peh, Basura Fernando

Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore

Fine-Grained Video Understanding Self-Supervised Learning Action Selection Impact Analysis

Nov 13, 2023cs.LG

DIRA-SS:Dynamic Domain Incremental Regularised Adaptation -- Self-Supervised

Autonomous systems (AS) often rely on Deep Neural Network (DNN) classifiers to operate in complex and dynamically changing environments. However, during operation, these classifiers may encounter domains that differ from those seen during development, causing performance degradation under distribution shift. Removing systems from operation for labelled data collection and retraining is often impractical, particularly when adaptation must occur quickly and at scale. This paper introduces DIRA-SS, a self-supervised extension of Dynamic Incremental Regularised Adaptation (DIRA) that enables online domain adaptation using only a small number of unlabelled target-domain samples. DIRA-SS augments an existing classifier with an auxiliary retraining branch and adapts the shared feature representation through a rotation-prediction task, while elastic weight consolidation regularises important source-domain parameters to reduce destructive updates. This allows the model to benefit from transfer learning without requiring classification labels during operation. We evaluate DIRA-SS on CIFAR-10C, CIFAR-100C, and ImageNet-C using ResNet architectures under severe common corruptions. The results show that DIRA-SS substantially improves performance over the non-adapted source model, achieves accuracy close to the supervised DIRA method, and outperforms existing unsupervised test-time adaptation baselines on ImageNet-C when using only 100 target-domain samples.

Abanoub Ghobrial, Kerstin Eder

University of Bristol, Bristol, UK

Domain Adaptation Self-Supervised Learning Transfer Learning Imagenet

May 31, 2023cs.LG

Training-Free Uncertainty Estimation for Embedding Models

Embedding models, often obtained via self-supervised learning, extract general-purpose representations from data. Quantifying the reliability of these representations is crucial, as many downstream models rely on them as input for their own tasks. To this end, we introduce a formal definition of representation reliability: the representation for a given test point is considered to be reliable if the downstream models built on top of that representation can, on average, consistently generate accurate predictions for that test point across various downstream tasks. However, accessing the downstream data to quantify the representation reliability is often limited or restricted for various reasons. We propose training-free methods for estimating the representation reliability without access to the downstream data. Our method is based on the concept of neighborhood consistency (NC) across distinct pre-trained representation spaces. The key insight is to find shared neighboring points as anchors to align these representation spaces before comparing them. We provide theoretical justifications for NC and develop two practical approaches: (1) directly computing NC when multiple pre-trained models are available, and (2) a perturbation-based NC (PNC), which creates synthetic ensembles from a single model through isotropic Gaussian noise, avoiding the computational cost of training deep ensembles. We further propose PNC-spread tuning, which systematically determines the perturbation magnitude by maximizing the spread of the PNC scores on a reference set. We demonstrate through comprehensive numerical experiments that our methods effectively capture the representation reliability with a high degree of correlation, achieving robust and favorable performance compared with baseline methods.

Young-Jin Park, Addison Kristanto Julistiono, Hao Wang +2

Massachusetts Institute of Technology · MIT-IBM Watson AI Lab

Uncertainty-Aware Classification Self-Supervised Learning Ensemble Uncertainty Approximate Nearest-Neighbor Search

Date pendingphysics.optics

A unified self-supervised framework for single-frame Fresnel CDI and overlapped ptychography

Ptychographic imaging at synchrotron and X-ray free-electron laser sources requires densely overlapping scans, which limits throughput and increases dose; extending coherent diffractive imaging to overlap-free operation on extended samples remains an open problem. We present a self-supervised inverse-mapping network for single-frame Fresnel coherent diffraction imaging (CDI) and overlapped ptychography with fixed, pre-estimated probes. The learned neural network reconstructs individual object patches from either one diffraction frame or several overlapping measurements at a time. In single-frame mode, the phase diversity provided by the curved-wavefront probe at the off-focus sample position removes the requirement for overlap constraints, enabling sparser scans and proportionally lower dose at fixed exposure. On synthetic line patterns, reconstructed amplitude SSIM exceeds 0.90 in single-frame mode with the curved probe and reaches 0.952-0.968 with overlap constraints. Optimization of the network via a Poisson negative log likelihood objective, rather than the more common mean absolute error, yields 10-fold improved photon-dose efficiency at doses below

10^5

photons per image, where shot noise typically limits resolution. In addition to these synthetic studies, we demonstrate robust single-frame reconstruction of extended samples using ptychographic datasets from APS and LCLS, with end-to-end reconstruction of a 10,304-frame workload approximately

36\times

faster than a highly optimized iterative solver. Together, these results unify single-frame Fresnel CDI and overlapped ptychography within one self-supervised framework, supporting dose-efficient, high-throughput imaging at modern light sources.

Oliver Hoidn, Steven Henke, Albert Vong +3

SLAC National Accelerator Laboratory, Menlo Park, California 94025, USA · Argonne National Laboratory, Lemont, Illinois 60439, USA

Image Reconstruction Single-Photon Avalanche Diode Self-Supervised Learning